At the end of May 2010, I attended – along with 150+ others, largely evaluation practitioners – the two-day conference on Evaluation revisited: improving the quality of evaluative practice by embracing complexity in Utrecht, The Netherlands. It has its own website here with more detailed information of the programme, presentations etc. This is the first part of my personal reflections on the conference which is part of a series of recent, development focused events on complexity which include the July 2009 workshop on How can complexity theory contribute to more effective development and aid evaluation? held at Panos in London, UK.
The central question of the Evaluation revisited conference was
what are the options for evaluative practice that respect the complexity of societal transformation while fulfilling quality standards?
The conference was based on an understanding of the duality of evaluation practice: one the one hand, the methodologies of the economist with RCTs (randomised control trials) and counterfactual analysis (a comparison of what has happened with what would have happened if the intervention had never taken place) and, on the other hand, a group of alternative methodologies with their roots in the social sciences.
How it was organised
The conference was spectacularly well organised and facilitated. And I’m not just saying this because many of my colleagues from Context, international cooperation were involved in the organisation. No, this was something that I think was recognised by all of the participants as one blogger notes on the website itself:
Conference was great! Really enjoyed it.
This also involved the use of lots of different methods and tools for facilitation. This made the conference fun and stimulating. One of these methods which was new to me was ritual dissent, a method designed to test and enhance proposals, stories and ideas. One of the unexpected side-effects of this was, short-term, an incredible team building experience with the group of people with whom I had been working.
One of the key themes of the conference was improving the status of the alternative approaches to evaluation, those with their roots in the social sciences. The organisers of the conference were espousing a new sort of evaluative practice, moving away from statistical, hard science approaches to evaluation to new methods – and often a combination of methods – which take a more participatory approach, a softer approach but which are still rigorous in their own way. In fact, one of the objectives of the conference was to develop a new definition of rigorous so that it could be applied to these methods. The conference advocated a new rigorous, diamond standard for evaluation which takes into account complexity, values, quality standards, and different methodological options.
Well I love games about words so I enjoyed following the developing definitions of rigorous evaluation. According to the conference, rigorous evaluation should comprise systematic, empirical, sceptical, and technically competent evaluation. But for me, rigorous is associated with harsh calculation, separation, death (rigour mortis). For exmaple, Longman’s online dictionary defines rigorous as:
- careful, thorough, and exact as in a rigorous analysis of defence needs or the rigorous standards required by the college
- very severe or strict as in rigorous army training
In the end, I thought that sequestering the economists’ terminology of rigour was not very helpful. Instead, I proposed the adoption of the word vigorous because it has implications of strength, active in mind and body, energy and determination. It also reflects more the power and dynamism of social engagement and the importance of volition – roughly meaning intentions – which is key to many development interventions. Volition, taken here from the RAAKS Glossary is defined as:
Volition emphasises both sense-making (creating comprehension and purpose) and commitment to stick to decisions that have been made. It also involves fluidity: an informed and thoughtful volition which is never in error and which is always subject to challenge and re-formulation. In addition, volition shows purpose and determination, even if no objects and results are specified in advance.
Vigorous is to life what rigorous is to death, and it is this which appeals to me as a stronger, more alive, more dynamic approach to evaluation. After all, we can’t beat the economists at their own game, and we can’t join them so let’s go for something which says more about our own espousal to development, let’s be vigorous.
For an impression of this vigorous conference, see the video below: