How to monitor and evaluate the impact of knowledge management initiatives? This is the central question of the study for Working Group 3. An intriguing and important question worth further research. On this blog we will share our insights and thoughts on this subject with you. Where to start? It appears important to be clear from the outset on what we mean with monitoring and evaluation on the one hand, and knowledge management initiatives on the other. And how do we see the context, the development paradigm we are working in?
Taking our perspective on the development paradigm, we see this as an approach that organisations take when they wish to bring about change in society, a value-based approach. The context is characterized by the significance of different stakeholders, whether these be donors, communities that may be the intended beneficiaries of our work, the employees of our organisations, or our trustees and board members (all variously known as ‘partners’!). All of these stakeholders have a real and tangible interest in and influence on the outcomes of our work. In this sense, despite the growing appreciation in the private sector of the ‘triple bottom line’ and the wider involvement of different stakeholders in public service delivery, the non-profit sector has a distinctive appreciation of a multiple ‘bottom-line’, and Anheier and others have articulated the law of ‘non-profit complexity’ to reflect this. The values perspective of the non-profit sector is differently articulated, and is sometimes contested. From our perspective we see it as encompassing democratic processes of participation and consultation aimed at achieving positive externalities for society in areas such as poverty, empowerment issues, HIV Aids and other areas that impact particularly in the poorest areas and communities of the world.
What is ‘knowledge management’? The literature suggests that this is more than simply an approach to learning. It is an all-encompassing approach to how an organisation (or nation, or community, etc) deals with the handling of information, knowledge and indeed wisdom, to advance its objectives – whatever they may be. This can include both explicit and tacit knowledge, and also encompasses the use of technology, databases, access to available knowledge through web-based systems, as well as social networks, formal and informal forums for dialogue, communities of practice and other socially-based learning opportunities, whereby structured learning programmes can fall somewhere between the technology and socially-based learning systems.
How do we see ‘monitoring and evaluation’? These are typically viewed as ‘processes’ of ‘measuring’ progress against predetermined goals and objectives, with various intentions (learning from what has happened, indicating success or failure, demonstrating results to funders, informing communities about project processes etc). Arising partly from a ‘log-frame’ perspective, typically many M&E processes have focused on data collection contributing to quantitative indicators (‘so many people attended the training workshops, of whom 66% were women’), through which project success or failure is determined. The limitations of this type of M&E process are obvious. We take the perspective that M&E processes can be much more than this and can contribute much greater value.
If we consider how M&E approaches can be developed to most usefully ‘measure’ progress in implementing KM processes, and then consider KM as involving various components, as identified above, we can see that a range of different methodologies will be appropriate – depending on where the emphasis is of the specific KM approach. How you monitor and evaluate the usefulness of communities of practice may be different from how you monitor the usefulness of a database established by a global policy network. Particular challenges exist in evaluating the effectiveness of KM strategies that focus specifically on tacit knowledge: how can we measure it, and even if we could, how could we measure how tacit knowledge is applied? And how can we measure the impact of socially based learning processes that rely on quality of engagement, purposeful dialogue, community support etc? What is your experience?
Filed under: knowledge management, Working Group 3 | Tagged: Add new tag, development paradigm, evaluation, km4dev, knowledge management, knowledge management initiatives, learning, monitoring |
The giraffe 
Hi Sibrenne, nice to see you blogging here! A thought-evocative posting, too. Two questions: first, what do you understand as the ‘development paradigm’? Second, what do you understand as ‘impact’?
I purposely ask this because ‘development’ is often problematised (for instance by authors such as Arturo Escobar – and more recently, and more popularly, yet less authentically, William Easterly) as a concept invented by Western post-colonial politics. This is quite an interesting and important perspective (I think) because it questions the very concept of development: it posits that the whole development discussion (‘developmentalism’) is in itself a Western one, viewed from a judgment of what ‘developed’ means, and how this should be done, often neglecting the opportunities and challenges which local ‘beneficiaries’ might have. But more importantly, it brings to the foreground issues of power: who determines the development agenda? I think particularly in a study focusing on M&E this could be of importance, because M&E is, in essence, about a value judgment: what is a succes, what a failure? But whose perspective is this judged from? Etc.
In terms of impact, this would mean that the whole ‘development paradigm’ is in a sense turned upside down, that participation in development ‘discourse’ becomes much more inclusive, and that unequal power dimensions are addressed… I think that this is, in essence, what KM in a development context *should* be about… but whether it is, is an issue (and of course differs from organization to organization); how this can be achieved, another; let alone, how one would go about measuring this…! Therefore I think the picture you added to the posting is a bit misleading, although I sense you did this on purpose to illustrate that this is how people often *see* M&E, but that it’s so much more complex than that, right?!
In short, many interesting factors which your study is linked with – too long and complex really for a blog post, more like a topic for a PhD… oh wait… it is 😉 There is obviously still a lot of ground that needs to be covered on this topic – so I will look forward to seeing your outcomes to this very important and timely study.
Hello Julie,
Thank you for your comment on our first blogpost! You raise two important questions, which make our thinking stronger as well. It made me curious to think about what we mean by a paradigm. And how important is it to know what we understand by ‘the development paradigm’. I think there are many different paradigms we use while working in the development sector. When you look at the pure concept of paradigm, the wikipedia says ‘ the term is used to describe the set of experiences, beliefs and values that affect the way an individual perceives reality and responds to that perception’. We tried to express our paradigm in the blogpost. From the sense that it is quite important to know from each other the beliefs and values laying behind the paradigm you use. The paradigm is not simply the current theory, but the entire worldview in which it exists, and all of the implications which come with it. We all have different worldviews, especially people with different backgrounds, from different cultures. The question is whether one is better than the other. I don’t think so. More important is to try to understand one another, talk about it, share your thoughts and beliefs. And to accept the fact that people have different in stead of inferior paradigms. But then we come to the next point: the greatest barrier to a paradigm shift, in some cases, is the reality of paradigm paralysis, the inability to see beyond the current models of thinking! And that is quite important because I agree with you that there might be more Western oriented paradigms saying something about the South!
This leads us to the question of what is impact. I think the picture shows exactly what is happening a lot in the world of M&E: we think measuring the impact is about collecting data about achievements, which seems not too complex. Many organizations have developed a certain competence for that. But the (learning) process of M&E has quite different requirements to be effective. Dealing with power relations, as you mention. A very important aspect! Talk about the question of what we mean by impact, define observable indicators beforehand, involve important stakeholders already in the process of defining indicators, make choices of your purpose of M&E, learn how to interpret figures you have collected (different worldviews might come in here), interact, communicate and learning from this. Even in an environment with stakeholders with different interests…