It isn’t an impossible task to monitor/evaluate (M&E) intangibles, knowledge or knowledge management (KM), but it requires a series of tough choices in a maze of possibles. This is what Simon Hearn and myself are discovering, trying to summarise, synthesise and build upon the two M&E of KM papers commissioned earlier, as well as the reflective evaluation papers by Chris Mowles.
We are still at the stage of struggling very much with how to set the ballpark for our study. So this is a good opportunity to briefly share a blogpost I wrote recently about this very topic, and to share some preliminary thoughts. If we get to engage your views it would certainly help us to get going.
In attempting to monitor knowledge and/or knowledge management, one can look at an incredible amount of issues. This is probably the reason why there is so much confusion and questioning around this topic (see this good blog post by Kim Sbarcea of ‘ThinkingShift’, highlighting some of these challenges and confusion).
With this confusion, it seems to me that if we want to make sense out of M&E of KM, we should look at factors that influence the design of M&E frameworks / activities, the approaches that result from it and finally the components that an M&E framework could touch upon.
Among the factors that seem to impact on the design (and later implementation) of M&E of KM, I would identify:
- The power play, influence, roles and responsibilities that precede M&E – let’s put this under a global ‘political economy‘ heading that includes: who finances / commissions / implements / contributes to / benefits from M&E of KM?
- The world view that dictates the design: from a theory of social learning to a cognitivist approach, what is shaping the view of knowledge management that one wants to monitor? This seems to have implications on the type of M&E approach and tools chosen. And perhaps it is at this junction that multiple knowledges are able to make the most serious contribution to the discussion?
- The purpose for carrying out M&E (obviously related to the above factors): accountability, learning, strategic reflection, capacity development etc. all these could also influence the choices made for a particular system or set of tools.
- The levels at which the unit observed operates, ranging from individual to societal, as described in the figure below…
Of course, in principle the knowledge management strategy (or overall strategy if there is no KM strategy in place) should influence choices to make but that kind of strategy itself would probably be subjected to the factors described above, so perhaps it is not a relevant factor.
The combination of the above is perhaps resulting in any combination of approaches on a spectrum that would span on the one end (in a caricatural way) very linear approaches to M&E of knowledge (management) and on the other end of the spectrum purely emergent approaches of ME of KM. Somewhere in the middle would be all the pragmatic approaches that combine elements from the two ends. If this holds a grain of truth, own observation is that the majority of development institutions seem to display one of these pragmatic in-between approach . So our spectrum could include:
- Linear approaches to monitoring of KM with a genuine belief in cause and effect and planned intervention;
- Pragmatic approaches to monitoring of KM, promoting trial and error and a mixed attention to planning (rather linearly) and observing how the reality affects the plan.
- Emergent approaches to M&E of KM, stressing natural combinations of factors, relational and contextual elements, conversations and transformations.
I forgot to mention in my original post that in spite of some wording, there is no implicit assumption that there is automatically one better end of the spectrum. They just differ in scope, ambition and applicability…
In the comparative table below I have tried to sketch differences between the three groups as I see them now, even though I am not convinced that in particular the third category is giving a convincing and consistent picture.
|Worldview||Linear approaches to M&E of KM||Pragmatic approaches to M&E of KM||Emergent approaches to M&E of KM|
|Attitude towards monitoring||Measuring to prove||Learning to improve||Letting go of control toexplore natural relations and context|
|Logic||What you planned – what you did – what is the difference?||What you need – what you do à what comes out?||What you do – how and who you do it with – what comes out?|
|Chain of key elements||Inputs – activities –outputs – outcomes – impact||Activities – outcomes – reflections||Conversations – co-creations – innovations –transformations – capacities and attitudes|
|Key question||How well?||What then?||Why, what and how?|
|Key approach||Logical framework and planning||Trial and error||Experimentation and discourse|
|Attitude towards knowledge||Capture and store knowledge (stock)||Share knowledge (flow)||Co-create knowledge and apply it to a specific context|
|Component focus||Information systems and their delivery||Knowledge sharing approaches / processes||Discussions and their transformative potential|
|I, K or? What matters?||Information||Knowledge and learning||Innovation, relevance and wisdom|
|Starting point of monitoring cycle||Expect as planned||Plan and see what happens||Let it be and learn from it|
|End point of monitoring cycle||Readjust same elements to the sharpest measure(single loop learning)||Readjust different elements depending on what is most relevant(double loop learning)||Keep exploring to make more sense, explore your own learning logic(triple loop learning)|
The very practical issue of budgeting does not come in the picture here but it definitely influences the M&E approach chosen and the intensity of M&E activities.
Finally, in relation with all the above, it seems easier to explain how a certain M&E framework has been built and the components it assesses. In the table below I tried to list the most obvious components that an M&E framework could encompass, in terms of the following sequence:
- Inputs, as in the resources invested and starting point of monitoring: What areas influence knowledge management and ought to be assessed in terms of their potential to influence positively or negatively any knowledge management initiative or any initiative involving the management of intangibles?
- Throughputs, as in the processes, activities and energies that are involved in any KM initiative.
- Outputs, as in the results of KM initiatives.
|Input (resources and starting point)||Throughput (work processes & activities)||Output (results)|
|- People (capacities)- Culture (values)
- Systems to be used
- Money / budget
|- Methods / approaches followed- Relationships
- (Co-)Creation of knowledge
- Rules, regulations, governance of KM
- Learning/innovation space
- Attitudes displayed
|- Creation of products & services- Appreciation of products & services
- Use/application of products & services
- Behaviour changes: doing different things, things differently or with a different attitude
- Application of learning (learning is fed back to the system)
Do you have any suggestion about all this? Does it make any sense? Does it help? Is it complete nonsense? Any suggestion would be helpful to make this synthesis piece a little easier to unpack…