(Originalmente publicado el 22 de Septiembre de 2006)
There is a wide agreement that the benefits of Communities of Practice are very hard to measure. Therefore CoP sponsoring is a difficult sell in corporate environments, on indeed anywhere that resources are managed. Intuitive evidence and personal experience of users suggest that a good CoP creates enormous value, but pinning it down in a financial report is another matter.
- Anecdote-driven. Best described theoretically by Richard McDermott , it has been extensively used in large industrial CoPs. It relies on capturing and documenting individual instances of value-adding actions directly related to the activity of the CoP, and translating those anecdotes into hard, conservative estimates of gained (or saved) revenue.
- Indirect measurement. Most popular in development agencies and government bodies [India. Canada]. It relies on estimates of the effects of CoPs through the evolution of the issues they were created to solve: how productivity has increased, how literacy or vaccination have risen since the CoP operates. Documentation may be story-driven or based on polls of users.
- Activity indexes. As exemplified by the Macuarium set of measurements  (aimed mostly at CoP health), it is possible to portray the relevance of a CoP through some of its metrics. Member participation, answers to issues raised, frequency of visit and even direct relevance qualification by users, all give a view of the CoP’s role in processes.
None of them is perfect. Anecdote-driven methods miss most of the impact, indirect measurement risks masking the effect of other factors (positive or negative), and activity indexes can conceal non-productivity-related activity.
Still, it is hard to improve on them for a number of reasons. Most of them are derived from the way resources have been traditionally managed.
- Human productivity management. The first and foremost issue is that human productivity in a knowledge-intensive job is extremely hard to measure. When measuring a process we can know how many people we’ve dedicated to it, and how many other resources are involved. But we can’t easily know what is the level of use of the people’s capacity. In most situations we don’t know just how hard that person is working, how intensive is the use of his/her faculties. Is that engineer taking more time to finish the project than he could? Is that doctor operating at maximum quality? How can we tell that a receptionist is more efficient than last week? How do you measure that copy-writer’s latest output?We can’t, because most of our measurement systems treat people as inputs, not work or quality of results. Most of our cost structures are still built (estimated, in truth) on people hired, not on time actually employed.
- Lack of a baseline. In the same vein, most processes are not described or measured in sufficient detail to appreciate a change in the time used to complete a task within them. In other words, the initial process structure and productivity levels are (usually) not known. Most companies can’t truthfully say they know what their real use of resources and costs are: they rely on estimates (“standard cost”, it is often called). Therefore it is ridiculous to ask for an identification of improvements: we can’t possibly tell the initial situation from the CoP-derived evolution, at least not with objective indicators, because we don’t know the initial situation.
This situation has driven many practitioners and theorists to avoid measurement like the plague, and even to ask for its need to be exorcised from the mind of “modern”, “aware”, “abundance-paradigm”- abiding managers. Unsurprisingly they have cut little ice with budget-aware organisations; and what they have gained with these arguments, was soon lost as the sponsor has been replaced with harder-headed types.
Still, the fact that something is impossible doesn’t mean that it shouldn’t be attempted. And anyway most purse-holding managers would refuse to be moved by that kind of arguments. So let’s look at ways to overcome that situation and find ourselves some objective indicators that can prove (or disprove) a CoP’s worth in a financial statement.
In these last years we’re witnessing the business world’s reaction to the above problems. And they include a couple of things that can be of very serious use:
- Quality management. In particular, Six Sigma methodologies… but also practically every other quality methodology, are useful in this context. They strive to shed light on to what is actually done in the organization, and by whom: they map the processes, the activities and the roles. In fact, it is plain that many of those efforts are no more that wishful thinking pasted onto chaotic realities… but, where they are really practised and management insists on their implementation, they provide a very close proxy for a baseline’s process structure.
- Workflow or BPM. In other words, the definition and management of the organisation’s processes through automated routines that link each point in the process (role) and notes the execution of each task, passing it on to the next point. For instance, in credit assessment circuits or expense approvals, but also in simpler tasks… and in more complex ones (support issues, letter-writing, customer proposal generation), a “workflow” provides an underlying automation that routes the necessary information from person to person, and meanwhile manages to capture real, objective times of execution: a close proxy for a baseline’s productivity levels.
In other words: armed with information on the processes and tasks in which a CoP user is involved, and a real, objective measurement of the user’s productivity, we can start to gather some systematic and objective measurements on productivity evolution.
Of course, this will never give us complete information on CoP returns. For three main reasons and a big externality:
- Time-based measurements don’t cover quality. The evolution in a worker’s output quality can’t be mapped this way; it will need a different measurement. Some times (credit assessment; sales proposals) it can be inferred from information available to the company (rates of default; closed sales and profit margin); sometimes it will have to depend on fuzzier customer-satisfaction measurements. Not all of them have easy translation into financial reports.
- Most CoPs deal on exceptions. In other words, CoP members consult each other on issues that represent deviations from standard procedure. While a worker can be relied upon to carry through the most usual situations, any number of reasons (technical issues, information issues, resource or supplier issues, customer relationship, internal channeling, criteria to be applied…) can cause doubts. These doubts are the food of CoPs, and their solution is their main product. In other words, CoP influence on productivity depends on the frequency of exceptions and their type: they need to be recorded, and baselined (we need to know the standard pre-CoP cost of dealing with them), if we are to measure CoP financial impact.
- There are other influences on productivity. The organisation will continually strive to improve; department leaders will work on this, policies will evolve, marketing programmes and training initiatives will roll through; not to mention new IT systems and changes in the market (changes in our customers’ perception of our quality can be due to changes in our competitors’ quality). What part of a worker’s increase in productivity (both efficiency and quality) is due to the CoP and which is due to other factors? Again, we are reduced to estimates and very complicated calculations, or to subjective, poll-based user measurement.
And last but not least:
- Not all benefits are aimed internally. Many organizations sponsor CoPs to benefit people or stakeholders outside its boundaries, where they can’t rely on direct objective measurements. Every advocacy group is an example, but also most open CoPs (where members belong to several organisations). Those CoPs are also frequently oriented at producing effects that cannot be sensibly measured in any economic terms (except maybe as substitutes for better-known sources of them), such as literacy increases or health (well-being of chronic patients, innovation in depressed areas). Even when they can, we must rely on external sources or subjective data.