A value judgement: Demonstrating the worth of knowledge-management

Berwin Leighton Paisner this year won an e-LOTIES award for its intranet-based know-how tool, the KITs. Director of knowledge management LUCY DILLON is therefore well placed to assess the value of KM and how law firms can best go about measuring the return on the considerable investment that KM projects demand.

The concept of demonstrating value in knowledge has been a recurring theme for knowledge managers for a number of years. As knowledge management has increasingly become a business imperative for many law firms, the need to demonstrate a return on the significant investment firms are making has become more relevant. However, the reality of how to achieve this has not been easy to grapple with.

This article will examine some of the challenges, that face anyone trying to adopt measures to demonstrate the value of knowledge-management initiatives, and issues to consider when establishing an appropriate methodology. It will then outline three practical models particularly applicable to the legal environment, which can be adapted to interpret and measure data to analyse the benefits of a knowledge-management programme.

Differing approaches to measurement

The difficulty, recognised by many management writers, with trying to assess the financial benefit of knowledge-management activities has led to two opposing views:

There are those who adopt the ‘you can’t see it, so you can’t measure it’ approach, which validly recognises the difficulty of trying to demonstrate, in numerical terms, a real return on investment on an intangible asset, such as knowledge. This school of thought acknowledges the fact that many have tried to do this unsuccessfully and has therefore concluded that the investment of time and effort is not worthwhile: the benefit is simply too uncertain;
There are those who have developed tangible outputs and metrics to demonstrate the added value of knowledge-management initiatives. Such metrics are increasingly becoming a management imperative. The difficulty with this approach is that statistics are a very blunt instrument with which to measure the benefits of intangible assets.

The right answer must be a combination of these two opposing views. Metrics are essential in the context of knowledge management, but they are notoriously difficult to interpret. If they are to be valuable, they need to be tied in with softer measures, which assist in better understanding them.

Some challenges

Some specific challenges await knowledge managers in trying to set up measures for their discipline. For example:

Traditional accounting practices are meaningless: typical auditing principles apply to tangible assets and the value of a firm’s intellectual capital is not yet measured in any systematic way. However, this is a developing area for auditors as the focus turns towards the ‘knowledge economy’; goodwill and intellectual-property rights are good examples of value being attributed to intangible assets;
Benefits from knowledge-management activities are blended with many other management disciplines. For example, the benefits of a more efficient business process could be claimed to be a result of better training (HR), faster desktop applications (IT), more appropriate clients (marketing) or more efficient transfer of knowledge (KM); and they may all be right. As knowledge permeates all law-firm activities, extracting and isolating its true value is difficult;
There is no such thing as one size fits all: knowledge-management metrics will be different for each and every organisation, as they should be linked to that organisation’s specific strategic objectives;

In a profession where knowledge is often still perceived as an instrument of power, extracting data relating to its growth is often patchy or even misleading.

Initial considerations

For those setting up a series of performance measures, or reviewing existing ones, there follow some initial considerations:

Identify the business need: it is much easier to demonstrate how knowledge management adds value in the context of the achievement of a business objective. Knowledge management should underpin a firm’s strategy. It is useful to ask practitioners themselves to articulate what will help them achieve their business goals – it promotes ownership of the initiatives and they will be able to help measure where the impact is being felt;
Take a snapshot of where you are: this is simple, but is often overlooked in the desire to move forward. Having a reference point to compare back to is essential;
Get the metrics right: there will initially be an element of trial and error in the selection of the metrics and you should remain flexible. The key is not to have too many metrics for each indicator and keep them simple: a good rule of thumb is three or four metrics for each indicator;
Action the results: you must be prepared to act on the results. Even when the outcome demonstrates that an investment of capital and time has not lived up to expectations. You should be prepared to salvage what you can, learn from the experience and move on;
Manage expectations: not just your own, but those of your management team and your knowledge workers. Measures cannot be interpreted overnight as time is required for patterns to emerge. At Berwin Leighton Paisner, we took many measures at the launch of our intranet, many of which for the first year looked fairly meaningless. However, looking back over the whole 12-month period, the data fell into some clear patterns, which enabled us to make changes and adapt our development to meet the needs of the users.

Three suggested methodologies

The following three measurement tools are particularly well adapted for use in a legal knowledge-management context:

1. List of key performance indicators (KPIs);

2. ARC ‘goals and themes’ (explained below);

3. Balanced scorecard.

List of KPIs

The British Standards Institute (BSI) in its ‘Guide to measurements in KM’1 conducted a survey among a series of (non-legal) knowledge organisations to establish a list of the most important objectives. These were:

Increased customer satisfaction/value;
Better employee attitude/morale/involvement;
Cost reduction/savings;
Better time to market;
Increased sales effectiveness;
Higher number of communities of practice;
Increased new product sales;
Higher number of knowledge-management initiatives;
Reduced employee turnover;
Amount of know-how on the system;
New business initiatives and product lines;
Commercialisation of knowledge products.

In any knowledge-intensive organisation, it is useful to split such metrics into quantative and qualitative measures, as different management approaches are required for each:

Quantative measures: these are numeric metrics and will give an insight into the maturity and effectiveness of systems (see box one above for some examples). These are simple to collate as they are within the knowledge manager’s control, once the audit mechanisms have been developed by the IT department. The benefit of such measures is that they are simple to read, and deliver a powerful message. The downside is that they are susceptible to multiple interpretations and, to make the meaning clear, they often require the back-up of softer qualitative data;
Qualitative measures: these are based on feedback and interpretation and will give an indication of the way knowledge-management efforts are being perceived, and of the reputation of the knowledge systems (see box one for some examples). These measures need to be collated alongside the quantative measures, to provide a context to the otherwise hard data. Qualitative measures can be obtained through feedback and survey; stories are a very powerful tool and will give far greater insight into performance than bare figures.

ARC model

This is a model outlined by Nigel Courtney of Courtney Consulting at a recent knowledge-management conference2, and provides a useful guide for year-on-year reporting on knowledge-management initiatives. ARC is an Austrian technology-research organisation, operating as a public/private partnership. Its mission is to create innovation in industry. Since 1999, it has published an annual report, which includes the outputs based on a model (figure one). ARC’s model is based on a number of defined ‘knowledge goals’, the performance of which is measured across the five ‘intellectual capital themes’. These outputs are analysed and commented upon in the report (figure two). ARC has observed that, as a direct result of its annual report, it has been able to secure more project revenue from the private sector, making it less reliant on public funding.

I have adapted the model for the legal sector (figure three). An annual report based on these goals and themes is straightforward to produce. The benefit of this model is that it ensures that each resource is considered and, where appropriate, used in the achievement of the knowledge goals. It promotes innovation in knowledge-management service delivery by encouraging knowledge managers to look strategically at the resources available.

The use of this model provides the measures. An analytical report will be required as well as an overall interpretation for each goal (figure two). This will include:

A numeric report providing results for each goal;
A narrative report providing an interpretation of the numbers;
An indication of whether the targets have been met and a comparison year on year. Please note the ‘smiley’ as a simple but effective graphic indication of achievement.

Balanced scorecard

This model (figure four) was developed in the early 1990s by Dr Robert Kaplan (Harvard Business School) and Dr David Norton3. Its aim is to provide a clear indication as to what companies should measure to balance the financial perspective. It enables organisations to clarify their strategy and translate it into action. On the website of the Balanced Scorecard Institute, Kaplan and Norton are quoted describing the innovative approach of the balanced scorecard: “The balanced scorecard retains traditional financial measures. But financial measures tell the story of past events, an adequate story for industrial age companies for which investments in long-term capabilities and customer relationships were not critical for success. These financial measures are inadequate, however, for guiding and evaluating the journey that information-age companies must make to create future value through investment in customers, suppliers, employees, process, technology and innovation.”

The balanced scorecard has proved popular with law firms in a number of management functions and has been particularly effective in the context of intangible assets, where financial measurement is not only difficult but also generally meaningless.

It allows managers to look at the firm’s vision from four perspectives (financial, client, learning and growth, and internal business processes), the non-financial perspectives complementing the financial measures. These clearly need to be adapted for the knowledge-management context (for example: ‘financial’ as a perspective is not always meaningful, whereas ‘efficiency’ may be better). For each perspective, the following metrics need to be identified:

Objectives: where do you want to get to?
Initiatives: how are you going to get there?
Measures: how do you know you are progressing towards your destination?
Targets: how do you know when you have arrived?

Each metric is associated to a question to ensure that the tasks are identified through a systematic and consistent thought process.

One of the major benefits of the balanced scorecard is its flexibility. It is infinitely adaptable, not only within many industry sectors and management disciplines, but also within different management levels. It operates equally well in the context of a strategy review for a multi-office law firm as in the annual business-planning process for a solo knowledge located in a satellite knowledge unit.

Demonstrating value

This article has set out a number of steps to demonstrate how knowledge management is bringing value to your firm. The process to adopt could be along the following lines:

Identify the firm’s business objectives;
Articulate how knowledge management can support their achievement;
Define measures of value for knowledge-management activities;
Use them;
Report on how these are adding value.

Demonstrating that knowledge management has a positive impact on the achievement of business objectives will make the process of seeking further investment for knowledge initiatives easier to manage.

References

British Standards Institute ‘Guide to measurement in knowledge management’, BSI 2003
Nigel Courtney’s presentation at the Ark Group conference, ‘Demonstrating the value of knowledge management’, March 2003
Balanced Scorecard Institute: http://balancedscorecard.com.org
Rusanow, G., Knowledge Management and the Smarter Lawyer, published by ALM Publishing, 2003

Lucy Dillon is director of knowledge management at Berwin Leighton Paisner. She can be contacted at lucy.dillon@blplaw.com