Insights from practice:Analysing the requirements of performance

Summary
Purpose – Performance measurement is a subject that has been high on the agenda for over two
decades. This article proposes making a contribution to this field by discussing how to deal
systematically with all the requirements a performance measurement system (PMS) should fulfil.
Design/methodology/approach – Different requirements suggested in the performance measurement
literature from the past 20 years have been analysed in order to structure the different tasks to conduct
when designing a PMS.
Findings – The article explains how to separate requirements that can be linked to a PMS and to an
individual performance measure. It also suggests three system classes depending on what
requirements a PMS fulfils. Finally a three-step procedure is proposed that describes how to evaluate
and improve an existing PMS in a company.
Practical implications – In practice, it is difficult to deal with numerous requirements simultaneously
when designing a PMS. The article supplies measurement practitioners with tools to identify any priority
important requirements.
Originality/value – Several new ideas to the field of performance measurement are introduced and
explained: the concept of system classes, classification of requirements and a simple three-step
procedure to evaluate and improve PMS.
Keywords Performance measures, Performance appraisal, Management information systems
Paper type Research paper
Introduction
A successful performance measurement system (PMS) is a set of performance measures
(i.e. a metric used to quantify the efficiency and effectiveness of action) that provides a
company with useful information that helps to manage, control, plan and perform the
activities undertaken in the company. The information retrieved from the PMS must in turn be
accurate, relevant, provided at the right time and easily accessible for the persons who need
it. Furthermore, the performance measures must also be designed to reflect the most
important factors influencing the productivity of the different processes that can be found in
the company. To design such a PMS is, of course, a difficult task and what that can be
considered to be the optimal PMS will also differ from case to case (Tangen, 2004a).
Various authors have discussed the design of performance measures and identified
numerous important requirements that performance measures should fulfil. Crawford and
Cox (1990), for example, believe that performance must be measured in ways that are easily
understood by those whose performance is being evaluated. They also propose that
measures should evaluate a group, not individual work. Moreover, graphs should be the
primary method of reporting performance data. Many authors emphasise that performance
measures should be derived from strategic objectives to ensure that employee’s behaviour
is consistent with corporate goals (Cross and Lynch, 1992; Kaplan and Norton, 1992; Dixon
et al., 1990; Bourne et al., 2003). Furthermore, measures should also provide timely, relevant and accurate feedback and be a part of a closed management loop (Globerson, 1985). It is
also desirable that both financial and non-financial performance measures are used (Kaplan
and Norton, 1992) and that short-, as well as long-term, results are considered (Tangen,
2004b). More recent studies (Neely et al., 2002) show that the PMS also should consider
other stakeholders besides the investors, such as employees, customers and suppliers. Not
all requirements are always compatible with each other, which makes compromise
unavoidable. An example of this problem is that performance measures, on the one hand,
should be designed to be as exact as possible, which may result in a very complex formula.
On the other hand, performance measures should be easy to measure and easy to
comprehend, which are arguments for using simple formulas.
As shown in the discussion above, there are numerous requirements to fulfil when designing
a PMS. Several compilations of what a PMS should fulfil can also be found in literature (Neely
et al., 1997; Franco and Bourne, 2003; Bauer et al., 2004; Najme et al., 2005). However, few
of these compilations provide any means of how to handle suggested requirements in
practice. The purpose of this paper is therefore to discuss how to deal with PMS
requirements systematically. An important distinction is made in the article between two vital
questions: ‘‘What should be measured?’’; and ‘‘How should it be measured?’’. It is
suggested that these two questions should be handled separately, which in turn means that
requirements fall into two categories:
1. System requirements. These represent criteria important from an overall system point of
view, such as: support strategy and selection of both financial and non-financial
performance (i.e. what to measure).
2. Measure requirements. These represent criteria important when designing an individual
performance measure, such as: have an appropriate formula and include necessary
specifications (i.e. how to measure).
Since measure requirements have been considered in a previous article (see Tangen, 2005),
this article is limited to questions that can be directly connected to the system requirements.
System classes
Before discussing the requirements the concept of system classes is introduced. This
concept has been developed in order to systematically deal with the requirements when
designing or improving a PMS. The idea is that a company should begin by designing a
simple yet useful PMS fulfilling the company’s basic needs instead of going directly to an
advanced PMS. In other words, it is suggested that a company should start with designing
the lowest class of a PMS (i.e. third class). Later when the company is completely prepared it
should progress to the next class (i.e. second class) and then consequently to the highest
class (i.e. first class).
The reason for introducing this concept is mainly that it is believed within this research that it
takes time to develop a PMS as well as that an organisation needs to build up experience of
performance measurement before being able to handle an advanced PMS. An impatient
company that directly attempts to reach the highest class of a PMS will almost surely fail. As
previously emphasised, designing a successful PMS is a difficult task. With little or no
previous experience it is likely that many performance measures will not be designed
appropriately and that the information retrieved from them will not be interpreted correctly.
Simply, ‘‘one needs to learn how to walk before how to run’’ and each class has its period of
experimentation and learning before being fully embedded into the organisation.
The concept of system classes also has some other advantages. First, companies rarely
want to design their PMS from scratch. Usually they are more interested in eliminating any
weaknesses in their existing PMS. This concept allows companies to keep their existing PMS
and focus on how to improve it. Instead of designing a completely new PMS that fulfils all
types of requirements, the system classes encourage companies to fulfil the specific
requirements within the class that their existing PMS belongs to. Second, all companies have
their own unique need concerning a PMS, meaning that some companies will strive for the
development of highly advanced PMS while others will be satisfied with rather simple ones.
Using the concept of system classes, a company can choose to stay at a certain system
class. For example, a company that is purely practising mass-manufacturing principles may
not need such an advanced PMS as an agile manufacturing company requires.
As described in Table I, three different classes of PMS are suggested:
1. Third class can briefly be described as PMS where mostly traditional performance
measures are used. The requirements of such a system are rather low, but it is important
to have control over the basic principles in performance measurement before moving up
to the next level.
2. Second class means that the PMS have a much more balanced view on performance
than the previous class. Non-financial measures are used, different time horizons are
considered and most levels within the company are overviewed by the PMS.
3. First class is the most advanced PMS, which means that very high standards are met,
ranging from the existence of an advanced information handling architecture to measures
that explain causal relationships across the organisation. These systems must also
include processes that enable them to evolve when necessary.
Even though each class has its own unique characteristics, it should be noted that there are
no exact defined borders between the classes. This can cause problems when trying to
decide (during an evaluation) what class an existing PMS belongs to. However, it is
recommended to always use the lower class if an existing PMS end up between two system
classes. Thus, a higher class must at all times still fulfil the requirements of a lower one. If
something is very wrong about the existing PMS and it is inducing behaviour that is
inconsistent with the strategy of the company, it may be necessary to start from the
beginning and design a completely new PMS. It is then suggested that the company should
strive for designing a new third class PMS.
Basic system requirements
Performance is a multidimensional term that has a variety of meanings depending on what
viewpoint you have. It is therefore not easy to give an exact definition of a ‘‘good’’ PMS.
Further, a ‘‘good’’ PMS may not actually result in better performance, since this demands that appropriate actions are taken based on the information from the PMS. However, actions
are also affected by numerous of other factors in a company, such as corporate culture,
availability of resources and motivation to change. Nevertheless, it is relatively easy to argue
that some basic requirements should always be fulfilled by a ‘‘good’’ PMS and, in this case,
regardless of system class:
B provide accurate information;
B support strategic, tactical and operational objectives;
B guard against sub-optimisation; and
B include a limited number of performance measures.
Provide accurate information
It is probably no necessary to point out, but the most important requirement of a PMS is of
course that it provides its users with accurate information. A PMS that to a majority gives
information that is distorted in some way or includes many errors will cause confusion and is
highly likely to have a negative influence on the performance of the company.
Support strategic, tactical and operational objectives
Another requirement, advocated by numerous of researchers in literature, is that the PMS
should be derived from the company’s objectives and supports the strategy of the company.
Otherwise, the PMS will encourage actions that have the opposite effect of what the PMS is
intended to induce. Further, equally important is that the strategic objectives, specified at
corporate level, are in turn translated into tactical and operational objectives to the lower
levels of the company. One must ensure that all parts of the company are striving towards the
same goal. However, at the same time it is necessary to remember that different parts of the
company have different needs and specified objectives at higher levels may be useless at
lower levels without appropriate transformation and additional work to detail them.
However, it must be noted that objectives will be defined differently in each system class. For
example, the objectives when dealing with a third class system will primarily have a financial
nature, while higher system classes demand that non-cost objectives are defined. Further,
both the importance and the complexity of defining clear objectives will increase when
dealing with a higher system class of PMS.
In addition, it is important to remember that objectives usually change over time and when an
objective changes, some performance measures must change too. There is, therefore, a
need for flexibility in the PMS, which provides a mechanism that ensures that the PMS at all
times is coherent with the objectives of the company.
Guard against sub-optimisation
As performance measures, by which employees are evaluated, greatly impact their
behaviour, an improper set of measures can lead to dysfunctional or unanticipated
behaviour (Fry, 1995). In other words, employees seeking to improve the measure of their
performance often make decisions that are contrary to the desires of management. It is,
for example, not rare that an improvement in one area leads to a deterioration in another
and in worst case also resulting in a decline in the performance from an overall point of
view. Skinner (1986) termed this phenomenon as the ‘‘productivity paradox’’ where
dysfunctional behaviour results from poor performance measures. In fact, there are many
examples where productivity improvement programs have failed mostly because they
actually hurt more than they help. For instance, to focus on direct labour and efficiency of
factory workers where direct labour costs is about 10 per cent of sales may not have a
significant effect on the total productivity. Unfortunately, organisations that have
behavioural ramifications in mind when designing performance measures are few and
far between. Often the problem is not that measures have designs that give inaccurate
information, but that the behaviours they are likely to induce in a particular setting are not
considered (Neely et al., 1997). A PMS must therefore guard against sub-optimisation,
possibly by establishing a clear link from the top of the company all the way to the
bottom, to ensure that employee’s behaviour is consistent with corporate goals.
Include a limited number of performance measures
An important aspect is that in order to take appropriate actions it is necessary to use a limited
amount of performance measures (Jackson, 2000). To produce unnecessary data is
expensive, and can lead to more harm than good. More measurement will require a greater
deal of time for analysis by managers, or, alternatively, it is a waste to collect data if they are
ignored. It is therefore important to pay attention to limiting the data requirements to both the
necessary detail and frequency, to consider whether the data are needed for a specific
useful purpose, and whether the cost of producing it is not higher than expected benefit
(Bernalak, 1997). A large number of performance measures also increase the risk of
information overload, meaning that it is practically impossible to distinguish information with
high importance from information with less value. Information overload can lead to their all
information being ignored instead of used. In other words, it is vital that ‘‘old’’ performance
measures, that are no longer of interest, are removed from the PMS.
Class-dependent system requirements
Bourne et al. (2003) describe five different aspects in performance measurement that have
changed significantly during the last 20 years: focus, dimensions, drivers, targets and
desired benefits. A similar classification will be used here when discussing what
requirements increase both in importance and in complexity depending on each system
class:
B requirements regarding used performance criteria;
B requirements regarding stakeholders;
B requirements regarding hierarchical levels;
B requirements regarding time-horizon; and
B requirements regarding information architecture.
A third class PMS requires that:
B Traditional performance criteria are used. The performance measures within this class will
to a majority be influenced by the traditional way of measuring performance (e.g. return
on investment (ROI), cash flow, efficiency). The performance measures control that
different costs are kept at a reasonable level.
B Internal needs are fulfilled. The performance measures are mainly focused on fulfilling the
internal needs of the organisation.
B Top hierarchical levels are included in the PMS. The performance measures should cover
at least the most important hierarchical levels of the organisation.
B The short-term time-horizon is considered. Important measures within the PMS should
include updated targets, which at least cover the short-term time-horizon.
B Some form of information architecture is achieved. The PMS should be designed in a way
so that information is easily retrieved as well as easily understood by those whose
performance is being evaluated.
A second class PMS requires that:
B Both financial and non-financial performance criteria are used. Performance should not
solely be seen from a traditional point of view. A second class PMS consist of various
types of performance measures (i.e. multi-dimensional) covering all the important
aspects that represent the performance of a company (e.g. cost, quality, delivery,
flexibility and dependability). There must in turn be a balance between the various
performance measures in the PMS.
B Internal and external needs are fulfilled. The performance measures are focused on
fulfilling both the intern needs of the organisation and external needs (such as from
suppliers and customers).
Most hierarchical levels are included in the PMS. Performance measures exist on most
hierarchical levels in the organisation and are linked to each other. There should be a
balance between global and local performance.
B Short- and long-term time-horizons are considered. Important measures within the PMS
should include updated targets, which cover both the short- and the long-term
time-horizon.
B A practical information architecture is achieved. Vital information goes to the right
persons with little delay.
A first class PMS requires that:
B The used performance criteria explain important causal relationships across the
organisation. The performance measures within this class must support sustainable
performance improvement and must therefore stand up to the highest standards. The
measures must not only have a multi-dimensional view on performance, they should also
be able to explain what factors that have an influence on the performance of the company
and indicate what to do in order to improve.
B The needs from all stakeholders are fulfilled. The PMS includes performance measures
considering the needs from all possible stakeholders needs (e.g. customers,
shareholders, competitors, suppliers, employees, the society).
B All hierarchical levels are included in the PMS. Performance measures exist on all
hierarchical levels in the organisation and clear links have been established between the
measures on the different levels.
B Processes have been created for natural evolution of the PMS. Secures that the PMS is
constantly updated when needed.
B An advanced information architecture is achieved. Performance measurement
databases and other reporting systems should be fully integrated to each other. The
information in the PMS is updated continuously and directly presented to the persons who
need it.
Improving a PMS
In order to deal with the above-presented requirements in practice when improving a current
PMS, the following simple three-step procedure is proposed. This is a similar approach to
identify weaknesses of PMS as the Performance Measurement Questionnaire proposed by
Dixon et al. (1990). However, that Performance Measurement Questionnaire does not
support the idea of system classes and does not consider all of the requirements presented
here (such as stakeholders):
1. Requirement evaluation. The first thing to do is a subjective evaluation of to what degree
the PMS fulfils different requirements. This is done with the system evaluation form
described in Table II. The idea with the form is to subjectively consider to what degree
each specified requirement is fulfilled by the PMS, on a scale between (1) to (7), where (1)
means that the requirement is not fulfilled at all and (7) that it is fulfilled to the highest
degree possible. It should be noted that there is no particular reason for choosing a
seven-graded scale here other than that this scale is easy to use and gives proper details.
2. System class determination. After the requirements have been analysed it is time to
determine the system class of the PMS (third class, second class or first class). This is
done by comparing the system evaluation form to the descriptions of the system classes
in Table I.
3. Revision of the PMS. Depending on the result of the previous steps, there are two different
strategies to follow when revising the PMS:
B Class completion strategy. This strategy is used if some defined requirements within
the system class have not yet been fulfilled to a satisfactory degree. The class
completion strategy simply means that the requirements within the class that can be improved, receive considerable attention until the PMS has completed all class
requirements.
B Class progress strategy. This strategy is used when all requirements within the system
class have been achieved to a satisfactory degree and the organisation is ready to
move up to a higher class. The class progress strategy will require more efforts for a
shorter period of time than the previous described strategy, since many new
requirements must be focused simultaneously and the PMS will be affected by major
modifications.
Case studies
The three-step procedure has been applied and tested empirically in three manufacturing
companies so far during this research (Table III). In short, the suggested requirements and
the three-step procedure showed several strengths when performing their tasks during the
case studies. First, the three-step procedure was rather simple to understand and to use,
which makes it suitable for assisting measurement practitioners. Second, the classification
of requirements made it easy to pinpoint the limitations and weaknesses of a current PMS as
well as show what to improve. For example, several main weaknesses were identified at one
of the company, even though their PMS had recently been severely modified. Third, the
procedure could also be used independently from the origin of PMS (such as if it had been
designed according to the balanced scorecard or other frameworks). Some limitations were
also identified. The system classes and the system evaluation form force its user to rely on
subjective analysis. Significant work is also required by the user, who for example must
identify how to make improvements.
Conclusions
There are numerous performance measures to select from when designing a PMS. This
gives the measurement practitioner many different options for designing a unique PMS.
However, from a practical point of view, there is often no obvious way to choose between
these performance measures and their abundance has caused much uncertainty (Tangen, 2003). It is also very difficult to fulfil all requirements suggested in literature when designing a
PMS. In this article, it is proposed that the question ‘‘What to measure?’’ and ‘‘How to
measure it?’’ should be handled separately in order to facilitate the design process. Further,
it is suggested that the former question should also be linked to different system
requirements depending on the complexity of the PMS (i.e. system class). Finally, a simple
procedure has been described that can be used when evaluating and improving an existing
PMS.

Comments