Conventional Benchmark Processes
Conventional benchmark processes measure and compare the performance of the IT services of one company with those of other companies. Performance combines both criteria for efficiency and effectiveness. The objective of benchmarking is to identify optimization potential, and to derive recommendations as to how performance can be optimized.
The widespread, traditional benchmark processes very often use simple indicators (known as KPI, or key performance indicators) to make comparisons. For example, one typical indicator is costs per user. If one compares the costs per user, one implicitly assumes that the costs are proportional to the number of users. However, this assumption does not take account of the following points:
- The costs generated by a user actually depend on his/her behavior. A power user generates higher costs than an occasional user. However, the intensity of use in different systems varies greatly: for example, a system with 90 percent occasional users can be very expensive but still generate very low costs per individual user.
- Costs depend not only on use but also on quality. If we only look at the costs per user, no allowance is made for differing quality requirements.
- Costs for batch operations are independent of the number of users.
- For every system there is a base amount which is independent of the number of users.
These are only a few points. In reality, a wide range of non-transparent assumptions are made from KPI processes.
The Use of Peer Groups
Traditional benchmark processes often use peer groups to produce comparability. For example, peer groups are formed when we limit ourselves to the same business sector, the same company size or the same quality requirements etc. This process has a series of disadvantages:
- It is implicitly assumed that SAP systems are also used in a similar manner in similar systems. From a statistical perspective, that is an unjustified bias, which can also be easily refuted by the available data.
- Various different aspects are important for various different problems; with these aspects, we are supposed to form various different peer groups. However, this leads to distortions and is far too complex for traditional processes.
- If, on the other hand, we apply a large number of criteria for forming peer groups as an obvious solution, the peer groups then become small, and the statements we get from the comparisons are therefore based on very little data. Conversely, if we select only one or a few criteria, we get peer groups with elements that are difficult to compare – an insoluble dilemma.