4.1 KPI Scoring
4.2 Setting Targets
4.3 Information Used To Measure Performance
4.4 Reviewing Performance - Overview
4.5 Reviewing Performance - Individual Responsibilities
4.1 KPI Scoring
Scoring Guidance
4.1.1 The Contractor can be awarded a score according to either a binary, numerical or subjective assessment of performance. Examples of how scores can be assigned to each of these categories are as follows:
4.1.2 Binary: here, there are only 2 options when assessing performance – acceptable or unacceptable. Therefore a score of "4" for acceptable and "0" for unacceptable should be applied.
4.1.3 Numerical: scores are linked to quantifiable, numerically-measurable KPIs, e.g.
- No delivery errors: 4 points
- 1 error: 3 points
- 2-3 errors: 2 points
- 4-5 errors: 1 point
- 6 or more errors: 0 points
4.1.4 When linking numerical measures to scores, it is important that consideration is given to what would constitute an acceptable level of performance by the Contractor.
4.1.5 Subjective: scores are linked to a subjective assessment of Contractor performance, e.g. a customer satisfaction rating, as follows:
- Excellent: 4 points
- Good: 3 points
- Acceptable: 2 points
- Poor: 1 point
- Very poor: 0 points
4.1.6 When utilising this type of measure, in order to minimise scoring subjectivity, it is important that clear guidance is provided to assessors explaining what level of service should lead to the award of each score ("Excellent", "Good" and so on).
4.1.7 In the "5 box" scoring system detailed above, the middle score would normally be deemed acceptable (i.e. 2-3 errors).
Weighting
4.1.8 It is probable that some KPIs are more vital to the success of a contract than others. In order to reflect this, performance measures can be ranked according to their relative importance via a process known as weighting.
4.1.9 As with the process of agreeing KPIs, all project team members should be involved in setting weightings. However, if the project team cannot agree on this then the Project Owner (the person within the Business Area with overall responsibility for the project) will make the final decision.
4.1.10 To achieve a common approach when determining weightings under Contract Assessment Summary (CAS), project team members should ensure that the sum of all weightings allocated totals 100.
Total CAS Score
4.1.11 In order to calculate a total CAS Score, simply aggregate the individual scores for each criterion assessed, as follows:
- Assessment for KPI 1 (Score awarded x Weighting) + Assessment for KPI 2 + Assessment for KPI 3 (And so on) = Total CAS Score
4.1.12 For all contracts under CAS, a common overall acceptable score of 200 has been set. This is broken down as follows:
- Average score per criteria (2) x Sum of weightings (100) = Acceptable Score (200)
4.1.13 Under binary and numerical/subjective assessment scoring, the maximum score per KPI is 4. Binary scoring should be kept to an absolute minimum to allow greater flexibility in assessing performance.
4.2 Setting Targets
4.2.1 Achieving an acceptable CAS score of 200 should be the minimum Scottish Parliamentary Corporate Body (SPCB) expectation. As well as helping achieve contractual compliance in this way, an underlying principle of CAS is to help facilitate continuous improvement in Contractor performance.
4.2.2 When setting performance targets, as well as aiming for a period-on-period increase in total CAS score, the SPCB should also target individual areas for improvement, e.g:
- in areas where past performance has been below an acceptable level; or
- where improvements would deliver tangible benefits to the SPCB.
4.2.3 Any targets set should be realistic and achievable. For example, in recognition of the usual learning curve at the start of a service contract, achieving an acceptable rather than excellent level of performance would be a reasonable objective.
4.2.4 In addition, an assessment needs to be made at the outset whether achievement of targets is to be linked to the Contractor’s level of remuneration.
4.2.5 If a decision is taken to "penalise" Contractors for failing to meet KPI targets by reducing their management fee, care needs to be taken that targets are not set at unrealistically high levels. Here, Contractors are likely to cover their losses by cutting corners elsewhere, which is likely to impact adversely on service levels to the SPCB.
4.3 Information Used To Measure Performance
4.3.1 Information used to measure performance can take different forms. It can be financial (e.g. cost per application processed) or non-financial (e.g. percentage of on-time deliveries)
4.3.2 It may be too expensive or time-consuming to measure a given aspect - time and resource implications must be considered. Any benefits associated with measuring a particular KPI may be outweighed by the resource commitment required to gather the information used to assess performance.
4.3.3 To free up the Contract Manager’s time from carrying out tactical monitoring activity, the Contractor wherever feasible should be required to supply management information on its own performance (this is known as “Contractor self-monitoring”).
4.3.4 The content, format and frequency of management information from Contractors should be incorporated into the contract at the outset wherever possible. It is obviously important that such information is received well in advance of key meetings, such as the CAS Review Meeting.
4.3.5 For certain business areas, such as Facilities Management (FM) and Business Information Technology (BIT), important user feedback on Contractor performance may be gleaned via:
- the Helpdesk facility (e.g. complaints);
- surveys/questionnaires.
4.3.6 To determine what information or data you need to collect, find out the following:
- What type of information is required to help measure the relevant KPI?
- What information do we currently gather? Will it meet our needs? At what effort and cost?
- What new information will need to be collected?
- Who will collect the data? Verify data? Analyse data?
- What problems might be encountered if data is collected this way?
- What existing data collection should be discontinued? Why?
- What computer hardware and software exists to assist data collection and analysis?
- What resources will be needed to manage performance data?
- What are the constraints to changing data collection (e.g. money, technology, tradition, politics, etc.)?
- How frequently will data be collected (monthly, quarterly, annually)?
4.4 Reviewing Performance - Overview
4.4.1 Although monitoring of Contractor performance is an ongoing activity, the frequency of formal reviews under CAS is dependant on the risk profile and amount of Contractor Performance Management (CPM) effort required for each contract. For example, business-critical contracts requiring significant CPM input may be measured quarterly whereas routine contract with limited CPM input may be measured yearly. The project team should agree what frequency is most appropriate for the particular contract.
4.4.2 For contracts requiring significant CPM input, business area and Contractor representatives (normally at contract/account manager level) may require to meet on a monthly basis to discuss on-going operational (including health & safety issues)/ service delivery matters. Indeed, especially during the early stages of such a contract, more frequent meetings (e.g. weekly) may be required.
4.4.3 On a quarterly/6-monthly/annual basis, depending on the profile of the contract, a CAS Review should take place to review:
- last period's CAS scores and set targets for next period;
- progress on total cost of ownership optimisation & other continuous improvement initiatives;
- internal customer feedback;
- instances of sub-standard performance, including Contractor Corrective Action Reports;
- whether KPIs (including scoring, weighting and measurement responsibility) remain appropriate or require to be amended (if so, they should be formalised into the contract at least annually).
4.4.4 The SPCB should be represented by the Contract Manager, Purchaser (if required) and any other appropriate people with relevant Contractor representatives (including regional/senior management) also attending. The Contract Manager should chair the meeting.
4.4.5 As well as being a forum to discuss the Contractor’s performance, the meeting should also provide an opportunity for the Contractor to comment on the SPCB’s contribution (good or bad) to the operation of the contract. Wherever appropriate, Responsible Purchasing issues should be a standing agenda item, for example, Health and Safety should be discussed if the Contractor provides the service on site).
4.4.6 The CAS Review Meeting should also be used to discuss and plan the long-term strategic direction of the contract, as well as addressing any relationship management issues.
4.4.7 CAS Reviews need not always be undertaken via a formal meeting. With the agreement of the Contractor, CAS Reviews can be undertaken by videoconference, telephone and/or e-mail wherever appropriate.
4.5 Reviewing Performance – Individual Responsibilities
4.5.1 In advance of the 1st reporting period under CAS, all relevant KPIs (including weightings, measurement methodology and assessment responsibility) should be incorporated into the KPI Scoring Sheet (13KB pdf) by the Contract Manager.
4.5.2 Where different project team members are responsible for assessing and scoring different areas of the Contractor’s performance, the same sheet should be used with the relevant sections identified to make clear who is assessing each criterion.
4.5.3 Assessors should then record their assessments of Contractor performance on the KPI Scoring Sheet, ensuring that scores awarded are in line with the agreed marking guidance.
4.5.4 Moreover, for audit and Contractor development purposes, full justification for scores awarded should be completed in the “Comments” section of the form. This is especially important in instances of sub-standard Contractor performance which may require to be escalated.