This is the third in my series of 10 commandments for payment by results.
Thy metrics shall be simple
In the last post in this series (Thy outcomes shall be few), I cautioned against payment by results schemes that set too many outcomes.
I argued that having a range of outcomes makes for a very complex commissioning system which ends up sucking a disproportionate amount of commissioner, and critically, provider time into monitoring rather than delivery. [You can read @AylesburyRisk recent post on complexity and Pbr]
Complexity is almost always the enemy of effective PbR because it inevitably results in unintended consequences.
Concern about possible unintended consequences results in worries about gaming the system which, equally inevitably, results in additional measures being taken to address these concerns which involves further complexity which…, well, you get the picture.
In this post, I want to make the case for simple, robust measures to assess how effective PbR schemes are in meeting their outcomes.
Complexity = expense
It is the experience of the drug recovery pilots which brought home to me the danger of complicated metrics.
The more complex the metrics, the more expensive they are to collect and validate.
PbR is supposed to be all about removing bureaucracy and promoting innovation (see the seventh Commandment).
The drug recovery metrics are so complex as they try to measure abstinence, treatment completion, injecting behaviour, accommodation problems, quality of life (and four more outcomes) that the payment system itself adversely affects service delivery.
One of the critical success factors in effective drug treatment is to engage with people seeking help as quickly as possible – because those caught up in addiction are often ambivalent and scared about change.
Therefore, a positive first contact with the treatment system is vital.
Unfortunately, in most of the drug recovery PbR pilots, the first contact is typically not with the treatment service but with a third party which is only assessing people to place them in the correct “complexity group” (their term) to set a payment tariff.
Binary v Frequency (again)
There has been a long-running debate in criminal justice circles about whether it is more appropriate to measure reoffending on a “binary” (simply whether an individual offender commits a further offence) or “frequency” (the total number of crimes committed by a group of offenders) basis.
Interestingly, the MoJ is currently proposing that the reducing reoffending contracts will have separate measures for both binary and frequency reoffending targets.
Each of these targets are linked to separate performance payments for successful providers.
However, providers will only get paid for meeting frequency targets if they meet their binary ones.
This is a classic case of complexity potentially leading to unintended consequences.
This binary “hurdle” could make for an iniquitous payment system – consider this scenario:
- Let’s assume a baseline position of a cohort of 1,000 offenders of whom 600 are expected to reoffend a total of 1400 times.
- The MoJ sets a target of 550 or fewer offenders to re-offend a total of 1200 times or less.
- Provider A gets paid for 550 offenders committing 1200 crimes.
- Provider B gets no payment for 551 offenders committing 1100 crimes.
The complications in this payment system could easily encourage providers to focus on those offenders who are at lower risk of reoffending in order to maximise their performance against the binary target.
The strength of good PbR commissioning is its alignment of commissioner and provider objectives.
A well-written PbR reoffending contract will incentivise providers to prioritise their work with the most persistent and prolific offenders who cause the most damage to local communities and incur the greatest costs to the public purse.
In my view, the current MoJ proposals risk achieving the opposite effect.
So, when setting your outcome metrics, remember the KISS approach:
KEEP IT SIMPLE, STUPID.
The next post will look at the Fourth Commandment of PbR:
“Thou shall not covet milestones”
Browse my comprehensive resource pack on Payment by Results.
Check out my Marmite Infographic for simple explanation of key PbR jargon.
One Response
You have identified a key conceptual flaw with PbR – that it requires simplicity in order to function half-way effectively.
But the social problems it is addressing are inherently complex. If you simply the process of outcome measurement, then you are not measuring the (inherently complex) impact of activity as its felt in people’s lives. You are simply measuring what is easily measurable. This means that providers are being paid against metrics that do not relate to improvements in people’s lives, they’re paid against what is measurable.
This is why all PbR creates perverse incentives, because it predefines what must be achieved for each person, rather than enabling the person themselves to specify what their desired outcome is.
So, for example, it doesn’t matter to a Work Programme provider that forcing someone into an unsuitable job may not be the best thing for themselves or their family, that’s what that provider will do. They’re not being paid to understand and address the needs of the client, they’re being paid to hit predefined (simplified) metrics.