Measuring and evaluating Payment by Results schemes

This is the final post in a series on the five principles of PbR commissioning set out in a recent Audit Commission report.

Principle 5: Effective and robust measurement and evaluation

The Audit Commission recommends that the measurement and evaluation mechanisms for PbR schemes are set up well in advance of the launch date.

Commissioners need to ensure that good quality data are available to inform payment decisions and a baseline of current performance or need is jointly agreed.

Most topically, the report recommends:

“Openness with data is also important, so that performance is visible.”

This is exactly what the Commons Public Account Committee recommended last week, saying that companies doing business with central government should be subject to the Freedom of Information Act and their performance and contracts should be examined by the government’s spending watchdogs.

Interestingly, the CBI  immediately responded critically, defending what they see as the right of companies to keep commercially sensitive information private.

This is an interesting debate, particularly because one of the driving forces behind PbR schemes is supposedly the desire to discover the most effective approaches to entrenched social problems and then replicate them. It is hard to see how this can happen if no data are made public.

The Commission emphasises that any milestones related to payment should be very carefully aligned with outcomes, although it stops short of recommending against proxy measures.

Unsurprisingly, the report pays considerable attention to the issue of “cherry picking” saying that not only must there be good data about the overall cohort of people at which the PbR scheme is targeted, but there must be a good understanding of subgroups to make sure that the provider does not merely concentrate on the easiest cases.

This is important to make sure that individuals with the most complex or challenging needs (who commonly cause the most damage or inflict the most costs on society) are not ignored.

Disappointingly, though, the report stops short of criticising the Government’s apparent insistence on measuring re-offending through a “binary” approach which actively encourages providers to both “cherry pick” and “park” any individuals who do re-offend for the basic reason that continuing to work with them will not generate income.


The Audit Commission helpfully sets out some of the key parameters of effective approaches to evaluation.

On the one hand, the report suggests that it may be important to develop a classification scheme but cautions against developing a too complex approach. I would also warn the classification schemes can become overly intrusive and damage the integrity of the provision itself; something which I think has happened in a number of the drug and alcohol recovery PbR pilots.

The Commission also advises that it is important to ensure that any comparison cohorts or control groups cover possible developments in the future, rather than being purely historical. For example, if we ever do return to a period of reasonable economic growth, it will be much easier for Work Programme providers to find their service users proper employment. PbR measurement and payment schemes need to allow for such developments (in either direction).

The report advocates an interesting idea by which peer groups can be used to help judge performance in competitive PbR schemes. Again, the Work Programme would be a suitable scheme for this approach since there are a number of different contracts delivered by a number of providers – although see the discussion above on whether the providers would agree to have their performance made public. Nevertheless, the Commission is keen on this approach on the basis that competitive schemes can provide a greater incentive to providers compared to single contracts.

The most important factor in developing an appropriate and reliable measurements mechanism for PbR schemes is a precise definition of the desired outcome. As soon as outcomes start to drift towards outputs and targets, there is a considerable danger of providers “gaming” the system and focusing on maximising their income without regard to what they are achieving in the real world. David Boyle has written a persuasive and  entertaining article on what he sees as the insuperable difficulties of getting this right.

Hidden in some of the complexities of developing an accurate monitoring and payment system is the clear realisation that the more thorough and reliable the evaluation, the more expensive the PbR scheme becomes.

The section concludes that the best evaluation would be a whole-life, whole-system approach which assesses the total cost and benefits to public services.

For instance, a successful scheme that prevented re-offending would save money for the criminal justice system (less police, courts and prison expenditure), and the Department of Work and Pensions as released prisoners found work and came off benefits. It could also improve quality and, potentially, length, of life – although this might incur more costs to the NHS in the long-term.

However, this level of evaluation is well beyond the scope of individual PbR schemes and, in any case, is extremely rarely commissioned in the UK.

And finally

Nonetheless, this Audit Commission report is by far the most useful document around for those thinking of commissioning PbR schemes.

The five key principles discussed in this blog are the right ones and the guidance around them is thoughtful and open-minded.

One of the key strengths of the report is that it approaches the key issues from first principles and does not merely focus on existing practice.

We have become used to this high-quality and extremely practical approach from the Audit Commission over the years – it is a great shame that the Government still intend to disband it.


Leave a Reply

Select Language
Ecstasy is popular again – and much stronger
Innovative practice in alcohol treatment and recovery
ONS drug related deaths overall
Deaths from legal highs
Andrew B apr 16
Interesting drug and alcohol facts April 2016
minimum wage
Higher wages reduce crime more than incarceration
The truth about why people do bad things
bugsy malone
Our punitive approach to children’s criminal records
Mental health and criminal justice
The difficulties of a split probation system
NAO TR performance
Performance of new probation system unclear
11 things I learnt from the 2015 offender statistics
limited competition
Limited competition for new probation market
paper clips
Myth: Only radical reforms can reduce crime
crime headlines
Why do we always think crime is rising?
3d rendering the empty room with open window
Time for a sensible debate on crime?
Close up of a line of British Police do not cross tape.
10 key facts from the latest crime figures
Unlocking prison performance
prison safety JC
How can we make our prisons safer?
Coates review graphic
Unlocking potential – Coates prison education review published
11 things I learnt from the 2015 offender statistics
Payment by Results
evolving tomatoes
The evolution of Social Impact Bonds
Payment by Results and international aid
What we know about payment by results
Finally Friday Social Media Innovation Why I Tweet
The law and social media
Unlock Youtube
Unlocking offenders’ doors to employment
digital govt
Digital government extends its reach
The law and social media
Wooden gallows rope and noose in St Augustine Florida
Global executions at a 25 year high
Balaclava girl
Crime accounts for 1.5% of world GDP
crazy laws FI
I’m arresting you for…

Keep up to date on drugs, crime & PbR

Every new post straight to your inbox