Measuring PbR outcomes – when worlds collide

Commissioning by outcomes is always right, public money should be focused on what we want to achieve – people in work, off drugs, no longer offending (as argued recently on this Blog). But measuring outcomes is difficult which is why, in the past, we have mainly settled for counting activity – the number of sessions delivered, action plans completed etc. instead. Measuring outcomes is particularly difficult in ‘payment by results’ schemes, where it is especially important to investors who carry the financial risk in Social Impact Bonds and other new ways of funding. Most PbR initiatives are large-scale programmes developing innovative approaches which are to be measured against ‘comparator cohorts’ in areas where the programme doesn’t exist. There is a great deal of protracted negotiation between government, providers and the financial middle men (such as Social Finance) to get a measuring system that is robust, accurate and does not take a disproportionate amount of funding to operate.

Just as these negotiations are being finalised for different Ministry of Justice and Youth Justice Board initiatives, there is an unexpected development threatening to de-rail the process. Last month’s riots saw a surge in lawlessness which will cause more than a blip in the statistics for re-offending rates and custodial sentences – the two most measured outcomes for crime-focused PbR schemes. The number of young offenders in custody jumped by more than 5% in a month according to official figures.  Unforeseen real world events threaten to undermine the neat theoretical models developed with actuarial tools. This is not the place to discuss whether the riots should really have been so unexpected, but I would argue they fit very neatly into Nicholas Taleb’s definition of a Black Swan:

Source: Wikipedia

“What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”

The YJB PbR pilot is about incentivising local authorities to reduce the total of (expensive) nights in custody by providing more effective interventions to young offenders. Unfortunately, the scheme starts next month despite the fact that all but one of the pilot areas have experienced large scale rioting. In Birmingham, one of the pilot sites, 28 young offenders are up before the courts for offences that make them very likely to receive custodial sentences according to the Guardian’s incredibly useful and publicly available database. These 28 young people were arrested for offences committed over two nights; in the previous year just 170 young offenders were sentenced to custody in Birmingham. According to the YJB, the PbR scheme will go live on 1 October regardless – the local authorities involved must be thanking their lucky stars that there is an optional break clause half way through the contract. The same difficulty will face all the MoJ pilots including the Justice Reinvestment programme. Because programmes are targeted at high crime areas, it is no surprise that all six pilot sites in this initiative (Croydon, Hackney, Lambeth, Lewisham, Greater Manchester and Southwark) were heavily affected by the riots.

The same concerns affect the largest and longest running PbR scheme – the work programme. What chance of meeting employment targets in a double dip recession? (See Alex Marsh for a detailed analysis).

Of course, we know from past experience of the work programme that when large providers (at least from the private sector) fail to deliver on PbR, contracts are re-negotiated and they get paid anyway as Polly Toynbee argues in some detail.

There are two fundamental principles to PbR outcome measures:

  1. They must be robust and challenging – substantially better than the status quo.
  2. They must be fair and (relatively) uncomplicated.

The riots have put these two criteria on a collision path.


Leave a Reply

Select Language
smartphone apps
12 Addiction Recovery Apps
Doctors mobilise for UN special session on drugs
drug seizures FI
Drug seizures down
Andrew Brown Nov 2015
Interesting things about alcohol and other drugs (Nov 2015)
Local criminal justice partnerships in disarray
New inquiry into restorative justice
RJ 2015 FI
Celebrate Restorative Justice Week 2015
map for RJ
Help map restorative justice services in England and Wales
Gove recommends outsiders as chief prison/probation inspectors
tyre inspection
New probation still a cause for concern
reoffending infographic FI
Latest re-offending figures sound alarm bells
losing race
What did failed prime bidders think of the probation TR competition?
police sns
Big fall in Stop and Search
drug seizures FI
Drug seizures down
cell door
Details of review into deaths in police custody
met police RSA 2
How to police London in 2020
Comparing the costs of public and private prisons
women of color behind bars
The Truth About Women of Color Behind Bars
Gove recommends outsiders as chief prison/probation inspectors
prison building game
Is building nine new prisons good news?
Payment by Results
Peterborough_prison 2
Peterborough Prison PbR pilot results improving, but still below target
unemployment line
The Work Programme – cheaper but not better
No Preview
Helping people with complex needs to lead less troubled lives
Finally Friday Social Media Innovation Why I Tweet
digital govt
Digital government extends its reach
Social media and crime
What do Apple, Facebook and Google know about you?
women of color behind bars
The Truth About Women of Color Behind Bars
US homicide
Murder rates across the world
college of policing demand FI
What do the police do?
sdp fi
Support, don’t punish women who use drugs

Keep up to date on drugs, crime & PbR

Every new post straight to your inbox