Measuring PbR outcomes – when worlds collide

Commissioning by outcomes is always right, public money should be focused on what we want to achieve – people in work, off drugs, no longer offending (as argued recently on this Blog). But measuring outcomes is difficult which is why, in the past, we have mainly settled for counting activity – the number of sessions delivered, action plans completed etc. instead. Measuring outcomes is particularly difficult in ‘payment by results’ schemes, where it is especially important to investors who carry the financial risk in Social Impact Bonds and other new ways of funding. Most PbR initiatives are large-scale programmes developing innovative approaches which are to be measured against ‘comparator cohorts’ in areas where the programme doesn’t exist. There is a great deal of protracted negotiation between government, providers and the financial middle men (such as Social Finance) to get a measuring system that is robust, accurate and does not take a disproportionate amount of funding to operate.

Just as these negotiations are being finalised for different Ministry of Justice and Youth Justice Board initiatives, there is an unexpected development threatening to de-rail the process. Last month’s riots saw a surge in lawlessness which will cause more than a blip in the statistics for re-offending rates and custodial sentences – the two most measured outcomes for crime-focused PbR schemes. The number of young offenders in custody jumped by more than 5% in a month according to official figures.  Unforeseen real world events threaten to undermine the neat theoretical models developed with actuarial tools. This is not the place to discuss whether the riots should really have been so unexpected, but I would argue they fit very neatly into Nicholas Taleb’s definition of a Black Swan:

Source: Wikipedia

“What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”

The YJB PbR pilot is about incentivising local authorities to reduce the total of (expensive) nights in custody by providing more effective interventions to young offenders. Unfortunately, the scheme starts next month despite the fact that all but one of the pilot areas have experienced large scale rioting. In Birmingham, one of the pilot sites, 28 young offenders are up before the courts for offences that make them very likely to receive custodial sentences according to the Guardian’s incredibly useful and publicly available database. These 28 young people were arrested for offences committed over two nights; in the previous year just 170 young offenders were sentenced to custody in Birmingham. According to the YJB, the PbR scheme will go live on 1 October regardless – the local authorities involved must be thanking their lucky stars that there is an optional break clause half way through the contract. The same difficulty will face all the MoJ pilots including the Justice Reinvestment programme. Because programmes are targeted at high crime areas, it is no surprise that all six pilot sites in this initiative (Croydon, Hackney, Lambeth, Lewisham, Greater Manchester and Southwark) were heavily affected by the riots.

The same concerns affect the largest and longest running PbR scheme – the work programme. What chance of meeting employment targets in a double dip recession? (See Alex Marsh for a detailed analysis).

Of course, we know from past experience of the work programme that when large providers (at least from the private sector) fail to deliver on PbR, contracts are re-negotiated and they get paid anyway as Polly Toynbee argues in some detail.

There are two fundamental principles to PbR outcome measures:

  1. They must be robust and challenging – substantially better than the status quo.
  2. They must be fair and (relatively) uncomplicated.

The riots have put these two criteria on a collision path.




2 Comments

  1. Frank Curran 9th October 2011
  2. Richard Thomson 27th October 2011

Leave a Reply

Select Language
Andrew B apr 16
Interesting drug and alcohol facts April 2016
drug overdose
Drug-related deaths continue to rise
weed on instagran
Using apps to buy and sell drugs
BBC LSD
Can LSD help with mental health issues?
bugsy malone
Our punitive approach to children’s criminal records
CMH FI
Mental health and criminal justice
E=mc2
A better way to count crime
Wooden gallows rope and noose in St Augustine Florida
Global executions at a 25 year high
UV FI
What do service users think of probation changes?
NAO TR1 FI
Transforming Rehabilitation faces big challenges
Female offender article 2
Restorative justice for women offenders
clinks state of sector 2016 FI
Offender voluntary sector faces funding crisis
Knife
Why the rise in recorded violence matters
Clinks rev doors PCC 2016
Police Commissioners and the voluntary sector
asylum seekers
The changing face of people smuggling
violence at work FI
Violence at work
330px-Main_gate_to_the_HM_Prison_Wormwood_Scrubs_in_spring_2013_(2)
Damning Wormwood Scrubs inspection report
Gove FI
The latest on prison reforms
EU prisons
11 things I learnt about prisons in Europe
Mark prison
The voice of probation
Payment by Results
shock
The unexpected consequences of payment by results
sign contract
Getting payment by results contracts right
rolls-royce-fi
Quality assurance in payment by results contracts
Finally Friday Social Media Innovation Why I Tweet
Unlock Youtube
Unlocking offenders’ doors to employment
digital govt
Digital government extends its reach
SocialMediaAndCrime-FI
Social media and crime
Wooden gallows rope and noose in St Augustine Florida
Global executions at a 25 year high
Balaclava girl
Crime accounts for 1.5% of world GDP
crazy laws FI
I’m arresting you for…
cities
The most violent cities in the world

Keep up to date on drugs, crime & PbR

Every new post straight to your inbox