Measuring PbR outcomes – when worlds collide

Commissioning by outcomes is always right, public money should be focused on what we want to achieve – people in work, off drugs, no longer offending (as argued recently on this Blog). But measuring outcomes is difficult which is why, in the past, we have mainly settled for counting activity – the number of sessions delivered, action plans completed etc. instead. Measuring outcomes is particularly difficult in ‘payment by results’ schemes, where it is especially important to investors who carry the financial risk in Social Impact Bonds and other new ways of funding. Most PbR initiatives are large-scale programmes developing innovative approaches which are to be measured against ‘comparator cohorts’ in areas where the programme doesn’t exist. There is a great deal of protracted negotiation between government, providers and the financial middle men (such as Social Finance) to get a measuring system that is robust, accurate and does not take a disproportionate amount of funding to operate.

Just as these negotiations are being finalised for different Ministry of Justice and Youth Justice Board initiatives, there is an unexpected development threatening to de-rail the process. Last month’s riots saw a surge in lawlessness which will cause more than a blip in the statistics for re-offending rates and custodial sentences – the two most measured outcomes for crime-focused PbR schemes. The number of young offenders in custody jumped by more than 5% in a month according to official figures.  Unforeseen real world events threaten to undermine the neat theoretical models developed with actuarial tools. This is not the place to discuss whether the riots should really have been so unexpected, but I would argue they fit very neatly into Nicholas Taleb’s definition of a Black Swan:

Source: Wikipedia

“What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”

The YJB PbR pilot is about incentivising local authorities to reduce the total of (expensive) nights in custody by providing more effective interventions to young offenders. Unfortunately, the scheme starts next month despite the fact that all but one of the pilot areas have experienced large scale rioting. In Birmingham, one of the pilot sites, 28 young offenders are up before the courts for offences that make them very likely to receive custodial sentences according to the Guardian’s incredibly useful and publicly available database. These 28 young people were arrested for offences committed over two nights; in the previous year just 170 young offenders were sentenced to custody in Birmingham. According to the YJB, the PbR scheme will go live on 1 October regardless – the local authorities involved must be thanking their lucky stars that there is an optional break clause half way through the contract. The same difficulty will face all the MoJ pilots including the Justice Reinvestment programme. Because programmes are targeted at high crime areas, it is no surprise that all six pilot sites in this initiative (Croydon, Hackney, Lambeth, Lewisham, Greater Manchester and Southwark) were heavily affected by the riots.

The same concerns affect the largest and longest running PbR scheme – the work programme. What chance of meeting employment targets in a double dip recession? (See Alex Marsh for a detailed analysis).

Of course, we know from past experience of the work programme that when large providers (at least from the private sector) fail to deliver on PbR, contracts are re-negotiated and they get paid anyway as Polly Toynbee argues in some detail.

There are two fundamental principles to PbR outcome measures:

  1. They must be robust and challenging – substantially better than the status quo.
  2. They must be fair and (relatively) uncomplicated.

The riots have put these two criteria on a collision path.

2 Comments

  1. Frank Curran 9th October 2011
  2. Richard Thomson 27th October 2011

Leave a Reply

Select Language
dark web FI
What drugs do people buy off the darknet?
Andrew Brown July 16
Interesting drug and alcohol facts July 2016
The extra mile road sign
The practicalities of desistance and recovery
two-wine-celebration-glasse
Drinking patterns in 2016
EM FI
Latest research on electronic monitoring
gavel court room
Early guilty pleas may force up prison population
reflections
Refreshing your perspective with peer research
women who kill dv
What happens to women who kill their abusers?
RW MoJ
Who are the new team at the Moj?
Liz Truss
Introducing new Justice Secretary Liz Truss
SUI RDA FI
Service user involvement with offenders
panel
Referral orders not achieving their potential
cybercrime mouse trap
Five key things about new crime stats
HOme Affairs prostitution
Should we decriminalise sex work?
paper clips
Myth: Only radical reforms can reduce crime
crime headlines
Why do we always think crime is rising?
PRT older
Older released prisoners set up to fail
PrisonReformLaunch2-v3
Justice Committee launches prison reform inquiry
Prison Officer recruitment
The rehabilitative workforce in future prisons
RW MoJ
Who are the new team at the Moj?
Payment by Results
sumo
Can the new Work Programme succeed?
SIBS SF FI
Social Impact Bonds: The early years
2
Making sense of payment by results
Finally Friday Social Media Innovation Why I Tweet
facebook cannabis
Do digital interventions for cannabis problems work?
AG FI
The law and social media
Unlock Youtube
Unlocking offenders’ doors to employment

Keep up to date on drugs, crime & PbR

Every new post straight to your inbox