Commissioning by outcomes is always right, public money should be focused on what we want to achieve – people in work, off drugs, no longer offending (as argued recently on this Blog). But measuring outcomes is difficult which is why, in the past, we have mainly settled for counting activity – the number of sessions delivered, action plans completed etc. instead. Measuring outcomes is particularly difficult in ‘payment by results’ schemes, where it is especially important to investors who carry the financial risk in Social Impact Bonds and other new ways of funding. Most PbR initiatives are large-scale programmes developing innovative approaches which are to be measured against ‘comparator cohorts’ in areas where the programme doesn’t exist. There is a great deal of protracted negotiation between government, providers and the financial middle men (such as Social Finance) to get a measuring system that is robust, accurate and does not take a disproportionate amount of funding to operate.
Just as these negotiations are being finalised for different Ministry of Justice and Youth Justice Board initiatives, there is an unexpected development threatening to de-rail the process. Last month’s riots saw a surge in lawlessness which will cause more than a blip in the statistics for re-offending rates and custodial sentences – the two most measured outcomes for crime-focused PbR schemes. The number of young offenders in custody jumped by more than 5% in a month according to official figures. Unforeseen real world events threaten to undermine the neat theoretical models developed with actuarial tools. This is not the place to discuss whether the riots should really have been so unexpected, but I would argue they fit very neatly into Nicholas Taleb’s definition of a Black Swan:
“What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”
The YJB PbR pilot is about incentivising local authorities to reduce the total of (expensive) nights in custody by providing more effective interventions to young offenders. Unfortunately, the scheme starts next month despite the fact that all but one of the pilot areas have experienced large scale rioting. In Birmingham, one of the pilot sites, 28 young offenders are up before the courts for offences that make them very likely to receive custodial sentences according to the Guardian’s incredibly useful and publicly available database. These 28 young people were arrested for offences committed over two nights; in the previous year just 170 young offenders were sentenced to custody in Birmingham. According to the YJB, the PbR scheme will go live on 1 October regardless – the local authorities involved must be thanking their lucky stars that there is an optional break clause half way through the contract. The same difficulty will face all the MoJ pilots including the Justice Reinvestment programme. Because programmes are targeted at high crime areas, it is no surprise that all six pilot sites in this initiative (Croydon, Hackney, Lambeth, Lewisham, Greater Manchester and Southwark) were heavily affected by the riots.
The same concerns affect the largest and longest running PbR scheme – the work programme. What chance of meeting employment targets in a double dip recession? (See Alex Marsh for a detailed analysis).
Of course, we know from past experience of the work programme that when large providers (at least from the private sector) fail to deliver on PbR, contracts are re-negotiated and they get paid anyway as Polly Toynbee argues in some detail.
There are two fundamental principles to PbR outcome measures:
- They must be robust and challenging – substantially better than the status quo.
- They must be fair and (relatively) uncomplicated.
The riots have put these two criteria on a collision path.
2 Responses
There are two potential impacts to the riots in relation to the YJB and MoJ JR pilots:
1. The short term distortion of the greater number of people being charged for offences
2. Longer term impacts on conviction outcomes because of potential changes in sentencing policy towards certain categories of offences committed after the riot period
Both pilots base their payment metrics on comparison against a baseline of previous year’s performance. In this case the first issue seems to me to be relatively straightforward to deal with: all convictions relating to offences committed on days on which riots occurred should be discounted along with the convictions for the corresponding dates in the baseline year. The second issue is more complex to deal with but, as the evidence for it is uncertain and anecdotal, perhaps we do not need to worry about it too much?
PBR is a great idea but unfortunately the methodology to make it statistically valid is unworkable. A vast range of indicators change the probability of re-offending. Here’s just a few biggies. Marital status, Postcode, family relationships, nature of criminal history, likelihood of being caught, speed of CJ system from offence to prosecution which is variable. etc etc etc etc etc. It is impossible to make a valid control group! While I’ve not seen the methodology to be fair, I’m convinced that if cherry picking prisoners can be managed out (and I’m not convinced it will be), that the resource deployed in the data collection and analysis will render services inefficient. Subsequently any valid statistical evidence will represent a fluke of bad or good results.