Commissioning by outcomes is always right, public money should be focused on what we want to achieve – people in work, off drugs, no longer offending (as argued recently on this Blog). But measuring outcomes is difficult which is why, in the past, we have mainly settled for counting activity – the number of sessions delivered, action plans completed etc. instead. Measuring outcomes is particularly difficult in ‘payment by results’ schemes, where it is especially important to investors who carry the financial risk in Social Impact Bonds and other new ways of funding. Most PbR initiatives are large-scale programmes developing innovative approaches which are to be measured against ‘comparator cohorts’ in areas where the programme doesn’t exist. There is a great deal of protracted negotiation between government, providers and the financial middle men (such as Social Finance) to get a measuring system that is robust, accurate and does not take a disproportionate amount of funding to operate.
Just as these negotiations are being finalised for different Ministry of Justice and Youth Justice Board initiatives, there is an unexpected development threatening to de-rail the process. Last month’s riots saw a surge in lawlessness which will cause more than a blip in the statistics for re-offending rates and custodial sentences – the two most measured outcomes for crime-focused PbR schemes. The number of young offenders in custody jumped by more than 5% in a month according to official figures. Unforeseen real world events threaten to undermine the neat theoretical models developed with actuarial tools. This is not the place to discuss whether the riots should really have been so unexpected, but I would argue they fit very neatly into Nicholas Taleb’s definition of a Black Swan:
“What we call here a Black Swan (and capitalize it) is an event with the following three attributes. First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”
The YJB PbR pilot is about incentivising local authorities to reduce the total of (expensive) nights in custody by providing more effective interventions to young offenders. Unfortunately, the scheme starts next month despite the fact that all but one of the pilot areas have experienced large scale rioting. In Birmingham, one of the pilot sites, 28 young offenders are up before the courts for offences that make them very likely to receive custodial sentences according to the Guardian’s incredibly useful and publicly available database. These 28 young people were arrested for offences committed over two nights; in the previous year just 170 young offenders were sentenced to custody in Birmingham. According to the YJB, the PbR scheme will go live on 1 October regardless – the local authorities involved must be thanking their lucky stars that there is an optional break clause half way through the contract. The same difficulty will face all the MoJ pilots including the Justice Reinvestment programme. Because programmes are targeted at high crime areas, it is no surprise that all six pilot sites in this initiative (Croydon, Hackney, Lambeth, Lewisham, Greater Manchester and Southwark) were heavily affected by the riots.
The same concerns affect the largest and longest running PbR scheme – the work programme. What chance of meeting employment targets in a double dip recession? (See Alex Marsh for a detailed analysis).
Of course, we know from past experience of the work programme that when large providers (at least from the private sector) fail to deliver on PbR, contracts are re-negotiated and they get paid anyway as Polly Toynbee argues in some detail.
There are two fundamental principles to PbR outcome measures:
- They must be robust and challenging – substantially better than the status quo.
- They must be fair and (relatively) uncomplicated.
The riots have put these two criteria on a collision path.