This is a guest post by Jack Cattell of Get the Data Ltd.
The context
After one year of payment by results (PbR) findings (part of the Transforming Rehabilitation (TR) programme), the trend across all Community Rehabilitation Companies (CRCs) is that the reoffending rate is down but the frequency of reoffending is up. In other words there are fewer re-offenders, but they are re-offending more (see here). Overall this has led to small increase in the number of reoffences under compared to probation performance in 2011. We also have a good understanding of the limitations of the PbR mechanism and potentially how to improve it. As many of you will know, there have been greater problems with TR than the appropriateness of the PbR mechanism. As such, the Ministry of Justice is redesigning the privatised probation service and will reopen the bidding process for probation services.
This begs the questions, should the PbR measure continue under these reforms, and if so, how can it be improved?
Should payment by results continue?
My short answer to this important question is “not yet”, and there are three things that led me to this conclusion.
- Stable reoffending rates
Proven reoffending rates are, in general, stubbornly stable. For example, Figure 1 describes the reoffending rate for adult custodial sentences and court orders since 2011 (the TR PbR baseline year).
Figure 1: The Proven Reoffending Rate of All Adult Custodial Sentences and Court Orders Between Jan 2011 and Sep 2016.
Source: Ministry of Justice, Proven Reoffending Statistics Quarterly Jul 2016 to Sep 2016
You can see that it has stayed close to 40% throughout the period. This is despite the inevitable disruption caused by the introduction of TR from mid-2014 and the implementation of significant cuts to the police budget in England & Wales. The overall reoffending rate of all offenders has remained at approximately 29% since 2005 (see here). And this is despite the moves to evidence based interventions since that time and the targeting of greater resources to those at high risk of reoffending. Of course, stubborn reoffending rates were the stated impetus for the TR programme, although the need for public service cuts must also have been a driver. As such we can only conclude nothing thus far has significantly reduced reoffending and perhaps we should keep trying to find something that will bring about such a change.
- Prioritisation of harm management
The National Offender Management Model (started in 2005)[1] asked the probation service to balance likelihood of reoffending and risk of harm when making resourcing decisions. When I conducted the Community Cohort Study[2], I concluded that the political imperative to manage of risk of harm often trumped the policy to reduce reoffending, and this potentially led to a culture more concerned with harm management than reducing reoffending. Events such as the Sonnex case[3] make this understandable but the importance of laying an audit trail to satisfy an inspector should not be at the expense of taking actions designed to change an offender’s behaviour.[4]
- Creating a rehabilitation culture
I see that the (understandable) preference for managing risk of harm is still seen in the Inspectorate’s reports. The 2017 annual report concluded the CRCs were failing on managing risk of harm, but shouldn’t they prioritise rehabilitation actions because PbR incentivises them to do so? The tension between harm management and rehabilitation is a real one, and the prioritisation of the latter will require a change in culture and new ways of working. I am sure that such changes will be seen by organisations as HMIP to be inherently risky, but anecdotal evidence that I picked up on is that the senior probation management take the reoffending rate more seriously than in the past. This does not mean that reoffending was unimportant to probation trusts, but it has been given more urgency since the advent of TR. The wider context to the implementation of TR might have meant that the CRCs have not reduced reoffending but it feels that for now the best option is to refine the PbR mechanism rather than throwing it away. If no reductions in reoffending come from an improved measure, then it will be time to think again.
How should payment by results operate?
I think it is fair to say that the jury is still out on whether a meaningful reoffending PbR mechanism can be designed for probation. The analyses I have completed and the blogs I have written on this website have identified the following limitations of PbR:
- OGRS is good, but we need to capture more differences between offenders.
The PbR binary results are adjusted to take account of differences in likelihood of reoffending using the OGRS score. Probation caseloads vary quarter to quarter, and year to year on factors are not captured in the score, such as the number with a drug addiction. However, these are important factors to understand why someone reoffends and these other factors might mean the score has less validity than when it was designed. It is also means two cohorts could have the same OGRS score but very different reasons behind their reoffending so you are not comparing like with like.
- We now know a 2011 baseline is too long ago
A lot has happened since 2011. I think it is fair to say that the context in which the CRCs are operating is different (evidence by the reoffending macro trends) and this undermines the validity of the PbR measure. It appears the Ministry of Justice agrees with this and it has chosen a more recent baseline for the frequency of reoffending measure.
- Frequency rates must also capture differences in offenders
Unlike the binary rate of reoffending, the frequency of reoffending rate is not adjusted for the likelihood of reoffending that a caseload presents. Analysis I have completed for CRCs shows that the caseloads are different today than in 2011 on factors associated with an increased numbers of offences, so such an adjustment is required in the PbR mechanism for it to be valid.
So what can be done instead? Rather than explaining detailed formulae etc, I present below ideas that help overcome these limitations.
More recent baseline: An easy way to improve the PbR measure would be to choose a baseline year much close to the start of the PbR contract -i.e. the proceeding year. This will mean the context of the baseline and of the PbR years will be similar and differences in reoffending should be less affected by changes caused by other agencies such as police activity. However, these problems will still exist under this arrangement.
Beat the average: My preferred new approach would be to compare a CRC’s performance against the average of all other CRCs. If the CRC has reduced reoffending significantly they can be paid a bonus. Offenders can be matched on OGRS criteria and other factors such as whether in accommodation to control for differences in cohort composition. This has the bonus of making PbR consistent with the impact evaluation designs the Ministry of Justice uses, for example the Justice Data Lab. This approach would only fail in the (depressingly) very unlikely event that all the CRCs reduce reoffending and I think it is safe to proceed.
[1] https://www.swmcrc.co.uk/wp-content/uploads/2010/05/offender_management_model_1.pdf
[2] https://www.gov.uk/government/collections/the-offender-management-community-cohort-study
[3] https://www.theguardian.com/uk/2009/jun/04/french-student-murder-sonnex-probation
[4] Admittedly, this statement is based on my experience
3 Responses
PBR is a failure and should be applied, also Probation cannot function as a business.
I have written about why I am sceptical that prison or probation services can make a significant impact on the reconviction rate, big enough to distinguish clearly from the rise and fall of the rate by a few per cent over time for no known reason, which that is about the size of effect that the services might realistically hope to have, on our current knowledge of ‘what works’ Given this, and performance to date, it seems to me illogical to go on pinning faith in PBR. There seem so many other more important things to worry about in contracts, such as the extent of offender/supervisor contact, ensuring adequate and timely information flows, that known criminogenic needs such as drug programmes and employment are met as far as possible. After all – it isnt that we dont know ‘wat works’ in reducing reoffending – we do – we just dont do it. Contracts should require providers to do it! Alongside that, PBR seems to me magical thinking – that somehow providers will discover some panacea that all the criminologists have missed all these years, if you just offer them a little bit more cash.
Thanks Julian. I agree that reoffending rates aren’t sufficiently linked to probation activity to make sense; still think we need an incentive to drive desistance work though; HMIP suggests there’s not much of that happening in CRCs or NPS.
Best Wishes
Russell