PbR figures for first cohort
Last week (27 October 2017) saw the release of the Transforming Rehabilitation (TR) Payment by Results figures for the October to December 2015 cohort. The overall result was encouraging, and defy the view that TR’s radical changes to probation, and the ensuing problems, would result in increased reoffending – though it is very important that I point out that this is just the first set of results of many and overall judgement should be reserved for at least a year. The reoffending rate for all CRCs was 45.6% compared to a 2011 baseline rate of 47.5%. I had to make some (conservative) assumptions to estimate the baseline rate but I think it is also safe to say that the difference was statistically significant, suggesting reoffending rates have reduced under TR. Please see the the note at the end of this blog to understand better how I completed the analysis.
CRC Performance
The chart below describes each CRC’s reoffending rate in relation to the baseline 2011 rate. The grey line represents the range of reoffending rates that would indicate no change from 2011 (the baseline confidence interval). If the CRC’s rate is outside this range, we are confident in statistically terms to state that the CRC’s performance was either better or worse than the reoffending rate achieved in 2011. The green bars represent the reoffending rates of CRCs that outperformed 2011, the orange bars represent those that performed the same as 2011 and the red bars present those that performed worse than 2011.
Thirteen of the CRCs beat the baseline rate. The best performing CRC was Cumbria and Lancashire, which beat the baseline rate by 8.2% (49.9% to 41.7%). [Editor’s note: Cumbria was the subject of a recent, very positive Probation Inspector’s report.] The nest best was Hampshire and the Isle of Wight which beat the baseline by 5.4% and the third best was Northumbria with a better rate by 4.3%. Two of the CRCs performed worse than the 2011 baseline. Warwickshire and West Mercia recorded a reoffending rate 3% worse than the baseline rate, and South Yorkshire’s rate was 2.8% worse. With most CRCs, however, outperforming the reoffending rate form 2011, the figures are a promising set of results.
Comparing CRC performance
Now that the baseline rates have been published, we can better understand how well each area was performing in 2011 and whether a CRC is now being asked to better good or bad performance achieved in that year. The chart below describes the difference between the actual baseline rate and the 2011 baseline’s OGRS score (in other words their expected rate of reoffending). A negative result in the chart means the area performed better in 2011 than the OGRS score expected.
2011 Baseline Performance
The charts highlights that six of the CRCs are being asked to beat better than expected performance in 2011 (in other words to be better than good). Whereas other CRCs, notably London and Wales, are being asked to outperform potentially poor performance in 2011. It it interesting that South Yorkshire and, Warwickshire & West Mercia – the two areas that recorded poor performance for TR – are being asked to beat good performance from 2011.
Merseyside and Cheshire & Greater Manchester, however, are equally being asked to beat good performance from 2011 and were able to do so for the October to December 2015 cohort. The OGRS score does not allow for area effects, which will exist and could explain the differences between the OGRS score and the baseline rate. It not possible now to conclude whether payment by results will be easier in some areas than others, but, going forward, I will monitor the impact of whether a CRC is being asked to perform better than good or poor performance from 2011 on their ability to achieve payment by results bonuses.
Editor’s Conclusion
I am very grateful for Jack Cattell’s technical skills in analysing the first cohort of offenders supervised under the government’s new private/public probation service created by the Transforming Rehabilitation (TR) initiative. Jack will be following the performance of the new private Community Rehabilitation Companies via the publication of reoffending rates every three months.
Many will be surprised at the apparent good performance of the CRCs (and it is important to remember that reoffending performance is linked to contract remuneration via the TR payment by results contracts).
However, I would like readers to bear a key fact in mind: These figures relate to the first three months of the CRCs operation when, by general consent (confirmed by HMI Probation reports) there was considerable chaos and confusion, as would be expected in moving from public to private ownership. Most CRCs have taken the best part of two years to implement fully their new models of operation.
There are two consequences to this fact:
- It is inappropriate to judge CRCs (either way) on this first set of figures and, in truth, we may have to wait another two years to get a reliable indicator of reoffending performance.
- It forces us to consider whether reoffending rates are a reliable way of measuring probation performance or whether other factors (such as policing) have too great an influence.
Nonetheless, the comparisons between CRCs will certainly be valid over time and I look forward to reading Jack’s analysis of the next set of figures at the end of January 2018.
Notes on analysis
The latest Ministry of Justice bulletin released more data than was previously available and I was able to complete a statistical analysis of the impact of TR. This could only be completed with making conservative assumptions that would make finding a statistically significant result less likely. The following actions were taken:
- I assumed the spread of offenders across CRCs in 2011 was exactly the same as it was in the October to December 2015 cohort. This would not be the case but any analysis would want to weight the two samples so they represented each other so the impact of this assumption is minimal.
- The 2011 sample size was assumed to be the same as that of the October to December 2015 cohort. The 2011 sample will be considerably bigger, so this assumption meant the standard error used for the analysis was larger than it should be.
- A t-test with unequal variances assumed was used to test the difference between the cohort’s and the baseline’s reoffending rate. The t statistic result was 4.6.
