Puzzling Performance? Unpicking TR Results after 6 months
At the end of January, the Ministry of Justice published the final results for a second cohort (January to March 2016), making 6 months of results available altogether. In this blog I am pleased to follow-up my initial analyses of the first payment by results figures. In that earlier piece I counselled that it was too early to come to any definitive conclusions, and in this article I am using careful analyses of the evidence to try and explain some of the seemingly puzzling results and to indicate where policy makers and practitioners might usefully direct their attention.
It was widely reported in the national press that the overall results are a blow for the government, with only 2 CRCs in line for bonus payments. But are they failing to cut reoffending? I found from the MoJ data that we would have expected, based on 2011 performance, there to be 25,375 re-offenders across the CRCs but there were in fact 24,218 (a drop of 5%).[1] In contrast, however, we would have expected 105,759 re-offences when in fact 112,284 were recorded across the CRCs (an increase of 6%).[3]
The results are caused by the two main measures of reoffending moving in opposite directions. For the two completed cohorts, the overall reoffending rate was 45% compared to the baseline rate of 47%. In contrast, the average number of re-offences across the CRCs was 4.6 compared to an expected rate of 4.2 – this represents an 11% increase.
Both results are explored in more detail below.
Reoffending rate[4]
The reoffending rate was lower than the 2011 baseline rate at all but three CRCs. At twelve of these CRCs the reduction was large enough for bonus payments to be paid (indicated by the green bars in the chart below). Cumbria & Lancashire’s reoffending rate was a substantial 7.9% lower, and Northumbria and Hampshire & Isle of Wight both recorded average reductions of over 5%. Staffordshire and West Midlands, South Yorkshire and Warwickshire, all recorded increases in reoffending compared to the baseline rate. None of these increases, however, were sufficient to trigger fines. Both South Yorkshire and Warwickshire & West Mercia were in danger of being fined, so performance has improved. The reoffending rate in both was approximately 3% lower in the January to March 2016 cohort than in the October to December 2015 cohort.
Frequency of reoffending
All but two of the CRCs’ frequency of reoffending rates are worse than the 2011 baseline. The largest increase is at Durham Tees Valley where re-offenders on average committed 6.3 offences compared to 4.7 at the baseline. The increase at South Yorkshire was similarly high (5.4 now compared to 4.0 at the baseline). The two CRCs where the frequency rate decreased were Merseyside (4.5 offences to 3.9) and Northumbria (5.15 to 5). Therefore, the Ministry of Justice state that only these two CRCs are on track to receive PbR bonus payments.
Why the difference?
The fact that the PbR results are moving in two opposite directions is puzzling. The reduction in the binary reoffending rate is of course a welcome change, whereas the frequency of reoffending increasing perhaps signifies more entrenched, prolific offenders which is a worry. An important question all of the CRCs are probably asking themselves is: why is this happening? Four potential explanations come to mind.
- The simplest explanation would be that the CRCs are causing the changes. The senior management teams in each CRC might have focused on reducing binary rates of re-offending and gave less attention to prolific offenders. CRCs should analyse their performance data to understand if they are prioritising resources inversely to how prolific an offender is expected to be.
- Second, feedback on my first blog of the TR reoffending rates suggested the observed reduction in reoffending rates was due to police cuts (the police positive outcome rate is going down for example). So how can the police increase the frequency of re-offending? We know that the police are charging fewer offenders and the crimes they commit are more likely to be harmful than in the past. This might signal that the police are prioritising harmful and prolific offenders and infrequent offenders of low harm offences are not being picked up.
- Third, a problem with the PbR frequency measure cannot be discounted. The measure excludes non-offenders so reducing the binary rate of re-offending could hurt you on the frequency rate. In the case of Cumbria & Lancashire, the binary rate of reoffending reduced by 7.9%. It would be safe to assume that reductions in reoffending are more likely to come from offenders expected to reoffend infrequently. The CRC’s 9% increase in frequency of reoffending would have been partly caused by the reduction in binary reoffending, assuming the remaining re-offenders’ behaviours did not change substantially.
- The final reason would be that the profile of the offenders has changed, with prolific offenders representing a greater proportion of the cohort. Due to a combination of multiple factors – e.g. fewer people are receiving a caution or court conviction than in 2011, fewer young people are entering the CJS, changing police behaviours – we might expect fewer people would present to court but those that do are more entrenched in their offending and require greater rehabilitative support.
Conclusion
If I had to choose a reason why we are seeing these results, I would focus my energy on understanding how the profile of offenders has changed since 2011. Many of the macro trends point to a change in the types of offenders presenting at court and my experience suggests that probation services are unlikely to cause a large increase in frequency of reoffending on their own. There is opposite evidence (the overall OGRS score is lower now than in 2011) and some CRCs have been criticised for their poor performance. I think this chart describes best the great variability in performance:
The chart describes the proportional increase or decrease in re-offences compared to expected number (based on 2011 performance). Three distinct groups emerge: the CRCs doing better than expected led by Merseyside, the middle group that are performing similarly to or a bit worse than 2011 and the seven CRCs where the number of re-offences is 18% or more higher than the expected number. A CRC should therefore investigate how its performance is interacting with the macro trends and whether it is making things better or worse. This could also provide the evidence to challenge the MoJ’s frequency of reoffending measure that does not make an adjustment for a changing profile of offenders.
[1] The expected number of reoffenders was calculated by: OGRS Adjusted Baseline Reoffending Rate x Number of offenders
[3] The expected number of re-offences was calculated by: Expected number of reoffenders x Baseline Re-offence Frequency
[4] The results I derived from the published cohort data are very slightly different to the MoJ’s published figures for the combined cohorts. I assume this is because of rounding to two decimals places in the reoffending rates and OGRS scores.
4 Responses
The analysis is most interesting. Could a further analysis be made in addition to the four points mooted, namely that re-offending is linked to ” deviant opportunities” and that the greater such “opportunities” are presented the greater the chance of re-offending. Like may factors influencing behaviour “opportunity” unfortunately,cannot be statistically measured. If this “opportunity” hypothesis stands, then time with offenders assiting them to grasp positive “opportunities” and avoid the opposite might be well worthwhile in reducing the re-offending rate
Indeed, Howard, I am with you on the time with offenders enabling a grasping of positive opportunities and avoiding negative: although I would say pro-social and anti-social. But time seems in short supply, despite the day and night cycle still being about 24 hours. Time with anyone looking to change behaviour patterns (including the motivating them so to do), generating alternative options and developing the incentive to grasp a different choice is key, it seems to me.
Thank you Jack for this response to the press reports, which I have been looking for; I will re-read and digest a little more! Meanwhile, might a longer repeat of the short-lived use of the European Foundation for Quality Measurement (EFQM) organisational assessment approach be useful? And what are the comparative figures for the work of the Probation element of the National Prison and Probation Service?
Hi Steve
I will look to produce a future blog on a CRC to NPS comparison (and see if the trends are similar). I am not familiar with the EFQM. it might form the basis for reviews of CRC performance and will look into it.