How is the Justice Data Lab doing?
New Philanthropy Capital, or NPC as it is now known, describes itself as a “charity think tank and consultancy” and was the prime architect of the Ministry of Justice’s Justice Data Lab.
The JDL was launched in April 2013 with the purpose of trying to establish the effectiveness of a range of interventions designed to prevent re-offending. To use the service, organisations simply provide details of the offenders who they have worked with and information about the services they have provided. The Lab then supplies aggregate one-year proven re-offending rates for that group, and, most importantly, that of a matched control group of similar offenders.
NPC has just (30 July 2015) published a review of the JDL to date, entitled “Under the microscope: Data, charities and working with offenders.”
How effective is the JDL approach?
The JDL is the only way that non government organisations can access information about whether they have an impact on reoffending without employing professional researchers to negotiate access to Police National Computer data. It is not a perfect solution as NPC points out:
- Reconviction rates do not tell us everything: the journey away from crime is long and complex and organisations can still contribute to it, sometimes significantly, without being able to show their impact in this kind of analysis. This is the nature of desistance.
- Reporting an average re-offending rate for a group of ex-offenders undoubtedly hides a range of successes and failures. For example, projects shown to be effective for some may still be useless or even harmful for others.
- The process of matching the control group cannot account for all factors, particularly when organisations are working with very difficult or complex individuals. For example, the Justice Data Lab is not really appropriate for organisations that target substance misusers because there is no variable on the PNC to match this sample with other substance misusers. Conversely, some organisations may get ‘false positives’ because their service users are less predisposed to re-offend in the first place.
- The laws of statistical reliability mean that organisations that have worked with larger numbers of people are more likely to get a definitive result. The minimum number of service users organisations can submit to the Justice Data Lab is 60, but even at this level, the findings are most likely to be inconclusive.
- The Justice Data Lab cannot answer more detailed questions such as why an intervention failed or worked, or the optimum type of level of intervention in different circumstances.
What have we learnt so far?
The short answer is that we haven’t learnt as much as we had hoped. There have been 125 analyses conducted so far. 29 of these have shown that the service is associated with a reduction in offending, 89 have shown inconclusive results (mainly because samples have been too small), and 7 have shown that services are associated with an increase in re-offending. Where results are positive, reductions in the re-offending rate have been around 1 to 10 percentage points.
Given these limitations, NPC’s further analysis is on fairly thin ground. However, the report does provide an interesting breakdown of intervention by sector, “comparing” the impact of providers from the voluntary, private and public sectors as well as educational institutions:
As you can see, the public sector comes out best, followed by the voluntary sector, even though the sample size is very small. And that is the major worry; when NPC was consulting with voluntary sector organisations working with offenders as part of their work to develop the JDL, there was a lot of enthusiasm for the project. However, fewer than 40 charities have used the Lab. This appears to be for a number of different reasons including:
- Concerns over data protection – some charities worried that they did not have offenders’ consent to share their information with the MoJ.
- Some organisations did not have sufficient information on the people they work with (even though the JDL requirements are pretty minimal).
- For a number of organisations, the JDL wasn’t able to match the the group of offenders they were working with accurately – if you’re working with people with very complex needs including mental health and substance misuse, it’s reasonable to measure your performance within the context of this very high risk (of reoffending) group.
- My sense is that a number of organisations saw the early results and wondered if it was in their best interests to get involved in an initiative which might show that they were ineffective.
This leaves the JDL in a tricky place; it has now been confirmed as a permanent resource designed to inform policy and funding around reoffending. However, if providers continue to vote with their feet and then submit their data, it will be of limited value. I can foresee a situation in the near future where the MoJ links its own funding to participation in the JDL and puts pressure on other funders to do the same.