The Justice Data Lab

The Justice Data Lab (JDL) was launched almost three years ago in April 2013 with the intention of allowing any organisation that works with offenders to find out the impact of their service on reoffending.

To use the service, organisations simply supply the data lab with details of the offenders who they have worked with and information about the services they have provided.

The justice data lab then supplies aggregate one-year proven re-offending rates for that group, and, most importantly,  that of a matched control group of similar offenders.

[divider]

Limitations

Many voluntary sector providers working with offenders were initially very excited about the prospect of proving the value of their work.  The JDL is the only way that non government organisations can access information about whether they have an impact on reoffending without employing professional researchers to negotiate access to Police National Computer data.

However, it is not a perfect solution because, as a review of the JDL by New Philanthropy Capital (NPC) pointed out:

• Reconviction rates do not tell us everything: the journey away from crime is long and complex and organisations can still contribute to it, sometimes significantly, without being able to show their impact in this kind of analysis. This is the nature of desistance.
• Reporting an average re-offending rate for a group of ex-offenders undoubtedly hides a range of successes and failures. For example, projects shown to be effective for some may still be useless or even harmful for others.
• The process of matching the control group cannot account for all factors, particularly when organisations are working with very difficult or complex individuals. For example, the Justice Data Lab is not really appropriate for organisations that target substance misusers because there is no variable on the PNC to match this sample with other substance misusers. Conversely, some organisations may get ‘false positives’ because their service users are less predisposed to re-offend in the first place.
• The laws of statistical reliability mean that organisations that have worked with larger numbers of people are more likely to get a definitive result. The minimum number of service users organisations can submit to the Justice Data Lab is 60, but even at this level, the findings are most likely to be inconclusive.
• The Justice Data Lab cannot answer more detailed questions such as why an intervention failed or worked, or the optimum type of level of intervention in different circumstances.

Perhaps the most important limitation of the JDL to date is that it has been under-used. There have been just 141 analyses conducted so far, an average of less than five per month, and almost two thirds of these produced inconclusive results. In fact only 32 different voluntary sector organisations have submitted data to the JDL.

There seem to be four main factors that put off initially enthusiastic voluntary sector providers:

1. Concerns over data protection – some charities worried that they did not have offenders’ consent to share their information with the MoJ.
2. Some organisations did not have sufficient information on the people they work with (even though the JDL requirements are pretty minimal).
3. My sense is that a number of organisations saw the early results and wondered if it was in their best interests to get involved in an initiative which might show that they were ineffective.
4. For a number of organisations, the JDL wasn’t able to match the the group of offenders they were working with accurately – if you’re working with people with very complex needs including mental health and substance misuse, it’s reasonable to measure your performance within the context of this very high risk (of reoffending) group.

[divider]

Overcoming limitations

The MoJ has now made substantial progress in overcoming the last of these limitations.

The latest report from the JDL (findings are published on the second Thursday of the month), on re-offending rates for the Langley House Trust is the first to use Offender Assessment System (OASys) data to calculate a more accurate comparison group.

OASys is a probation assessment tool used to understand the risks and needs of offenders linked to their offending. Until now, organisations working with offenders with complex needs couldn’t use the Justice Data Lab, because there was no data from which to derive a suitable comparison group with similarly complex characteristics.

If your organisation works with drug using offenders with very high reoffending rates, it is not fair for your performance to be compared to the reoffending rate of another group of offenders of a similar age and number of previous offences but who aren’t driven to offend on a daily basis by the need to get funds to buy drugs. The inclusion of the OASys data to a great extent overcomes this problem and so makes the assessment of impact more accurate. (I say “to a great extent” because, in my experience, OASys is more accurate on issues such as alcohol and drug use or homelessness than it is, for example, on mental health issues.

In the case of the Langley House Trust, the link to OASys data enabled the MoJ to provide a more detailed picture of the profile of its service users, according to needs such as housing, substance and alcohol use. The impact analysis showed that Langley House Trust’s work is associated with statistically significant reductions in re-offending, which reflected earlier reports from the Justice Data Lab, only this time with more confidence about the comparison group used.

I do hope that more voluntary sector organisations will now feel confident enough in the JDL’s methodology to use what is an excellent opportunity to test the effectiveness of their work at no cost.