Measuring recovery
The biggest challenge of the drug recovery payment by results pilots has been agreeing a simple but robust measurement system to report on whether new approaches to drug treatment are successful in helping problem drug users stay clean and lead constructive and fulfilling lifestyles.
There was an extensive public consultation about what these outcome metrics should be and, even though they have recently been updated, it’s far from clear whether they will work in practice.
The big challenge in developing PbR outcome metrics for drug recovery is that recovery typically takes one – two years or longer to achieve.
It’s not financially realistic to expect providers to work with no payment for this length of time.
For this reason, the drug recovery pilots set interim measures as well as long term outcomes.
But critics say that once interim measures are introduced, providers will merely work towards these like any other targets and the outcome-driven approach which is the whole point of PbR is lost.
A Social Impact Bond approach
Given the difficulty of the challenge of getting recovery outcome metrics right, last week’s report by Social Finance on how recovery from addiction might be measured as part of a social investment contract was very welcome.
Social Finance is an international organisation that champions the use of Social Impact Bonds (SIBs) to fund initiatives to tackle entrenched social problems via a payment by results approach. They organised the SIB which bankrolls the Peterborough Prison ONE Project designed to reduce re-offending by released prisoners.
They commissioned leading addiction experts Professor John Strang and Dr Samantha Gross from the National Addiction Centre to undertake this study.
Outcome metrics and desirable outcomes
The Social Finance report takes a broadly similar view on outcomes to the drug recovery pilots although there is considerable difference in the detail.
It advocates measuring not just treatment progress but “life progress” for a two year period after the start of treatment.
The report advocates metrics covering three main domains:
- Substance use (where they propose measuring both abstinence and continuing levels of use)
- Offending
- Employment
Substance Use
The report proposes measuring reduction in use of the primary substance(s) of dependency compared with an initial baseline taken at the start of treatment.
It proposes reporting intervals at 3,6,12,18 and 24 months and that use should be measured via self-report confirmed by a biological test. (This is a huge issue and one that I will explore in a separate blog post next week.)
It suggests a tiered payment structure:
Getting better – “a reliable reduction in frequency of use” compared to baseline. Measured on an individual basis.
Even better – “a reliable reduction in frequency of use” since last measurement. Measured on an individual basis.
Abstinence – “a verified self report of zero use”. Measured on a cohort basis.
Offending
The report proposes measuring the “reduction in the total number of reconviction events” achieved by a cohort of people in recovery in the two year period from the start of treatment.
In plain English, this means that Social Finance favour a “frequency” approach – paying providers for every offence that drug users did not commit compared to the number of offences they would have been expected to commit.
They suggest that actual reconviction (as measured by the Police National Computer) is compared either to the performance of a comparison cohort or to what would be expected from an aggregated of the Offender Group Reconviction Score.
OGRS (pronounced “Ogres” as in Shrek) are calculated using the number, type and seriousness of previous offences and the age that an individual started offending.
Most criminologists and desistance experts will applaud the decision to opt for a frequency rather than binary measure of re-offending. Binary models – favoured by the Government – mean that payments are only made if an individual does not re-offend. Since most long term offenders tend to give up crime over a period of time, offending less often and less seriously, the binary model is regarded as too crude an instrument to measure real-life progress.
Employment
Again, the report goes for a tiered payment in the domain of employment with a larger payment for finding and sustaining a job and a smaller payment for being “employment ready” which they envisage measuring through sustained voluntary or part-time work.
Tariffs
The report steers away from recommending what level payments should be set at for achieving these outcomes – although you can be assured that Social Finance will have done rigorous calculations on the subject since the tariff is the critical factor in setting up a Social Impact Bond which rewards both providers and investors.
The authors do, however, helpfully lay out key criteria for tariff levels which they recommend should be paid on a staggered basis to reflect positive changes during a treatment journey and should:
- Encourage a long-term perspective
- Provide a more capital efficient model (in other words money comes in earlier and can be recycled)
- Discourage parking by rewarding continual progress
- Reflect the increased difficulty of working with the most chaotic service users who cost the most to the state.
- Be weighted so that they do not simply reflect relative cost savings to government – to incentivise progress to the service provider, not just savings to government.
Conclusion
One of the real merits of this report is that it takes great care to make its decision making process about which outcome metrics to adopt completely transparent.
Different approaches are explored and pros and cons discussed, enabling the reader to take a more informed view on whether to agree or not with the authors’ conclusions.
Social Finance are also very honest that the report is merely a hypothetical starting point to the development of effective outcome metrics.
Their next step is to collect data to form a baseline and determine some of the practical problems of matching data sets and correlating self-reported levels of use with empirical tests.
They freely admit that they expect their thinking to be substantially influenced as they move into real world trials.
