Measuring the impact of arts and mentoring interventions with offenders

HMPPS toolkit to measure intermediate outcomes to reduce reoffending from arts and mentoring interventions.

Share This Post

Share on twitter
Share on facebook
Share on linkedin
Share on email

Her Majesty’s Prison & Probation Service has just (22 March 2019) published a useful Toolkit to measure intermediate outcomes to reduce reoffending from arts and mentoring interventions. Authored by Mike Maguire, Emma Disley, Mark Liddle, and Rosie Meek with Nina Burrowes, the main output was a “robust but user-friendly” instrument designed to measure the outcomes of mentoring or arts interventions for offenders. The focus was on ‘intermediate outcomes’, defined as measurable changes in individuals that are directly or indirectly associated with reductions in reoffending. These outcomes may, of course, also produce other social and individual benefits, independent of any demonstrable effect on offending.

A theoretically informed 29-item questionnaire, named the Intermediate Outcomes Measurement Instrument (IOMI), was produced through an iterative process of literature searches, consultation, piloting and analysis. IOMI has not undergone full validity and reliability testing, but the results of initial testing have been encouraging.
The IOMI forms part of a broader toolkit for service providers to assist them in evaluating their work. The toolkit includes: guidance on the administration of IOMI and how it could be used as part of wider evaluations; a data entry tool to facilitate the collection and analysis of IOMI data; and a tool for the collection and presentation of cost data.

While the IOMI was developed for use in arts or mentoring interventions with adult male offenders, it is likely that it will have wider applicability to other intervention or groups. Its main purpose is to provide evidence of offenders’ progress towards desistance, which is not necessarily reflected in standard reoffending rates. This could be particularly valuable for capturing any contribution to desistance made by individual interventions which are not expected to reduce reoffending on their own. You can see a page of the IOMI instrument in the image below.

Context

The preliminary research for the project found strong interest among providers in a tool of this kind, with providers admitting that they relied too much on anecdotal evidence when attempting to convince commissioners of the effectiveness of their work. At the same time, some resistance was encountered, particularly among front-line staff and volunteers, to the idea of introducing ‘measurement tools’. This was largely because of concerns that voluntary agencies often lack the capacity to collect and analyse significant amounts of data. So, while there was clearly an appetite for measurement, there was also a demand for simplicity and succinctness in any evaluation tool.

Validity

The IOMI itself is theoretically informed and is anchored in careful reviews of evidence from research literature, as well as extensive consultation with providers of mentoring and arts programmes. The results of all three stages of testing were highly encouraging. There were strong indications that the instrument has face validity, internal consistency, stability and the potential to reflect change across most dimensions. At the end of the research period, it was concluded that the IOMI is in a form that is likely to require little further amendment, apart from possible changes or additions to the questions in the ‘resilience’ dimension.

The authors, however, do emphasise that an instrument of this kind inevitably requires extensive testing over longer periods of time (including the collection and analysis of reconviction data on those who have completed the questionnaires) before confident claims can be made for its validity and reliability as a measure of psychological change, or for its robustness as a proxy for reoffending.

Potential uses

The authors make clear that the tool is not yet sufficiently validated for it to provide a benchmark to compare provider performance. However, the tool does have considerable potential to be used in the following ways:

  • Alongside other evidence, helping bidders to make a case to commissioners for the effectiveness of particular interventions or practices.
  • Helping organisations improve their services by finding out more about the outcomes that they are producing for clients.
  • Offering an extra resource for research into what works in offender rehabilitation, especially into the effectiveness of small-scale and less mainstream interventions such as arts programmes.

Share This Post

Share on twitter
Share on facebook
Share on linkedin
Share on email

Related posts

Payment by Results
Getting the right outcomes for payment by results

Not only do outcomes therefore have to be accurate and realistic but commissioners need to consider that providers will quite reasonably de-prioritise work which is not governed by an outcome.

Alcohol & Drugs
Drug treatment helps recovery but is not enough on its own

Measuring drug recovery is problematic, to say the least. Recovery from drug dependence is, like desistance from crime, rarely a linear process and typically includes lapse and relapse over many years. Different people choose different recovery goals: some people remain abstinent from all substances for life; others continue to use occasionally; or replace drug dependence with a reliance on alcohol.

Payment by Results
Payment by results – the devil really is in the detail

PbR is simple in theory…

Payment by results is quite a straightforward concept. Its chief attraction lies in its ability to incentivise providers to deliver exactly what a commissioner wants. For example, any PbR contract concerned with reducing reoffending should ensure that organisations receive the biggest payments when they succeed in getting prolific offenders to give up crime. This saves the commissioner – the Ministry of Justice – and the country money and is to the benefit of everyone in society.
However, getting the contract right in practice is proving rather more challenging – indeed, I’ve yet to go to a PbR event where at least one speaker hasn’t said: “The devil is in the detail.”

Leave a Reply

Your email address will not be published.

keep informed

One email every day at noon