This is a guest post by Andrew Smith, a Research Associate at the Policy Evaluation and Research Unit where he runs www.reducing-reoffending.uk

Finding new solutions

The need to find solutions to new and enduring challenges has been at the forefront of scientific progress and the improvement of living conditions. This quest has often accelerated during times of unprecedented change, danger or novel problems. Writing during one of the greatest public health challenges in generations, it is clear that the scale of the pandemic is being met by an overwhelming scientific effort to find a vaccine that works. And this effort is largely underpinned by one research method: the randomized controlled trial (RCT).

Whilst medical trials are a familiar example of the use of experimental methods, RCTs are also used in many non-medical fields, including criminal justice, to help policy makers to understand what works. For example, the US National Institute of Justice provides details of nearly 100 programmes that have been deemed ‘effective’ as a result of evidence from trials. This evidence is based on mathematical logic and statistical methods and many policymakers and researchers see trials as being a method par excellence when it comes to being certain about whether or not a programme or intervention works.

Randomised Controlled Trials

But as my colleagues and I point out in a recently published paper, the quantitative basis of RCTs (the very thing that makes them so effective in the quest to understand what works) is also a significant limitation. RCTs tell us ‘what worked’, and not necessarily how it worked. Neither do they tell us how well the programme was implemented, nor whether it will work elsewhere or in the future. To answer these sorts of questions different forms of evidence can be really helpful. That doesn’t mean giving up on RCTs. In other sectors researchers are starting to combine different kinds of evidence much more effectively as part of a ‘multi-method’ approach. So, alongside the RCT they may use qualitative methods such as interviews and focus groups to better understand the mechanisms within an intervention that cause an outcome or the importance of different contextual factors. They might also develop a program theory about how exactly the intervention is expected to work. But when we reviewed recent criminological RCTs we struggled to find the kinds of methodological innovations that are becoming commonplace in sectors such healthcare, education and international development.

The idea behind a RCT or ‘trial’ is quite simple: test some sort of intervention (perhaps a new pharmaceutical drug or a new programme of cognitive behavioural therapy) by administering it to one group of people and comparing the outcome with another group who don’t get the intervention. As long as the groups are large enough, the trial is well conducted, and (importantly!) individuals are randomly assigned to the groups, the difference in outcomes between the groups (e.g. health, educational attainment, reoffending) is wholly and directly attributable to the intervention. Using trials in this way helps researchers, evaluators and policy makers to systematically test interventions in order to understand what works in any given field. This approach has gained considerable traction in government and policymaking circles around the world. In the UK, the government’s nine independent What Works centres and affiliates largely base their approach on commissioning and summarising evidence from RCTs and experimental methods. They cover policy areas such as crime reduction, local economic growth and education, which account for more than £250 billion of public spending.