This is the final post in a series on the five principles of PbR commissioning set out in a recent Audit Commission report.
Principle 5: Effective and robust measurement and evaluation
The Audit Commission recommends that the measurement and evaluation mechanisms for PbR schemes are set up well in advance of the launch date.
Commissioners need to ensure that good quality data are available to inform payment decisions and a baseline of current performance or need is jointly agreed.
Most topically, the report recommends:
“Openness with data is also important, so that performance is visible.”
This is exactly what the Commons Public Account Committee recommended last week, saying that companies doing business with central government should be subject to the Freedom of Information Act and their performance and contracts should be examined by the government’s spending watchdogs.
Interestingly, the CBI immediately responded critically, defending what they see as the right of companies to keep commercially sensitive information private.
This is an interesting debate, particularly because one of the driving forces behind PbR schemes is supposedly the desire to discover the most effective approaches to entrenched social problems and then replicate them. It is hard to see how this can happen if no data are made public.
The Commission emphasises that any milestones related to payment should be very carefully aligned with outcomes, although it stops short of recommending against proxy measures.
Unsurprisingly, the report pays considerable attention to the issue of “cherry picking” saying that not only must there be good data about the overall cohort of people at which the PbR scheme is targeted, but there must be a good understanding of subgroups to make sure that the provider does not merely concentrate on the easiest cases.
This is important to make sure that individuals with the most complex or challenging needs (who commonly cause the most damage or inflict the most costs on society) are not ignored.
Disappointingly, though, the report stops short of criticising the Government’s apparent insistence on measuring re-offending through a “binary” approach which actively encourages providers to both “cherry pick” and “park” any individuals who do re-offend for the basic reason that continuing to work with them will not generate income.
Challenges
The Audit Commission helpfully sets out some of the key parameters of effective approaches to evaluation.
On the one hand, the report suggests that it may be important to develop a classification scheme but cautions against developing a too complex approach. I would also warn the classification schemes can become overly intrusive and damage the integrity of the provision itself; something which I think has happened in a number of the drug and alcohol recovery PbR pilots.
The Commission also advises that it is important to ensure that any comparison cohorts or control groups cover possible developments in the future, rather than being purely historical. For example, if we ever do return to a period of reasonable economic growth, it will be much easier for Work Programme providers to find their service users proper employment. PbR measurement and payment schemes need to allow for such developments (in either direction).
The report advocates an interesting idea by which peer groups can be used to help judge performance in competitive PbR schemes. Again, the Work Programme would be a suitable scheme for this approach since there are a number of different contracts delivered by a number of providers – although see the discussion above on whether the providers would agree to have their performance made public. Nevertheless, the Commission is keen on this approach on the basis that competitive schemes can provide a greater incentive to providers compared to single contracts.
The most important factor in developing an appropriate and reliable measurements mechanism for PbR schemes is a precise definition of the desired outcome. As soon as outcomes start to drift towards outputs and targets, there is a considerable danger of providers “gaming” the system and focusing on maximising their income without regard to what they are achieving in the real world. David Boyle has written a persuasive and entertaining article on what he sees as the insuperable difficulties of getting this right.
Hidden in some of the complexities of developing an accurate monitoring and payment system is the clear realisation that the more thorough and reliable the evaluation, the more expensive the PbR scheme becomes.
The section concludes that the best evaluation would be a whole-life, whole-system approach which assesses the total cost and benefits to public services.
For instance, a successful scheme that prevented re-offending would save money for the criminal justice system (less police, courts and prison expenditure), and the Department of Work and Pensions as released prisoners found work and came off benefits. It could also improve quality and, potentially, length, of life – although this might incur more costs to the NHS in the long-term.
However, this level of evaluation is well beyond the scope of individual PbR schemes and, in any case, is extremely rarely commissioned in the UK.
And finally
Nonetheless, this Audit Commission report is by far the most useful document around for those thinking of commissioning PbR schemes.
The five key principles discussed in this blog are the right ones and the guidance around them is thoughtful and open-minded.
One of the key strengths of the report is that it approaches the key issues from first principles and does not merely focus on existing practice.
We have become used to this high-quality and extremely practical approach from the Audit Commission over the years – it is a great shame that the Government still intend to disband it.
