This is the fifth in a series of posts on the National Audit Office’s recent report: Outcome-based payment schemes: government’s use of payment by results. The report’s overall conclusion is that because of the lack of a clear evidence base, the PbR approach is currently laden with risk.
How effectively has government monitored PbR?
The NAO makes the point that it is intrinsic to the definition of outcomes for a PbR scheme that they can be validated in a reliable way. The DWP let the contracts for the Work Programme before it had established a robust way to validate whether individuals using the scheme were in sustainable employment. As a result, DWP made £11 million in invalid payments in the early years of the scheme.
The NAO found that commissioners frequently neglected to put sound performance monitoring systems in place before a PbR scheme starts. They note that the offender rehabilitation pilots were hampered because the MoJ had not established effective data sharing between its prison and probation systems and the police national computer when piloting began, causing expensive temporary workarounds. This was also the case for the drug and alcohol recovery PbR pilot schemes.
A central problem in monitoring PbR schemes is that the true test of whether a service is working is in whether it achieves its outcomes, but outcomes can typically only be measured fully two years or longer into the operation of the contract. The challenge for commissioners is how to have sufficient monitoring data to indicate serious early problems whilst not introducing a whole host of targets and outputs which basically sabotage the purpose of a PbR approach.
It could be argued that there is no need for quality assurance systems because if providers are not delivering a good quality service, they will not achieve their outcomes and not get paid. But in reality, it’s not acceptable for people to receive a sub-standard service over a prolonged period of time.
Of course, if providers do not achieve their outcomes, commissioners are in a strong position. They can then either renegotiate the contract or simply terminate it and give it to another provider. This latter approach was behind the DWP’s thinking in making the Work Programme a competitive exercise with two to three prime providers operating in each of its 18 contract areas. The DWP has taken contracts off low-performing providers and transferred them to high-performing ones twice, in August 2013 and March 2015.
In addition to looking at the effectiveness of individual PbR schemes, the NAO is keen to encourage government to evaluate the overall effectiveness of payment by results as a delivery mechanism. This makes sense since the current evidence base for PbR is very limited.
The NAO’s favoured approach is to measure the impact of PbR commissioned interventions against “counterfactuals” – what happens when the interventions isn’t provided (see my PbR dictionary here). However, this is only really ethically possible for pilot schemes, otherwise using a counterfactual control group effectively denies vulnerable people access to a service.
It is interesting to note that the Public Accounts Committee has criticised the DWP for not using control groups to ascertain whether the Work Programme has any impact beyond what would happen naturally with people looking to find jobs on their own, without a government-funded intervention.
For those of you interested in the technicalities of counterfactuals and control groups, here’s the NAO’s table showing the different sorts of comparators:
The NAO makes a very strong recommendation not only that all government payment by results schemes should be evaluated but that these evaluations should be shared across government in order to develop a more robust evidence base for PbR; indicating whether and how this commissioning model should be utilised.