This is the fifth in a series of posts on the National Audit Office’s recent report: Outcome-based payment schemes: government’s use of payment by results. The report’s overall conclusion is that because of the lack of a clear evidence base, the PbR approach is currently laden with risk.
How effectively has government monitored PbR?
The NAO makes the point that it is intrinsic to the definition of outcomes for a PbR scheme that they can be validated in a reliable way. The DWP let the contracts for the Work Programme before it had established a robust way to validate whether individuals using the scheme were in sustainable employment. As a result, DWP made £11 million in invalid payments in the early years of the scheme.
The NAO found that commissioners frequently neglected to put sound performance monitoring systems in place before a PbR scheme starts. They note that the offender rehabilitation pilots were hampered because the MoJ had not established effective data sharing between its prison and probation systems and the police national computer when piloting began, causing expensive temporary workarounds. This was also the case for the drug and alcohol recovery PbR pilot schemes.
A central problem in monitoring PbR schemes is that the true test of whether a service is working is in whether it achieves its outcomes, but outcomes can typically only be measured fully two years or longer into the operation of the contract. The challenge for commissioners is how to have sufficient monitoring data to indicate serious early problems whilst not introducing a whole host of targets and outputs which basically sabotage the purpose of a PbR approach.
It could be argued that there is no need for quality assurance systems because if providers are not delivering a good quality service, they will not achieve their outcomes and not get paid. But in reality, it’s not acceptable for people to receive a sub-standard service over a prolonged period of time.
Of course, if providers do not achieve their outcomes, commissioners are in a strong position. They can then either renegotiate the contract or simply terminate it and give it to another provider. This latter approach was behind the DWP’s thinking in making the Work Programme a competitive exercise with two to three prime providers operating in each of its 18 contract areas. The DWP has taken contracts off low-performing providers and transferred them to high-performing ones twice, in August 2013 and March 2015.
[divider]
Evaluating effectiveness
In addition to looking at the effectiveness of individual PbR schemes, the NAO is keen to encourage government to evaluate the overall effectiveness of payment by results as a delivery mechanism. This makes sense since the current evidence base for PbR is very limited.
The NAO’s favoured approach is to measure the impact of PbR commissioned interventions against “counterfactuals” – what happens when the interventions isn’t provided (see my PbR dictionary here). However, this is only really ethically possible for pilot schemes, otherwise using a counterfactual control group effectively denies vulnerable people access to a service.
It is interesting to note that the Public Accounts Committee has criticised the DWP for not using control groups to ascertain whether the Work Programme has any impact beyond what would happen naturally with people looking to find jobs on their own, without a government-funded intervention.
For those of you interested in the technicalities of counterfactuals and control groups, here’s the NAO’s table showing the different sorts of comparators:
Conclusion
The NAO makes a very strong recommendation not only that all government payment by results schemes should be evaluated but that these evaluations should be shared across government in order to develop a more robust evidence base for PbR; indicating whether and how this commissioning model should be utilised.
2 Responses
There is also a question of what service if being provided by the contracted out service. The ‘black box’ approach of the Work Programme, for example, assumes a medical model – that the provider is ‘treating’ its clients like doctors treat patients. Yet, finding work is down to the behaviour of the individual – the effectiveness of their jobsearch – not the treatment they receive from the provider.
Therefore, the service provided by the provider should be about changing the behaviour of the client – maximising the effectiveness of the jobsearch of all the clients not getting them a job. That would suggest a franchise model rather than ‘black box’ contracting out. Without measures to determine the level of service for each individual (e.g. fortnightly signing to ensure they are looking effectively for work) then the problem of ‘creaming and parking’ will always be there.
So, obviously, the key to effective PBR is determining what exactly the people are being paid for. In most cases, including the Work Programme I think that this requires, at least, a minimum level of service for each client/customer. There is also a question – difficult for providers to deliver – of how providers can enforce the requirements that they agreed to undertake.
Thanks for your comment, Bill. Personally, I think that there’s little point in PbR without a black box approach – unless you allow flexibility, government might as well just keep commissioning in the same way. I do think there’s value in minimum standards to protect vulnerable service users, but that can be measured by mystery shopper and other approaches which are more accurate and not bureaucratically cumbersome.
Interested in what others think?