Keep up-to-date with drugs and crime

The latest research, policy, practice and opinion on our criminal justice and drug & alcohol treatment systems
Search

Arrested development: Why individual PbR will not work

Share This Post

Saturday’s newspapers sent shockwaves through all ranks of police officers who are waiting for Tom Winsor’s review of pay and conditions to be published.

The Telegraph announced:

“Police could be given performance-related pay for first time”

While the Star went with:

“Cops collar cash”

It’s not sure how much credence we should give to these reports, but the Telegraph and Star do make substantially similar claims:

“cash incentives for high-performing police officers who can successfully fight crime”

“police are set to pocket bonuses for the number of arrests they make”.

We should know for certain soon since the Winsor report is scheduled for publication tomorrow morning.

I’m feeling slightly guilty about the whole situation since I wrote a post a couple of weeks ago in which I idly speculated that it would be theoretically possible to pay probation officers on an individual payment by results basis – never expecting that anyone would seriously propose a similar approach for probation, police or anyone else.

Until last Saturday.


Fortunately, we have good evidence that individual PbR schemes just don’t work.

Earlier this week I wrote about a Freakonomics case study in which a Washington Emergency Room Doctor, Craig Feied, turned around a failing ER by installing a super-efficient computerised information system.

The system generated so much data it was used for medical research and an assessment of how good individual ER doctors were.

What the case study made very clear was that attempting to judge doctors on a payment by results basis just didn’t work. PbR is all about outcomes – improved patient health for doctors and, at least in part, increased detection and arrest rates for police officers.

When researchers tried to evaluate the effectiveness of doctors by their patient outcomes, it quickly became clear that this was a pointless exercise.

An assessment of ER doctors’  performance on a patient outcome basis was rejected for a wide range of reasons, all of which would be relevant to paying police on their clear-up rates:

Selection bias – patients aren’t randomly assigned patients.

The profile of people attending ERs varies markedly throughout different times of the day and the days of the week. In the same way, we would expect officers on duty in town centres on Friday and Saturday nights to make more arrests than those on the same duty on a Tuesday afternoon.

Sometimes the better doctors have higher patient death rates.

The sicker you are, the more likely you are to seek out the best cardiologist. In the same way, a more experienced and skilled officer may defuse a confrontation, rather than nicking everyone in sight.

Once individuals know that they are being measured and paid on performance, they start adjusting the way their work to fit.

This is perhaps the most worrying aspect of performance-related pay. A doctor who knows he is being paid on patient outcomes may start “creaming” – selecting low risk patients and rejecting those with more serious complaints who are most in need of treatment but who are most likely to reduce his/her outcome rates.

What would be the equivalent for police officers? More arrests for possession of cannabis? Less stop and search? Forced deployment to jobs where performance related pay bonuses are not likely?

Payment by results is a great opportunity to focus public services on outcomes that make a difference. A chance to break free from a culture where work priorities are driven by targets, Key Performance Indicators etc., rather than the needs 0f the public.

A key component of successful PbR schemes is that they focus different teams, departments and organisations on how they can most effectively collaborate for the greater good.

Individual performance-related pay completely undermines this approach.

As any study of Bankers’ bonuses will show.


Share This Post

Related posts

Payment by Results
Can payment by results improve outcomes?

The idea is that by commissioning outcomes rather than outputs, commissioners allow provider to work in any way they see fit, safe in the knowledge that if the outcomes are not achieved, they do not have to make payment. But do PbR schemes achieve better outcomes?

Commissioning
It’s time we did something about commissioning

Reform argues that the current system does not encourage innovation or quality. Whether provision is public or private it is typically a local monopoly with limited or no incentives to improve performance. Too often national and local commissioners prioritise price over effectiveness.

Payment by Results
The 6th Commandment of Payment by Results: Profit shall not be thy God

One of the most controversial aspects of payment by results in the UK has been the way the funding model has been used to outsource public services and open the market up to private providers, typically the sort of global companies who deliver the Work Programme. Many people are opposed in principle to the idea of public services generating profit for multinationals. On the other side of the argument are those that see the introduction of business sense and commercial acumen as a key way of reducing cost and driving innovation. But is financial profit the only measure of success?

Payment by Results
The 2nd Commandment of payment by results: Thy outcomes shall be few

Most payment by results pilot schemes are targeted at entrenched social problems. These problems – troubled families, long term unemployment, re-offending and drug dependency – are complex by nature. They require a coordinated response which addresses a wide range of issues. PbR funded interventions are a natural commissioning approach to tackle complex problems. However, PbR schemes quickly run into trouble when the outcomes themselves become complex.

Payment by Results
1st Commandment of Payment by Results: Thou shalt commission for a single purpose

1st Commandment of Payment by Results: Thou shalt commission for a single purpose. PbR schemes are often sabotaged by trying to achieve too many objectives. The Transforming Rehabilitation project is likely to suffer because it wants to reduce reoffending at the same time as cutting costs, transferring risk and privatising the probation service.

3 Responses

  1. Quite agree with you. Recall failed performance related pay for NHS managers in late 80’s. Favored certain groups with more predictable work and working hours over others. Was very divisive. This scenario much more worrying.

  2. Just about every aspect of the Winsor Review is flawed and displays muddled, confused and incorrect thinking. Performance pay in the police would undoubtedly lead to widespread abuse in the power of arrest and undermine the basic duty of a police constable to apply the law without fear or favour.

  3. Good article. I would like to add some nerdy comments if I may.

    Individual payment by results has its place. In selling double-glazing, for example. It’s a pretty clean binary measure. Sell loads, get paid loads – your contribution to the business is tangible and duly rewarded. But sell nothing? No bonus. Unless you work in the banking sector, where pay still appears confusingly performance-unrelated.

    But in health, crime, prisons, probation? That might require a bit more thought.

    In Probation, if you base a PbR scheme on reoffending outcomes, as is suggested, you must first ensure a big-enough cohort size (I will return to this).

    Reoffending rates in AnyTrust are 32%. You are offered payment if you reduce them by five percentage points – to 27%. AnyTrust might think, OK, it’s possible, with freedom to innovate, but would understandably fear interference from external factors such as a steep improvement in police clear-up rates or a changing caseload including more high-risk-of-reoffending offenders.

    In this scenario, if AnyTrust remained at 32% reoffending rate, but with a much more demanding caseload, this should be recognised – and it is, by using an adjusted baseline.

    There are different models. The one used to contextualise local adult reoffending rates – the predicted probability of reoffending score – takes account of a whole host of variables including age, gender, index offence type, age at first conviction, number of previous convictions etc.

    You can mitigate to some degree the influences outside your control by using an adjusted baseline against which to judge success. In the example above, your adjusted baseline might be 37% (taking into account these circumstances). Your achievement in holding rates at 32% would be acknowledged, and paid for.

    In reality you might want to go further and split your cohort, setting separate, achievable outcome reoffending targets on what you know would constitute success in each area.

    As an example, young men (18-24) with an index offence of theft or burglary have much higher than average reoffending rates – in AnyTrust they can be up to 55%, more like short-sentence prisoner release rates than the overall Trust average of 32%. It would be insane to expect AnyTrust to reduce this specific cohort’s reoffending rates to anywhere near the Trust average. There is a clear and well-documented danger here of “parking”.

    But if you use the adjusted baseline, you realise this group has a “predicted reoffending rate” of 58% – an expectation of their reoffending rate based on some of the variables mentioned above.

    You might then set a realistic and achievable target of 50% for this group. You might develop a payment structure that allowed for different “offender types” and therefore encouraged appropriate interventions across the board.

    This is unlikely to be popular with ministers, who seem to like talking about “reoffending rates being unacceptably/stubbornly high” at every available opportunity. They might baulk at having to justify paying out on reducing reoffending to “only” 50% – though this would (and should) be considered a relative success.

    To return to cohort size – as long as the group is large enough, this adjusted baseline, or predicted rate, gives a pretty useful idea of the reoffending rates you might expect to see from particular subsets of offenders. It’s never miles away from reality.

    But it’s important to keep it as a rough indicator of the behaviour of a big group – to some extent it seems humans behave quite predictably in group situations.

    The same cannot be said for individuals. Paying probation officers on an individual PbR basis, using reoffending outcomes as the measure of success, is fraught with danger.

    Every individual on AnyTrust’s caseload has a “predicted probability of reoffending (PPoR) score”. By calculating the mean average of this score across big cohorts, you arrive at the adjusted baseline figure.

    But in looking at individual scores, this “minority report” can be emphatically wrong. To cherry-pick two examples, a 27-year-old man on AnyTrust’s caseload with a PPoR score of 95% had no proven reconviction over the following 12 months, while a 36-year-old man with a 4% PPoR did. In other words, the reoffending predictor felt (based on all the evidence) that this 36-year-old was a remote 25/1 shot to reoffend. But 25/1 shots do come in every now and again.

    PbR can work with reoffending as a measure of success, as long as the design is clever and realistic. But beware small numbers, take steps to avoid creaming and whatever you do, don’t try and predict what individual human beings are going to do next (even double-glazing saleswomen!)

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe

Get every blog post by email for free