Category: Data

Joy MacKeith argues that Payment by Results can cause as many problems as it addresses.  Management by Results, which supports ongoing learning and collaboration, is the missing middle way between ignoring outcomes on the one hand, and linking them to financial incentives on the other.

In early September I was privileged to participate in the fifth Social Outcomes Conference, organised by the Government Outcomes Lab at Oxford University. Contributions from both academics and practitioners from all over the world made for a very rich debate in which everyone had their eye on the prize of improving social outcomes.

The debate got me thinking about the limitations of Payment by Results and an alternative – an approach I am calling Management by Results.  This blogpost explains the difference between the two and how Management by Results has the potential to unlock performance improvement.

Why I am a fan of an outcomes approach

In the old days we didn’t measure outcomes.  We counted inputs and outputs.  We collected case studies.  Occasionally we commissioned evaluations or user surveys.  Then came the outcomes revolution.  I have been part of that revolution, spending much of the last 20 years helping organisations to measure their outcomes.

I am a fan because I have seen that defining, measuring, and managing outcomes enables service providers to create services with a clarity of purpose, identify issues and gaps, and ultimately improve what they deliver for service users. It undoubtedly is a good thing for organisations to focus on outcomes.

But what happens when financial imperatives are introduced into the equation?  What happens when a project or organisation’s survival becomes dependent on evidencing that they have achieved certain outcomes?

Why I’m wary of linking outcomes with financial incentives

In the employment sector where Payment by Results (PbR) has been in operation for some time the consequences are quite well documented (Hudson., Phillips, Ray, Vegeris & Davidson, 2010^[1]).  Organisations can be incentivised to focus narrowly on the specific targets which are linked to payment and ignore everything else.

This can lead to a narrowing of their work with individuals (just making sure they get a job rather than working on longer-term issues such as addiction or mental health problems that are likely to impact on their ability to keep the job for example).  It can lead to short-termism with less focus on long-term impact and sustainability.  It can lead to ‘cherry picking’ of clients who are most likely to achieve the target (also called ‘creaming’) and not ‘wasting resources’ on those who are not likely to achieve the target within the timescale of the project (also known as ‘parking’).

The fact that there are widely used terms for these kinds of gaming practices reflects the fact that these perverse incentives are widely recognised and understood. In the financial sector Goodhart’s Law^[1] that any financial indicator that is chosen by government as a means of regulation becomes unreliable is well accepted. In the words of the anthropologist Marilyn Strathern “When a measure becomes a target, it ceases to be a good measure”.^[2]

In addition to this, there are other more subtle but nevertheless powerful impacts.  In Triangle’s work helping organisations to measure their outcomes we have seen time and again that when the impetus for this measurement is commissioner requirement, the organisation is likely to see outcomes as something that is done for the commissioner rather than something they own.

The result is that the quality of the data collected is poorer and the service provider just passes it on to the commissioner rather than mining this outcomes gold for learning and service development.  This is very unfortunate because sending outcomes information to commissioners doesn’t improve outcomes, whereas using it to better understand delivery does.

Another impact of PbR is that it focuses attention on the work of the service provider in isolation as opposed to looking at how the service delivery system as a whole is working. In practice often it is the network of service provision that achieves the outcome rather than a single provider.

Finally, in the market for social outcomes, providers find themselves in competitive rather than collaborative relationships, which can make system-wide cooperation and information sharing more difficult.

The missing middle way
There were several speakers at the recent GoLab conference who argued that financial incentives can work – if they are done well.  I am writing primarily from personal experience rather than extensive research and I trust that what they say is true.  I am also aware myself of PbR contracts and Social Impact Bonds that have been sensitively implemented with all parties understanding the risks and the funding mechanisms carefully designed to build the right incentives.

My concern is that too often the approach isn’t done well and also that the alternative of MbR is not recognised and considered.  In our enthusiasm to embrace outcomes we have gone from one extreme of not talking about or measuring outcomes at all, to the other extreme of linking payment to outcomes.  Between these two poles there is a middle ground – a third way which can unlock the potential of outcome measurement without so many of the downsides.

So what does Management by Results look like and how is it different from Payment by Results?

The Management by Results mindset
Both MbR and PbR involve identifying and measuring outcomes.  But in MbR the emphasis is on the service provider using this information in the management of the service to identify strengths, weaknesses and issues to be addressed.  Whereas in PbR the emphasis for the service provider is on using the information to secure the funding the organisation needs to survive.

For commissioners MbR means requiring the service provider to measure their outcomes and then drawing on that information to assess their performance.  But crucially in MbR the commissioner draws on other information as well and has room for judgement.  PbR is black and white.  Target achieved = good, payment made. Target not achieved = bad, no payment made.

MbR allows for greater subtlety and a more rounded assessment.  The commissioner looks at the data, but they also look at the organisation’s narrative about the data.  Is it a coherent narrative? Are they learning from their data and using the lessons to improve service delivery?  What do others say about the service?  What do you see if you visit and what do service users have to say?

The commissioner draws on all this information to make their assessment.  Of course, life would be a lot easier if you didn’t have to do this and could reduce a project’s effectiveness to a few numbers.

But you can’t.

There is always a wider picture, for example in the employment sector, what is happening in the service user’s personal life, what is happening in the local economy, what other services  the person is receiving and what impact are they are having. The numbers have a part to play but they are never the whole answer.

How Management by Results changes the questions and supports learning
An organisation that is managing by results will take a systematic approach to collecting and analysing outcomes data and will then use that data for learning and accountability.  The job of the manager is to ask: “Why did this work – what good practice can we share?”  and “Why didn’t this work, what do we need to change and where can we learn from others?”

The job of the commissioner or investor is to assess “Is this organisation taking a sensible and systematic approach to measuring its outcomes? And is it learning from its measurement and continually changing and improving what it does?” PbR encourages hiding of poor results and exaggeration of positive results as well as the creaming and parking described above.  This positively hinders learning and obscures what is really happening.

MbR encourages collaboration between service provider and commissioner in identifying and achieving their shared goals.  PbR obscures these shared interests by incentivising service delivery organisations to prioritise their own survival.

The table below summarises the differences:

Management by results is not easy but it is worth the effort

Management by Results is not easy.  At Triangle we support organisations to implement the Outcomes Star and in practice this means that we are supporting them to build a MbR approach.  This involves forging new habits, behaviours and organisational processes, creating new interdepartmental links, new reports and new software.

It isn’t easy and it takes time, even for the most willing and able.  But we also see the benefits for those that stick with it – managers with a much better handle on what is happening in their services, who can pinpoint and address issues and share good practice as well as evidence achievements.

I believe that if the sector put more energy, funding and research into supporting organisations to manage by results, it would really start to unlock the potential to not only measure, but also improve outcomes.

What do you think?