Monday, 10 June 2013

Measuring Outcomes: Good, Bad and Ugly

I've been comparing two recent publications, both from the Department of Health. The differences in their approach to measuring outcomes are interesting and instructive.

The first, actually a set of publications, is the Outcomes Frameworks 2013-14*. There are three: NHS, Adult Social Care, and Public Health (in development). Each defines a range of high-level outcome measures linked to national strategic priorities for these services.

There are a number of things I like about these frameworks:
  • Each contains a manageable number of high-level indicators grouped into ‘domains’, rather like the presentation of a balanced scorecard.
  • The measures are outcome-focused. That is, they address the critical question that defines outcomes: "What changes?" (or if you prefer "What difference have we made?"). They don't just measure activity or benchmarks at a point in time.
  • Many are perception-based, using the experience of patient/clients rather than just clinical indicators.
  • They are continually evolving. Recognising they are far from perfect, each annual version is improved from the previous one and anticipates further developments for the future.

The second document is Mental Health Payment by Results Guidance for 2013-14. To an extent this is also ‘work in progress’, but its aim is to rationalise the commissioning of mental health work based on symptoms or type of illness. These are defined as a number of clusters, currently 21 with a few conditions excluded from the system.

Whilst the intention is clear, the practice is more difficult, not least because every mental health patient is an individual, and conditions are much harder to ‘pigeonhole’ than physical health. Harder still is objectively assessing improvement, and here the paper recommends a number of “quality and outcome measures” without standardising on a particular method.

Even these options are mixed in terms of their relevance as outcomes. Some patient-based assessment scales are sound, though arguably over-generic. But other measures relate to activity, or even completeness of records/planning, and these are not outcomes; they do not measure what changes for the patient or client.

The ugly part though is the title of the paper. Calling it “Payment by Results” is misleading, at least at the current stage of development. It is payment by category, not by results. It worries me that this may give people the impression they are achieving outcomes when they are not.

I believe that making a real difference to people's lives, be it in health and social care or other types of support, relies fundamentally on understanding what we mean by outcomes. It isn't about levels of activity, standard practice or monitoring. It's about what difference we make to the quality of life for users of those services, and this is what we should be measuring.

*The NHS Outcomes Framework 2013/14, November 2012 
The Adult Social Care Outcomes Framework 2013/14, November 2012
Improving Outcomes and Supporting Transparency: Part 1A  A public health outcomes framework for England, 2013-2016, January 2012