Thursday, 11 April 2013

Secret Statistics and Lies

The saying "lies, damned lies and statistics" is attributed to Disraeli in the 19th century. But it seems truer than ever today.

Examples abound, and the NHS is particularly prone. The furore around children's heart surgery at Leeds seems to stem from the quality of statistics rather than the quality of surgery or care. And the Mid-Staffordshire experience shows (if we ever doubted it) that one way to reduce mortality rates is to reclassify patients as ‘palliative care’ – i.e. they were expected to die anyway.

And should anyone challenge this approach, the solution – up to now – has been to dismiss them and pay them off with a gagging order. What a shame that the government simply proposes to ban gagging clauses rather than solving the underlying problem.

A former advisor to the Bank of England made the observation known as Goodhart's Law, summarised as “When a measure becomes a target, it ceases to be a good measure”. This happens because it's generally much easier to manipulate statistics than to actually improve performance!

Hardly surprising. If "what gets measured gets done" (another cliché, but true), it naturally follows that if you set people targets, they will aim to achieve these targets, not necessarily the intention behind the targets.

So should we just give up and abandon all performance indicators? No, I’m a believer in an alternative approach, that of deliberately designing performance measures to influence the way people behave. Every potential measure should be 'destruction tested', to see how it can be fiddled, before implementation. Only those performance measures that pass this test should be used.

For the NHS this surely means greater emphasis on patient (and family) satisfaction, rather than clinical success rates. I'm not just talking about PROMs (patient reported outcome measures), but a much broader patient view of how well the NHS delivers its core function of care. Clinicians and managers should be incentivised to deliver a service that patients acknowledge as excellent.

If you doubt this, just ask yourself which is the better dentist: one whose work is technically brilliant but very painful, or one whose work is perfectly adequate and pain-free? I know which one I would rather visit.

Of course such a system would need careful design to ensure it is truly objective. But this is possible, and it offers the hope of creating a genuinely patient focused service for the future.