Wednesday, 5 February 2014

Can We Trust Statistics?

It's a question that's been asked a lot lately. Late last year the government reported that its Troubled Families programme was "on track", despite statistics that showed huge variation in results reported by different Local Authorities. More recently, the National Audit Office found wrong and inconsistent recording in waiting time data for elective surgery. Then just last week new figures showed wide variations in recording and classifying rape allegations by police forces across England and Wales.

These kind of figures are used to assess performance, make decisions, and in some cases justify funding. So what’s behind these apparent problems, and what can we do about it?

Let me make a confession. Umpteen years ago, as a junior civil servant, I endured the weekly ritual of "doing the stats". Well before computerisation, this was a manual count of work processed and outstanding. The figures were meaningless to me; counting outstanding work was particularly tedious and nothing was ever done about the backlog anyway. So my solution was to make the figures up – or at least an educated guess rather than actually counting.

Does that sort of thing still happen? Well, even with today's IT systems, it's still people who input data and tell the system what to count. So you decide!

There's a common factor in all these problem cases: those compiling the data have little or no ‘ownership’ of it. In other words, if it makes no difference to how they work, what incentive do they have for compiling it accurately? The old cliché "what gets measured gets done" has an unfortunate corollary: if you set people numeric targets, they will aim to achieve those targets – not necessarily the intention behind the targets!

Some people suggest we should just give up and not try to collect any performance data. I suggest an alternative approach: think about how data recording might affect the way people behave, and design your reporting systems with this in mind. So for example, measure waits for elective surgery in terms of patients’ perception of their waiting time rather than the hospital's perception of it. This might encourage better patient communication as well as helping to improve what really matters to patients.

An element of independent verification helps too, and this need not be bureaucratic. But returning to the original question, if you're unsure whether to trust any statistics, try asking yourself who is recording the data and what their incentive is to get it right. That may tell you all you need to know.

No comments:

Post a Comment