Wednesday, 21 September 2016

Shooting the Messenger



A leaked report on the Government’s ‘Troubled Families’ programme made the news a few weeks ago. The evaluation, by consultants Ecorys, apparently showed “no discernible impact” on the unemployment, truancy and criminality that this much-hyped programme is supposed to address. DCLG’s reply was that there is “not yet a finished report” – which means that the leak is a draft and they’re now trying to get the authors to change it.

The outcome doesn’t surprise me; I’ve been critical of this scheme in earlier blogs. Its success criteria are vague, data looks highly unreliable, and its ‘payment by results’ element positively incentivises misreporting. Most of all, it judges success purely based on savings to the public purse, with virtually no account taken of what the families themselves think!

Based on experience, I now expect things to go very quiet whilst ministers attempt to ‘spin’ bad news, and it could be that the whole scheme quietly disappears without trace. But this also raises a wider question: what happens to evaluation reports generally when their findings are unwelcome?

It’s an occupational hazard of the evaluation business. Organisations that run social programmes ask for independent evaluations, but still expect the results to prove favourable for them. Fortunately, in most cases they’re right. People know enough about what works for most social programmes to deliver the value they expect, to beneficiaries and others. Unfortunately though, that isn’t always the case. From time to time, projects fail – or at least don’t achieve the success they anticipated. The challenge then is how an independent evaluator reporting this avoids criticism, blame or pressure to distort the facts – classic ‘shoot the messenger’ responses.

A recent NPC briefing Reporting When Things Don’t Go to Plan recognises this issue. It considers why this can happen and offers a number of practical suggestions, including asking the right questions, maintaining trust and transparency, and emphasising learning. This is sound advice although not a complete answer, and the Troubled Families example is a case in point. Here, politics trumps everything: the need for Government not to be seen to fail overrides anything we might say about openness, integrity and learning.

Similar issues can apply to projects run by charities and social enterprises. Sadly, considerations such as reputation, career futures or simply pride sometimes lead people to put desired outcomes ahead of objective assessment. I’ve come to terms with this over the years and offer the following thoughts to minimise ‘shoot the messenger’ risk:


  • Learn as much as possible about the organisation and its projects from the start, and respond accordingly (in worst-case scenarios, I’ve turned down “poisoned chalice” evaluations)
  • Be honest: unsound or misleading evaluations can damage everyone concerned – not least the people the project is trying to help
  • Communication is everything; if it’s going wrong, everyone should be aware of this at the earliest possible moment, and have the chance to react
  • Look for improvement: if it doesn’t work at the moment, what would make it work? Evaluation reports should always include recommendations, even where the project is working well already.
 
This still isn’t a perfect solution, but I believe that an organisation that resists learning and change will not survive in the long run. At least, that works in the private and third sectors – if only it were true of government as well!