Wednesday, 22 November 2017

How Happy are EU?



The Office for National Statistics (ONS) recently released its latest figures on personal wellbeing in the UK*. We now have five years of continuous data since this national “happiness survey” was launched in 2012. It’s also the first survey to be based on a full year of data since last year’s EU referendum result. Hence the terrible pun in the title above – sorry!

The survey, of around 158,000 people across the UK, asks four questions on personal wellbeing:
·         Overall, how satisfied are you with your life nowadays?
·         Overall, to what extent do you feel the things you do in your life are worthwhile?
·         Overall, how happy did you feel yesterday?
·         Overall, how anxious did you feel yesterday?

Responses to the first three questions show a further small increase that continues the gradual trend of improvement since 2012. Only the fourth question, on anxiety, shows a marginal worsening over the past two years. (See below – note that these graphs do not start at zero.)


 

ONS commentary offers some insight into these trends. People’s wellbeing is affected by socio-economic factors such as employment and income, but also by other issues such as health and personal relationships. More in-depth analysis is also produced by the What Works Centre for Wellbeing (https://www.whatworkswellbeing.org).

The ONS report also mentions Brexit, and it’s tempting to interpret trends in this context. Certainly, ONS data provides no evidence that the Brexit vote has made us less happy as a nation. But the data doesn’t prove the contrary either. Firstly, the exit itself hasn’t happened yet; secondly, even if it had, the survey alone doesn’t prove cause and effect because of the many other factors in play. So the jury is out on this one – and I believe is likely to remain so for many years to come!

But for me there is a more fundamental question. Why does so much of the continuing argument around Brexit focus on “the economy”? Surely personal wellbeing – how satisfied we are with our lives – is what really matters.

Sure, money has an influence on happiness, but it’s not the only thing. For most people, it’s not even the most important thing. And indicators of “the economy” generally measure regional or national wealth, not how fairly that wealth is distributed or how it affects individuals. I believe that continued work to understand trends in wellbeing, and how we can influence them, is a far better use of time and brain-power than arguing about Brexit.

*https://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/measuringnationalwellbeing/july2016tojune2017

Wednesday, 28 June 2017

The Power of Perspectives

I’m a great believer in understanding different points of view. It’s a vital part of evaluation, and not bad advice for life too. As Robert Burns said “…to see ourselves as other see us”.

Different perspectives affect the way charities work as well. Over time, I’ve formed a view of the contrasting ways in which funders, charities and beneficiaries themselves can see things. I’m talking mainly about smaller charities that work with people, and that receive funding from commissioning, charitable trusts or sponsorship (as opposed to public donation).

As an evaluator, I’m sometimes asked to provide independent evaluation of a specific project or activity, to provide assurance or validation to the funder. That funder’s perspective, put simplistically, can be summarised by first diagram. It provides funding to a charity, which then carries out a project, which in turn benefits the people it works with. In the old days, this was measured by the number of people seen, number of contacts, or even just opening hours – outputs, in other words. Increasingly (and quite rightly), funders now want to know about outcomes – what different the project has made to its beneficiaries, and possibly to others as well.


All well and good, but the charity itself may have a different perspective. It’s rare for a charity to have a single source of funding, or indeed to be running just one project. Many draw funding from a range of different sources and run several projects or activities, sharing some staff and central functions between them. Of course, they can account for expenditure to different funders, but accounting for outcomes may not be so simple, particularly where beneficiaries are part of more than one project. Here the question of “attribution” arises: what outcomes, or how much impact, comes from each project?

But there’s a further perspective still, that of the beneficiary him/herself – the person who needs help. It’s not uncommon for that person to have a choice of support services, from different organisations (both voluntary and statutory), and there may even be further options that they are unaware of. They may well be receiving support from more than one organisation, let alone more than one project. Not only that, but all this takes place in the context of real life, where events outside the control of any service can have a big impact on the person’s wellbeing. Assessing the outcome of any one piece of this ‘jigsaw’ now becomes even more problematic.

So, what’s the solution? Firstly, I believe that third perspective must be the starting point. It’s why we’re doing this, surely. Rather than “what outcomes does the project achieve?” a better question might be “what life changes do people experience, and how does the project contribute to this?”. This may need some change of mind-set from funders and charities: less of “our funding” or “our charity”, more of “our contribution to better outcomes”.

There’s much to be done, but I do see signs that this need for collaboration is being recognised. Social Return on Investment (SROI) captures this thinking, as do ideas like shared measurement and impact management. If we maintain this momentum, I believe there’s a future where charities, working together and in partnership with funders, can achieve even greater social impact and value.

Wednesday, 15 March 2017

Post-Truth Statistics

Last year, “£350 million per week to the EU” hit the headlines – not to mention the side of a bus – as the cost of the UK’s contribution. The Remain camp was quick to rubbish this figure as it failed to take any account of what the UK gets in return for this. But the figure still sticks in people’s minds.
 

More recently, the Remain lobby (or more accurately, the pro-migrant lobby) hit back with a figure of £328 million per day as the contribution that migrants make to the UK economy. Unfortunately, this figure is equally spurious. It’s based simply on dividing UK GDP by the proportion of working people who are non-UK born. Apart from earnings being just one aspect of GDP, this assumes that none of these jobs could be done by someone else, for example a UK-born worker who is currently unemployed. Far from proving a point, the statistic risks pandering to the “foreigners taking our jobs” brigade.
 

Don’t get me wrong; I believe the UK economy (and society as a whole) gains a huge amount, in diversity and skills as well as labour, from non-UK born workers. But fatuous statistics are certainly not the way to prove it.
 

So where does this rubbish come from? I look on it as an aspect of our current “post truth” world, where the most bizarre claims can achieve credibility simply by being repeated often enough. I blame a combination of social media and the acquiescence of mainstream media, both of which seem to prioritise impact over integrity. That plus the fact that anything with a number attached sounds authentic, even if in reality it’s completely made up.
 

Therein, in my view, lies a problem for serious analysis, such as Social Return on Investment (SROI). This is a rigourous methodology for establishing social value, hence can be time-consuming and costly to get right. Why would you want to bother with this when you can virtually make it up and grab bigger headlines? It goes without saying that a lot of serious academic research is also at risk from the same problem.
 

I wish I had an answer to this, but it isn’t easy. We need somehow to educate the public out of the mindset that believes that figures = facts, and into a healthy scepticism of anything the media tells them. In the meantime, I think I’ll claim that the value of my consultancy is 500 times the price my clients pay, and hope it goes viral!
 

*PS: That quote in the text box is widely attributed to Churchill, but I can find no evidence that he actually said it!

Wednesday, 18 January 2017

Impact Management - New or Recycled?


Perhaps I’m getting old, but I have this feeling of déjà vu. It’s been prompted by the launch by Access – The Foundation for Social Impact – of their new Impact Management Programme. (See http://accessimpact.org/ for details.) The principle is that charities and social enterprises should not just be measuring their impact and social value, they should actively work to increase it.
 

I must confess it hadn’t seriously occurred to me that you might measure impact without trying to improve it. I’ve always believed that the purpose of any measurement or evaluation should be twofold:
(a)    To demonstrate what has been achieved; and
(b)    To find ways to improve that achievement further
That’s why, in over 25 years, I’ve never written an evaluation report that doesn’t include recommendations – and I’m sure I never will.
 

Of course, recommending change and making it happen are two different things. So, quite rightly, the Impact Management Programme is looking for skills in facilitating culture change, leadership development, planning, and product/service development from the consultants who will act as its support providers.
 

But that’s really no different from change and improvement for other purposes. If you’re looking to improve customer service, increase profit margins or even save the environment, managing change involves much the same thing – leadership, culture, planning, processes/systems and so forth. Hence, for example, approaches such as Lean and Systems Thinking are primarily process-based, but still need ‘soft skills’ around leadership and change management to implement them.
 

So, allow me to present my ‘Theory of Everything’:

What I’m saying is that to pursue any aspect of improvement you need to be able to (a) measure the outcomes you achieve and (b) build and measure your capability to enable improvement. To paraphrase an old political cliché: “Tough on results, tough on the causes of results”. Check this against any improvement model you choose: EFQM Excellence Model, Balanced Scorecards, Lean, PQASSO (for the Third Sector), and now Impact Management. They may emphasise different aspects of the ‘theory of everything’ – social value for example focuses on non-profit results aspects. But they all fit with this overall way of thinking.
 

So does that mean I’m against the Impact Management initiative? No, absolutely not. The medium is as important as the message, and I endorse any approach that helps get these concepts across. Different methods – languages if you like – appeal to different people, and we need a diversity of models and communication to engage diverse organisations.
 

If Impact Management helps to drive change and improvement in the context of social value, then it has my full support.

Wednesday, 2 November 2016

Shared Impact


It seems to me that Newton’s First Law of Motion applies to people as well. The principle is that a body remains at rest or in motion until acted on by some external force. For us as individuals, this means that we keep doing what we are doing (or not doing) until something makes us change.

Trouble is, real life is not a science laboratory where we can isolate the effect of a single ‘intervention’ (excuse the jargon). In life, we’re subject to a host of external influences all the time – some good, some bad. So if you’re a Third Sector organisation working to improve people’s health and well-being for example, it can be hard to distinguish the effect of what you’re doing from everything else that’s going on in their lives. 


This is an important issue in evaluation, the technical term being ‘attribution’. Even if you can measure the change that someone has achieved – improved health and well-being let’s say – and you know that you’ve helped them achieve that change, how much of that outcome is due to you rather than other influences? Even if yours is the only agency working with them (which is often not the case anyway), has a contribution also been made by medical treatment, by their friends or family, perhaps even by national health publicity that they’ve seen?


Social Return on Investment handles this by estimating what percentage of change is due to the intervention being evaluated. It’s not precise of course, but there are ways of making it quite robust, and a 2011 publication by New Economics Foundation, Small Slices of a Bigger Pie, goes into detail on this. But even this needs caution where change occurs in stages. Suppose for example that as a result of your work, someone is motivated to attend health screening which diagnoses a condition that is then treated successfully. Clearly the treatment produced that health improvement, but it would not have happened without your intervention, so how much of the outcome can you claim credit for?


I’d make two points. The first is the need to fully understand the outcomes achieved, by developing a Theory of Change (see my earlier blog here). What exactly has changed for the individual, and how has that happened?


Second, just how important is measuring your unique contribution anyway? Surely it’s the overall outcome that you’ve helped to achieve that matters most. Yes, I appreciate the practical issues in terms of demonstrating value to funders, but I believe this is a short-term view. Achieving real and lasting change needs collaboration, not competition. Measuring shared impact, and understanding how this is achieved by different organisations working together, is the way to create better outcomes for everyone.

Wednesday, 21 September 2016

Shooting the Messenger



A leaked report on the Government’s ‘Troubled Families’ programme made the news a few weeks ago. The evaluation, by consultants Ecorys, apparently showed “no discernible impact” on the unemployment, truancy and criminality that this much-hyped programme is supposed to address. DCLG’s reply was that there is “not yet a finished report” – which means that the leak is a draft and they’re now trying to get the authors to change it.

The outcome doesn’t surprise me; I’ve been critical of this scheme in earlier blogs. Its success criteria are vague, data looks highly unreliable, and its ‘payment by results’ element positively incentivises misreporting. Most of all, it judges success purely based on savings to the public purse, with virtually no account taken of what the families themselves think!

Based on experience, I now expect things to go very quiet whilst ministers attempt to ‘spin’ bad news, and it could be that the whole scheme quietly disappears without trace. But this also raises a wider question: what happens to evaluation reports generally when their findings are unwelcome?

It’s an occupational hazard of the evaluation business. Organisations that run social programmes ask for independent evaluations, but still expect the results to prove favourable for them. Fortunately, in most cases they’re right. People know enough about what works for most social programmes to deliver the value they expect, to beneficiaries and others. Unfortunately though, that isn’t always the case. From time to time, projects fail – or at least don’t achieve the success they anticipated. The challenge then is how an independent evaluator reporting this avoids criticism, blame or pressure to distort the facts – classic ‘shoot the messenger’ responses.

A recent NPC briefing Reporting When Things Don’t Go to Plan recognises this issue. It considers why this can happen and offers a number of practical suggestions, including asking the right questions, maintaining trust and transparency, and emphasising learning. This is sound advice although not a complete answer, and the Troubled Families example is a case in point. Here, politics trumps everything: the need for Government not to be seen to fail overrides anything we might say about openness, integrity and learning.

Similar issues can apply to projects run by charities and social enterprises. Sadly, considerations such as reputation, career futures or simply pride sometimes lead people to put desired outcomes ahead of objective assessment. I’ve come to terms with this over the years and offer the following thoughts to minimise ‘shoot the messenger’ risk:


  • Learn as much as possible about the organisation and its projects from the start, and respond accordingly (in worst-case scenarios, I’ve turned down “poisoned chalice” evaluations)
  • Be honest: unsound or misleading evaluations can damage everyone concerned – not least the people the project is trying to help
  • Communication is everything; if it’s going wrong, everyone should be aware of this at the earliest possible moment, and have the chance to react
  • Look for improvement: if it doesn’t work at the moment, what would make it work? Evaluation reports should always include recommendations, even where the project is working well already.
 
This still isn’t a perfect solution, but I believe that an organisation that resists learning and change will not survive in the long run. At least, that works in the private and third sectors – if only it were true of government as well!