Wednesday, 2 November 2016

Shared Impact


It seems to me that Newton’s First Law of Motion applies to people as well. The principle is that a body remains at rest or in motion until acted on by some external force. For us as individuals, this means that we keep doing what we are doing (or not doing) until something makes us change.

Trouble is, real life is not a science laboratory where we can isolate the effect of a single ‘intervention’ (excuse the jargon). In life, we’re subject to a host of external influences all the time – some good, some bad. So if you’re a Third Sector organisation working to improve people’s health and well-being for example, it can be hard to distinguish the effect of what you’re doing from everything else that’s going on in their lives. 


This is an important issue in evaluation, the technical term being ‘attribution’. Even if you can measure the change that someone has achieved – improved health and well-being let’s say – and you know that you’ve helped them achieve that change, how much of that outcome is due to you rather than other influences? Even if yours is the only agency working with them (which is often not the case anyway), has a contribution also been made by medical treatment, by their friends or family, perhaps even by national health publicity that they’ve seen?


Social Return on Investment handles this by estimating what percentage of change is due to the intervention being evaluated. It’s not precise of course, but there are ways of making it quite robust, and a 2011 publication by New Economics Foundation, Small Slices of a Bigger Pie, goes into detail on this. But even this needs caution where change occurs in stages. Suppose for example that as a result of your work, someone is motivated to attend health screening which diagnoses a condition that is then treated successfully. Clearly the treatment produced that health improvement, but it would not have happened without your intervention, so how much of the outcome can you claim credit for?


I’d make two points. The first is the need to fully understand the outcomes achieved, by developing a Theory of Change (see my earlier blog here). What exactly has changed for the individual, and how has that happened?


Second, just how important is measuring your unique contribution anyway? Surely it’s the overall outcome that you’ve helped to achieve that matters most. Yes, I appreciate the practical issues in terms of demonstrating value to funders, but I believe this is a short-term view. Achieving real and lasting change needs collaboration, not competition. Measuring shared impact, and understanding how this is achieved by different organisations working together, is the way to create better outcomes for everyone.

Wednesday, 21 September 2016

Shooting the Messenger



A leaked report on the Government’s ‘Troubled Families’ programme made the news a few weeks ago. The evaluation, by consultants Ecorys, apparently showed “no discernible impact” on the unemployment, truancy and criminality that this much-hyped programme is supposed to address. DCLG’s reply was that there is “not yet a finished report” – which means that the leak is a draft and they’re now trying to get the authors to change it.

The outcome doesn’t surprise me; I’ve been critical of this scheme in earlier blogs. Its success criteria are vague, data looks highly unreliable, and its ‘payment by results’ element positively incentivises misreporting. Most of all, it judges success purely based on savings to the public purse, with virtually no account taken of what the families themselves think!

Based on experience, I now expect things to go very quiet whilst ministers attempt to ‘spin’ bad news, and it could be that the whole scheme quietly disappears without trace. But this also raises a wider question: what happens to evaluation reports generally when their findings are unwelcome?

It’s an occupational hazard of the evaluation business. Organisations that run social programmes ask for independent evaluations, but still expect the results to prove favourable for them. Fortunately, in most cases they’re right. People know enough about what works for most social programmes to deliver the value they expect, to beneficiaries and others. Unfortunately though, that isn’t always the case. From time to time, projects fail – or at least don’t achieve the success they anticipated. The challenge then is how an independent evaluator reporting this avoids criticism, blame or pressure to distort the facts – classic ‘shoot the messenger’ responses.

A recent NPC briefing Reporting When Things Don’t Go to Plan recognises this issue. It considers why this can happen and offers a number of practical suggestions, including asking the right questions, maintaining trust and transparency, and emphasising learning. This is sound advice although not a complete answer, and the Troubled Families example is a case in point. Here, politics trumps everything: the need for Government not to be seen to fail overrides anything we might say about openness, integrity and learning.

Similar issues can apply to projects run by charities and social enterprises. Sadly, considerations such as reputation, career futures or simply pride sometimes lead people to put desired outcomes ahead of objective assessment. I’ve come to terms with this over the years and offer the following thoughts to minimise ‘shoot the messenger’ risk:


  • Learn as much as possible about the organisation and its projects from the start, and respond accordingly (in worst-case scenarios, I’ve turned down “poisoned chalice” evaluations)
  • Be honest: unsound or misleading evaluations can damage everyone concerned – not least the people the project is trying to help
  • Communication is everything; if it’s going wrong, everyone should be aware of this at the earliest possible moment, and have the chance to react
  • Look for improvement: if it doesn’t work at the moment, what would make it work? Evaluation reports should always include recommendations, even where the project is working well already.
 
This still isn’t a perfect solution, but I believe that an organisation that resists learning and change will not survive in the long run. At least, that works in the private and third sectors – if only it were true of government as well!

Thursday, 28 July 2016

Well-Being and Inequality

The Office for National Statistics (ONS) recently published the results of its fifth annual survey of personal well-being in the UK*. So are we getting happier, and – more important – what can we learn from this?

Well, the data shows a steady improvement in life satisfaction, happiness and ‘feeling things are worthwhile’ since 2011 – although not significantly over the past year for the latter two. ‘Anxiety’ actually got slightly worse last year, although again not to a significant degree. But taken over the five years since 2011, yes apparently we’ve been getting happier.


The headline figures are averages, so don’t show ‘polarisation’ (i.e. whether the number of both high and low scores is increasing). However, supporting analysis indicates this is not happening. At least at national level, fewer people are reporting the lowest levels of life satisfaction as more people report higher levels. Regional analysis shows Northern Ireland at the top of the “happiness league” and London at the bottom, although the difference is small.


So what’s behind all this? ONS’s own research suggests a combination of personal factors (health, employment, relationships) and external factors (the economy, security) as the main contributors. The 2016 survey pre-dates both the EU referendum vote and more recent terrorist violence, so it will be interesting to see whether these affect trends next year.


However, a recent blog for the What Works Centre for Wellbeing** raises an interesting correlation, not about well-being per se but about local inequalities. Its graph, reproduced here, suggests that more ‘unequal’ local authority areas – i.e. those where there is greater polarisation – were more likely to vote to leave the EU. This is consistent with the theory that those who feel disadvantaged are more likely to want change, expressed in this case through the Brexit vote. And it is perceived disadvantage, rather than more tangible aspects such as income disparity, that matters here – which is what well-being measurement captures.

Source: What Works Centre for Wellbeing https://whatworkswellbeing.org

The blog concludes that “more research will be needed”, a common get-out for researchers but in this case I agree. Lots of arguments have been made (after the event of course) as to why Brexit won, and this is just one idea. But for me, the underlying question is not why people voted to leave the EU but why the result came as a surprise. Did we really fail to understand people’s attitudes to this degree?


I’ve commented before on the need to understand cause-effect and address well-being issues more prominently in government policy, and I believe this is more important now than ever. As a society we need to understand why people vote – and indeed act – as they do, and tackle the underlying causes of discontent. Because if we don’t, then Brexit may be the least of our worries.


*http://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/measuringnationalwellbeing/2015to2016
**https://whatworkswellbeing.org/2016/07/08/what-wellbeing-inequalities-tell-us-about-the-eu-referendum-result/

Well-Being and Inequality

The Office for National Statistics (ONS) recently published the results of its fifth annual survey of personal well-being in the UK*. So are we getting happier, and – more important – what can we learn from this?

Well, the data shows a steady improvement in life satisfaction, happiness and ‘feeling things are worthwhile’ since 2011 – although not significantly over the past year for the latter two. ‘Anxiety’ actually got slightly worse last year, although again not to a significant degree. But taken over the five years since 2011, yes apparently we’ve been getting happier.


The headline figures are averages, so don’t show ‘polarisation’ (i.e. whether the number of both high and low scores is increasing). However, supporting analysis indicates this is not happening. At least at national level, fewer people are reporting the lowest levels of life satisfaction as more people report higher levels. Regional analysis shows Northern Ireland at the top of the “happiness league” and London at the bottom, although the difference is small.


So what’s behind all this? ONS’s own research suggests a combination of personal factors (health, employment, relationships) and external factors (the economy, security) as the main contributors. The 2016 survey pre-dates both the EU referendum vote and more recent terrorist violence, so it will be interesting to see whether these affect trends next year.


However, a recent blog for the What Works Centre for Wellbeing** raises an interesting correlation, not about well-being per se but about local inequalities. Its graph, reproduced here, suggests that more ‘unequal’ local authority areas – i.e. those where there is greater polarisation – were more likely to vote to leave the EU. This is consistent with the theory that those who feel disadvantaged are more likely to want change, expressed in this case through the Brexit vote. And it is perceived disadvantage, rather than more tangible aspects such as income disparity, that matters here – which is what well-being measurement captures.



The blog concludes that “more research will be needed”, a common get-out for researchers but in this case I agree. Lots of arguments have been made (after the event of course) as to why Brexit won, and this is just one idea. But for me, the underlying question is not why people voted to leave the EU but why the result came as a surprise. Did we really fail to understand people’s attitudes to this degree?


I’ve commented before on the need to understand cause-effect and address well-being issues more prominently in government policy, and I believe this is more important now than ever. As a society we need to understand why people vote – and indeed act – as they do, and tackle the underlying causes of discontent. Because if we don’t, then Brexit may be the least of our worries.


*http://www.ons.gov.uk/peoplepopulationandcommunity/wellbeing/bulletins/measuringnationalwellbeing/2015to2016
**https://whatworkswellbeing.org/2016/07/08/what-wellbeing-inequalities-tell-us-about-the-eu-referendum-result/

Wednesday, 22 June 2016

Give Me Strength (of Evidence)!

I’m asked a lot about evidence in evaluation. For example, how much evidence does our charity need to prove we’re achieving the outcomes our funders expect? What sample size do we need for our survey to be valid? Some published ‘evidence standards’ (e.g. NESTA, EEF*) don’t really help. The problem is, they focus on statistical methods such as Randomised Controlled Trials, which are way beyond the means of many third sector organisations.
 

Go back to a basic definition of ‘evidence’: “that which tends to prove or disprove something”. This means it’s less about specific thresholds, more about a collection of findings which together present a convincing case. So think of evidence as a collection of elements if you like – an ‘evidence bag’; the more you have, the better. Some examples:
 

Theory of Change: In simple terms, this is cause and effect. Can you explain – clearly and convincingly to others – how and why the action you are taking achieves the outcomes you are looking for?


Stories (qualitative feedback): Case studies and other types of narrative feedback are important. Partly because heart-warming stories can influence the public to support fundraising, but also because they can help to prove – through personal accounts – that your Theory of Change is valid.


Hard data (quantitative feedback): These are the surveys or improvement scales that put numbers to what you achieve. Where possible these should cover both the number of people you make a difference to, and extent of improvement they achieve. The umpteen ways of doing this form a subject in themselves, but to begin with, make sure you’re asking relevant questions (the R in SMART – see previous blog).
 

Oh yes: Sample Size. Yes, there are statistical formulae that allow you to work out ideal sample sizes depending on the ‘confidence level’ you want. But in most situations it’s more important to make sure you’ve got a representative sample from all of the people and groups you’re working with. The failures as well as the successes.
 

RCTs (Randomised Controlled Trials): The ultimate in hard data is comparison with a control group who have not experienced your activity of support. But in many socially-based situations this is virtually impossible. To be valid, both the subjects and the control group should have nothing else that changes in their lives apart from your “intervention”. And anyway, would you deny support to someone who needs it just so that they can be part of an RCT control group?
 

Independence: Whether it’s surveys, stories or other feedback, it helps to show they are unbiased. Data collected independently generally has a better chance of avoiding people saying what they think you want to hear, or being too polite to criticise.
 

Triangulation: This fancy term just means getting evidence from more than one source. If a service user says you’ve made a difference, that’s good. If their GP and their family also say you’ve made a difference, that’s even better!
 

Finally, don’t forget the most important question of all: what are you going to do with the information? Learning and improvement should be fundamental to any evaluation. A key test of whether you have sufficient evidence is whether it gives you the confidence to make decisions for the future.


*National Endowment for Science, Technology and the Arts: Standards of Evidence, and Education Endowment Foundation: Security of Findings

Wednesday, 25 May 2016

How SMART Are You?

Allow me a mild rant. I don’t often get annoyed, but this point bugs me – particular when I see influential people repeating the error.

Do you know what SMART, as in SMART targets, stands for? Well, if you said yes, it’s:
S – Specific
M – Measureable
A – Achievable
R – Realistic
T – Time-bound
…then think again.


What’s the difference between Achievable and Realistic? Surely if a target is achievable it must be realistic and vice-versa? I suppose at a push you could argue that a realistic target is one that is measuring things in the right way, hence it’s rational and relevant.


But hang on, there’s several other R’s in there – Right, Rational, Relevant. And these are much more sensible interpretations of the R in SMART. Is the target, and how we’re measuring it, relevant to what we’re trying to achieve? Using ‘Relevance’ as the ‘R’ incorporates the vital principle of cause and effect.


To take an absurd example, would you measure children’s education through their shoe size? Why not? It meets all of the SMART criteria listed above, and can be expected to increase over time as children’s feet grow. Of course, it fails because the measure is nonsense; I’m aware of no research that links shoe size to academic performance. The measure is completely irrelevant.

A more pertinent example was recently published by the Centre for Health Economics, which looked at mortality (i.e. the % who die) of patients admitted to hospital at weekends rather than weekdays. Mortality for patients admitted at weekends is known to be higher, so does this mean we need to improve hospital care at weekends (“24-hour NHS” and all that)?


Well, the paper shows that fewer patients are admitted at weekends, and those that are tend to be more seriously ill, so are more likely to die anyway. It’s not poorer care that makes the difference, it’s sicker patients. Trying to reduce mortality for weekend admissions is therefore the wrong target, because it misses the point about cause and effect. (Jeremy Hunt please note.)


The moral is that handy acronyms (SMART or otherwise) are no substitute for real understanding. Understanding the outcomes that you’re trying to achieve and what works to achieve those outcomes. Only with that understanding can you formulate sound evaluation measures and targets.


PS: If you said SMART stands for Specific, Measurable, Attributable, Realistic, and Time-Bound, then I’ll allow that too. Shows you’re thinking!

Wednesday, 27 April 2016

The Value of Charities

Can competition between charities undermine their value? It’s a question – perhaps a controversial one – that sometimes bothers me.

A recent New Philanthropy Capital (NPC) report Untapped Potential looks at the role of charities in health and social care. This is undoubtedly a field where charities can make a real difference, and the potential is widely acknowledged – not least in the NHS Five-Year Forward View. The NPC report is well worth reading, and two key conclusions are:

  • The health and care system needs to ensure funding and contracting arrangements support collaboration between charities and other providers, and minimise the burdens on providers.
  • The charity sector needs to be wary of ‘asking for more’ in a resource-starved system. Frame conversations in terms of added value, not just the resources required.

These are themes most are aware of and many are actively pursuing. The vast majority of charities know that they need to demonstrate real outcomes, not just outputs, to engage commissioners and other funders. This produces a welcome focus on how charities can maximise their impact, and much current debate centres on the best methods and tools to help them do this. This risk however is that, in demonstrating their value, charities will seek to out-do each other and create competition where there should be collaboration.

When I first heard that Leeds alone had around 3,000 charities, it had me wondering. Too many? Surely there can’t be 3,000 different ‘good causes’? Maybe so, but there are many more than 3,000 people who benefit from these charities’ activities. And all of them are individuals, for whom choice could be a critical consideration. Certainly in the mental health field, where I’ve completed several evaluation projects, any “one size fits all” approach would be completely wrong.

Those who run charities are generally passionate people. They believe in what they’re doing and want to show that their approach delivers results. But surviving and thriving when funding is tight inevitably means an element of competition, whatever the source of that funding. I’ve seen instances where there’s a fine line between providing choice and competing to provide essentially the same service. I can think of some great examples of collaboration between charities (even mergers in a few cases), but I’ve also seen cases where these opportunities have been missed.



Charities provide value from many perspectives: service users, families, wider communities, their volunteers, public services and more. But I believe value should focus first and foremost on the people they aim to help. It should be less about “what value can my charity add?” and more about “what added value do people need, and how can my charity contribute to that?” Please, let’s not create the impression – however misleading – of charities putting their own interests above those of the people they serve.