[Editor: We have something a bit different for you today. A guest post from Caroline Fiennes, Director of Giving Evidence, who has today published a new report that explores the ways in which UK mental health charities use evidence in the development of their services. We thank Caroline for this eye-opening piece of work and look forward to working with her more in future.]
UK non-profits delivering mental health services are not great at producing or using scientific evidence.
This is the main finding of a new study by Giving Evidence. We interviewed 12 such organisations to understand their ‘evidence system’, i.e., how evidence is:
- Shared, both ‘outbound’ from them and ‘inbound’ to them – and stored
Mental health research: inaccessible, irrelevant and unusable?
These nonprofits talked of their growing interest in being evidence-based and focusing on impact (and we don’t doubt them) but in practice it’s not happening consistently. Some charities said that they struggle to find and use external research about what is effective in treating or preventing mental health conditions when designing their programmes.
One reason given is the difficulty of accessing, interpreting and applying academic / independent research – for example, much academic research is behind paywalls, so charity staff sometimes resort to sneaking into their former universities to read it, and certainly much of it is pretty unintelligible to non-researchers. Another is the claim that there isn’t much research which is relevant, although that claim is disputed by some experts and researchers.
However, charities delivering mental health services seem laudably interested in the views of their service users. Three-quarters of the charities we interviewed regularly collect user feedback, and over half have done so on a large scale.
About half of these organisations are producing (or funding production of) impact evaluations, i.e., investigations of the causal effects of their interventions, and many of these seem to be simple pre/post studies, which are open to considerable errors. It may be just as well that not all of them are producing such evaluations, because doing unbiased evaluation research is a specialism which most service delivery organisations don’t have. Instead, they should (we would argue) be using reliable research from elsewhere, which few are.
One charity said that:
Evidence for us is what our users say works…that is enough for us
This concerns us, because the human mind is often misled about what works and only rigorous research can reveal the reality. Happily some of the non-profits which are involved in producing evaluation research are doing so in partnership with reputable research institutions.
Undervalued and underfunded?
Sadly some charities we spoke to seem to be being forced to produce low quality research. Several told us that funders and commissioners require ‘evaluations’ of services but only put towards them budgets too small to allow for reliable research (e.g., with adequate sample size). Most were only £5-10k, and a few were £20-30k.
For example, one charity said that is has dozens such budgets a year, which is very frustrating because individually, those budgets only allow for research that is essentially pointless, but collectively they could enable something insightful.
Adding to the knowledge pool?
About half of these charities are producing the kind of research or impact evaluations which could be useful to other organisations. Plus, reportedly, “every contract specifies different outcomes, which makes it a nightmare to aggregate” (charity interviewee) and also prevents comparisons. We didn’t have resource to look at the quality of that research, i.e., to see whether it is reliable and useful. However, dissemination of that is weak, and that’s not really the charities’ fault; there’s no incentive for them to do so, and few channels anyway.
One charity said that part of the reason they don’t publish much is that:
We don’t want competitors to pick this [our intervention] up.
We have encountered this in other sectors and this is a major problem (not of the charities’ making).
On the upside, amongst the charities that do produce this kind of material, we found no evidence of selective publication: we had thought we might find that material which is flattering is more often published and unflattering material isn’t, which creates publication bias, but we did not find this.
Brutal under-funding of mental health
This is all in a context of brutal under-funding. Mental health accounts for 23% of the UK disease burden, but gets only 13% of the NHS budget and 5% of the UK health research budget. Moreover, charitable giving to mental health is very low: it’s only £714 for every adult with mental health problems whereas donkeys get £2,047 each.
Using evidence-based mental health research to find out what works
Giving Evidence has long said that most charities should not produce causal research, which requires expertise that they don’t have and don’t need, but rather should get good at hearing from their target users about what they want and think of what they’re getting, and then finding and using causal research about what works in addressing it. That seems to be the case for charities delivering mental health services. Some organisations help with this, such as the Centre for Mental Health, and the Mental Elf.
We recommend that mental health charities work towards (and are funded and incentivised to work towards) finding and applying the relevant rigorous research, and working with specialist researchers to produce research where none already exists. We expect to work with some mental health charities on this.
Buckland L, Fiennes C. (2016) An Exploration of the Evidence System of UK Mental Health Charities (PDF). Giving Evidence, 11 Jul 2016.
@Mental_Elf really useful view @carolinefiennes & seems we can learn from the sector on how to listen & involve service users
Exploring the mental health charity sector evidence system https://t.co/dcvZa5HyCD #MentalHealth https://t.co/dNXDCG5Aht
Today @carolinefiennes explores the evidence system of UK mental health charities https://t.co/4tfVeDXbo7
@Mental_Elf Thanks! C.
@Mental_Elf paywall is crucial issue.Charities underfunded resources used to plug gap in services but not enough budget.
@Mental_Elf Great article & now following @carolinefiennes ?
@Patient_Leader @Mental_Elf Thanks!
@Mental_Elf @carolinefiennes Very thought provoking article. How do small MH charities gather “evidence” rather than local knowledge ?
@Mental_Elf @carolinefiennes Commissioners of VCS services often not competent in own use of evidence so little drive for quality evaluation
@Mental_Elf @carolinefiennes Commissioners will look for ‘good enough’ level of evidence (if any) to make decision to fund.
Most mental health charities fall well below the resource levels required to fund research. I can’t see how it could be considered our responsibility. We cannot conduct research unless we are funded to do so, and as you recognise, few funders do this, and even then it is unrealistic to expect all but the very largest projects to include a research component.
@Mental_Elf it’d be good to see an increase of partnership working to strengthen one another, learn and assist in bridging gaps such as this
@Mental_Elf there’s alot to be learnt from vol sector but alot others could give too given current lack of resources
Mental health charities are picking up the pieces of a totally underfunded mental health sector due to cuts!
Exploring the mental health charity sector evidence system https://t.co/e1QmZOAgLc
Can we be sure that mental health services developed & provided by some charities are doing more good than harm? https://t.co/4tfVeDXbo7
@Mental_Elf in my experience they certainly provide a missing link…
@Mental_Elf interesting but must be set in context of woeful underinvestment / provision appropriate timely statutory services
@Mental_Elf v.interesting. Do you know how to find out about drug research for GAD? Not much on Web?
@Mental_Elf A real concern, ++ NHS provisions farmed out to third sector contractors. What do we really mean by ‘Any Qualified Provider’ ?
An excellent article. Charities must think very very carefully about providing direct services. This being said, the evidence base in many areas is still “thin” so some testing in flight – under carefully controlled conditions – may help sufferers and add to the knowledge base. The real problem however, is that many charities simply do not have a rigorous servuce development and intellectual base. But here again, in an appalling public health service vacuum and austerity impacted community sector, some service is often better than none. Most sufferers would rather be in a programme and on a pathway in an empathetic context than be isolated without support and sitting on an absurd waiting list.
When mental health services started to focus on the concept of “recovery” some non-statutory services were at the forefront of innovative services, however commissioners started to introduce in their service level agreements the “need” for evidence based interventions eg CBT, use of PQ9 and GAD7. One mans IAPT based CBT can be another’s poison. Research and evidence base in mental health is widely recognised as difficult, tainted by the influence of drug companies and schools of therapy with the “gold standard” of RCT questionable, . If we go back to “recovery” principles and people defining their recovery and what’s helpful we need to consider focusing more on personal experience, so called anecdotal evidence.
Exploring the mental health charity sector evidence system https://t.co/4DXNC31XLP via @Mental_Elf
This blog and report is less about charities DOING research and more about them USING research evidence to inform the way their services are developed and run. For example, if a charity had a choice to make about setting up a service for young people with common mental health conditions and were deciding whether to set up an Art Therapy programme or an Outward Bounds programme, it would make sense to look at the published academic research that assesses the safety and efficacy of those programmes before making a decision. It is this use of the research evidence that Caroline Fiennes is suggesting happens much less than it should. I’d be interested in everyone’s experience of this. Cheers, André
Caroline Fiennes here: report co-author. To be clear, we were simply interested to find out what is currently happening (this report was designed to be descriptive, not prescriptive). But obviously we’re interested in what happens next and how to improve the system, and here André’s right: it would seem the case that charities can make better use of the evidence that already exists – even though it is (surprise!) incomplete and imperfect. We don’t say that filling all the research gaps is charities’ responsibility; in fact, we say quite the opposite. What most charities should get good at is finding & using research, not generating it. See http://www.giving-evidence.com/oops for more on that.
It’s a great blog. “Charities delivering mental health services seem laudably interested in the views of their service users”.
It would be really good if those producing evidence, including systematic reviews, were similarly interested in the views of patients, service users, and service providers so they are more likely to produce timely, relevant and accessible evidence which will help charitable organisations improve the health services they provide.
I Chair a Charity, and as Andre says, it’s up to Charities, ideally, to explore the evidence, when decision-making about what to invest in delivering. The problem with this, as I’m sure many know, is that “s/he who pays the piper, calls the tune”; and as much as Charities might prefer to refer to evidence & trials, about what’s likely to work best, but they need to maintain their income. Charities might suggest particular outcome measures, but they will use the outcome measures specified in Contract; though sometimes, that’s negotiable.
There’s a lot, I think in what Michael says about Recovery work and anecdotes. Anecdotal information provides an intuitive starting point for exploring small scale local outcomes, perhaps to be explored in the context of a meta analyses.In particular, which arrangements are likely best delivered in the Statutory Sector and where Charities fit in overall economy of delivery is not really on the cost effectiveness/ cost benefit agenda. Moreover, information about this, is not being systematically collected, let alone analysed.
We also need to be clear on the step before which is are services clear on what they are trying to deliver, statutory services have traditionally been assess, treat mainly symptom reduction, discharge. “recovery” focus is quality of life with or without symptoms and also a value base with a focus on Individual goals, hope etc. I would suggest evaluation of whether the values base is being adhered to is priority, see IMROC’s work – http://www.imroc.org/
How UK mental health charities use evidence in the development of their services: guest blog on @Mental_Elf https://t.co/4YyuClkUMU
Exploring the mental health charity sector evidence system https://t.co/MzZQTl3xBl
Mental health charities!
How do you use research to shape your services?
@carolinefiennes https://t.co/4tfVeDXbo7 https://t.co/Dty6YHXXu0
Interesting article by @carolinefiennes
on how charities use evidence in the development of their services https://t.co/7xJq328krz
UK non-profits delivering mental health services are not great at producing or using scientific evidence https://t.co/4tfVeDXbo7
RT @Mental_Elf: Great guest blog today from @carolinefiennes on how #MentalHealth charities use evidence-based research, or not https://t.c…
Many mental health charities not using academic research available about effectiveness of different interventions https://t.co/4tfVeDXbo7
https://t.co/5oeXZpiNLE #evidencebased within the #mentalhealth #voluntary sector @RayFWalker @JR_MerseyCare @irisbenson100 @donarobbo
Top story: Exploring the mental health charity sector evidence system https://t.co/Hc2FFwTNdT, see more https://t.co/oT5ZtD4hhQ
Don’t miss: Exploring the mental health charity sector evidence system https://t.co/4tfVeDXbo7 #EBP
Exploring the mental health charity sector evidence system. (Thanks for the mention of @CentreforMH) https://t.co/RwgP6wRiU5 via @sharethis
Charitable giving to mental health so low: £714 for every adult w/ mental health problems; donkeys £2,047 each https://t.co/JWDdKbqy6u
@Mental_Elf V interesting piece exploring mental health charity sector evidence system https://t.co/JWDdKbqy6u
Mental health is 23% of UK disease burden but gets 13% of NHS budget & 5% UK health research budget https://t.co/JWDdKbqy6u
Have often thought the reports major charities provide should be subject to more external scrutiny. If researchers wanted to publish similar material it would have to go through peer review. Some reports are hugely biased; yet hugely influential.