The Mental Elf

No proof that 85% of mental health apps accredited by the NHS actually work

Depression Apps: in theory

It’s an inconvenient reality that while demand for psychological services seems to be forever increasing, NHS resources designated for the treatment of mental health problems are going the other way. The unfortunate, but inevitable result is that unmet need for NHS mental health services has reached an unprecedented level (Cooper, 2014).

To illustrate the situation, monthly referrals to community mental health teams increased over 13% in 2013, and 16% in the case of crisis services (The Mental Health Policy Group, 2015), yet roughly 200 full-time NHS mental health doctors, and 3,600 nurses were lost over the same period (Cooper, 2014).

This has meant that despite the ‘new NHS standard’ whereby 95% of those with a mental health issue are to be seen within 18 weeks (NHS England, 2015), 1 in 10 are waiting over a year before receiving any form of treatment, with 1 in 2 waiting over 3 months (We Need To Talk Coalition, 2013).

While this is a problem in itself, a seemingly much bigger problem is the reality that 1 in 6 of those on waiting lists for mental-health services are expected to attempt suicide, 4 in 10 are expected to self-harm and 6 in 10 will likely to see their condition deteriorate before having the opportunity to see a mental health professional (Cooper, 2014).

So how should the NHS and its patients respond? With the widespread availability and increased reliance upon smart phones, one increasingly popular suggestion is the use of apps. Our own experiences can tell us that apps are relatively inexpensive and widely available, but probably most importantly, the use of an app by one person won’t prevent another using the same service at the same time. Given the history of long waiting lists for mental health services this is a highly desirable trait for future NHS services, which could result in flexible, user-led healthcare delivery.

MIND have reported that just 50% and 13% of people currently have a choice as to when and where they receive therapy respectively (We Need To Talk Coalition, 2011), and as such, it’s also possible that the wider-spread use of apps could extend the reach of traditional mental-health services to those who, for one reason or another, are not currently able to engage with treatment. Examples may include the teenager who is too anxious or stigmatised to discuss his condition face-to-face, the armed forces serviceman for whom a desire for anonymity is paramount (Murphy & Busuttil, 2014), or the single-mother who struggles to schedule an appointment around her childcare and work commitments.

With 1 in 10 people with a mental health problem waiting over a year for treatment, perhaps apps can help cut the queues?  
With 1 in 10 people with a mental health problem waiting over a year for treatment, perhaps apps can help cut the queues?

Depression Apps: in practice

Regrettably, the reality is that there’s currently a considerable gap between the benefits that apps may provide in theory, and what they are likely to deliver in practice.

Taking a look at the NHS Apps Library, there are currently 27 mental health apps accredited for use by patients, with a total of 14 designated for the treatment/management of the symptoms of depression and anxiety. Upon close inspection, terms such as ‘control stress’, ‘increase wellbeing’, ‘beat depression’ and ‘improve mood’ are frequently listed benefits from downloading, and often purchasing these apps.

Worryingly though, just 4/14 are able to provide any tangible evidence of outcomes, as reported by real-world users, to substantiate their claims, while just 2/14 make use of NHS-validated performance measures including the Generalised Anxiety Disorder 7 (GAD-7) questionnaire, which is routinely used to assess the effectiveness of other NHS-accredited treatments, including counselling and cognitive-behavioural therapy.

As a result, we are currently facing an open question regarding the true-effectiveness of the remaining 85% (12/14) of NHS-accredited mental health apps.

The majority of depression and anxiety apps accredited by the NHS Apps Library have no tangible evidence of outcomes to substantiate their claims.
The majority of depression and anxiety apps accredited by the NHS Apps Library have no tangible evidence of outcomes to substantiate their claims.

Room for concern?

In 2013, a review of mHealth apps found that from 2003-2013 just 32 articles were published regarding depression apps, compared with a total of 1,536 available for download (Martínez-Pérez et al, 2013). While this finding of a high availability, but low underlying evidence-base is concerning, it could arguably be expected from the open and largely unregulated free-markets that are the app stores. In contrast, the apps under consideration here are the beneficiaries of a ‘seal of approval’ from a world-leading healthcare system, and as a result, the expectation is that they are of significantly greater quality.

This is a worrying situation. Most of us would acknowledge that there is a perceived implicit level of quality that comes with accreditation or association with the NHS, with reputation and legitimacy of sources known to be highly correlated with app downloads (Dennison et al, 2013). Considering that 3 in 10 individuals with an untreated mental health issue currently opt to pay for treatments privately (We Need To Talk Coalition, 2011), the recommendation of, purchase and use of apps that are yet to demonstrate any objectively measurable benefits to users, is not only a potential waste of money, but could also potentially have a compounding and devastating effect on levels of anxiety in those with the greatest need and the least access to effective NHS-led mental health services.

The NHS Apps Library has closed after just 2 years.
The NHS Apps Library has closed after just 2 years.

Moving forward

On the large part, the National Health Service provides a regulatory framework that is second to none, with the attention to detail, rigour and emphasis on safety, clinical quality and cost-effectiveness, seeing NICE tokened as the ‘4th hurdle’ to market access. Unfortunately it would appear that this same level of rigour has not been applied to the apps that the NHS has, until now, recommended to patients.

Although the NHS is pushing for a 21st century approach to healthcare, it’s important that this isn’t achieved through a dilution of quality, and fortunately it would appear that the NHS are now taking this subject more seriously. As of October 16th, the NHS Choices Health Apps Library will officially cease to exist, with the National Information Board considering how alternative models for assessing and regulating health apps may be put in place, and ultimately how quality control can be improved.

In the meantime, until such a framework exists, it’s imperative that those considering downloading mental health apps take a moment to weigh up the available evidence, in order to ensure that apps don’t result in more harm than good. Whilst the app store is often slim on technical information, and sifting through medical publications is far from ideal, there are some clear indications of quality to look out for:

  • Firstly, apps supported by a mental health practitioner are on average more than twice as effective as those from non-practitioner led developers (Richards & Richardson, 2012)
  • Secondly, apps with approval from other well-established regulatory bodies including the US Food and Drug Administration (FDA), can act as an intermediary quality control and help separate those apps which offer users hope, and those which offer real proven solutions
  • Thirdly, are apps forthcoming with the information they provide? It’s easy enough to say ‘this app beats depression’ but do they offer any proof to turn this from what is essentially marketing into evidence of clinical effectiveness?

Finally, it’s worth re-enforcing, that not all mental-health apps are created equally, and that some, designed with clinical quality and effectiveness in mind, are providing real solutions and support to their users. One such app, ‘Big White Wall boasts recovery rates of 58%, which contrasts with the 44% exhibited by the NHS’s flagship ‘Increasing Access to Psychological Therapies’ (IAPT) initiative over the same period, demonstrating that if done properly, apps really can improve people’s mental health, at a low cost and from the comfort of their own home.

https://www.youtube.com/watch?v=m0Tw0u_wEJY

Links

Primary paper

Leigh S, Flatt S. App-based psychological interventions: friend or foe? Evidence Based Mental Health 2015 doi:10.1136/eb-2015-102203

Other references

Cooper, C. (2014) Thousands attempt suicide while on NHS waiting list for psychological help. Independent, 16 Sep 2014.

A manifesto for better mental health (PDF). The Mental Health Policy Group, 2015.

Guidance to support the introduction of access and waiting time standards for mental health services in 2015/16 (PDF). NHS England, 2015.

We still need to talk A report on access to talking therapies (PDF). We Need To Talk Coalition, 2013.

We need to talk: getting the right therapy at the right time (PDF). We Need To Talk Coalition, 2011.

Murphy D, Busuttil W. (2014) PTSD, stigma and barriers to help-seeking within the UK Armed Forces (PDF). J R Army Med Corps 2014;0:1–5. doi:10.1136/jramc-2014-000344

Generalised Anxiety Disorder Assessment (GAD 7). Patient website, last accessed 12 Oct 2015.

Martínez-Pérez B, de la Torre-Díez I, López-Coronado M. (2013) Mobile Health Applications for the Most Prevalent Conditions by the World Health Organization: Review and Analysis. J Med Internet Res 2013;15(6):e120 DOI: 10.2196/jmir.2600

Dennison L, Morrison L, Conway G, Yardley L. (2013) Opportunities and Challenges for Smartphone Applications in Supporting Health Behavior Change: Qualitative Study. J Med Internet Res 2013;15(4):e86 DOI: 10.2196/jmir.2583

Richards D, Richardson T. (2012) Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329-42. doi: 10.1016/j.cpr.2012.02.004. Epub 2012 Feb 28. [PubMed abstract]

Add a comment
  • Mental health apps for young people: an evidence-free zone? - National Elf Service

    Mental health apps for young people: an evidence-free zone? - National Elf Service

    8 years ago
    […] No proof that 85% of mental health apps accredited by the NHS actually work […]
  • Barriers to evidence-based practice for app-based psychotherapies

    Barriers to evidence-based practice for app-based psychotherapies

    9 years ago
    […] 2013). Even with respect to NHS-accredited app-based psychological interventions, historically, as few as 15% have been backed by data to corroborate claims of effectiveness (Leigh et al, […]
  • No Evidence for the Mental Curing APP | wongkaiwong

    No Evidence for the Mental Curing APP | wongkaiwong

    10 years ago
    […] http://www.nationalelfservice.net/mental-health/depression/no-proof-that-85-of-mental-health-apps-ac… […]
  • Evidence-based research & fast paced technology development #Mindtech2015

    Evidence-based research & fast paced technology development #Mindtech2015

    10 years ago
    […] a recent Mental Elf blog by health economist Simon Leigh (who is here today) showed that 27 mental health apps were […]
  • Is clinical research essential to develop a good mental health app? #mindtech2015 | The New Mental Health

    Is clinical research essential to develop a good mental health app? #mindtech2015 | The New Mental Health

    10 years ago
    […] in a discussion at the Mental Elf site about the lack of clinical evidence for many popular depression apps I said: “At the moment, rather than being a true partner The NHS is more like an in-law, tutting […]
  • Mental_Elf

    Mental_Elf

    10 years ago
    @MarkOneInFour cites our recent blog https://t.co/QhrKDb5UzL #mindtech2015
  • DrBastianSeidel

    DrBastianSeidel

    10 years ago
    RT @pash22: @TcameronTodd @DrBastianSeidel Mental health apps have a woeful evidence-base https://t.co/F30waFzOVl via @Mental_Elf
  • pash22

    pash22

    10 years ago
    @TcameronTodd @DrBastianSeidel Mental health apps have a woeful evidence-base https://t.co/F30waFzOVl via @Mental_Elf
  • Maite_San

    Maite_San

    10 years ago
    RT @Mental_Elf: Are you developing mental health apps in a truly collaborative way? Share your experiences on our blog >>> http://t.co/WCZg…
  • App_Insight_

    App_Insight_

    10 years ago
    Very interesting article (and others' comments are possibly even more interesting). I'm an app developer and thought I would drop in my two penneth worth... A 'traditional' app developer view might be something along the lines of "if you want better apps you need innovation, and to get innovation you need many apps, many attempts and many failures (ideally with the failures occurring rapidly) so their demise leads to a better next generation of apps - and if you try to regulate too much you will strangle innovation..." I think I probably agree with these "let a thousand flowers bloom" and "let the cream rise to the top" approaches - BUT not at the cost of putting out apps that do harm… So how do we improve quality of the apps? The usual regulatory approach would be to test the app against a series of standards - probably at the end, or close to the end of the development process - and it sounds like this will continue to be needed in some form. How about at the other end of the development process where people are dreaming dreams and putting together app development projects? It is not uncommon for me to be involved in (non-health related) app development projects where some of the basics haven’t been thought through - which usually means the resulting app didn’t end up as good as it might otherwise have. I bet that if a health-related app development project has input from clinicians, users and developers, has developed other partnerships as needed, has spent time on the reasons for the app, is clear on what the app should do, has sorted out metrics for success, AND processes for testing the app - then it’s much more likely to create a better app. I’m slightly involved in a thing called Code4Health (https://code-4-health.org) as a sub-contractor running introductory coding workshops around the country. I really like the things they are doing around building communities of innovation and should think what they do will be of interest to others here.
  • RedFlanno

    RedFlanno

    10 years ago
    RT @meta4RN: Mental health problem? There's an app for that... but it is it any good? http://t.co/6a9gQgIIHh #mHealth HT @Mental_Elf
  • edtangent

    edtangent

    10 years ago
    http://t.co/lshH5WoHDi @VictoriaBetton
  • Aislinn Bergin

    Aislinn Bergin

    10 years ago
    The mention of Happtique in the primary paper is interesting considering what happened to them has also happened to the NHS Apps Library – a publication reveals security flaws in the apps they endorse and they slowly bow out of existence (http://mobihealthnews.com/28165/happtique-suspends-mobile-health-app-certification-program/ and http://www.ukauthority.com/news/5698/nhs-england-closes-health-apps-library). A potential user has far more of an issue with whether an app is going to leak their health data (which by the way we would pay $59.46 to protect https://hbr.org/2015/05/customer-data-designing-for-transparency-and-trust?utm_campaign=Socialflow&utm_source=Socialflow&utm_medium=Tweet) than whether it has an evidence-base behind it. The reality is that we find it easy to download, critique and delete an app (although dropping out of an IAPT appointment appears to be nearly as easy!). As Leigh mentions in the paper, engagement can be incredibly important when it comes to achieving outcomes and technology has many tricks up its sleeve. Others have already pointed out the impossible dilemma that faces all stakeholders in this field of trying to provide evidence or regulation at the rate of innovation and uptake. For developers therein lies the issue of financing, for researchers the issue of development, and for the NHS the issue of both. I can see many ways in which technology helps in the area of mental health, not just by providing an intervention but also providing an adjunct to therapy, a pathway to an appropriate service, and a way to improve mental health literacy. This means accepting outcomes other than validated metrics, although they of course play a role. In a population that goes online before going to a doctor (http://www.pharmafile.com/news/497975/uk-patients-self-diagnose-rather-visit-gp), must wait for months to access psychological therapies, and who often wants to just deal with it themselves (http://www.ncbi.nlm.nih.gov/pubmed/23931656) I believe there is a role for apps other than those delivering evidence-based interventions. But can we prevent harm? I believe the greatest harm comes from a lack of security and safety. The nocebo effect is used to describe when therapy or a supposedly harmless placebo has a harmful effect. A mental health app has the potential to cause harm through a failure to consider the unintended consequences and through actively causing a negative thought or behaviour to be reinforced. It might also cause someone to consider it an alternative and not find appropriate help. How can this even be detected? In the real world all our regulations and research often fails to account for actual usage. Some attempts have been made to find a solution for health apps in general – the British Standards Institute (http://shop.bsigroup.com/forms/PASs/PAS-2772015/), the Digital Health and Care Alliance, the European Commission (https://ec.europa.eu/digital-agenda/en/mhealth), the MHRA – but most are faced with exactly the same problem that the FDA and NHS face. It is reliant on the developer’s intended use. The use of words like ‘anxious’ and ‘stress’ to describe our general day to day wellbeing places mental health as a concept far from the concept of mental illness (this was also discussed in CMO Annual Report 2013) and opens up apps to an avoidance of regulation, even within a structure that purports to cover it. Does this mean that all NHS accreditation of mental health apps is doomed to fail? No, but it requires a different approach. Rather than focusing on finding evidence for an apps efficacy for a specific condition (and thus placing us into another decade of quackery and avoidance of responsibility before research can catch up) it should be based on the need it can provide for. Those outside of mental health services who need encouragement to seek help should have apps designed to provide this and their value appropriately weighted – many apps with moderated forums already do this. Support on the days without therapist contact. An app that can help them keep on top of the CBT skills they learn. Somewhere they can track their physical and mental health. Evidence where evidence is due – if an app delivers an intervention it should be evidence-based but there are already studies out there that have proven how effective certain functions are (for instance self-monitoring in http://www.biomedcentral.com/1471-244X/13/312). It is also important that their users, possibly measured through reviews or usage analytics, accept them. But most important is that they are intentionally aimed at a relevant and appropriate population as a way to help avoid any possible nocebo effects.
  • 1in4Network

    1in4Network

    10 years ago
    RT @clarkmike: NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co…
  • DrAnnaLCox

    DrAnnaLCox

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • nchadborn

    nchadborn

    10 years ago
    RT @clarkmike: NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co…
  • DBadenoch

    DBadenoch

    10 years ago
    RT @clarkmike: NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co…
  • 121Therapy

    121Therapy

    10 years ago
    RT @clarkmike: NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co…
  • Mental_Elf

    Mental_Elf

    10 years ago
    RT @clarkmike: NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co…
  • clarkmike

    clarkmike

    10 years ago
    NHS Health apps library currently closed - data privacy issues, no proof MH apps work - @Mental_Elf takes a look http://t.co/iiRNqImNt1
  • RobertaBuchan1

    RobertaBuchan1

    10 years ago
    RT @ideas4thefuture: No proof most NHS accredited mental health apps actually work http://t.co/yq0SKSmkY3 via @Instapaper
  • ideas4thefuture

    ideas4thefuture

    10 years ago
    No proof most NHS accredited mental health apps actually work http://t.co/yq0SKSmkY3 via @Instapaper
  • rorymb

    rorymb

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • rorymb

    rorymb

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • CatherineD76

    CatherineD76

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • allartmarkets

    allartmarkets

    10 years ago
    No proof that 85% of mental health apps accredited by the NHS actually work https://t.co/HSlivpxYjM
  • ideas4thefuture

    ideas4thefuture

    10 years ago
    No proof most NHS accredited mental health apps actually work http://t.co/yq0SKSmkY3
  • Eleni Chambers

    Eleni Chambers

    10 years ago
    Eleni Chambers liked this on Facebook.
  • AriesStudyUCL

    AriesStudyUCL

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • BuffDavis

    BuffDavis

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • Anna Moore

    Anna Moore

    10 years ago
    There's a recent NICE guideline/review on effectiveness of digital in adolescents that shows a positive effect. Difficulty is that for some it really works and you get a good result (the analysis from BWW shows this I think) but for others it's really not effective. This means average effect size in a pop can either look modest or impressive depending on how you cut the data. Suspect that trick is working out what works for whom. Issue regarding validity/efficacy a challenge. Tension between developing innovation in timescale that enables a startup to launch app and test, and getting enough users so that evaluation is possible and things can be iteratively improved. On one hand we really need innovation in NHS on other we prefer for RCTs before we decide it's good enough. I've no idea what's right, but it seems to me that until we create environment that enables innovation we won't ever get to point where we have great solutions that help with some of our current challenges.
    • 1 reply
    • Simon Leigh
      Simon Leigh 10 years ago
      Hi Anna, I think these are all very good points around the issue, thank you for getting involved. Regarding average treatment effect, it may be that we are able to identify which sub groups of individuals, whether by age, or specific type of mental illness etc. are more conducive to successful outcomes using apps. One possibility regarding the selective reporting of results is a kind of clinical audit prior to NHS approval. If apps are used and downloaded through the android/apple app store and not the NHS health apps library, there is potential that data reported could be contrasted against sales data to gain an estimate of what proportion of total available results are currently being reported. Alternatively if these apps are obtained via other means this obviously becomes much more difficult. R.e. the desire for RCTs and strong credible evidence, there is that clear trade off between getting things to market for the potential gains to users right now, and the alternative of being certain before hand that these apps are effective before widespread use (or that they at least 'first do no harm'). The European medicines agency (EMA) have proposed ways around similar problems for promising and novel, yet currently un-assessed/un-regulated pharmaceuticals. Their requirements are along the lines of, if market access is granted early, all evidence resulting from use within the healthcare system is transparent and ready for analysis by regulators in the form of real-world evidence (RWE), perhaps a similar system would work here. NICE are now increasingly accepting RWE as a valid form of evidence due to increased validity compared to often cherry picked RCT participants, and are becoming less reliant on the gold standard RCT. I also believe the cancer drugs fund will soon trial/are in the process of trialling a similar initiative, so there are some possibilities there for a similar initiative. I think one of the most important first steps from the NHS would be to openly announce what is acceptable and what isn't and provide some form of framework such that developers know what is expected from the start. This way we can design the development of apps more efficiently and give clear guidance to developers as to which apps (and what level of underlying evidence) are likely to receive accreditation by the NHS. I suppose they're currently not sure exactly what it is the NHS want given the variable standards (in terms of evidence generation and design) shown in those accredited so far. Kind regards, Simon
      • @markoneinfour
        @markoneinfour 10 years ago
        I think for developers the most important thing is being able to cost the process into the overall cost of development. Guidance will help to clarify this greatly. It's exceptionally important that NHS can illustrate at least guide price for the process of evaluation and testing. The only way that investment is really viable is if costs like this can be confidently quantified. It will also, perhaps, bring a sensible discussion to the fore about where the investment for such services and apps might come from and what, if any, the value of the potential market actually is. I don't say this from a cut throat business point of view; I say this from a good five or six years experience of not being able to give very definitive advice about the business models for delivering mental health relevant services in a sustainable way. @markoneinfour
  • sonicresolution

    sonicresolution

    10 years ago
    RT @vinylarm: No proof most NHS accredited mental health apps actually work http://t.co/aHv4EtCE8e #Apps #MentalHealth #EvidenceBase #Depre…
  • ArianePSY

    ArianePSY

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • bromleydiabetic

    bromleydiabetic

    10 years ago
    RT @vinylarm: No proof most NHS accredited mental health apps actually work http://t.co/aHv4EtCE8e #Apps #MentalHealth #EvidenceBase #Depre…
  • vinylarm

    vinylarm

    10 years ago
    No proof most NHS accredited mental health apps actually work http://t.co/aHv4EtCE8e #Apps #MentalHealth #EvidenceBase #Depression
  • adthompson4

    adthompson4

    10 years ago
    RT @SalfordRD: An interesting read, showing need for more research into health apps | #askforevidence | http://t.co/VhRA862CAv https://t.co…
  • LeslieCannold

    LeslieCannold

    10 years ago
    RT @meta4RN: Mental health problem? There's an app for that... but it is it any good? http://t.co/6a9gQgIIHh #mHealth HT @Mental_Elf
  • Sue Sibbald @BPDFFS

    Sue Sibbald @BPDFFS

    10 years ago
    I use apps not ones the NHS have on their list but ones I have tried myself or my peers have recommended . I take them or leave them as I please. Honestly it doesn't always take a professional to tell me what's good for me I can make that decision. Apps are out there people are using them and meanwhile the NHS trudges miles behind ... We are just experimenting ourselves I like it better that way .
  • OTBeth131

    OTBeth131

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • HStonebank

    HStonebank

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • JeffreyGuterman

    JeffreyGuterman

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • MarieLouiseLu

    MarieLouiseLu

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • MichaelFarrellE

    MichaelFarrellE

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • amal_elan

    amal_elan

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • MHFEwhatsnew

    MHFEwhatsnew

    10 years ago
    RT @Mental_Elf: Don't miss: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQBbXq #EBP
  • loueymarge

    loueymarge

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • JoeJudgePsy

    JoeJudgePsy

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • soniajohnson

    soniajohnson

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • gamingthemind

    gamingthemind

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • The Mental Elf

    The Mental Elf

    10 years ago
    The Mental Elf liked this on Facebook.
  • Bethan Mair Edwards

    Bethan Mair Edwards

    10 years ago
    Bethan Mair Edwards liked this on Facebook.
  • LizHughesDD

    LizHughesDD

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • 100prayingwomen

    100prayingwomen

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • aijeria

    aijeria

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • And_Cipriani

    And_Cipriani

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • CSETDartmouth

    CSETDartmouth

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • managing_stress

    managing_stress

    10 years ago
    RT @Mental_Elf: Are you developing mental health apps in a truly collaborative way? Share your experiences on our blog >>> http://t.co/WCZg…
  • CatalyseC

    CatalyseC

    10 years ago
    RT @Mental_Elf: Are you developing mental health apps in a truly collaborative way? Share your experiences on our blog >>> http://t.co/WCZg…
  • samillaiee

    samillaiee

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • ruth_blackburn

    ruth_blackburn

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • samillaiee

    samillaiee

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • samillaiee

    samillaiee

    10 years ago
    RT @Mental_Elf: Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.c…
  • Anon

    Anon

    10 years ago
    This is a very interesting article and I think it is a good thing that light is finally being shone on the shambolic NHS health apps library which I have been involved with since it's launch several years ago. I was involved with a team that was one of the first apps on the library, in fact we were one of the launch apps. The on boarding process and validation of apps was awful with very little if any that I could see clinical input. It was all very rushed in time for the NHS Innovation Expo this year and at the time I was surprised at the lack of clinical evidence needed to get onto the library. I do know that we had to answer a lot of questions and do a lot of paperwork but since the launch it seems more and more apps have been added with less and less information so I wonder what process they had to go through to get onto the library and how rigorous it was. Since then I have heard the library referred to the "NHS' best kept secret" because any news of publicity of it has been absolutely buried and the NHS stopped talking about it at all about a year after launch. Since then we have had talk of accreditation and kite marking but this seems to change on a monthly basis making it very difficult for both consumers (to know what apps are safe and effective) and developers (to know what hoops they will have to jump through). I'm also curious as to what the article means by clinically validated as many of the tools on the library are not clinical interventions and so do not necessarily need to go through the same rigorous testing. Also what standard of clinical evidence was required for apps to pass the study's test? If it was an RCT then that potentially makes the statistics in this article very misleading as many innovations and tech products are not suitable for this kind of evaluation - something that has been argued for a long time now. As Mark has already pointed out, one of the main issues is that the majority of people involved in this kind of work from the NHS' side of things do not know what they are looking for
    • 1 reply
    • Simon Leigh
      Simon Leigh 10 years ago
      Hi, I found your comment around seemingly unending goalpost shifting interesting. I imagine a lot of this was ultimately down to the NHS jumping in at the deep end and then 'learning' as they progressed with the apps library, unfortunately though it is still unclear exactly how apps were appraised before acceptance into the library. To answer your question, the methodology of the analysis was simple, and we tried to keep things as broad and inclusive as possible. We looked at what apps claimed they did/could do, on the NHS apps library, both android, and apple app stores and any websites dedicated to these apps. We then simply looked for 'any' evidence to corroborate these statements, whether a report that stated 100 people used the app and 50 had an X% reduction in the symptoms of depression, or much more rigorous in terms of an RCT or an observational study. We then looked (for any study where evidence was provided) at the quality of this evidence in comparison to the quality of evidence we may expect if evaluating other NHS approved activities, including antidepressants, counselling etc. Our criteria for NHS approved/generalisable metrics were that evidence must be in the form of a generalised anxiety disorder -7 question scale (GAD-7), WEMWBS or PHQ-9, as these are commonly used within the NHS and are the metrics of choice for comparator and substitute therapies provided by the NHS' flagship Increasing Access to Psychological therapies (IAPT) initiative. I hope this helps clarify things. Kind regards, Simon
  • viscidula

    viscidula

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • samillaiee

    samillaiee

    10 years ago
    RT @Mental_Elf: Are you developing mental health apps in a truly collaborative way? Share your experiences on our blog >>> http://t.co/WCZg…
  • sonsonoax

    sonsonoax

    10 years ago
    RT @cgblanch1: La mayoría de las apps para depresión y ansiedad acreditadas por el NHS no tienen ninguna evidencia de su utilidad http://t.…
  • BINGBINGBLOG

    BINGBINGBLOG

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • @markoneinfour

    @markoneinfour

    10 years ago
    This is a useful article. I've contended for a while that the NHS is in the mire when it comes to assessing the affective elements of apps and online services. It doesn't really know what it's look for, how to look for it or even how to understand it if it finds it. There's an attitude of refusing to meet the developer world half way; almost looking forward to the opportunity to say 'you've got that wrong' tutting and shaking its collective professional head while failing to acknowledge that it has done nothing to help anyone do anything 'right'. At present the interface between mental health and tech development is poor at best. Clinicians consistently undervalue the profession of app development and user-centred design, assuming that 'anyone can do it' whilst often failing to put into the public domain useful knowledge that that would enable developers to bring their skills to bear upon problems and challenges in mental health in an evidence informed way. It's correct that many mental health apps are crap. But much of just about anything is crap. Judgement of some individual examples should not be used as a ruling on the potential of a field. At present there is a wall between the people doing the tech thinking (who will often tend to work in for profit business) and those who 'know' about mental health who will tend to work within the NHS, charity sector or academic institutions. Scepticism about the benefits of technology leads to a stunted, awkward interaction between those sides where the common ground cannot be established to develop useful, measurable apps and services. There is often a refusal on the public sector side this relationship to discuss business and revenue models for apps, and a refusal to discuss the financial challenges of trialling and iteratively developing mental health apps and services. At present, often the only way of robustly trialling mhealth apps is to find a university department that a) has funding and b) is already thinking a long the same lines. The problem or challenge is that the tech possibility runs far quicker than the mental health world's ability to get it's head around it and translate that into research and funding decisions. As the holders of both the purse strings, and the knowledge assets for mental health, the NHS should become an investor of first choice if it is serious about getting the mental health apps and digital services that it appears to want. It can bring the knowledge assets and the framework for interacting with possible trial subjects and with existing intelligence and can smooth path for people to carry out proper user-centred app design by working with people who actually experience particular conditions and illnesses. At the moment, rather than being a true partner The NHS is more like an in-law, tutting and shaking its head that tech is doing it wrong but giving no guidance as to how mental health apps might do it right. @markoneinfour (dashed this off on the train, may return to it later)
    • 1 reply
    • Andre Tomlin
      Andre Tomlin 10 years ago
      Great post Mark! :-) I completely agree with your point about more joined up working. The best apps come from true collaboration with a mix of service users, professionals, developers, researchers and funders. So who is developing apps in this way? Let's hear some inspirational stories. Cheers, André
  • TayloredPsychol

    TayloredPsychol

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • EllenGrimas

    EllenGrimas

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • cgblanch1

    cgblanch1

    10 years ago
    La mayoría de las apps para depresión y ansiedad acreditadas por el NHS no tienen ninguna evidencia de su utilidad http://t.co/Tzf2LtB5va
  • MigraineAgain

    MigraineAgain

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • jselanikio

    jselanikio

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • NeedMoreCourage

    NeedMoreCourage

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • brhaspati108

    brhaspati108

    10 years ago
    RT @meta4RN: Mental health problem? There's an app for that... but it is it any good? http://t.co/6a9gQgIIHh #mHealth HT @Mental_Elf
  • Girl_Interrupt_

    Girl_Interrupt_

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • MHNEtweets

    MHNEtweets

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • ClinicalMonkey

    ClinicalMonkey

    10 years ago
    RT @Mental_Elf: What makes a good, scientifically credible, well-validated health app? http://t.co/WCZgEQBbXq http://t.co/9Q3PEaMoTK
  • CatalyseC

    CatalyseC

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • t_unit2010

    t_unit2010

    10 years ago
    RT @Mental_Elf: Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.c…
  • NeedMoreCourage

    NeedMoreCourage

    10 years ago
    @Mental_Elf Curious, what is their approval process & surely such apps are medical devices which have strict criteria? Who defines if meddev
  • ClaireBentley2

    ClaireBentley2

    10 years ago
    RT @meta4RN: Mental health problem? There's an app for that... but it is it any good? http://t.co/6a9gQgIIHh #mHealth HT @Mental_Elf
  • NeedMoreCourage

    NeedMoreCourage

    10 years ago
    RT @Mental_Elf: Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.c…
  • BipolarCONBRIO

    BipolarCONBRIO

    10 years ago
    RT @Mental_Elf: Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.c…
  • wschroederMMFT

    wschroederMMFT

    10 years ago
    RT @Mental_Elf: Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.c…
  • Mental_Elf

    Mental_Elf

    10 years ago
    Most NHS accredited depression & anxiety apps have no tangible evidence of outcomes to substantiate their claims http://t.co/WCZgEQBbXq
  • Anon Ymous

    Anon Ymous

    10 years ago
    "...it’s imperative that those considering downloading mental health apps take a moment to weigh up the available evidence, in order to ensure that apps don’t result in more harm than good." It may be imperative, but, from my own experiences, this is not a population likely to be able to do so. It's hard enough simply trying to get information on local mental health services online, nevermind wading through a mire of apps all promising benefits and evaluating there respective positives, negatives and evidence bases.
    • 1 reply
    • Andre Tomlin
      Andre Tomlin 10 years ago
      So how should we support the general public to become more discerning and savvy about app quality? Or are you saying that this is just impossible from the word go? How do people assess quality right now? Simon's point is about the trust we all put in the NHS brand, which when attached to a collection of apps, heavily implies that these apps are safe and effective. Unfortunately this is not the case. I think the general public are a lot more discerning about health websites than they were 10 years ago. We have all learnt from initiatives like DISCERN, even if these are skills we've picked up surreptitiously along the way: http://www.discern.org.uk/discern_instrument.php I guess the world of apps is still so young that we are still searching for the best ways to critically analyse them. For me, the core principles are the same. We need to be assessing accessibility, usability and reliability. Cheers, André
      • Anon Ymous
        Anon Ymous 10 years ago
        My comment is only that it is un-realistic to believe that someone in the throws of what could be their first episode of depression is well suited, or able, to evaluate (on their own) the relative evidence-based merits of different options for apps to self-manage their condition. If you've barely got the energy or willpower to take a shower, or get dressed, are you really going to have the energy to research multiple apps? If you already have huge doubts about your own ability to do things, are you going to trust in your own ability to 'choose the right app' among the hundreds claiming effectiveness, when you're feeling completely hopeless anyway? I would argue that it's pretty unlikely not. A rubber stamp of approval from something like the NHS, or MIND, makes these choices far easier. Sources you're likely to be looking through, or for, anyway. Which is why I whole-heartedly agree with Simon's point about trust in the NHS and people's assessment of the value of an app they advise. Trust which, depending on the app, may be misplaced. My feeling is that this is an ideal area in which the NHS, through the NIHR, should be investing in. They should be producing an app and evaluating it as rigorously as they are able. In partnership with someone, preferably. I think there is enormous, positive scope for such an app to help those with depression, or other mental health issues. I would agree that the general public are far better at assessing these kinds of things than they were ten years ago, but that doesn't necessarily apply to someone with depression. It may not be that they are able; they might just not care.
  • DBadenoch

    DBadenoch

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • JigsawDublin15

    JigsawDublin15

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • mndsci

    mndsci

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • bobjerunkle

    bobjerunkle

    10 years ago
    @LouMatter @forestimaginatn @IsmaelVelasco @towndigitalhub http://t.co/wyvrlRuQNw
  • DrIanDawe

    DrIanDawe

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • michaeldalili

    michaeldalili

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • MFargas_ICCR

    MFargas_ICCR

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • yayayarndiva

    yayayarndiva

    10 years ago
    No proof most NHS accredited mental health apps actually work http://t.co/O3eQRJBSW0 via @rightrelevance thanks @mental_elf
  • painoccthy

    painoccthy

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • deborahapthorp

    deborahapthorp

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • raluca_lucacel

    raluca_lucacel

    10 years ago
    No proof that 85% of mental health apps accredited by the NHS actually work https://t.co/vXb6GQ2g1f via @sharethis
  • 121Therapy

    121Therapy

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • C_2me

    C_2me

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • felly500

    felly500

    10 years ago
    but...some proof that 15% of mental health apps accredited by the NHS DO actually work? http://t.co/pOjIBRYGBk
  • felly500

    felly500

    10 years ago
    "No proof that 85% of mental health apps accredited by the NHS actually work" http://t.co/pOjIBRYGBk
  • annelisedd

    annelisedd

    10 years ago
    RT @Mental_Elf: 1 in 10 people with a #MentalHealth problem wait over a year for treatment. Perhaps apps can help cut the queues? http://t.…
  • PWGTennant

    PWGTennant

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Bethan Davies

    Bethan Davies

    10 years ago
    Bethan Davies liked this on Facebook.
  • Hampshire Healthcare Library Service

    Hampshire Healthcare Library Service

    10 years ago
    Hampshire Healthcare Library Service liked this on Facebook.
  • HHLibService

    HHLibService

    10 years ago
    Be wary of claims made by mental health apps http://t.co/cDkZhyTrWo
  • BioethicsUoM

    BioethicsUoM

    10 years ago
    No proof that 85% of mental health apps accredited by the NHS actually work - http://t.co/jTv6UblRvN #mentalhealth
  • Andre Tomlin

    Andre Tomlin

    10 years ago
    Thank you Simon for this great blog summarising an important area. It's certainly true that this message needs to reach the people who are downloading and using apps, which they no doubt presume are reliable. I hope that our readers investigate this issue further and read the free full-text article available on the Evidence-Based Mental Health Journal website: http://ebmh.bmj.com/content/early/2015/09/16/eb-2015-102203.full You mention the demise of the NHS Apps Library, which was recently announced. Within mental health, the apps library has been replaced by a page on the NHS Choices website which brings together a number of leading online mental health services (e.g. Big White Wall, Buddy, FearFighter, Ieso, Kooth, Silvercloud and Sleepio). See more here: http://www.nhs.uk/conditions/online-mental-health-services/pages/introduction.aspx I'm not sure how many of these apps use validated performance measures and have high quality research supporting their safety and effectiveness, but it seems to me that this list has more of a feeling of robust and reliable digital services than the 27 mental health apps previously accredited and included in the old NHS Apps Library. What do you think? It will be interesting to see how the NHS take this forward. I note that Alexia Tonnel from NICE is giving a keynote speech at the MindTech conference in December on the National information Board’s plans to evaluate digital mental health products and make accredited products available via NHS Choices: http://www.mindtech.org.uk/mindtech-annual-conference.html There are a couple of consultation documents on this for readers who want to find out more: https://www.nationalelfservice.net/cms/wp-content/uploads/2015/10/NIB1.2-slides-June-2015_for-consultation.pdf https://www.nationalelfservice.net/cms/wp-content/uploads/2015/10/Workstream-1.2-roadmap-long-version-June-2015.pdf Cheers, André
    • 1 reply
    • Simon Leigh
      Simon Leigh 10 years ago
      Hi André, I share your enthusiasm for engaging with potential users of such apps. While a great number of apps are proving to be very effective (Big White wall and Leso digital health have demonstrated recovery rates far in excess of those achieved by IAPT), the are just as many, if not more, that have little if any credible evidence underpinning them. I would agree that the new approach to offering mental health apps through NHS choices appears more rigourous, and the apps included, more effective than those included in the NHS apps library. However, it would still of great benefit if a structure was formalised sooner rather than later, much like with NICE's health technology assessment of pharmaceuticals and other health generating products, such that developers can have clear indications as to what constitutes a 'good, scientifically credible, well-validated app', and also understand what is classed as an acceptable evidence of effectiveness. Thanks, Simon
  • couns_direct

    couns_direct

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • DochkaH

    DochkaH

    10 years ago
    @Mental_Elf Great blog today! But worrying news...
  • DochkaH

    DochkaH

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • emmalken

    emmalken

    10 years ago
    @Mental_Elf levels of digital participation & more potential to increase health inequality?? Many considerations...
  • StanKutcher

    StanKutcher

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • marionpfallon

    marionpfallon

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • castrocloud

    castrocloud

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • 12ebartram

    12ebartram

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • opiumia

    opiumia

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • SchrebersSister

    SchrebersSister

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Stroppybrunette

    Stroppybrunette

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • Mental_Elf

    Mental_Elf

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • CCreatingSpace

    CCreatingSpace

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • BelindaJPlatt

    BelindaJPlatt

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • AmirSariaslan

    AmirSariaslan

    10 years ago
    RT @And_Cipriani: App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http:…
  • HannanRuth

    HannanRuth

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • And_Cipriani

    And_Cipriani

    10 years ago
    App-based psychologcal interventions: friend or foe? Free @EBMentalHealth http://t.co/7GRxtwPKXq + blog @Mental_Elf http://t.co/7gHAWeuMgU
  • SCEW86

    SCEW86

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Adelina Comas-Herrera

    Adelina Comas-Herrera

    10 years ago
    Adelina Comas-Herrera liked this on Facebook.
  • CG_Sheikh

    CG_Sheikh

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • JohnBaker_Leeds

    JohnBaker_Leeds

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • AlresfordBear

    AlresfordBear

    10 years ago
    @Dr_JB_Kirkbride @Mental_Elf Interesting. Because many people have become heroes in retailing their inclusive development.
  • 121Therapy

    121Therapy

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • LaurPozzi

    LaurPozzi

    10 years ago
    RT @iVivekMisra: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/QFoWPYi22s #MentalHealth http://t.…
  • egosyntonically

    egosyntonically

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • MichaelGFollan

    MichaelGFollan

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Psydocsar

    Psydocsar

    10 years ago
    Be wary of claims made by mental health apps: http://t.co/m5669RzVCt
  • markomanka

    markomanka

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • iVivekMisra

    iVivekMisra

    10 years ago
    No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/QFoWPYi22s #MentalHealth http://t.co/itLusXLGxV
  • PowysTHBLibrary

    PowysTHBLibrary

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Dr_JB_Kirkbride

    Dr_JB_Kirkbride

    10 years ago
    No guaranty of 'appiness, suggests piece from @Mental_Elf https://t.co/1NNdebPNYp
  • CoyneoftheRealm

    CoyneoftheRealm

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • AdelinaCoHe

    AdelinaCoHe

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • drjennybarnett

    drjennybarnett

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • JosefienMHF

    JosefienMHF

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • allan_conor

    allan_conor

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • PSCRuth

    PSCRuth

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY
  • Sean_Hornby74

    Sean_Hornby74

    10 years ago
    RT @Mental_Elf: No proof that 85% of mental health apps accredited by the NHS actually work http://t.co/WCZgEQSMOY