Sometimes, while strolling through woodlands for example, I become overwhelmed by questions such as “what exactly is mental disorder?”

In the dark hours of sleepless nights however a darker question occurs: “Is it hubristic to believe that the complex experience of mental phenomena and distress can be affected by chemical substances and talking therapies?”

Occasionally research articles are published that suggest I am not alone in experiencing this form of existential angst. One such article has just been published in JAMA Psychiatry (Huhn et al 2014) looking at the efficacy of pharmacological and psychological treatments in the treatment of mental disorder.

Study aims

1. What are the effects of pharmacotherapy and psychotherapy on the experience of mental disorder?
2. Which areas of treatment have received the least research attention?
3. Are there methodological differences in the conduct of pharmacotherapy and psychotherapy trials that might impact on the interpretation of the evidence base?

Methods

• A search strategy was used to identify systematic reviews and meta-analyses addressing the treatment of various mental disorder diagnoses (a meta-meta analysis?)
• Identified trials within the meta-analyses were assessed for evidence of bias according to:
  • Appropriate randomisation
  • Allocation concealment
  • Blinding (outcome assessment as blinding not possible for therapists in psychotherapy trials)
  • Missing outcome data
  • Control group nature (placebo, treatment as usual, ineffective treatment vs waiting list or no treatment)

Statistical analysis

The following analyses were undertaken:

• Outcome measures were compared in terms of standardised mean difference [SMD] (calculated mean difference divided by standard deviation)
• Sample size comparison and methodological comparison between trials using two quantitative measures
• Subgroup analysis (e.g. between control group nature, any follow-up data etc)
• Mean baseline severity difference between psychotherapy and pharmacotherapy trials (only possible in depressive disorder as insufficient trials in other diagnoses)

The authors emphasise that this should be viewed as exploratory research and not a confirmatory direct comparison between methods.

Results

Pharmacotherapy or psychotherapy vs placebo or no treatment

• Overall SMD was 0.50 (95% Confidence interval 0.41 to 0.59)
• Few treatments demonstrated a large effect size (SMD of more than 0.8)
• Psychotherapy treatments trended towards greater effect size than pharmacotherapy (not statistically significant)
• The number of participants in psychotherapy trials was less on average than for pharmacotherapy (2,507 vs 3,623 [calculated as a statically significant difference]).

Pharmacotherapy vs psychotherapy

• Trend favouring psychotherapy; this was statistically significant only in the case of relapse prevention in depression and bulimia
• Pharmacotherapy outperformed psychotherapy in dysthymia and schizophrenia

Combined pharmacotherapy and psychotherapy

A trend was observed favouring combination of therapies, this became significant in the case of:

• Major depressive disorder
• Social phobia
• Bulimia
• Schizophrenia (CBT augmenting antipsychotic treatment)

Research quality comparison

Meta-analysis quality

No significant difference in the quality of meta-analyses was detected, excepting in that conflicts of interest were declared in all pharmacotherapy meta-analyses but only 63% of psychotherapy reviews (statistically significant difference).

Quality of individual included trials

The following differences were observed in terms of trial methodology (statistically significant differences only described):

• The population sizes of psychotherapy trials were lower than for pharmacotherapy (mean population 89 vs 191)
• Pharmacotherapy trials were more likely to use blind outcome assessment (97.7% vs 44.5%)
• Intention to treat analyses were conducted more often in pharmacotherapy trials (68.7% vs 45.1%)
• Withdrawal rates and reasons were reported more often for pharmacotherapy trials (62.2% vs 36.3%)
• Withdrawal rates were lower in psychotherapy trials (21.3% vs 31%)
• Follow up data were reported more often in psychotherapy than pharmacotherapy trials (54.9% vs 4.9%)

Analysis of impact of methodological quality

Few meaningful subgroup comparisons could be made to assess the impact of methodology on findings. However in psychotherapy trials studies with contemporaneous controls (placebo, treatment as usual etc) vs waiting list controls produced effect sizes favouring waiting list control studies mean effect size 0.46 vs 0.86 (statistically significant difference). Contemporaneous control conditions were used in only 60% of the psychotherapy trials compared with 100% of the pharmacological trials.

Publication bias, follow up assessment and baseline severity differences

• There was evidence detected of publication bias in 18% of psychotherapy meta-analyses vs 38% pharmacotherapy (not statistically significant)
• Follow-up data reported for psychotherapy trials tended to suggest stability of outcome
• The baseline severity of depression was 3.6 points lower in psychotherapy than pharmacotherapy trials (statistically significant)

Conclusions

The authors concluded:

Many pharmacotherapies and psychotherapies are effective, but there is a lot of room for improvement. Because of the multiple differences in the methods used in pharmacotherapy and psychotherapy trials, indirect comparisons of their effect sizes compared with placebo or no treatment are problematic. Well-designed direct comparisons, which are scarce, need public funding. Because patients often benefit from both forms of therapy, research should also focus on how both modalities can be best combined to maximise synergy rather than debate the use of one treatment over the other.

Discussion

This study uses robust methods to offer an exploratory appraisal of the literature assessing efficacy in pharmacotherapy and psychotherapy for the treatment of a range of mental disorders. The authors clearly state that limited conclusions can be drawn in terms of direct comparisons between treatment types; instead suggesting that this work be used to illustrate gaps in the research literature that urgently require investigation.

So is it hubristic to believe that psychological and pharmacological treatments can affect the experience of mental distress? The average treatment effect size in this study (0.50) can be described as a “moderate effect size” (Cohen 1988). The authors cite their earlier work comparing this effect size with that seen in the treatment of “physical” disorders of 0.45 (Leucht 2012), suggesting perhaps that the treatment of mental disorder is no more hubristic than any other medical practice? The interpretation of effect sizes presented as standardised mean differences, as in this paper, can be difficult however – Cohen’s interpretations are commonly used, but they are just rules of thumb. An alternative could be the use of natural units, for example the clinical global improvement scale (Furakawa 2014) and this could be something that future meta-analyses consider reporting?

The most important finding from this study however appears to be the differences in methodology between psychotherapy and pharmacotherapy trials with suggestion of influence on the outcomes from clinical trials. For example, the comparison of active treatment vs waiting list control in this paper suggested the existence of a nocebo effect – that when placed on a waiting list participants may “wait” before getting better – this effect has been described recently in other systematic reviews (Furukawa et al 2014), which have been covered here on the Mental Elf.

Finally the authors highlight the scarcity of research assessing the role of combined pharmacological and psychological therapies – more research needed.

Standardisation of methods, declaration of researcher bias, clear reporting of outcomes;  some more themes that seem to becoming leitmotifs for my existential ramblings on woodland strolls.

Links

Huhn, M., Tardy, M., Spineli, L. M., Kissling, W., Förstl, H., Pitschel-Walz, G., et al. (2014). Efficacy of Pharmacotherapy and Psychotherapy for Adult Psychiatric Disorders. JAMA Psychiatry (epub ahead of print) [PubMed abstract]

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. L. Erlbaum Associates. [Google Books]

Leucht, S., Hierl, S., Kissling, W., Dold, M., & Davis, J. M. (2012). Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses. The British Journal of Psychiatry, 200(2), 97–106. doi:10.1192/bjp.bp.111.096594 [PubMed abstract]

Furukawa, T. A. (2014). How can we make the results of trials and their meta-analyses using continuous outcomes clinically interpretable? Acta Psychiatrica Scandinavica. doi:10.1111/acps.12278 [PubMed abstract]

Furukawa, T. A., Noma, H., Caldwell, D. M., Honyashiki, M., Shinohara, K., Imai, H., et al. (2014). Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatrica Scandinavica, doi:10.1111/acps.12275 [PubMed abstract]