If you’re reading the Mental Elf, I am going to make the assumption that you value evidence and believe it should inform clinical practice. But evidence is not black and white, and unfortunately, sometimes when studies contradict one another it can be hard for those of us, such as myself, who are mere statistical mortals to work out who to believe.
In 2015, a meta-analysis by Johnsen and Friborg rocked the cognitive behavioural therapy (CBT) boat by claiming CBT was becoming less effective over time. They reported a significant negative relationship between year of publication and the effect size of CBT for depression.
There are conceptual ideas that support this finding, such as ‘the winner’s curse’ in which the first study to report a significant effect demonstrates an effect size larger than that observed in replication attempts (Ioannidis, 2008). But not all researchers were convinced.
The current paper by Cristea et al. (2017) published in the Psychological Bulletin identifies a number of shortcomings in the original meta-analysis and sets out to clarify whether CBT really is becoming less effective over time by running a series of alternative meta-analyses.
Cristea et al. used the same inclusion criteria as Johnsen and Friborg for selecting studies for their meta-analyses, with the exception that they only included randomised controlled trials (RCTs) and also included trials not published in English:
- “Pure” CBT interventions
- Aimed to treat depression
- Delivered face-to-face by therapists trained in CBT
- Adult participants with depression
- Pretreatment Beck Depression Inventory (BDI) score over 13.5
- No acute physical illness, bipolar disorder or psychotic disorder
- With reported outcomes on the BDI (version I or II) and/or Hamilton Rating Scale for Depression (HRSD)
Cristea et al.’s methodology differed from Johnsen and Friborg’s in a number of notable ways:
- Cristea et al. only included RCTs whereas Johnsen and Friborg also included non-randomised trials.
- Cristea et al. included between-group effect sizes (i.e. looking at the difference in depression scores at the end of the intervention between the CBT group and the control group) as well as within-group effect sizes (i.e. pre-intervention to post-intervention differences), whereas Johnsen and Friborg used within-group effect sizes rather than between-group effect sizes.
- Cristea et al. ran one set of analyses where they preferentially used completer analyses (i.e. only those still in the trial at the end) if a trial included them, and another where they preferentially used intention to treat (ITT) analyses (including everyone randomised into a part of the study whether they finished the study or not). In contrast, Johnsen and Friborg only examined completer analyses.
- Cristea et al. included all subgroups in studies with multiple groups (not just the group with the most severe depression which is what Johnsen and Friborg did).
- Cristea et al. assessed trial quality using 4 out of 5 items of the Cochrane Collaboration Risk of Bias tool rather than the Randomised Controlled Trial Psychotherapy Quality Rating Scale used by Johnsen and Friborg. Cristea et al, did not include the fifth item, selective outcome reporting, as they felt it was difficult to assess this when so many older trials did not register their methods in advance, but as an aside, see this interesting project on selective outcome reporting for interest: http://compare-trials.org).
The studies included in the meta-analysis were found by examining a continuously updated database of 1,756 papers on the psychological treatment of depression (Cuijpers et al, 2008).
Cristea et al. ran one set of analyses that was very similar to Johnsen and Friborg. For example, it only included RCTs included in Johnsen and Friborg’s analysis, and it only used within-group effect sizes. They also ran a second set of analyses that included all eligible trials and also used between-group effect sizes.
To be frank, the results section of this paper was at times hard for me to follow as I am far from an expert in meta-analyses. What is clear is that Cristea et al. found an additional 30 eligible RCTs.
They conducted multiple analyses and found that year of publication was only a significant predictor when including the RCTs found by Johnsen and Friborg and looking at:
- Within-group effect sizes on the BDI (both completer and ITT analyses)
- Within-group effect sizes using the HRSD (completer analyses only)
- Single meta-regression models were performed (and not a full multiple meta-regression)
Year of publication was not a significant predictor for any of the analyses that included between group effect sizes and all of the eligible trials found by Cristea et al.
Cristea et al. concluded:
the association of year of publication with depression outcomes was unstable and fleeting, present only under certain limited conditions, and completely absent in others. Specifically, this relationship was almost completely absent in the within-group ESs [effect size] analyses on the HRSD and in all analyses that employed between-group ESs.
Strengths and limitations
A major strength of this new review is that it is a more complete analysis of the RCTs that have been conducted to examine CBT for depression. Cristea et al. have sought to reduce sources of bias by using between-group effect sizes and looking at both completer analyses and intention to treat analyses.
However, I think it is important that we start to recognise the potential conflict of interest of authors in psychotherapy trials, as has been stressed in drug trials. Indeed, Cristea and Cuijpers, two of the authors of this new analysis, have published a paper on the impact of investigator background on trial findings (Cristea et al. 2017). I would have liked it to be mentioned somewhere in the paper that some of the authors are trained cognitive behavioural therapists who might have a conflict of interest when examining whether the effectiveness of CBT is declining over time.
This updated meta-analysis by Cristea et al. included 30 new trials and aimed to reduce sources of bias. Cristea et al. conclude that their new analyses shows that CBT is not reliably declining over time and they suggest that the finding by Johnsen and Friborg is spurious.
Interestingly, in the same issue of Psychological Bulletin, Johnsen and Friborg (2017) respond to Cristea’s review and suggest that many of the decisions made by Cristea et al. are not justified. The issue also includes another re-analysis of Johnsen and Friborg’s meta-analysis by Ljótsson et al. (2017) who argue that there was a decline in CBT effectiveness between 1977 and 1995, but that this levelled off from 1995 onwards. So, depending on who you believe, CBT is either declining over time, only declined between 1977 and 1995, or is as effective as it has always been.
Such fights between researchers makes me worry for clinicians and patients trying to engage in evidence-based care. Whatever the truth of the matter is, CBT remains one of the most effective treatments in our arsenal against depression, and I don’t think anyone should stop referring their patients for CBT, stop delivering CBT, or opt not to receive CBT on the basis of this evidence.
Cristea IA, Stefan S, Karyotaki K, David D, Hollon SD, Cuijpers P (2017). The effects of cognitive behavioral therapy are not systematically falling: A revision of Johnsen and Friborg (2015). Psychological Bulletin 143, 3, 326-340. [PubMed abstract]
The COMPare Trials Project. Goldacre B, Drysdale H, Powell-Smith A, et al. www.COMPare-trials.org, 2016.
Cristea IA, Gentili C, Pietrini P, Cuijpers P (2017). Is investigator background related to outcome in head to head trials of psychotherapy and pharmacotherapy for adult depression? A systematic review and meta-analysis. PLoS ONE 12(2): e0171654.
Cuijpers, P., van Straten, A., Warmerdam, L., & Andersson, G. (2008). Psychological treatment of depression: A meta-analytic database of randomized studies. BMC Psychiatry, 8, 36.
Friborg O, Johnsen TJ, (2017). The effect of cognitive-behavioral therapy as an antidepressive treatment is falling: Reply to Ljótsson et al. (2017) and Cristea et al. (2017). Psychological Bulletin 143, 3, 341-345. [PubMed abstract]
Ioannidis, JPA (2008). Why most discovered true associations are inflated (PDF). Epidemiology, 19, 5, 640-648.
Johnson TJ, Friborg O (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: A meta-analysis. Psychological Bulletin, 141, 747–768. [PubMed abstract]
Ljótsson B, Hedman E, Mattson S, Andersson E (2017). The effects of cognitive-behavioural therapy for depression are not falling: A re-analysis of Johnsen and Friborg (2015). Psychological Bulletin 143, 3, 321-325. [PubMed abstract]
- By Ftiercel – Pirate-shipchestsScimitar with background, CC BY 3.0, Link
- Department of Foreign Affairs CC BY 2.0