If you’re reading the Mental Elf, I am going to make the assumption that you value evidence and believe it should inform clinical practice. But evidence is not black and white, and unfortunately, sometimes when studies contradict one another it can be hard for those of us, such as myself, who are mere statistical mortals to work out who to believe.
In 2015, a meta-analysis by Johnsen and Friborg rocked the cognitive behavioural therapy (CBT) boat by claiming CBT was becoming less effective over time. They reported a significant negative relationship between year of publication and the effect size of CBT for depression.
There are conceptual ideas that support this finding, such as ‘the winner’s curse’ in which the first study to report a significant effect demonstrates an effect size larger than that observed in replication attempts (Ioannidis, 2008). But not all researchers were convinced.
The current paper by Cristea et al. (2017) published in the Psychological Bulletin identifies a number of shortcomings in the original meta-analysis and sets out to clarify whether CBT really is becoming less effective over time by running a series of alternative meta-analyses.
In 2015, Johnsen and Friborg reported a significant negative relationship between year of publication and the effect size of CBT for depression.
Methods
Cristea et al. used the same inclusion criteria as Johnsen and Friborg for selecting studies for their meta-analyses, with the exception that they only included randomised controlled trials (RCTs) and also included trials not published in English:
Inclusion criteria
- “Pure” CBT interventions
- Aimed to treat depression
- Delivered face-to-face by therapists trained in CBT
- Adult participants with depression
- Pretreatment Beck Depression Inventory (BDI) score over 13.5
- No acute physical illness, bipolar disorder or psychotic disorder
- With reported outcomes on the BDI (version I or II) and/or Hamilton Rating Scale for Depression (HRSD)
Cristea et al.’s methodology differed from Johnsen and Friborg’s in a number of notable ways:
- Cristea et al. only included RCTs whereas Johnsen and Friborg also included non-randomised trials.
- Cristea et al. included between-group effect sizes (i.e. looking at the difference in depression scores at the end of the intervention between the CBT group and the control group) as well as within-group effect sizes (i.e. pre-intervention to post-intervention differences), whereas Johnsen and Friborg used within-group effect sizes rather than between-group effect sizes.
- Cristea et al. ran one set of analyses where they preferentially used completer analyses (i.e. only those still in the trial at the end) if a trial included them, and another where they preferentially used intention to treat (ITT) analyses (including everyone randomised into a part of the study whether they finished the study or not). In contrast, Johnsen and Friborg only examined completer analyses.
- Cristea et al. included all subgroups in studies with multiple groups (not just the group with the most severe depression which is what Johnsen and Friborg did).
- Cristea et al. assessed trial quality using 4 out of 5 items of the Cochrane Collaboration Risk of Bias tool rather than the Randomised Controlled Trial Psychotherapy Quality Rating Scale used by Johnsen and Friborg. Cristea et al, did not include the fifth item, selective outcome reporting, as they felt it was difficult to assess this when so many older trials did not register their methods in advance, but as an aside, see this interesting project on selective outcome reporting for interest: http://compare-trials.org).
The studies included in the meta-analysis were found by examining a continuously updated database of 1,756 papers on the psychological treatment of depression (Cuijpers et al, 2008).
Cristea et al. ran one set of analyses that was very similar to Johnsen and Friborg. For example, it only included RCTs included in Johnsen and Friborg’s analysis, and it only used within-group effect sizes. They also ran a second set of analyses that included all eligible trials and also used between-group effect sizes.
There were significant methodological differences between this new meta-analysis (Cristea et al, 2017) and the previous study (Johnsen and Friborg, 2015).
Results
To be frank, the results section of this paper was at times hard for me to follow as I am far from an expert in meta-analyses. What is clear is that Cristea et al. found an additional 30 eligible RCTs.
They conducted multiple analyses and found that year of publication was only a significant predictor when including the RCTs found by Johnsen and Friborg and looking at:
- Within-group effect sizes on the BDI (both completer and ITT analyses)
- Within-group effect sizes using the HRSD (completer analyses only)
- Single meta-regression models were performed (and not a full multiple meta-regression)
Year of publication was not a significant predictor for any of the analyses that included between group effect sizes and all of the eligible trials found by Cristea et al.
This new analysis suggests that the effects of CBT for depression are not declining over time.
Conclusions
Cristea et al. concluded:
the association of year of publication with depression outcomes was unstable and fleeting, present only under certain limited conditions, and completely absent in others. Specifically, this relationship was almost completely absent in the within-group ESs [effect size] analyses on the HRSD and in all analyses that employed between-group ESs.
Strengths and limitations
A major strength of this new review is that it is a more complete analysis of the RCTs that have been conducted to examine CBT for depression. Cristea et al. have sought to reduce sources of bias by using between-group effect sizes and looking at both completer analyses and intention to treat analyses.
However, I think it is important that we start to recognise the potential conflict of interest of authors in psychotherapy trials, as has been stressed in drug trials. Indeed, Cristea and Cuijpers, two of the authors of this new analysis, have published a paper on the impact of investigator background on trial findings (Cristea et al. 2017). I would have liked it to be mentioned somewhere in the paper that some of the authors are trained cognitive behavioural therapists who might have a conflict of interest when examining whether the effectiveness of CBT is declining over time.
Summary
This updated meta-analysis by Cristea et al. included 30 new trials and aimed to reduce sources of bias. Cristea et al. conclude that their new analyses shows that CBT is not reliably declining over time and they suggest that the finding by Johnsen and Friborg is spurious.
Interestingly, in the same issue of Psychological Bulletin, Johnsen and Friborg (2017) respond to Cristea’s review and suggest that many of the decisions made by Cristea et al. are not justified. The issue also includes another re-analysis of Johnsen and Friborg’s meta-analysis by Ljótsson et al. (2017) who argue that there was a decline in CBT effectiveness between 1977 and 1995, but that this levelled off from 1995 onwards. So, depending on who you believe, CBT is either declining over time, only declined between 1977 and 1995, or is as effective as it has always been.
Such fights between researchers makes me worry for clinicians and patients trying to engage in evidence-based care. Whatever the truth of the matter is, CBT remains one of the most effective treatments in our arsenal against depression, and I don’t think anyone should stop referring their patients for CBT, stop delivering CBT, or opt not to receive CBT on the basis of this evidence.
CBT remains one of the most effective treatments in our arsenal against depression.
Links
Primary paper
Cristea IA, Stefan S, Karyotaki K, David D, Hollon SD, Cuijpers P (2017). The effects of cognitive behavioral therapy are not systematically falling: A revision of Johnsen and Friborg (2015). Psychological Bulletin 143, 3, 326-340. [PubMed abstract]
Other references
The COMPare Trials Project. Goldacre B, Drysdale H, Powell-Smith A, et al. www.COMPare-trials.org, 2016.
Cristea IA, Gentili C, Pietrini P, Cuijpers P (2017). Is investigator background related to outcome in head to head trials of psychotherapy and pharmacotherapy for adult depression? A systematic review and meta-analysis. PLoS ONE 12(2): e0171654.
Cuijpers, P., van Straten, A., Warmerdam, L., & Andersson, G. (2008). Psychological treatment of depression: A meta-analytic database of randomized studies. BMC Psychiatry, 8, 36.
Friborg O, Johnsen TJ, (2017). The effect of cognitive-behavioral therapy as an antidepressive treatment is falling: Reply to Ljótsson et al. (2017) and Cristea et al. (2017). Psychological Bulletin 143, 3, 341-345. [PubMed abstract]
Ioannidis, JPA (2008). Why most discovered true associations are inflated (PDF). Epidemiology, 19, 5, 640-648.
Johnson TJ, Friborg O (2015). The effects of cognitive behavioral therapy as an anti-depressive treatment is falling: A meta-analysis. Psychological Bulletin, 141, 747–768. [PubMed abstract]
Ljótsson B, Hedman E, Mattson S, Andersson E (2017). The effects of cognitive-behavioural therapy for depression are not falling: A re-analysis of Johnsen and Friborg (2015). Psychological Bulletin 143, 3, 321-325. [PubMed abstract]
Photo credits
- By Ftiercel – Pirate-shipchestsScimitar with background, CC BY 3.0, Link
- Department of Foreign Affairs CC BY 2.0
[…] Battle of the meta-analyses: is CBT becoming less effective over time? […]
Hi! Thank you for blogging about this. However, some things are not very clear. First of all, the original (Johnsen & Friborg) was not just another meta-analysis, this is not about disagreement on academic things. In fact many of the problems were quite evident from the beginning, as I evidenced in a blog post actually here on the Mental Elf: https://www.nationalelfservice.net/treatment/cbt/crisis-of-faith-instead-of-cbt-we-should-be-worrying-about-meta-analyses/. (I would have really appreciated if you had cited that because it underscored some clear problems with the meta-analysis). So what we did was not simply reversing some choices we happened to disagreed on, it was correcting some that were highly problematic and in some cases wrong. There is, for example, a list of consistency errors within the authors’ own data (we didn’t even insist on them but they were not few). Sure errors happens, but when they accumulate to a certain degree, this is no longer a question of diverging opinion. Secondly, we said this in the paper, other commenters have said this, I have said this repeatedly: noone is debating that the effects of CBT have not declined over time from the seminal studies of Rush or Beck in the 70s and 80s. It is unreasonable (and naive) to held CBT to the need of having to prove it can reproduce effect sizes of 4 just because a small study in the 70s showed that. Had Johnsen & Friborg only made that claim, the paper would have never been hyped or published. The main point is about the decline being linear and systematic, which has led some people to wonder (of course completely erroneously, but they were not few), in how many years CBT’s efficacy will get to 0 (If I am not mistaken, even the original authors said this). Finally, it is great that comments on our paper held to such high standards of scrutiny (conflict of interest because authors are certified in CBT). However, this is just a re-analysis that got none of the publicity and unjustified hype the original paper did. I really think such scrutiny should be put to good use also in the case of such hyped papers. This is in fact the most interesting angle of the story: regardless of what you think about about CBT, how could everyone so uncritically and enthusiastically embrace a meta-analysis with so many problems (many, I repeat, evident from the first reading) and such exaggerated (or at least hard to prove) claims? Maybe because one just liked the results?
After a few decades of so-called CBT, I am convinced, now so more than ever, that law-suit avoiding doctors have simply touted this semi-helpful (at best, mystical at worst) therapy as some wondrous panacea to the suffering and ill-informed masses deliberately being deprived of highly effective and well-proven pharmaceutical therapies.