Cognitive Behavioural Therapy for Psychosis (CBTp) has negligible effects on psychotic symptoms in randomised controlled trials where raters are ‘blind’ (or masked) to what treatment the participants are receiving (Jauhar et al, 2014).
One of the few CBTp studies that has reported significant changes in psychotic symptoms using “masked” raters also reported parallel changes in brain function and connectivity (Kumari et al 2011; Mason et al 2015), although in a non-randomised trial. Now of course, if a treatment for psychotic symptoms works to reduce symptoms, then we would expect to see changes in brain function.
In the original study, Kumari et al, (2011) compared 22 people diagnosed with schizophrenia before and after they had received 16 sessions of CBTp with 16 patients who had received treatment-as-usual (TAU). All underwent Functional MRI (fMRI) while simply deciding if faces expressing fear, anger, happy or neutral expressions were male or female. Scanning revealed altered activation in several brain areas including the amygdala when viewing angry expressions and crucially, this activity correlated with symptom improvement following CBT. Such findings have been used to support the notion that CBTp changes brain activity, although of course, almost everything we do changes brain activity…including reading this sentence.
This latest study by Mason et al (2017) claims that the functional brain changes following 16 sessions of CBTp predict recovery in schizophrenia up to 8 years later – a bold assertion.
Almost everything we do changes brain activity, even looking at this picture of a kitten.
Methods
Although the abstract mentions 22 participants, (from the original Kumari et al study in 2011) the data are from 15 of their original patients diagnosed with ‘paranoid schizophrenia’, who were followed up 7-9 years later.
Data collected from the TAU control group from the earlier studies were not included in the analyses here; a serious weakness.
Electronic case notes were reviewed and monthly symptom levels recorded and then participants allocated to full/partial or no remission outcomes. The ratings were made by the paper’s first author and so, ratings were obviously made knowing that patients had received CBT. This underscores not only the issue of blind assessment (which is known to be a major uncontrolled inflator of the effects of apparent therapeutic success in CBTp research: Jauhar et al 2014), but again the necessity of having a control group; both for ratings and imaging data.
Participants themselves completed the Questionnaire about the Process of Recovery (QPR,31) measuring domains such as hope or empowerment; so, again unblinded (self)assessment. An fMRI was performed including assessment of regional brain activity during tasks measuring reactions to angry and happy faces as well as connectivity between regions of interest.
Multiple statistical tests were used; one-tailed (which increases the chance of finding a significant result) and given the small number of participants it is unclear if this was sufficiently considered when checking if these results were significant. A Bonferroni correction was said to be applied, which may mitigate this concern. It is unclear if the proposed analysis was pre-registered before the data was analysed, which would reduce concerns that “fishing” for positive results took place.
Results
Many tests were performed and only a few appeared to be statistically significant:
- Long-term psychotic symptoms were predicted by changes in prefrontal connections during prosocial facial affective processing
- Long-term affective symptoms were predicted by threat-related amygdalo-inferior parietal lobe connectivity
- Higher subjective ratings of recovery at long-term follow-up were predicted by greater increases in dorsolateral prefrontal cortex connectivity with amygdala
The authors’ three key results above are interpreted as positive; however none actually reach conventional levels of acceptable statistical significance (where p <=0.05; but are p=0.06 both in 1) and in 2); and p=.052 in 3, although the latter is reported as p=.05). Leaving aside debates about the use of the terms ‘marginally significant’ or ‘approaching significance’, all three results are nonsignificant.
Furthermore, some of the presented results are clearly driven by the influence of outlying data from single individuals. For example, the third claim that increased dorsolateral prefrontal cortex (DLPFC); amygdala connectivity is associated with higher levels of subjective recovery. In this example, one person scores abnormally highly on the change in amygdala-DLPFC connectivity (see Figure 2 in the paper).
The same is true for the claimed increase in connectivity between dorsolateral prefrontal cortex (DLPFC) and postcentral gyrus when processing prosocial facial affect predicted reduced levels of positive psychotic symptoms, where one individual is clearly an outlier.
It is clear that with such a small sample the authors have not attended to the presence of obvious extreme outlying data points from single individuals (indeed, reanalysis of the data [albeit imperfect after extracting data-points from the scatterplots] removing the outliers indicates that the data are uncorrelated).
Most press coverage of this study repeated the unsubstantiated claims in the press release.
Strengths and limitations
Strengths
- The study authors did use a validated method to measure case notes symptoms
- There were attempts to assess the long-term consequences of an intervention
Limitations
- Small sample, multiple tests, one-tailed tests. The authors claim that “the final sample size remained adequate” and reference Friston (2012) – who it should be noted also asserts in the same paper “…if you cannot demonstrate a significant effect with sixteen subjects, it is probably not worth demonstrating”
- No control groups for comparison: the lack of control group makes it impossible to attribute any changes to CBT rather than changes in medication or even aspects of lifestyle
- Insufficient data on what happened in 8 years between studies apart from symptom ratings as derived from case notes (which will undoubtedly be written by many individuals with varying experience and backgrounds and in varying degrees of detail, or not as the case may be)
- No repeated assessment of symptoms over the 8 years using the same measure (PANSS) as in the original study.
Small sample, no control, insufficient data, lack of repeated assessment: this study has a significant number of weaknesses.
What does it all mean?
This study seems to have generated some media buzz with the one of the authors making bullish assertions about CBTp. It is difficult however to see what the fuss is about. How do we know if any changes over the time are even due to CBTp received 8 years earlier or the result of medication, or changes associated with age or indeed changes caused by anything else? Unfortunately, the study lacks the key ingredient of a control group; either healthy controls or participants with psychosis who received “treatment as usual”.
The study raises many questions that perhaps eluded the reviewers:
- Why does the abstract talk about 22 participants when the follow-up data of symptoms applies to only 15 participants?
- Why do the authors claim results are positive when they don’t meet conventional level of statistical significance?
- Why did the authors check reliability of clinician ratings for only 8 rather than the full 15 patients?
- Why did two raters disagree so much on what counted as remission? Why did they not minimally comment upon outlying data points when they are so evidently influential in the figures?
- Other minor anomalies also appear to be present, e.g. how did the authors arrive at 20 degrees of freedom for the analyses in Table 1?
The lack of data on what happens during the 8 year period creates a chasm in any causal chain back to CBTp, what happened in the person’s lives, did they form close social bonds with others that changed how they reacted to people or did they take regular medication that helped?
In one sense it’s trivially true that everything we do changes brain function and unfortunately this study does not allow us to either rule CBT in or everything else out.
This study adds little to our knowledge about the safety and efficacy of CBT for psychosis.
Links
Primary paper
Mason L, Peters E, Williams SC, Kumari V. (2017) Brain connectivity changes occurring following cognitive behavioural therapy for psychosis predict long-term recovery. Translational Psychiatry, 7(1), p.e1001.
Other references
Friston K. (2012) Ten ironic rules for non-statistical reviewers. Neuroimage 2012; 61: 1300–1310
Jauhar S, McKenna PJ, Radua J, Fung E, Salvador R, Laws KR. (2014) Cognitive-behavioural therapy for the symptoms of schizophrenia: systematic review and meta-analysis with examination of potential bias. The British Journal of Psychiatry, 204(1), pp.20-29.
Kumari V, Fannon D, Peters ER, Sumich AL, Premkumar P, Anilkumar AP, Andrew C, Phillips ML, Williams SC, Kuipers E. (2011) Neural changes following cognitive behaviour therapy for psychosis: a longitudinal study. Brain, p.awr154.
Mason L, Peters ER, Dima D, Williams SC, Kumari V. (2015) Cognitive behavioral therapy normalizes functional connectivity for social threat in psychosis (PDF). Schizophrenia Bulletin, p.sbv153.
This is a very interesting blog.
If any treatment works, it makes sense to try and understand the neurobiological basis for this. This should work for CBT.
Reading the blog made me go through the paper, and the media articles. I am a neuroscientist and clinican, and had not come across these sorts of studies before.
Reading the paper, it is very difficult to see how this could ever be replicated.
The original study did not compare the two groups directly, CBT vs TAU, and the corrections used were uncorrected. Very few studies now use these corrections, as they do not control adequately for multiple comparisons. The TAU group was underpowered to detect an effect in the original study, so it is difficult to make anything of the study from an imaging perspective.
The follow-up study here is unconventional at best-it is post hoc and exploratory, involving placing of seeds in various areas and correlating these with ourcome measures.
The outcome measures look pretty strong-monthly symptoms ratings and subjective well being. However, the symptoms ratings are rated by the lead investigator and taken from monthly case record reports of psychotic and affective symptoms. I have worked around the world, and have yet to see monthly documentation of psychopathology that would be as adequate as a PANSS for example. The authors rightfully show the inter rater reliability, which is very poor for good outcome-agreement of good outcome was achieved in 3/8 of the 15 people. If you look at the plots, as the bloggers point out, you can see that the correlations are tenuous, and any rating change inters of outcome would lead to a less (non) significant result. Stepwise regression as well, used here, is unconventional.
It is really hard to follow the placing of seeds, and determine that the outcome measures were reliable or valid.
In terms of outcome, surely other things apart from CBT may have affected outcome after 8 years? There is nothing on medication status, substance misuse, life events, co morbid physical health, or anything else that would affect wellbeing or psychotic or affective symptoms. It may be there, but it is very difficult to read this paper and make sense of it all.
Written as it is, I do not see how it could be replicated. This is surely the point of science, and given that this paper deals with a crucial issue-psychotherapy for psychosis, the data could be made available for analysis, as occurs with most imaging journals nowadays.
The Journal, Translational Psychiatry, is well recognised and respected, and one can only assume that the peer reviewers have let it down badly on this occasion. It would have been absolutely fine for this to be a negative fMRI study-there are enough negative pharmaco fMRI studies around.
As a scientist I would ask that people please pause for a moment and think before conveying information to the general public, especially in imaging. This blog is rightfully critical of the methods used, and inferences drawn, and it is difficult to see how this study advances knowledge, which is all rather depressing. Thank you, Mental Elf, and Drs Laws and Huda.