Can network meta-analysis decide the best psychosocial intervention for bipolar disorder?


There is uncertainty and controversy around the best psychosocial interventions for bipolar disorder.

Network meta-analysis (NMA) is becoming increasingly employed to try and resolve such uncertainty. NMA can be an ideal tool for achieving data synthesis (Leucht, 2016), however for the approach to be valid the relevant data must meet exact formal criteria; such methodological requirements of NMA need to be more widely appreciated.


Mary Lou Chatterton, Emily Stockings, Michael Berk, Jan J. Barendregt, Rob Carter and Cathrine Mihalopoulos have entered published data identified by systematic review from psychosocial interventions for bipolar disorder into a network meta-analysis (Chatterton et al, 2017) focused on:

  • Relapse to mania or depression,
  • Medication adherence, and
  • Symptom scales for mania, depression and Global Assessment of Functioning (GAF).
Principle of the use of indirect evidence in network meta-analysis

Principle of the use of indirect evidence in network meta-analysis. Reproduced from Leucht, 2016.


Forty one trials were identified and the interventions were grouped as:

  • Cognitive Behaviour Therapy (CBT),
  • Psychoeducation alone,
  • Psychoeducation in combination with CBT,
  • Psychoeducation and Personalized Real-time Intervention for Stabilizing Mood,
  • Family focused psychotherapy and
  • Carer-focused interventions.

Networks were created for the different outcomes described in the methods section and are illustrated in the online supplementary material (PDF). They are all sparse and not well connected, in that most of the comparisons are not between different interventions but with the common control condition known as ‘treatment as usual’ or TAU. Ranking of the different treatments was then made for the different treatment outcomes using relative risk (for events) or effect size (g: for scaled outcomes).

The analysis suggested that:

  • Carer-focused interventions significantly reduced the risk of depressive or manic relapse;
  • Psychoeducation alone and in combination with cognitive behavioural therapy (CBT) significantly reduced medication non-adherence;
  • Psychoeducation plus CBT significantly reduced manic symptoms and increased GAF;
  • No intervention was associated with a significant reduction in depression symptom scale scores.


The authors rightly highlight the desirability of more standardized outcome measures in trials of psychosocial interventions and highlight the challenge of treating depression in bipolar disorder, but they are cautiously confident in endorsing their findings as a basis for practice and indeed policy. We are not so sure that their confidence is really justified.

Can we be confident in the findings of this network meta-analysis? 

Can we be confident in the findings of this network meta-analysis?

Strengths and weaknesses

In 2014, NICE recommended psychological treatments as the primary modality of treatment in primary care for bipolar depression, as equivalent to medication in the management of bipolar depression in secondary care and sharing equal importance with medication in the long term (Kendall 2014).

An earlier blog reviewed a study demonstrating the bias apparent in the handling of the relevant evidence by NICE (Jauhar et al, 2016). In particular, the use of multiple parallel meta-analyses and subsequent cherry-picking of possible positive findings was highlighted. In principle, NMA should take us beyond that kind of approach and the present study appears to do so.

Was the network meta-analysis valid?

Network meta-analysis has been developed to synthesise evidence across a network of randomised trials, assess the relative effectiveness of several interventions and rank treatment options. This statistical method is based on the simultaneous analysis of direct evidence (which compares treatments within the same study) and indirect evidence (comparing interventions across different studies using a treatment in common). Indirect evidence is important because, alone, it allows comparison of treatments that have not been compared directly and, in combination with direct evidence, it increases the precision of the estimates (producing so-called mixed evidence).

NMA is becoming very popular in the scientific literature, but unfortunately this success too often goes together with an increased risk of poor methodological quality. NMA requires that the subject data meet various criteria the most basic of which is transitivity. In other words, before carrying out a NMA (and in order to be confident that final results are correct and clinically informative) the authors should answer “yes” to the following questions:

  1. Can any patient within the network be randomised to any of the treatments included in the network?
  2. Is it possible to imagine a mega trial where patients can be randomised to any of the interventions included in the network?

If yes to both, transitivity is preserved. If no, it is not. In fact the answer appears to be ‘no’ here, because some studies required patients to be “euthymic” and other studies simply required that they could give consent (and so might be highly symptomatic, in an episode of depression or even treatment resistant). In such situations, having access to the full protocol usually helps because authors report why they think the assumption of transitivity holds. We searched but we could not find the document (in PROSPERO there is only a very short and not really informative summary). We would be grateful to Mary Lou Chatterton and colleagues if they could clarify their position.

When it's done well, network meta-analysis can be a powerful tool, but as with all research designs, bias can creep in if the methodology is not water-tight.

When it’s done well, network meta-analysis can be a powerful tool, but as with all research designs, bias can creep in if the methodology is not water-tight.

A further critical issue is the shape of the network. Ideally, networks should be well connected and these are not. This means that the analysis depends heavily on indirect comparisons via TAU, which is problematic. The choice of a fair comparison treatment is difficult in psychotherapy trials. When the active treatment is superior to TAU, no specificity can be claimed for the content and TAU is often very poorly specified. Any network so dependent on indirect comparison is likely to be unstable.

NMA allows consistency to be examined formally (i.e. statistically): so if A beats B and B beats C, does A beat C? If the answer is yes, there is no inconsistency in the loop between A, B and C. If not, the loop is inconsistent. In this paper there are no data/tables that report this, and the authors did not perform local tests of inconsistency (even when the global test is OK, there might be important local inconsistencies, which must be appraised in order to evaluate the affected mixed estimates). Despite the best efforts of investigators to construct a consistent network, statistically significant inconsistency may arise, which should be investigated when found. Systematic review protocols list potential sources of heterogeneity, possibly using them to form more homogeneous subgroups of studies and generate hypotheses for effect modifiers. Similarly, network meta-analysis should describe in the protocol a clear strategy to deal with inconsistency. There is nothing like that here and, indeed, the reporting of the statistical analysis is inadequate.

A lack of reporting of the statistical analysis in this review make it impossible to be confident in the findings.

A lack of reporting of the statistical analysis in this review make it impossible to be confident in the findings.

Finally, the quality of the evidence is crucial to interpret the results from a network meta-analysis. Guidance exists on how to rate the quality of evidence supporting treatment effect estimates obtained from NMA and simply assessing the risk of bias of individual studies (as the authors did and reported in detail in the Appendix) is not enough. Methods developed by the GRADE working group have standardised the procedure and showed that the quality of evidence supporting NMA estimates can vary from high to very low across comparisons (quality ratings given to a whole network are uninformative and likely to mislead). In any NMA, quality of evidence is likely to be different from estimate to estimate; therefore the GRADE ratings need to be attempted for all primary outcome estimates. For these elements not to be addressed in a NMA published in a peer-reviewed journal is disappointing; the authors deserved more searching editorial scrutiny.

The problem of interpreting psychotherapy trials

It is possible to carry out a methodologically sound NMA, but we are left with the generic problems for psychotherapy trials of small scale, allegiance bias, lack of blinding, demand characteristics when assessing outcomes and publication bias (Flint et al, 2015). This is certainly not something a systematic reviewer can change; however, they should be discussed and appraised in any attempt at evidence synthesis of psychological therapies.


‘Adverse reactions’ to psychological treatment are not collected systematically and hence are under-appreciated (Nutt and Sharpe, 2008). A consideration only of efficacy is inherently unbalanced in the perspective it gives to treatment selection.


NMA does offer a powerful method for synthesising data and avoiding bias in the selection of evidence. However to be valid, it sets high demands on the statistical methods employed and the data included. This is not widely enough appreciated and the present publication illustrates some of the limitations. Its conclusions may be correct, but we remain concerned that the evidence supporting them is actually less than compelling.

The review conclusions may be correct, but we remain concerned that the evidence supporting them is actually less than compelling.

The review conclusions may be correct, but we remain concerned that the evidence supporting them is actually less than compelling.

The views expressed in this blog are those of the author and not necessarily those of the NHS, the NIHR or the Department of Health.



Primary paper

Chatterton ML, Stockings E, Berk M, Barendregt JJ, Carter R, Mihalopoulos C. (2017) Psychosocial therapies for the adjunctive treatment of bipolar disorder in adults: network meta-analysis. The British Journal of Psychiatry Feb 2017, DOI: 10.1192/bjp.bp.116.195321 [Abstract]

Other references

Is the NICE guideline for bipolar disorder biased in favour of psychosocial interventions?

Flint J, Cuijpers P, Horder J, Koole SL, Munafò MR. (2015) Is there an excess of significant findings in published studies of psychotherapy for depression? Psychol Med 45: 439-446.

Kendall T, Morriss R, Mayo-Wilson E, Marcus E. (2014) Guideline Development Group of the National Institute for Health and Care Excellence. Assessment and management of bipolar disorder: summary of updated NICE guidance. BMJ. 2014 Sep 25;349:g5673.

Leucht S, Chaimani A, Cipriani A, Davis JM, Furukawa TA, Salanti G. (2016) Network meta-analyses should be the highest level of evidence in treatment guidelines. Eur Arch Psychiatry Clin Neurosci. 2016 Sep;266(6):477-80.

Nutt DJ and Sharpe M. (2008) Uncritical positive regard? Issues in the efficacy and safety of psychotherapy. Journal of psychopharmacology (Oxford, England) 22: 3-6.

Photo credits

Share on Facebook Tweet this on Twitter Share on LinkedIn Share on Google+
Mark as read
Create a personal elf note about this blog
Profile photo of Andrea Cipriani

Andrea Cipriani

Andrea is NIHR Research Professor at the Department of Psychiatry, University of Oxford, and Honorary Consultant Psychiatrist at Oxford Health NHS Foundation Trust. His main interest in psychiatry is evidence-based mental health and his research focuses on the evaluation of treatments in psychiatry, mainly major depression, bipolar disorder and schizophrenia. He has carried out many systematic reviews, meta-analyses and randomised controlled trials in psychopharmacology, however in the past few years he has also been investigating relevant issues in epidemiological psychiatry and public health, like patterns of drug consumption, risk of serious adverse events (most of all, suicide and deliberate self harm) and implementation of treatment guidelines. His interest in the methodology of evidence synthesis has now a specific focus on individual patient data network meta-analysis and data science, trying to assess the validity, breadth, structure and interpretation of innovative statistical and machine learning approaches to better inform the decision-making process between patients and clinicians and personalise treatment indications in routine clinical care. Andrea has been working closely with world class academic institutions in the UK, Europe, US, Canada, Japan, China and Australia, and also with important organisations, such as the National Institute for Health and Clinical Excellence in the UK, the Istituto Superiore di Sanità in Italy, the United Nations in Vienna and the World Health Organization (WHO) in Geneva. Together with the Department of Mental Health and Substance Abuse at WHO he has co-authored a manual on psychopharmacology, which provided evidence-based information to health care professionals in primary care especially in low- and middle-income countries. This manual is part of the Gap Action Programme of the WHO and is distributed by WHO as a reference source to assist physicians working in the primary health care through increasing their knowledge and improving their routine clinical practice in using evidence-based medicines for mental disorders. Andrea is currently Editor-in-Chief of BMJ Mental Health (; he is also on the Editorial Boards of The Lancet Psychiatry, the Australian and New Zealand Journal of Psychiatry and Bipolar Disorders.

More posts - Website

Follow me here –

Profile photo of Guy Goodwin

Guy Goodwin

Professor Guy Goodwin is Senior Research fellow at the University of Oxford, Oxford, UK, where he was previously WA Handley Professor of Psychiatry. He completed his medical degree and DPhil in Physiology at the University of Oxford and, following his training in psychiatry, became a Clinical Scientist and Consultant Psychiatrist at the Medical Research Council (MRC) Brain Metabolism Unit at the Royal Edinburgh Hospital, Edinburgh, UK. Professor Goodwin’s research interests are in the treatment of bipolar disorder and the application of neuroscience in understanding the neurobiology of mood disorders, with a focus on developing new treatments. He has been a lead investigator in clinical trials for bipolar affective disorder, including the BALANCE and CEQUEL studies. He works with industry in developing preclinical models of psychotropic drug action in humans. Professor Goodwin has served as a member of the Wellcome Trust Neurosciences Panel, the Council of the British Association for Psychopharmacology, the Clinical Fellowships Panel and Advisory Board of the MRC, and INSERM’s ANR panel. He was previously President of the British Association for Psychopharmacology and is a Fellow of the American College of Neuropsychopharmacology, Fellow and current President of the European College of Neuropsychopharmacology (ECNP) and a National Institute for Health Research Senior Investigator.

More posts - Website

Follow me here –