Recently Storebø et al (2015a) published a Cochrane systematic review on the efficacy and tolerability of methylphenidate (MPH) in children and adolescents for the treatment of attention deficit hyperactivity disorder (ADHD), a summary of which was then published in the BMJ (Storebø et al, 2015b).

In recent years, numerous systematic reviews and meta-analyses have consistently demonstrated the high degree of efficacy of stimulants, such as MPH, for the treatment of ADHD. Consequently, those making treatment recommendations, such as the UK NICE guidelines (NICE, 2008), have identified MPH as a first-line treatment for school-age children with moderate to severe ADHD.

Hence, the new Storebø et al review has generated controversy as it has challenged the quality of this prior evidence-base and suggests that there is significant uncertainty about the effectiveness of methylphenidate for the treatment of ADHD.

Methods

The authors report a Cochrane systematic review and meta-analysis of randomised controlled trials (RCTs) of methylphenidate for the treatment of children and adolescents (age 18 years or younger) with ADHD.

Major databases were searched and 185 trials were included in the review:

• 38 were parallel-group trials
• 147 were cross-over trials.

The primary outcome reported in the BMJ paper was ADHD symptoms rated by teachers. Other secondary outcomes included general behaviour and quality of life.

Risk of bias

Risk of bias in trials was recorded using the Cochrane ‘risk of bias’ (RoB) tool. The authors added an additional domain of ‘vested interests’ to the RoB tool identifying trials with industry involvement. In effect, every trial that was funded by a manufacturer of the drug, authored by someone with ties to a manufacturer or where this was unclear was considered to have a ‘high risk’ of bias; irrespective of the scientific quality of the trial, i.e. even if it had a low risk of bias in the other methodological RoB domains.

Quality of evidence was summarised using the GRADE approach.

Results

• The Storebø et al Cochrane review and meta-analysis reports a large effect size (SMD -0.77) for teacher reported ADHD symptoms, indicating clinically meaningful improvement with methylphenidate. These findings are in line with previous systematic reviews.
• Furthermore, the authors report additional benefits of methylphenidate in improving:
  • General behaviour (SMD -0.68, 95% CI -0.78 to -0.60)
  • Quality of life (SMD 0.61, 95% CI 0.48 to 0.80)

However, Storebø et al cast serious doubt on quality of the evidence and certainty of these effects. They identify 96.8% of included trials as ‘high risk of bias trials’ and assessed all outcomes as ‘very low quality’ using GRADE.

Conclusions

The authors conclude in their BMJ paper that:

Given the risk of bias in the included studies and the very low quality of outcomes, the magnitude of effects [of methylphenidate] is uncertain and the strength of evidence is insufficient to guide practice.

The accompanying BMJ commentary by Mina Fazel (2015) states that the author’s:

Findings are potentially important but confusing for clinicians and affected families, thanks to poor overall quality of the evidence.

Strengths and limitations

Storebø et al have conducted a large and comprehensive systematic review and meta-analysis of the effects of methylphenidate in children and adolescents with ADHD. The main limitation of the study concerns the idiosyncratic methods used to assess study bias and quality of evidence. These methods, although reported under the aegis of a Cochrane review, deviate significantly from standard Cochrane methodology and result in an exaggeration of study bias and excessive downgrading of the quality of evidence.

The most misleading and potentially damaging part of the paper relates to way the quality of studies is assessed. Storebø et al adopt the GRADE approach, which includes the Cochrane risk of bias tool (RoB). However, the authors deviate from standard Cochrane methodology in two important ways:

1. Firstly, their idiosyncratic broadening of RoB domains to include ‘vested interests’. The Cochrane Handbook recommends that rather than presuming bias on the basis of industry involvement it is preferable to assess whether there are any reasons to believe that vested interests may have led to bias in each trial individually. If this were the case, then the bias would be coded under the standard RoB tool domains. Adding an additional ‘vested interests’ category runs the risk of ‘double counting’ bias.
2. Secondly, in summarising risk of bias, the authors combined the ‘uncertain’ and ‘high’ risk of bias categories and reported these all as ‘high’ risk of bias trials. Using this approach, the authors categorised 96.8% of trials as ‘high risk of bias trials’ when the standard Cochrane RoB methodology would categories just over one third (37%; 69/185) of trials as being in the high risk of bias category. For the 19 trials contributing to the primary outcome of teacher rated ADHD symptoms; less than a third of trials (31%; 6/19) had a high risk of bias using the standard Cochrane RoB tool. If the author’s ‘vested interests’ domain is included in the RoB tool, then 63% (12/19) would have a high risk of bias. These figures are significantly lower than the 96.8% of studies having high risk of bias reported as the headline figure in the BMJ paper.

Turning to GRADE, the author’s misapplied the rules for downgrading (and upgrading) evidence as recommended in the Cochrane Handbook.

1. Firstly, risk of bias is just one of five factors contributing to a GRADE assessment. A high risk of bias would typically lead to downgrading by one or occasionally two points. The authors downgraded all studies assessing ADHD (teacher) outcomes by two points and then by an additional point (total downgrade by 3 points) for study heterogeneity (I²). Thresholds for the interpretation of I² can be misleading, since the importance of inconsistency depends on several factors. The Cochrane Handbook gives a rough guide to interpretation of I² as follows: ‘0% to 40%: might not be important; 30% to 60%: may represent moderate heterogeneity (depending on the magnitude and direction of effects)’. In this review, I² = 37% for the 19 trials contributing to the primary outcome is of questionable significance and would not be expected to result in a downgrading using GRADE.
2. Finally, GRADE can also be upgraded where there are large effect sizes and narrow confidence intervals: (SMD) -0.77, 95% confidence interval (CI) -0.90 to -0.64, represents a large effect estimated with moderate to high precision. In summary, the authors were excessively stringent in downgrading all GRADE scores by 3 points (all GRADE scores were ‘very low’ quality).

Finally, the substantive question remains whether presumed ‘vested interests’ bias materially affected the results. The key subgroup analyses presented in the full Cochrane review (Storebø et al, 2015a), compares RCTs at high versus low risk of bias. Crucially, this distinction (using the author’s own ratings of study bias) failed to find a significant difference in effect size (χ²1=2.43, p=.12).

Summary

The outcomes reported in this review show that methylphenidate is a highly effective, safe and generally well-tolerated treatment for ADHD, with findings similar to those of previous meta-analyses.

However, the idiosyncratic approach used by the authors for assessing quality of evidence deviates significantly from the standard Cochrane method and as a result, exaggerates the risk of bias assessment and excessively downgrades the quality of evidence.

Crucially, the authors themselves showed in the full Cochrane review (but did not report this in the BMJ paper) that ‘vested interests’ bias did not materially affect the results. Therefore, the author’s interpretation of the results and conclusion that the ‘strength of evidence is insufficient to guide practice’ is misleading and potentially dangerous as it could undermine the confidence of practitioners, children and parents in what is an effective and generally safe treatment.

Conflicts of Interest

None

Links

Primary paper

Storebø OJ, Ramstad E, Krogh HB, Nilausen TD, Skoog M, Holmskov M, Rosendal S, Groth C, Magnusson FL, Moreira-Maia CR, Gillies D, Buch Rasmussen K, Gauci D, Zwi M, Kirubakaran R, Forsbøl B, Simonsen E, Gluud C. (2015a) Methylphenidate for children and adolescents with attention deficit hyperactivity disorder (ADHD). Cochrane Database of Systematic Reviews 2015, Issue 11. Art. No.: CD009885. DOI: 10.1002/14651858.CD009885.pub2.

Other references

Storebø OJ, Krogh HB, Ramstad E, et al. (2015b) Methylphenidate for attention-deficit/hyperactivity disorder in children and adolescents: Cochrane systematic review with meta-analyses and trial sequential analyses of randomised clinical trials. BMJ 2015;351:h5203

Fazel M. (2015) Methylphenidate for ADHD (Editorial). BMJ 2015;351 :h5875

NICE. (2008) Clinical Guideline 72: Attention deficit hyperactivity disorder: Diagnosis and management of ADHD in children, young people and adults. NICE. London, 2008.

Photo credits

• Rachel Caiano CC BY 2.0
• Travis CC BY 2.0
• Steve Johnson CC BY 2.0
• Jenny Downing CC BY 2.0