We have been monitoring outcome switching in five top journals, and writing letters to correct the record wherever we have found misreporting. You can read more about our project here, here and here. One peculiar response has been: “you’ve found so much misreporting, in so many trials! Your findings cannot be credible!”. This argument seems to have been used, for example, by Annals in their responses here and here (comment #2) saying they will not engage with our letters pointing out their misreporting.

So are our findings exceptional? Are we the only people to have found a problem? No. The phenomenon of academic journals permitting outcome switching has been studied at length, and it is now extremely well documented. The only thing we have done, which seems to have solicited some odd responses, is correct the record on individual trials, rather than simply publish overall prevalence figures.

For the avoidance of any doubt, here is a walk through the some of the recent literature on the prevalence of outcome switching.

A systematic review in the journal BMC Medicine included 27 analyses that examined selective outcome reporting. As you will hopefully know, systematic reviews are an extremely important form of evidence, because they set out to include all the relevant literature on a topic. Overall in this systematic review the median proportion of trials with a discrepancy between the registered and published primary outcome, was 31% (range 0 % [0/66] to 100 % [1/1], IQR 17–45 %). Thirteen percent of the published articles introduced a new primary outcome, 9% registered a primary outcome but did not report it, 7% registered a primary outcome but reported it as a secondary outcome, 5% published a primary outcome described as a secondary one in the registry and 8% included a primary outcome whose timing differed between the registry and published article. Note that COMPare assesses both primary and secondary outcomes.

Furthermore, in these trials the outcomes weren’t being switched at random: the switches tended to favour more flattering or positive findings. Eight of the analyses assessed the impact of outcome switching and collectively, the presence of outcome switching repeatedly favoured a statistically significant result within the publication. (The review found no association between the frequency of discrepancies and funding source, journal type, or when the trials were published).

Since that systematic review completed its search, a new paper has been published in PLOS ONE. This is worth a read, because it identifies 137 RCTs over a 6 month period from the exact same five high impact medical journals that COMPare is currently looking at (The Lancet, British Medical Journal, New England Journal of Medicine, Annals of Internal Medicine and Journal of American Medical Association). Of the included trials: 18% had some form of discrepancy relating to the primary outcome; 15% changed the definition of the primary outcome; and, worst of all, 64% of the trials had discrepancies between pre-specified and reported outcomes for non-primary endpoints.

The results from the BMC systematic review and the PLOS ONE analysis are broadly consistent with COMPare’s findings. So far, out of a total of 792 pre specified outcomes assessed, 301 (38%) were not reported: you can see a rolling total, all figures for each trial, and links to the underlying raw data, on the front page of www.COMPare-trials.org. We will be publishing prevalence figures, breakdowns, and other aspects of the study in an academic paper shortly.

So why does all this matter? You need look no further than the classic example of study 329, a clinical trial carried out in adolescents to compare the efficacy of the antidepressant paroxetine against placebo. This trial was sponsored by GlaxoSmithKline, and when it was published in 2001 the trial report presented 4 outcomes favouring paroxetine. The authors reported that the drug was “generally well tolerated and effective”. Clinicians were undoubtedly impressed by this positive finding, and 2 million prescriptions were issued for children and adolescents were issued in 2002 alone.

But the trial report in the academic journal didn’t tell the real story about this paroxetine trial, because the outcomes had been switched. Concerns soon emerged about the safety profile and trial reporting of paroxetine, and specifically about selective reporting of outcomes in study 329. It transpired that the original trial protocol included 2 primary outcomes and 6 secondary ones: none of these gave a flattering result for the drug. But digging deeper into the trial data: 19 additional outcomes were measured, making 27 outcomes in total. Of these, only 4 gave a flattering result for paroxetine, and yet when it came to the trial report, all of those 4 were reported as if they had been the main outcomes of the trial all along. More worryingly, the re-analysis revealed “clinically significant increases in harms, including suicidal ideation and behaviour and other serious adverse events in the paroxetine group”. If the stated pre-planned outcomes had been provided, without any outcome switching, in the open and transparent fashion that COMPare is calling for, we would all have had a clearer sense of the benefits and risks of this treatment.

This is no small issue. Medical research is not an abstract academic pursuit. Alongside being researchers, we (Ben and Kamal) are both practising clinicians and, like all doctors, we use the results of clinical trials to make informed decisions with our patients about which treatment to use. When the data is wrong, this has very real implications for clinical care. At the very least doctors will be advocating treatments which fail to meet our patient’s expectations, or failing to use the best treatment from a range of options. At worst, we will be causing patients actual harm.

So, is outcome switching bad? Undoubtedly. And is outcome switching commonplace? Yes, it is, as you can see. So it makes no sense for journals to suggest otherwise, and it makes no sense for journals to evade their responsibilities around policing this issue, especially because so many of them have publicly pledged that they will uphold high standards of trial reporting.

That is why we are trying to address this problem, through the COMPare Trials project: correcting the record, holding individual trialists and journals to account, and producing worked contemporaneous examples of outcome switching, for discussion, to drive the issue forward.

We hope that the journals will engage, and we hope you will help us, to help them, to do the right thing!

(Dr) Kamal Mahtani, (Dr) Ben Goldacre.