Read the COMPare papers:

  • A prospective cohort study correcting and monitoring 58 misreported trials in real time
  • Qualitative analysis of researchers’ responses to critical correspondence on a cohort of 58 misreported trials

COMPare

Tracking switched outcomes in clinical trials

  • Methods
  • Results
  • Team
    • SPEAKING DATES
  • FAQ
  • Blog

The Lancet and COMPare: why journals should address outcome switching themselves

April 29, 2016 - by Aaron Dale 3 Comments

As you will hopefully know by now, we have been writing to five leading medical journals to correct the record on misreported trials. We have sent 58 letters in total, and the responses have been varied: some (such as the BMJ) have issued rapid corrections, whereas others (such as JAMA and NEJM) have refused to publish any correspondence at all. The Lancet have taken a mixed approach: of 19 letters of correction we submitted they have so far published 8, accepted a further 3 for future publication, and rejected 2; with 6 still under editorial review dating back as far as 5 months.

However, there is an extra feature, around an important issue: who should take responsibility for misreported outcomes in clinical trials? We think journals have a clear set of roles and responsibilities here, not least because so many are listed as endorsing the CONSORT guidelines, which explicitly state that all prespecified outcomes should be reported. The Lancet have somewhat absented themselves from this discussion. Our published correction letters have, in most cases, been accompanied by letters from other researchers raising methodological criticisms of the trial in question. In all cases, our published letters have been accompanied by an author’s reply. Where this reply has reiterated significant misrepresentations or misunderstandings around pre-specification of outcomes, we have written a follow-up letter to the journal to point these out. However, each of these subsequent letters has been rejected by the Lancet editors; and our efforts to get the Lancet editors themselves to give a view, on misreported outcomes in their journal, have not yet had a reply. [Read more…]

Lessons in Outcome Switching: The ERASE Trial

March 22, 2016 - by Aaron Dale 1 Comment

Over 6 weeks the COMPare team analysed 67 trials in the top 5 medical journals to assess the ongoing prevalence of outcome switching in clinical trials, and to find out why it persists. Our methodology can be found here. In this series we will be highlighting the salient lessons learned from this process by describing the analysis of several trials in detail. Each study highlights different ways in which outcomes can be misreported, leading to undetected flaws in the evidence presented in clinical trial reports.

Here we describe our coding of the ERASE trial [1]. The context and overview of this study is described in this editorial. This study was 1 of 13 randomised control trials published in JAMA that we analysed. Of these, 11 contained outcome switching and required a letter of correction. All were rejected by JAMA’s editors; more on that, including full correspondence with JAMA’s editors, here.

 

Our summary analysis of outcome reporting in the ERASE trial was that:

3/3 pre-specified primary outcomes were correctly reported

21/37 pre-specified secondary outcomes were correctly reported

16/37 pre-specified secondary outcomes were not reported at all

6 additional outcomes were reported that were not pre-specified

 

[Read more…]

Tagged With: jama

How did NEJM respond when we tried to correct 20 misreported trials?

February 25, 2016 - by Aaron Dale 14 Comments

It is well established that academic journals routinely permit outcome switching in the trial reports that they publish, despite public commitments to address this problem. For six weeks from October 2015, the COMPare team analysed every RCT published in the top 5 medical journals to check if they had correctly reported their pre-specified outcomes. Where we found discrepancies between the outcomes that were pre-specified and those reported, we wrote a letter to that journal to correct the record. The responses we have received from journals have been extremely varied: from full public engagement and transparent corrections in the BMJ, to inaccurate and concerning responses in the Annals of Internal Medicine.

New England Journal of Medicine (NEJM) is the highest impact factor journal in medicine. So far we have analysed 23 of their trials. To NEJM’s credit, of 32 primary outcomes pre-specified across these papers, 31 were correctly reported. However, 16 publications added in a total of 65 additional outcomes that were not pre-specified in the protocol or trial registry, and only 2 of these were declared as novel. Furthermore, a total of 160 pre-specified secondary outcomes spread across these publications were either not reported, or were reported incorrectly. Our full results and raw data are available here. Only 3 trials were perfectly reported.

[Read more…]

Tagged With: nejm

How to cite us

Prior to publication of the first paper on COMPare, please cite this project as:

The COMPare Trials Project. Goldacre B, Drysdale H, Powell-Smith A, et al. www.COMPare-trials.org, 2016.

Or alternatively:

The COMPare Trials Project. Ben Goldacre, Henry Drysdale, Anna Powell-Smith, Aaron Dale, Ioan Milosevic, Eirion Slade, Philip Hartley, Cicely Marston, Kamal Mahtani, Carl Heneghan. www.COMPare-trials.org, 2016.

University of Oxford logo Nuffield Department of Primary Care Health Sciences logo Centre for Evidence-Based Medicine logo

Copyright © 2019 · Bennett Institute for Applied Data Science