As you will hopefully know by now, we have been writing to five leading medical journals to correct the record on misreported trials. We have sent 58 letters in total, and the responses have been varied: some (such as the BMJ) have issued rapid corrections, whereas others (such as JAMA and NEJM) have refused to publish any correspondence at all. The Lancet have taken a mixed approach: of 19 letters of correction we submitted they have so far published 8, accepted a further 3 for future publication, and rejected 2; with 6 still under editorial review dating back as far as 5 months.
However, there is an extra feature, around an important issue: who should take responsibility for misreported outcomes in clinical trials? We think journals have a clear set of roles and responsibilities here, not least because so many are listed as endorsing the CONSORT guidelines, which explicitly state that all prespecified outcomes should be reported. The Lancet have somewhat absented themselves from this discussion. Our published correction letters have, in most cases, been accompanied by letters from other researchers raising methodological criticisms of the trial in question. In all cases, our published letters have been accompanied by an author’s reply. Where this reply has reiterated significant misrepresentations or misunderstandings around pre-specification of outcomes, we have written a follow-up letter to the journal to point these out. However, each of these subsequent letters has been rejected by the Lancet editors; and our efforts to get the Lancet editors themselves to give a view, on misreported outcomes in their journal, have not yet had a reply.
We view this cessation of public discussion as problematic for two key reasons. Firstly, these author replies often portray important misunderstandings, on the part of trialists, about selective reporting and the mechanisms to prevent it. Open discussion on these issues would likely help to reduce the prevalence of such misreporting, which is a common source of bias in clinical trials. But there is a second issue, perhaps even more important, around who should take responsibility and police outcome switching. In their actions, the Lancet seem to suggest that this is a matter for trialists. We think journals should take responsibility themselves, to ensure that outcomes are correctly reported in the trials they publish. This is especially so in the case of the Lancet, because they are prominent public supporters of initiatives to improve reporting standards, such as the REWARD project on research waste, and the CONSORT guidelines on trial reporting.
We will now begin sharing our full correspondence with the Lancet on the trials for which our follow-up letters were rejected. First, we cover the PETIT2 trial: our initial letter, the author’s reply, and our response inviting a view from the Lancet editors.
COMPare to the editors, 06/11/2015:
Dear Editor,
Your recent publication “Eltrombopag for children with chronic immune thrombocytopenia (PETIT2): a randomised, multicentre, placebo-controlled trial” [1] reports outcomes that are different to those registered prior to trial commencement [2].
There were 22 pre-specified secondary outcomes. Of these, 14 are reported in the paper; while 8 are not reported anywhere in the publication. In addition, the paper reports 1 new outcome (“concomitant drugs for immune thrombocytopenia”), which was not pre-specified, without flagging it as such. For clarity, Tables 2 to 4 have been considered to represent the main trial results tables, with secondary outcome measures split between them.
The Lancet has endorsed the CONSORT guidelines on best practice in trial reporting, which aim to reduce the risk of incorrect outcome reporting [3]. Our team is currently tracking all trials in five top journals to monitor and signpost incorrect outcome reporting, as part of the COMPare project [4]. We have attached a corrected results table (Table 1) reporting the outcomes that were pre-specified for the trial where those are available, or with the words “Data not reported” where those pre-specified outcomes are not reported anywhere.
Many thanks,
Ioan Milosevic, Eirion Slade, Henry Drysdale on behalf of the COMPare project team.
- Grainger JD et al., Eltrombopag for children with chronic immune thrombocytopenia (PETIT2): a randomised, multicentre, placebo-controlled trial, Lancet 2015; 386: 1649–58
- Clinical Trials Archive, https://clinicaltrials.gov/archive/NCT01520909/2012_03_09
- Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869
- COMPare project website www.COMPare-Trials.org
Authors’ response, 23/01/2016:
We thank Ioan Milosevic and colleagues for their inquiry regarding the PETIT2 trial [1] We agree that the CONSORT statement for reporting of randomised trials is an important guide for communicating trial results, and we used these guidelines when drafting our report. Because of word limits, we presented only the key findings that would be of particular interest to clinicians who treat children with immune thrombocytopenia. Additionally, results are publicly available on ClinicalTrials.gov (number NCT01520909) and the GlaxoSmithKline clinical study registry (number 115450).
We wish to clarify that the study endpoints in the protocol were not different from those reported in our Article [1]. All the primary and secondary endpoints in the Article were the same as those in the protocol, including “reduction or discontinuation of concomitant drugs for immune thrombocytopenia”, which was a secondary endpoint. Vital signs and clinical laboratory values were part of the safety assessment. Clinical laboratory values related to liver function adverse events were reported in the text and were also provided in the Article1 appendix (table S2). The results of the ophthalmic examination were reported in the text.
We did not report the five pharmacokinetic endpoints in our Article1 but will include them in a separate publication, in which these data will be combined with similar data from the phase 2 PETIT study.
COMPare response: Addressing Misreporting Of Pre-Specified Trial Outcomes in The Lancet, 05/02/2016
Dr Grainger replies [1] to our letter [2] noting misreporting of PETIT2 [3].
He states the reported outcomes reflect the 2011 protocol [4]. This protocol was updated in February 2015, with no record of amendments. We therefore compared reported outcomes against the only source of pre-specified outcomes pre-dating trial commencement [5].
He questions our classification of the pre-specified ophthalmic outcome as “unreported”. This was pre-specified as “visual acuity, intraocular pressure, lens opacity measured using the AREDS scale”. This outcome was not reported, nor were various others, and their absence was not discussed.
He states “Reduction or discontinuation of concomitant drugs” for ITP was pre-specified and reported. We agree. “Concomitant drugs” for ITP (Table 4) is different, and was not pre-specified. As per CONSORT: additional outcomes are reasonable, but should be declared as such.
He argues space constraints prevent all outcomes being reported. CONSORT recommends identifying unreported outcomes, giving reasons. Furthermore, non-prespecified outcomes are routinely added to trial reports, including PETIT2.
Outcome switching remains prevalent [6,7], despite widespread support for guidelines like CONSORT. COMPare-Trials.org monitors outcome switching in five top journals to help identify misapprehensions and structural issues driving misreporting, and provide worked examples for discussion.
Since The Lancet have a longstanding positive commitment to improving reporting standards, lead the REWARD campaign on research integrity, and endorse CONSORT, we would welcome their perspective on why undeclared outcome switching in PETIT2 (and others) was apparently not addressed prior to publication; whether they now view outcome switching as acceptable; or whether they disagree that it has happened here.
Ben Goldacre, Ioan Milosevic, Henry Drysdale, Kamal Mahtani and Carl Heneghan, on behalf of the COMPare Trials team.
References
[1] Grainger JD, Eltrombopag for chronic immune thrombocytopenia – Authors’ reply, The Lancet 2016; 387: 336-337
[2] Milosevic I, Slade E, Drysdale H, Eltrombopag for chronic immune thrombocytopenia – Correspondence, The Lancet 2016; 387: 336
[3] Grainger JD, Locatelli F, Chotsampancharoen T et al, Eltrombopag for chronic immune thrombocytopenia (PETIT2): a randomised, multicentre, placebo-controlled trial, The Lancet 2015; 386: 1649-1658
[4] PETIT2 Trial Protocol, https://www.gsk-clinicalstudyregister.com/study/115450#ps, accessed 26 Jan 2016
[5] PETIT2 Trial Archived Registry Entry, https://clinicaltrials.gov/archive/NCT01520909/2012_03_09, accessed 26 Jan 2016
[6] Fleming, Padhraig S., Despina Koletsi, Kerry Dwan, and Nikolaos Pandis. “Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals?” PLoS ONE 10, no. 5 (May 21, 2015): e0127495. doi:10.1371/journal.pone.0127495.
[7] Jones, Christopher W., Lukas G. Keil, Wesley C. Holland, Melissa C. Caughey, and Timothy F. Platts-Mills. “Comparison of Registered and Published Outcomes in Randomized Controlled Trials: A Systematic Review.” BMC Medicine 13 (2015): 282. doi:10.1186/s12916-015-0520-3.
In conclusion
At present, many of those reading trial reports in academic journals will reasonably assume that outcome switching has been checked for and excluded by the large professional editorial teams on leading journals. If this is known not to be done, then one solution is disclosure: journals could more prominently warn their readers to be wary of outcome switching, and publicly state that they do not address such misreporting at an editorial level. A better option would be to address the issue, and ensure all outcomes are reported. This approach is neither expensive nor onerous. Trials are conducted, at enormous financial cost, only because they are less vulnerable to bias and confounding than other forms of research. Reporting outcomes completely, as required by CONSORT, is a zero-cost intervention to reduce bias in clinical trials, and it therefore makes little sense to permit outcome switching to persist.
The Lancet is one of the world’s leading medical journals: it sets the tone for the community, and plays an extremely influential role in generating the evidence base on which clinical decisions are made. It is good that some of our correction letters have been published, and discussion with the authors encouraged. However, by rejecting rebuttals on important misunderstandings or erroneous comments around outcome switching, and by declining to express a view on shortcomings against the high standards they have set themselves, we feel the Lancet is missing an opportunity to host an important and informative debate on this issue, to the benefit of all concerned.
Ben Goldacre , Aaron Dale, and Henry Drysdale on behalf of the COMPare team
Should not as a matter of policy incompletely reported trials be excluded from Evidence Based Medicine guidelines? If journals fail their fiduciary duty to vet reports for sources of bias, they mislead their readers and become part of the problem. The disclosure of switched outcomes reporting in clinical trials is much needed. Hopefully, Compare will periodically publish a list of the clinical trials and the journals involved, together with its correspondence with the offending authors and journal editors. In publishing such a list, besides being alerted to the sources of biased information, readers will want to learn the nature and extent of the possible bias contained in the published reports.
This highlights a number of important issues about clinical trial reporting. While “outcome switching” is probably not the major issue that leads to a distortion of the clinical evidence base (a misunderstanding of p values, as discussed here: https://www.forbes.com/sites/davidgrainger/2015/01/29/why-too-many-clinical-trials-fail-and-a-simple-solution-that-could-increase-returns-on-pharma-rd/#4d8d3c275f67 most likely has that honour), it is nevertheless entirely appropriate that COMPare-trials.org holds journals (and by extension principal investigators to account on this issue).
The inadequate response from a number of leading journals – most notably JAMA and NEJM, but also Lancet, highlights the central issue: clinical trials rely on patient (or volunteer) participation, and the moral payback for their participation is open reporting of the resultant data. Is it, then, appropriate at all for commercial for-profit publishers to police that activity?
The observations of COMPare-trials.org suggests that the self-regulation through peer-review and the oversight from commercial publishers is manifestly failing. This is unsurprising: both researchers and publishers have competing priorities, which are misaligned with the interests of the community from whom patients and volunteers are drawn. This conflict leads to the current situation, where the participants give lip-service to open reporting, by notionally adopting guidelines such as CONSORT, but then fail to properly implement them. An unbiased reader might observe that the editors see their role as protecting the trial authors from enquiry by organisations such as yours, rather than holding them to the highest standards the community of participants has a right to expect.
While your approach is commendable, the only long term solution is to move away from a situation where peer-review editorial boards and commercial publishers are responsible for the oversight. We should transfer oversight of reporting to the Research Ethics Committees who approved the trial designs in the first place, with a clear understanding that failure to comply with the reporting standards will (a) be made public and (b) disbar the investigators from further clinical research activities.
Indeed, this is a major issue but a key actor must not be ignored: The reviewer who fail to do its job properly. PubMed Common allows to post comments about articles. It is a way to raise issues. Last a specific website with a hall of SHAME could be another response