As you will hopefully know by now, COMPare has been examining all trials published in five major journals to check if they have misreported their results by failing to report their prespecified outcomes, or by adding outcomes that weren’t prespecified. Outcome switching is widely recognised as a bad thing; however it continues to be widespread, despite various promises from journals that they have already addressed the issue.

Previous studies have measured the prevalence of outcome switching, finding it to be extensive, and quietly published academic papers giving an overall summary figure for how often this problem occurs. At COMPare we decided to go one step further: we have written a letter for publication on every trial that misreported its outcomes. Our reasons for doing this were twofold. Firstly, we wanted to test whether the mechanisms of science for correcting the record would function correctly. Secondly, we thought that the responses of the system, or the journals, might help shed light on why this problem has persisted, despite extensive public promises that it had already been addressed. You can see for yourself here the process of assessing a trial, and the kinds of misreporting we have found.

In this piece we describe the responses we have had from Annals of Internal Medicine, the world’s 4th highest impact factor medical journal. Five of the 67 trials we analysed were published in Annals, and all of these reports contained misreported outcomes, to varying degrees. We pointed out these errors in online comments for publication, on all the relevant trials. We expected that Annals editors would publish our corrections, encourage authors to address their errors, and report the prespecified outcomes as per best practice.

What happened next was surprising to many in the clinical trials community. The Annals editors initially responded online, with a lengthy criticism of our methods and a simple dismissal of the errors found: however, as you can read here, this editors’ response was almost entirely factually incorrect, and contains a number of extremely concerning views on clinical trial reporting.

Since then, there have been further concerning developments.

1. The Annals editors have now refused to appropriately publish corrections or responses to their long piece, and appear to have worked carefully to prevent readers of the journal seeing their errors: they have published factually incorrect critiques of COMPare’s methods and results in the full journal, and then refused to publish a response setting out their errors.

2. The Annals editors have actively discouraged authors from responding to the misreporting that COMPare has identified, in online comments signed by The Editors beneath COMPare’s corrections. This contravenes ICMJE guidance which states that authors should be encouraged to respond to serious criticism of published work.

We believe this represents a breach of publication ethics by Annals. We have had an exchange of letters with Annals editors on these issues, encouraging them to engage in open discussion, but they have ceased replying. In keeping with our commitment to open science, we are now making our most recent letter – which summarises the salient issues – available online in full below.

We will shortly be publishing our academic paper summarising the extent of misreporting in these five journals, and describing their response to corrections, to help inform wider understanding of whether academic journals are functioning well as the venue for sharing the results of clinical trials. We would welcome any suggestions from the wider academic community of additional strategies to address the high prevalence of outcome switching, and journals’ responses to date, and continue to hope that Annals will reconsider their approach.

COMPare team to the editors of Annals, 08/03/2016:

Dear Annals,

we are writing to express serious concerns about recent editorial decisions at Annals which we feel may represent a breach of publication ethics. Specifically, in summary, we are concerned that Annals editors have: (1) issued a response to COMPare that contains multiple errors (2) refused to publish responses appropriately (3) actively encouraged trialists who have misreported trials in Annals to ignore their duty to respond to correspondence raising legitimate concerns.

We believe that this and other activity from Annals represents a serious breach of various best practice and ethical standards, most importantly ICMJE (where Prof Taichman of Annals is currently secretary). Specifically, we consider Annals are breaching ICMJE recommendations on correspondence in three respects:

“Medical journals should provide readers with a mechanism for submitting comments, questions, or criticisms about published articles, usually but not necessarily always through a correspondence section or online forum. The authors of articles discussed in correspondence or an online forum have a responsibility to respond to substantial criticisms of their work using those same mechanisms and should be asked by editors to respond.” [ICMJE].

This sets out a clear duty to facilitate and encourage serious discussion around criticisms research. We have concerns about Annals’ editors posts on three trials [screenshot1, link1, screenshot2, link2, screenshot3, link3]. We have raised substantial and legitimate concerns in our letters on these three trials, specifically by demonstrating that outcomes have been switched, with reference to publicly accessible material. However, in the comments section beneath all of these trials, rather than encouraging the authors to respond to these substantial criticisms and evidence of outcome switching (as required under ICMJE recommendations) Annals editors have actively discouraged these trialists from doing so.

This is the text from Annals editors:

“Though we share COMPare’s overarching goals to assure the validity and reporting quality of biomedical studies, we do not agree with their approach. Until the COMPare Project’s methodology is modified to provide a more accurate, complete and nuanced evaluation of published trial reports, we caution readers and the research community against considering COMPare’s assessments as an accurate reflection of the quality of the conduct or reporting of clinical trials. Given our ongoing concerns about COMPare’s review methods, we do not believe that COMPare’s comments on Pianko et al (Ann Intern Med. 2015; 163:809-817) merit a response.” [Annals editors]

We regard this as a remarkable and unjustifiable comment from editors of a leading academic journal. We are a team of clinicians and academics who have identified misreporting of pre-specified outcomes in several trials. The rate of outcome switching we have identified, through a reproducible and robust process, is consistent with the existing findings from an extensive published literature on the prevalence of outcome switching in academic journals [1] [2]. All the discrepancies we have described are verifiable using publicly accessible documents which we have identified and referenced. Annals have made no attempt to address this clear evidence of misreporting, and have instead apparently told authors that there is no need to respond to these legitimate concerns on misreporting.

If the editors of Annals genuinely believe that we and others have incorrectly identified misreporting in Annals, then they should explain how, with reference to the publicly accessible documents pertaining to these trials, as we have done; and they should encourage Annals authors to respond to evidence of misreporting.

The ICMJE recommendations continue:

“In all instances, editors must make an effort to screen discourteous, inaccurate, or libellous comments.”

“Responsible debate, critique and disagreement are important features of science, and journal editors should encourage such discourse ideally within their own journals about the material they have published. Editors, however, have the prerogative to reject correspondence that is irrelevant, uninteresting, or lacking cogency, but they also have a responsibility to allow a range of opinions to be expressed and to promote debate.”

We believe that Annals editors’ repeated refusal to publish criticisms of their own work in the journal represents a serious breach of ICMJE guidance. It seems likely that they also reflect a conflict of interest in decision-making at the journal, as Annals editors are currently refusing to engage with legitimate criticism of their own position, avoiding engaging with good evidence that they have made factually incorrect statements, and preventing readers having access to a public discussion on the problem of outcome switching.

The 850 word letter from Annals in response to COMPare contains a number of serious factual errors and internal inconsistencies; it attempts to both justify and deny outcome switching; and it makes concerning but internally contradictory statements about Annals’ commitment to good reporting standards. These errors and other concerning statements are outlined here.

Not only have Annals editors failed to screen demonstrably inaccurate and factually incorrect statements from their own 850 word letter; they have also refused to permit publication, in an accessible location, to our brief 450 word response setting out their errors. The editors of Annals have published their own 850 word letter in three places: in the full paper edition of the journal (March 1st 2016); as a prominent standalone online piece; and as a comment buried in the online only comments section of one single trial. Annals have only permitted COMPare to respond in the obscure online comments section of one single trial.

It is clear that this will deprive Annals’ readers of access to a clear explanation of why Annals’ editors criticisms are unfounded and inaccurate. Furthermore, the actions of Annals editors will deprive readers and the academic community more broadly of an open and transparent two-way discussion on the importance of outcome switching, and its prevalence in leading journals. It is clear from ICMJE guidance that Annals should actively encourage responses such as ours. Furthermore, it is also clear from ICMJE guidance that Annals’ editors should have responded to our substantial criticisms of their prominent 850 word piece, published by Annals in three locations. Far from engaging, instead it seems clear that Annals editors have sought to restrict access to our criticisms and concerns.

We believe the response from Annals to COMPare may represent an abuse of platform and a breach of publication ethics. We are very concerned that the current position of Annals editors will perpetuate the well-documented problem of outcome switching in the pages of Annals and, because of their influential position, help to perpetuate the same misreporting more broadly in the academic literature.

It is notable that other journals have not responded in the same terms as Annals. For example, The BMJ have already issued rapid corrections, while The Lancet have published numerous letters in the full edition of the journal, with ongoing and informative correspondence with individual trialists.

Our objectives through COMPare are simple: to document the prevalence of outcome switching in leading journals; to explore the reasons why this problem has persisted, despite extensive public commitments to resolve it from journal editors; and to explore and develop mechanisms to reduce the prevalence of this problem. The impact that misreporting can have on patient care is well documented.

We therefore strongly encourage you to reconsider your decisions; to publish COMPare’s response in the appropriate locations, both online and in the journal; and to encourage trialists to engage with clear evidence of outcome switching, as other journals have done. We very much hope you will reconsider and move forward, in a spirit of open discussion and improving reporting standards.

Yours,

Carl Heneghan, Kamal Mahtani, Ben Goldacre on behalf of the COMPare Trials team.

[1] Jones, Christopher W., Lukas G. Keil, Wesley C. Holland, Melissa C. Caughey, and Timothy F. Platts-Mills. “Comparison of Registered and Published Outcomes in Randomized Controlled Trials: A Systematic Review.” BMC Medicine 13 (2015): 282. doi:10.1186/s12916-015-0520-3. https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-015-0520-3

[2] Fleming, Padhraig S., Despina Koletsi, Kerry Dwan, and Nikolaos Pandis. “Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals?” PLoS ONE 10, no. 5 (May 21, 2015): e0127495. doi:10.1371/journal.pone.0127495. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0127495