It is well established that academic journals routinely permit outcome switching in the trial reports that they publish, despite public commitments to address this problem. For six weeks from October 2015, the COMPare team analysed every RCT published in the top 5 medical journals to check if they had correctly reported their pre-specified outcomes. Where we found discrepancies between the outcomes that were pre-specified and those reported, we wrote a letter to that journal to correct the record. The responses we have received from journals have been extremely varied: from full public engagement and transparent corrections in the BMJ, to inaccurate and concerning responses in the Annals of Internal Medicine.
New England Journal of Medicine (NEJM) is the highest impact factor journal in medicine. So far we have analysed 23 of their trials. To NEJM’s credit, of 32 primary outcomes pre-specified across these papers, 31 were correctly reported. However, 16 publications added in a total of 65 additional outcomes that were not pre-specified in the protocol or trial registry, and only 2 of these were declared as novel. Furthermore, a total of 160 pre-specified secondary outcomes spread across these publications were either not reported, or were reported incorrectly. Our full results and raw data are available here. Only 3 trials were perfectly reported.
We therefore sent 20 letters to NEJM to correct the record on the individual trials that misreported their pre-specified outcomes in the pages of NEJM. None of these letters have been published. At COMPare, we set out to establish why outcome switching persists in academic journals, despite public commitments to best practice. The responses we have received from journals in the course of the project shed important light on the answers to this question. In their correspondence with us NEJM dismiss concerns on outcome reporting and openly state that they do not regard it as problematic. In accordance with our commitment to open science, because we are citing and analysing journal correspondence in our forthcoming research paper, and in order to drive forward discussions on how to fix the problem of outcome switching, we are sharing our correspondence with the editors of NEJM below.
From COMPare to NEJM, 4/11/2015
Dear Editor,
Your recent publication “A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention” [1] reports outcomes that are different to those initially registered [2].
There were ten pre-specified secondary outcomes, of which four were reported in the paper; while six are not reported anywhere in the publication.
The New England Journal of Medicine has endorsed the CONSORT guidelines on best practice in trial reporting, which aim to reduce the risk of selective outcome reporting [3]. Our team is currently tracking all trials in five top journals to monitor and signpost incorrect outcome reporting, as part of the COMPare project [4].
Attached is a corrected results table (Table 1)*, giving the results for the pre-specified outcomes, or the words “data not reported”.* We hope you will publish this letter to ensure those using the results of this trial are aware of the discrepancies.
Yours faithfully,
Henry Drysdale, Aaron Dale and Ben Goldacre on behalf of the COMPare project team.
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.[1] Chen A et al, A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention, N Engl J Med 2015;373: 1618-26.
[2] Trial registry entry: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12612000625875
[3] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.
[4] COMPare project website www.COMPare-Trials.org
*for clarity we have not reproduced that table here
Below is the email we received from Deputy Editor Prof Dan Longo, in response to that letter (the first of 20 that we sent to NEJM). To set this correspondence in the broader context of ongoing discussions around open science and reproducibility, Prof Dan Longo is lead author of the widely discussed and controversial recent NEJM editorial on “research parasites”.
From NEJM to COMPare, 17/11/2015
Dear Mr. Drysdale,
I am sorry that we will not be able to print your recent letter to the editor regarding the Damian article of 22-Oct-2015. The space available for correspondence is very limited, and we must use our judgment to present a representative selection of the material received. Many worthwhile communications must be declined for lack of space.
Furthermore, I gather that you and your COMPare colleagues intend to make this audit a regular activity. The New England Journal of Medicine finds some aspects of CONSORT useful but we do not, and never have, required authors to comply with CONSORT. Rather we view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors.
Thank you for your interest in the Journal.
Sincerely,
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
To NEJM from COMPare, 19/11/2015
Dear Professor Longo,
Thank you for your email, and for the clarification. The aim of the COMPare project is not to assess compliance with CONSORT, but rather to assess prevalence of outcome switching in clinical trials. Specifically, we aim to detect outcomes that are pre-specified but not reported, as well as outcomes that are novel but not declared as such. We are using the CONSORT guidelines as we believe that the methods of outcome reporting they represent are important and widely accepted.
We strongly believe that outcome switching in trials reduces the quality of evidence used to make clinical decisions. In carrying out this project we hope to reduce the prevalence of the problem and to ensure that, when a trial’s outcomes have been switched, those reading the trial report are aware of this fact.
Best wishes,
Henry Drysdale, Ben Goldacre and Carl Heneghan
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.
From NEJM to COMPare, 21/11/2015
Dear Mr. Drysdale,
Thank you for your follow-up letter. Space constraints for articles published in the Journal do not allow for all secondary and other outcomes to be reported. Any interested reader can compare the published article, the trial registration and the protocol (which was published with the article) with the reported results to view discrepancies; thus your letter does not add substantively to the published record.
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
From COMPare to NEJM, 25/11/2015
Dear Professor Longo,
Thank you for your reply.
You mention space limitations as a justification for selective outcome reporting, and non-reporting of pre-specified outcomes; however our monitoring, and our letters, have regularly identified trials where additional non-prespecified outcomes are reported (and not declared as novel), including several trials published recently in the New England Journal of Medicine [1,2,3].
You also mention that the data needed to detect misreported outcomes are available in the public domain. We agree: indeed this is how we are conducting our work. However, readers reasonably assume that trials published in high-impact journals have been correctly reported. Furthermore, from our extensive experience it takes between 30 minutes and 5 hours to obtain the relevant documents, cross-check the pre-specified and reported outcomes, and assess whether or not these have been reported correctly. Misreported outcomes are therefore highly likely to go undetected, and the evidence used to make clinical decisions remains distorted.
Selective reporting of outcomes is widely regarded as misleading. It undermines the validity of findings, particularly when driven by statistical significance or the direction of the effect [4], and has memorably been described in the New England Journal of Medicine as “Data Torturing” [5]. When pre-specified outcomes are left unreported, the trial report is more likely to overestimate treatment benefits, and more likely to exaggerate the significance of the results that are reported [6]. Even where outcomes are left unreported arbitrarily, regardless of strength or direction of finding, permissiveness around poor reporting gives cover to those who engage in selective reporting of more positive outcomes to exaggerate findings.
Can we please ask you to reconsider publishing our letters, in order to let us correct the record on individual trials; and also to reconsider your policy on permitting trial reports to switch outcomes between the protocol/registration and publication?
Best wishes,
Henry Drysdale, Carl Heneghan and Ben Goldacre, on behalf of the COMPare team.
Centre for Evidence Based Medicine,
Nuffield Department of Primary Care Health Sciences,
University of Oxford.[1] Chen A et al, A Phase 3 Randomized Trial of Nicotinamide for Skin-Cancer Chemoprevention, N Engl J Med 2015;373: 1618-26.
[2] Kaul U et al, Paclitaxel-Eluting versus Everolimus-Eluting Coronary Stents in Diabetes, N Engl J Med 2015;373:1709-19.
[3] Ellis, S. G. et al, Everolimus-eluting bioresorbable scaffolds for coronary artery disease, N Engl J Med 2015;373:1905-15.
[4] Dwan K et al, Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. Jun 24; 11(6):e1001666. doi: 10.1371/journal.pmed.1001666
[5] Mills JL, Data torturing, N Engl J Med.1993;329: 1196-1199.
[6] Chan A et al, Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles, JAMA. 2004;291(20):2457-2465.
We received no further reply to this email. However, we have received the following replies from NEJM in reply to our subsequent letters correcting misreporting of pre-specified outcomes. These replies from NEJM are all identical.
From NEJM, 1/12/2015
Dear Mr. Hartley,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
Dan L. Longo, M.D.
Deputy Editor, New England Journal of Medicine
Professor of Medicine, Harvard Medical School
From NEJM, 5/1/2016
Dear Mr. Slade,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
John A. Jarcho, M.D.
Deputy Editor, New England Journal of Medicine
From NEJM, 28/1/2016
Dear Mr. Slade,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
Julie R. Ingelfinger, M.D.
Deputy Editor, The New England Journal of Medicine
Professor of Pediatrics, Harvard Medical School
From NEJM, 29/1/2016
Dear Mr. Milosevic,
Thank you for your letter. The publication you cite is the summary report of a complex study. In it, the study findings are described with fidelity to the study protocol, which is posted at NEJM.org as a supplement to the publication. We consider the publication to be consistent with the scientific and clinical intent of the trial as described in the protocol; we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record.
We appreciate your interest in the New England Journal of Medicine.
Sincerely,
John A. Jarcho, M.D.
Deputy Editor, New England Journal of Medicine
We think the correspondence above speaks for itself, but there are some concerning themes in here that warrant signposting, and discussion.
“We do not, and never have, required authors to comply with CONSORT”.
While it is true that NEJM does not state that it requires compliance with CONSORT guidance, the journal is listed as endorsing the CONSORT statements on clinical trial reporting. This is a matter that we will be taking up with CONSORT, along with various other changes and concerns that have arisen during the course of COMPare.
“We view each piece individually and add the data as appropriate based on the judgment of the peer reviewers, the statistical reviewers, and the editors.”
This seems to state that NEJM is not only unconcerned by outcome switching, but that they actively request it from trialists, according to their own judgement. This seems contrary to good trial reporting practice, and is a clear source of potential bias which the review process is supposed to eliminate. We would be interested to know whether all readers of NEJM are aware that this form of covert outcome switching is permitted in NEJM, and whether they regard it as problematic that this statistical and interpretative context for reported effects is not clearly signposted in the journal.
“Any interested reader can compare the published article, the trial registration and the protocol..”
This is true, but as discussed above, this is an impractical protective factor. We have found the process of checking is long and laborious, and it often takes several hours to identify the outcomes that were specified before trial commencement and then compare those against the outcomes reported. We believe there is an expectation amongst the readers of journals that this kind of checking is conducted as part of the peer review process.
“Space constraints for articles published in the Journal do not allow for all secondary and other outcomes to be reported.”
We agree that space constraints may be a potential problem with print journals. However, at COMPare we have repeatedly found large numbers of additional non-prespecified outcomes reported in journal articles, including repeatedly in NEJM. Furthermore, there are no space constraints to reporting pre-specified outcomes online versions or appendices, and we have coded this as correctly reported where it has occurred (although we think it would be more informative for the reader if journals signposted these from the main body of the text).
In Conclusion
The New England Journal of Medicine is one of the most influential and widely read medical journals, with an impact factor of 55.8. Outcome switching is an important source of bias. It seems NEJM are unwilling to have the record corrected on individual trials when outcome switching is identified, and do not regard outcome switching as problematic (in fact, they may even actively solicit it). Most importantly, readers may not be aware of these issues, and therefore will take reported outcomes at face value.
As previously discussed, we don’t see a problem with trialists switching their outcomes where there are good reasons to do so, as long as this is openly declared in the trial report, as per CONSORT guidelines. Furthermore, we don’t think every trialist switching their outcomes is setting out to deliberately mislead readers, or exaggerate their findings. But a culture of permissiveness around outcome switching leads to lower standards, and it gives cover to those who are setting out to mislead. That’s why we have clear guidelines and reporting standards in academic medicine.
As ever we would welcome any suggestions or views on the best strategies to address this problem.
Ben Goldacre, Henry Drysdale, Ioan Milosevic, Aaron Dale, Philip Hartley, Eirion Slade, Anna Powell-Smith, Kamal Mahtani and Carl Heneghan, on behalf of the COMPare Trials Team.
Put each letter up on PubPeer and on PubMed Commons. More and more people have the PubPeer add on that alerts the reader when doing a PubMed search of the existence of a comment on PubPeer. This way your work is clearly in the public domain. People can then draw their own conclusions.
Thanks, we are overdue on doing this and will get onto it soonest.
Write a paper on outcome switching in clinical trials comparing and contrasting the prevalence at the various high impact journals and submit it for publication to the journal that ‘allows’ it least. Set the cat amongst the pigeons!
This is an excellent idea.
It would seem to me that the first question would be to find out if Dr.’s Dimitri, Young or Drazen concur and support the opinion of presented by Dr.’s Longo and Jarcho. If so it may suggest that science is not the key influence in editorial decisions at the NEJM.
We have had a large number of identical replies from NEJM, so I don’t think there’s any doubt that this is the coordinated and centrally agreed response.
Given the “research parasites” editorial, I’m sure Dr. Drazen supports dr Longo’s position.
In case anyone else is wondering, that comment was made here:
https://www.nejm.org/doi/full/10.1056/NEJMe1516564
Journals, especially those at the top of the tree, aren’t really configured to admit error or doubt, hence the minuscule possibilities for correspondence . If they conceded this point, they would have to generate many corrections, which would look really bad. To avoid admitting error multiple times, any excuse will do (and that is what you are getting).
However, the aims of this excellent campaign are such plain good sense that it is bound to be effective in the longer term. Keep up the good work! If publishing bad stuff creates bad publicity, people will stop in the end. In that respect, posting material on PubMed Commons (whence it will be automatically mirrored on PubPeer and its plugins) is likely to be quite effective. In particular, such comments will stick to the paper and journal forever rather than fade with the news cycle. One would just need a link to each case. A relevant example is the campaign led by Amanda Capes-Davis on studies using incorrectly identified cancer cell lines.
Endorsing CONSORT guidelines but not desiring to apply them is quite fancy footwork. They clearly hire a better class of politician at the NEJM.
Only 5 hours to obtain the documents and check reported outcomes ? That means you should have plenty of time to look at PACE trial and write to the Lancet.
It is alarming that most of the trials were found to have switched outcomes and what s even more disturbing is the stance of the journal editors who do not feel responsible for alerting the readers and getting the records corrected. Considering the fact that the results are used by doctors and policy makers, journals should insist on accurate reporting of outcomes in the papers they publish. Would it not be in the best interest of the journals as well if the papers they publish meet ethical standards?
What about JAMA?
great article- well done on making them respond- you beat them hands down too- just quietly.
The health economist Uwe Reinhart has allegedly observed that health care prices are “chaos behind a veil of silence.” In the place that I could find he states, “The high variance of health care prices in the United States can be explained in good part by the opacity of these prices. Both government and the private sector have done their best to maintain that opacity.” https://economix.blogs.nytimes.com/2013/03/29/u-s-health-care-prices-are-the-elephant-in-the-room/?_r=0.
And so it is with medical journals that fail to disclose sufficient information by which readers can assess the reliability and validity of published reports. Instead, readers are expected to have faith in the reputational authority of the medical journal. Religion urges embrace of faith. Empirical science eschews faith in favor of demonstrated evidence. The guiding principle of inquiry is skepticism rather than reliance on authority.
Ben Goldacre’s “How did NEJM respond when we tried to correct 20 misreported trials?” provides a window into the workings of the NEJM in this regard. It seems that almost every week a new study surfaces about the unreliability of medical journal reports and the harm done to patients as a result. Just one example is Mark Wilson’s recent Indian Journal of Medical Ethics report documenting how the NEJM failure to publish what it knew about the dangers of Vioxx was responsible for many preventable Vioxx-related illnesses and deaths. https://ijme.in/index.php/ijme/article/view/2407/4974
Is there a solution in sight? In his comment to the aforementioned Goldacre report, Boris says, “If publishing bad stuff creates bad publicity, people will stop in the end.” Apparently, the crescendo of bad publicity is not yet enough to modify the bad behavior of medical journal publishers. It will probably take a significant loss of subscription income to bring about change for the better.
Ironically, medical journals are held to a lesser standard than tour agencies under the U.S. common law duty to discover and warn of anticipated dangers. See: Dickerson, T.A. (1981). Travel Law. New York, NY: Law Journal Seminars-Press, ©1981- Retrieved from https://www.nycourts.gov/courts/9jd/TacCert_pdfs/Dickerson_Docs/Duty_To_Warn_Of_Danger_The_Chinese_Tick_Case4.pdf. Arguably, the subscriber to a medical journal seeks knowledge of vetted medical science much like a tourist taking a tour to discover and learn about new places. Doesn’t the medical journal subscriber deserve fair warning about the known dangers of relying on published reports of new scientific findings?