The COMPare Trials research project assessed 5 trials published in Annals of Internal Medicine for misreported outcomes – a well-documented and common source of bias. We found that all 5 trials published in Annals over the 6 week period misreported their prespecified outcomes, and so we sent a correction letter on each trial.

We also identified a systemic problem. None of these trials had a publicly accessible protocol dated before trial commencement, so we took the prespecified outcomes from the trial registry entries, and compared these against those published in the report. This is not necessarily a problem: trial registers were set up explicitly for the purpose of preventing selective reporting of outcomes, with support from august bodies such as ICMJE, WHO, and legislation across various territories.

In response, Annals’ editors claimed that the registry entries are often “outdated, vague or erroneous”, and that the discrepancies we found are accounted for by trial protocols that cannot be accessed publicly, but are reviewed by editors behind closed doors. We view this as a very outdated approach, lacking in transparency, and a problematic view of trial registers.

Then we discovered a further problem. One Annals trial reported in its “Reproducible Research Statement” that the protocol was available on request: we therefore requested this protocol from the trial authors. We were passed on to the commercial sponsor, who replied informing us we would need to sign a Non-Disclosure Agreement, committing us to secrecy and preventing us from discussing any discrepancies or methodological issues in this document. The full story can be read here.

This raised two main issues. The first is that the outcomes in the registry entries should match those in the trial protocol, regardless of whether the protocol is publicly available. Trial registers were specifically set up as a publicly accessible source of such information, and it is the responsibility of the trialists to ensure they are complete and up to date. If journal editors publicly announce that they routinely ignore registries, then this only adds to the risk that the information on them is inadequately managed: we should recall that one of the key drivers for more complete trial registration was journals holding the line, and stating through the ICMJE in 2005 that they would only consider trials for publication if they had been properly registered.

But there is a second issue: these protocols should not be inaccessible. They contain a huge amount of important information about the trial, and these protocols should be published alongside the trial report. We are in the midst of a reproducibility crisis in science, and transparency in the research community is the only way to identify hidden sources of bias.

We wrote to Annals’ editors to raise our concerns on this withheld protocol, and their misleading Reproducible Research Statement, on 29th March 2016 (archive copy). After a second letter from us on 19th April 2016 (archive copy), Annals issued a corrected reproducibility statement for this trial.

Also on April 19th, Annals’ editors took an extremely positive step towards improving transparency, by making a series of changes to their “Instructions to Authors”. All trials submitted must now include a protocol with dated amendments, and Annals say they plan to publish these alongside the trial report if it is accepted. Assuming that compliance is managed appropriately, this will provide increased transparency, allowing readers to more easily check for discrepancies between prespecified and reported outcomes, and whether any changes are justified in the trial report. It is a clear improvement over Annals’ previous approach of reviewing inaccessible protocols behind closed doors which was, as described above, lacking in transparency and extremely outdated.

While this is exactly the kind of result we were hoping for at COMPare – increased transparency to address ongoing structural flaws in science – it is odd that Annals’ editors made this amendment quietly, whilst publicly dismissing COMPare’s methods without evidence to support their claims, defending their current inappropriate and untransparent methods (which they have now reviewed and amended), and resisting constructive positive discussion.

We hope that Annals’ editors will now also move forward and engage in an open and constructive discussion of the misreporting we have identified in all 5 trials assessed in Annals; their dismissal of trial registries; and possible improvements to their processes for identifying and preventing outcome switching in trial reports.

 

Henry Drysdale and Ben Goldacre, on behalf of the COMPare trials team.

 

The relevant section of Annals’ amended instructions is given in full below, with the recent changes in bold.

3. Data Sharing and Reproducible Research

Authors should be prepared to provide original study data and statistical code if requested by the editors. The editors may request access to study data during the peer-review process and may cease consideration of a manuscript if the authors cannot or will not provide the data. These data will be treated confidentially and not shared beyond the editorial staff without specific permission from the authors.

As of April 19, 2016, Annals requires authors submitting manuscripts reporting the results of a clinical trial to submit a copy of the study protocol with all dated amendments. If the manuscript is accepted for publication, the protocol will be published as a supplement to the article on annals.org. If necessary, Annals will consider publication of protocols that redact proprietary information from introduction and background sections.

To encourage transparency and reproducible research (Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med, Mar 2007; 146: 450-453.) Annals publishes a statement with every original research article and systematic review/meta-analysis indicating the authors willingness to share the following items with the public:

– Study protocol (original and amendments)

– Statistical code used to generate results

– Dataset from which the results were derived

For articles reporting clinical trials submitted after April 19, 2016, the Reproducible Research Statement for the study protocol will include “published as supplemental material to this article”.

Annals strongly encourages but does not typically require the sharing of these items unless the article is reporting the results of a clinical trial in which case sharing of the protocol is required (see above). However, we do require that authors state their willingness to share, and any conditions for sharing. Access to these items may range from completely unrestricted (e.g., free availability of all the items via posting on an open-access Web site) to restricted (e.g., availability of certain portions of the items to approved individuals through written agreements with the author or research sponsor).