The COMPare Trials research project assessed 5 trials published in Annals of Internal Medicine for misreported outcomes – a well-documented and common source of bias. We found that all 5 trials published in Annals over the 6 week period misreported their prespecified outcomes, and so we sent a correction letter on each trial.

We also identified a systemic problem. None of these trials had a publicly accessible protocol dated before trial commencement, so we took the prespecified outcomes from the trial registry entries, and compared these against those published in the report. This is not necessarily a problem: trial registers were set up explicitly for the purpose of preventing selective reporting of outcomes, with support from august bodies such as ICMJE, WHO, and legislation across various territories.

In response, Annals’ editors claimed that the registry entries are often “outdated, vague or erroneous”, and that the discrepancies we found are accounted for by trial protocols that cannot be accessed publicly, but are reviewed by editors behind closed doors. We view this as a very outdated approach, lacking in transparency, and a problematic view of trial registers. [Read more…]

As you will hopefully know by now, we have been writing to five leading medical journals to correct the record on misreported trials. We have sent 58 letters in total, and the responses have been varied: some (such as the BMJ) have issued rapid corrections, whereas others (such as JAMA and NEJM) have refused to publish any correspondence at all. The Lancet have taken a mixed approach: of 19 letters of correction we submitted they have so far published 8, accepted a further 3 for future publication, and rejected 2; with 6 still under editorial review dating back as far as 5 months.

However, there is an extra feature, around an important issue: who should take responsibility for misreported outcomes in clinical trials? We think journals have a clear set of roles and responsibilities here, not least because so many are listed as endorsing the CONSORT guidelines, which explicitly state that all prespecified outcomes should be reported. The Lancet have somewhat absented themselves from this discussion. Our published correction letters have, in most cases, been accompanied by letters from other researchers raising methodological criticisms of the trial in question. In all cases, our published letters have been accompanied by an author’s reply. Where this reply has reiterated significant misrepresentations or misunderstandings around pre-specification of outcomes, we have written a follow-up letter to the journal to point these out. However, each of these subsequent letters has been rejected by the Lancet editors; and our efforts to get the Lancet editors themselves to give a view, on misreported outcomes in their journal, have not yet had a reply. [Read more…]

At COMPare we have been checking for outcome switching in five major journals, and then writing correction letters wherever trials have been misreported, to see whether journals respond appropriately. Along the way we have identified and documented many serious failings.

Today we describe a new issue COMPare has identified. While editors at Annals of Internal Medicine maintain a public commitment to the modern era of trials transparency and reporting integrity, we have already established that in reality the journal preferentially uses unpublished protocols, which are often inaccessible to other readers, behind closed doors, to assess reporting accuracy on issues such as outcome switching. This is a surprising and concerning choice by Annals editors: to use inaccessible documents is itself concerning. Moreover, trial registers were specifically set up over a decade ago to address this need, and to address selective outcome reporting, with legislation from national governments alongside near universal support and endorsement from the medical and academic community, regulators, academic journals, and organisations such as the International Committee of Medical Journal Editors and the World Health Organisation. To undermine and abandon this resource, as Annals editors seem to have done, seems a remarkable choice, and one that warrants extensive public discussion and debate.

More concerningly, we have found that Annals has published a false claim about the accessibility of the one trial protocol that we have so far requested, and have so far failed to correct the record when this false claim has been pointed out. [Read more…]

As you will hopefully know by now, COMPare has been examining all trials published in five major journals to check if they have misreported their results by failing to report their prespecified outcomes, or by adding outcomes that weren’t prespecified. Outcome switching is widely recognised as a bad thing; however it continues to be widespread, despite various promises from journals that they have already addressed the issue.

Previous studies have measured the prevalence of outcome switching, finding it to be extensive, and quietly published academic papers giving an overall summary figure for how often this problem occurs. At COMPare we decided to go one step further: we have written a letter for publication on every trial that misreported its outcomes. Our reasons for doing this were twofold. Firstly, we wanted to test whether the mechanisms of science for correcting the record would function correctly. Secondly, we thought that the responses of the system, or the journals, might help shed light on why this problem has persisted, despite extensive public promises that it had already been addressed. You can see for yourself here the process of assessing a trial, and the kinds of misreporting we have found.

In this piece we describe the responses we have had from Annals of Internal Medicine, the world’s 4th highest impact factor medical journal. Five of the 67 trials we analysed were published in Annals, and all of these reports contained misreported outcomes, to varying degrees. We pointed out these errors in online comments for publication, on all the relevant trials. We expected that Annals editors would publish our corrections, encourage authors to address their errors, and report the prespecified outcomes as per best practice.
[Read more…]

There seems to be some resistance and misunderstanding from journal editors and the academic community on outcome switching, so we thought it would be useful to give an overview of what various authoritative bodies have to say on the topic. In this series, we will cover ICMJE, CONSORT, the FDA, and the Word Of God. Today, we look at the FDA.

[Read more…]

We have been monitoring outcome switching in five top journals, and writing letters to correct the record wherever we have found misreporting. You can read more about our project here, here and here. One peculiar response has been: “you’ve found so much misreporting, in so many trials! Your findings cannot be credible!”. This argument seems to have been used, for example, by Annals in their responses here and here (comment #2) saying they will not engage with our letters pointing out their misreporting.

So are our findings exceptional? Are we the only people to have found a problem? No. The phenomenon of academic journals permitting outcome switching has been studied at length, and it is now extremely well documented. The only thing we have done, which seems to have solicited some odd responses, is correct the record on individual trials, rather than simply publish overall prevalence figures.

For the avoidance of any doubt, here is a walk through the some of the recent literature on the prevalence of outcome switching. [Read more…]

There seems to be some resistance and misunderstanding from the journal editor and academic community on outcome switching, so we thought it would be useful to give an overview of what various authoritative bodies have to say on the topic. In this series, we will cover CONSORT, the FDA, ICMJE and the Word Of God. Today, we look at ICMJE.

[Read more…]

This is a long post on a response we have received from the editors of Annals of Internal Medicine, one of the world’s leading medical journals. We hope you will read it, as their approach to this issue raises some extremely important issues.

The COMPare project sets out to address the problem of outcome switching in clinical trials. We wanted to move on from simple studies reporting the prevalence of this issue, and instead we are exploring what happens when you try to correct the record on individual trials. From the outset, the key aspect of this work was: how will journals respond? In advance of our first academic publication on the project, consistent with our open data approach, we are now beginning to share the responses we have had from journals. We think they provide important qualitative information on the reasons why this problem has persisted for so long. They are also highly varied: the BMJ, for example, demonstrated best practice by immediately issuing a correction when alerted to misreporting of pre-specified outcomes in their journal. Other journals have not responded in the same way.

This blog post is about a long response to COMPare written by the editors of Annals [1], after we sent a series of correction letters on trials misreported in that journal. The editors have published their own response as a letter in the online edition of the journal, and it will soon appear in the paper edition, alongside some of our letters. We believe their response reveals a set of important misunderstandings about the issue of outcome switching, from the editors of the 4th highest impact factor journal in medicine.

[Read more…]