COMPare aims to fix the ongoing problem of outcome switching in clinical trials by assessing individual trials for misreported outcomes, openly sharing our results, and then submitting correction letters to the journals concerned. From the outset, the question on our minds was: how will the journals respond? We’ve now had a range of responses, from a range of journals, and one striking feature is the diversity of approaches to these reporting errors being pointed out. Here is an example of best practice, from the BMJ.
We assessed the REEACT trial [1] for outcome switching back in November, and it scored pretty well. It wasn’t perfect, but a much higher standard of outcome reporting than the majority of trials we’ve looked at. In short, four of the pre-specified outcomes [2] were not reported or mentioned anywhere in the BMJ publication: ICD-10 score (itself a poorly pre-specified outcome) at four months; and EQ-5D at 4, 12, and 24 months. Another subtler issue was that PHQ-9 was reported as a dichotomous outcome (PHQ-9>10), and this was not pre-specified. Creating novel dichotomous outcomes increases the risk of false-positive results; if the cut-off is not pre-specified, trialists are free to choose any value, with each cut-off creating a different outcome.
As usual, we wrote a letter to the editors [3] describing these errors, to correct the record on the misreported outcomes. A few days later, the authors of REEACT posted a response [4] addressing our concerns, but fundamentally disagreeing. The cause of our disagreement seems to arise from a misunderstanding on the part of the trialists about what constitutes a pre-specified outcome. They wrote that “there is no inconsistency between our trial report and our pre-specified trial protocol which has been in the public domain for several years”. However, the protocol they cite as “pre-specified” was published in December 2012 [5], when the trial start date is May 2009. That’s three years earlier! For the missing EQ-5D score, they stated that the failure to report this was “not an omission on our part, but reflects a separation of clinical and economic results which sometimes happens (even in the BMJ)”. On this, we disagree: publishing quality of life data in a separate academic paper might well be done sometimes, but it is in no sense a recognised universal convention, and we therefore think that readers should have been told that an extra three secondary outcomes were pre-specified but left unreported, and that these would be reported elsewhere. This would have been consistent with the widely recognised CONSORT guidelines on trial reporting [6].
We outlined these concerns in more detail in a further rapid response to the trial on the BMJ site [7], and despite the authors’ initial response, for the first time, we are delighted to say that our letters worked. Last week the authors, to their very great credit, posted a correction on the REEACT trial in the BMJ [8], in which they correct the record on the missing ICD-10 and EQ-5D outcomes. They also posted a further response [9]: while this still contains minor disagreements with our conclusions about outcome reporting in their trial, they have broadly acknowledged the initial errors, and corrected them swiftly and clearly.
We think this demonstrates clear best practice, a commitment to good science, and serious consideration of the problem at hand. It is exactly what readers of journals would expect from authors and journals who learn of misreporting in their publications, and it sets the standard for the other journals we have assessed.
We now have a range of responses and correspondence from all journals in our study that we will be transparently sharing on this site, and publishing as part of the academic output from our project. Some of them take a very different approach to this example of best practice from the BMJ, but we are confident that all journals share a common commitment to high reporting standards, and we are optimistic that the scientific community will see progress on this issue.
Henry Drysdale and Ben Goldacre, on behalf of the COMPare team.
References:
[1] Gilbody S et al, Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial), BMJ 2015;351:h5627.
[2] REEACT trial registry entry: https://www.controlled-trials.com/ISRCTN91947481/91947481?link_type=ISRCTN&access_num=ISRCTN91947481, last accessed 20/01/2016.
[3] Drysdale H et al, Discrepancies between pre-specified and reported outcomes in Gilbody et al: https://www.bmj.com/content/351/bmj.h5627/rr-12, BMJ Rapid Response 17/12/2015, last accessed 20/01/2016.
[4] Gilbody S et al, REEACT is entirely consistent with CONSORT and the authors exceeded reporting requirements of CONSORT and COMPare: https://www.bmj.com/content/351/bmj.h5627/rr-13, BMJ Rapid Response 21/12/2015, last accessed 20/01/2016.
[5] REEACT Trial Protocol version 12, 10/12/2012: https://www.nets.nihr.ac.uk/__data/assets/pdf_file/0003/51438/PRO-06-43-05.pdf, last accessed 20/01/2016
[6] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.
[7] Drysdale H et al, Changes to pre-specified outcomes must be declared and explained in the published trial report: https://www.bmj.com/content/351/bmj.h5627/rr-14, BMJ Rapid Response 12/01/2016, last accessed 20/01/2016.
[8] Correction on the REEACT trial, published 12/01/2016: https://www.bmj.com/content/352/bmj.i195, last accessed 20/01/2016.
[9] Gilbody S et al, Final response from the authors on the integrity of outcome reporting in the REEACT trial, https://www.bmj.com/content/351/bmj.h5627/rr-15, BMJ Rapid Response 14/01/2016, last accessed 20/01/2016.
As a trialist, I think the work you are doing is important, but I also wonder if it generating sensationalistic headlines. I am certainly concerned about investigators switching primary outcomes to suit their hypothesis. That sort of thing must be stamped out. The issue around non-primary outcomes is tricky. My sense is that there is huge variation between how investigators apply the terms “secondary” and “other” outcomes. Unfortunately, they are used interchangeably, and sometimes there are so many secondary outcomes listed, they could not possibly all be listed in print, although they can now be included in online appendices. I think that many investigators would be unclear about how publication of secondary outcomes in an appendix would impact on ability to publish these as an independent paper. It is problematic when outcomes are switched or hidden to change the interpretation of results but I, perhaps naively, wonder whether this is less often the case than attempts to get the primary message into print with the limited space available. There is yet another problem. A colleague of mine told me of the urgency with which his group published the results of a trial in NEJM. All of the primary analyses were carried out but certainly there wasn’t the extent of exploration that might be deemed problematic by your project. The investigators were not trying to misrepresent their results in any way. There are all sorts of reasons there is pressure to get things published rapidly. Even with the intention of publishing secondary results in secondary papers, it is inevitably more difficult to get these published in a secondary paper, especially if the secondary results are less “exciting” than the primary. And, of course, after a few rejections of a secondary paper, investigators often move onto something else, not out of an agenda to hide anything that would change results, but just because of priorities, What I do agree is that there could be online repositories where tables of all the stated outcomes of a trial could be presented, if not published in a journal.