How is COMPare funded?

No specific funding has been sought for COMPare. The PI Ben Goldacre receives academic funding from the Laura and John Arnold Foundation to do work in the field of scientific integrity. The other two supervisors, Kamal Mahtani and Carl Heneghan, are funded by teaching and other small non-industry grants. The majority of the analysis and coding is currently done by 5 medical students at the University of Oxford, who work for free.

Don’t journals already check for outcome switching?

Journals generally say that they do, but our findings – and those from previous work on the subject – show that they do not.

Are you publishing on this?

Yes, we are currently submitting a series of academic journal publications on the quantitative and qualitative data we have collected.

There are often legitimate reasons for changing outcomes during a trial. Does COMPare consider this “outcome switching”?

No. We agree that outcomes can change for good reasons after a trial has started. However, consistent with the CONSORT guidelines, these changes must be declared and explained in the trial report. Where changes have been declared, we consider the outcomes in that trial to be correctly reported.

Where do you get the information on pre-specified outcomes?

Our gold standard for finding pre-specified outcomes is a trial protocol that pre-dates trial commencement, as this is where CONSORT states outcomes should be pre-specified. However this is often not available, in which case, as a second best, we get the pre-specified outcomes from the trial registry entry that pre-dates the trial commencement. Where the registry entry has been modified since the trial began, we access the archived versions, and take the pre-specified outcomes from the last registry entry before the trial began. As in the previous FAQ point, consistent with CONSORT, if such changes are made after the start of the trial, but they are discussed and declared in the paper, then we consider the outcomes in the trial to be correctly reported. If there is no registry entry or protocol from before the trial began, then there are no pre-specified outcomes at all.

On one of your assessments, you score outcomes as “non-prespecified”, however they are declared in the protocol published with the trial. Why did you not consider this protocol when assessing the trial?

If the protocol was published after trial commencement, it cannot, by definition, contain pre-specified outcomes. If the outcomes in this protocol differ from those originally specified before the trial, then these changes must be declared in the publication reporting the trial results.

Trial protocols are often updated and published after trial commencement, for good reasons. Do you take into account protocol amendments when assessing outcome reporting?

Protocols published after the trial start date cannot, by definition, contain pre-specified outcomes. We regard changes to pre-specified outcomes as completely normal, however they must be declared in the publication reporting the trial results. Where updates or amendments to the original outcomes have been declared, we consider the outcomes in that trial to be correctly reported.

What if some outcomes are reported in a different publication?

This is fine, as long as this fact has been declared in the trial publication. For example, if a trial says “here we are reporting A B and C, in a subsequent paper we will report X Y Z” then the outcomes X Y Z are not considered as unreported, and they are removed from the denominator.

What’s next?

Phase one of our project was writing the correction letters. We are now moving on to phase two: discussing journals’ responses, and publishing commentary on specific aspects of coding individual trials, to illustrate the mechanics of misreporting by using concrete examples.

Where are your data?

We take an “open lab books”, “open science” approach. All of our data are shared in full on this site, including our internal coding sheets that list all the pre-specified outcomes, the presence or absence of each pre-specified outcome in the trial report, and the novel non-pre-specified outcomes. We are also openly sharing responses from journals, and our responses back to them.

This is a huge amount of work!

Yes it is. It takes several hours on average to obtain the pre-specified outcomes, and then compare them against each trial report. We believe this work could be done more efficiently, and should be done, by journals themselves.

I disagree with your coding of my trial! Can I tell you why, or comment on it?

We will happily correct any errors, please email [email protected] and do bear in mind the FAQ issues above. We will gladly publish comments about coding or journal responses on our blog. However please do bear in mind the following: we have set out to correct the record on incorrect outcome reporting through correction letters to the journal that printed the error. This is to ensure that those using the results of trials to inform their own clinical decision making and research are able to access information on the reporting error. We therefore think it is most appropriate that informed discussion on individual trials switching their pre-specified outcomes should take place in the pages of the journal that published the trial.

Can I help?
Yes, by raising awareness of outcome switching, and individual trials that have misreported their outcomes; and by working within your field to reduce the prevalence of the problem, by supporting CONSORT and other initiatives. We are currently considering how best to expand the model. In particular we are interested in working with other groups who want to “own” a series of journals, for example in their own medical specialty. Please do get in touch if you are interested. [email protected]