What Can COVID Teach Us about Research Integrity?

The pre-Covid world faced many issues including extreme politics, nationalism and racism; huge wealth disparities were widespread. The Covid pandemic, with more than 160 million cases and 3.3 million deaths reported worldwide to date,^[1] amplified these problems through disinformation and conspiracy theories and affecting impoverished and essential workers most of all. Research and publishing likewise had their weaknesses highlighted by the Covid pandemic.

Peer review, the lynchpin of scholarly publishing, depends on experts willing to devote volunteer time to carefully evaluate what is and is not reported, often while juggling their own research, a busy clinical schedule and other responsibilities. Editors must identify red flags and missing elements while similarly juggling multiple responsibilities and the many deadlines publishing a journal involves. With the onset of the Covid pandemic, however, time was a luxury no one had—not researchers, who could not instantly plan and conduct well-designed multicentre studies, enrol sufficient numbers of patients and check and recheck data; not editors and reviewers, who had less time to carefully review research and ask for additional information as necessary; not clinicians, who sought answers in the myriad but inadequate observational studies (freely available, at least)^[2] to decide which best applied to their patients; and least of all, patients, who were getting ill and dying while their families and society in general were distraught and clamouring for information.

The Covid crisis had several consequences for research. First, researchers studied what information was readily available to them, inundating journals with submissions. Second, the volume of research submitted to journals overtaxed editors and reviewers,^[3] and the latter used different review criteria to avoid slowing time to publication.^[4] At least one major journal accepted some research without external peer review.^[5] Some journals inadvertently published duplicate submissions, as they quickly went through peer review to publication simultaneously.^[6] Third, the volume of research published overwhelmed clinicians and others trying to find the best available evidence,^[7] which was not up to usual standards.^[8]

What were the consequences of a rushed and overloaded system? To date, 109 articles have been retracted, plus an additional 12 retracted due to journal error, 4 retracted and reinstated and 6 Expressions of Concern issued.^[9] Two of the most notable retractions were published in the Lancet^[10] and N Engl J Med,^[11] journals generally not lacking in resources. In brief, vascular surgeon Sapan Desai and co-authors published two observational studies ostensibly of the Surgisphere database, one concluding that hydroxychloroquine for Covid was associated with an increased risk of death^[10] and another that patients with cardiovascular disease were more likely to die from Covid.^[11] When asked for more information, as stated in the retraction notices, all authors other than company founder and CEO Sapan Desai were ‘unable to complete an independent audit of the data’^[12] and ‘all the authors were not granted access to the raw data and the raw data could not be made available to a third-party auditor’.^[13]

Could these issues have been identified before acceptance? A spokesperson for N Engl J Med said, ‘We have limited experience with reviewing or publishing studies like this one, which used a large database based on electronic medical records. The reviewers and editors asked the authors questions about the data sources and data validity. The editors accepted the authors' responses, rather than asking for help from reviewers with expertise in this type of data. In the future, our review process of big data research will include reviewers with such specific expertise.’^[14] However, other researchers found that had existing reporting guidelines for research using medical databases been adhered to, the study limitations would have been apparent.^[15] Given these issues, should peer review and reporting be more transparent? Suggested guidelines describing the specific peer review conducted are rarely used.^[16] Publishing the peer reviews themselves could make their strengths and limitations more apparent.^[17]

Furthermore, retraction is not the end of the story: research frequently continues to be cited after it has been retracted, without indicating that the studies were retracted.^[18] According to the N Engl J Med website, the Surgisphere study has been cited 382 times to date.^[11]

Preprints are one way to enable sufficient time for peer review while not delaying access, and they have served as both a lifeline and a source of misinformation during Covid.^[19] As a lifeline, some badly-needed descriptive information about the clinical characteristics of Covid was quickly made available.^[20] At the same time, some research, notably the use of hydroxychloroquine and azithromycin for Covid, was widely disseminated before the study was peer-reviewed and found to be of poor quality (fortunately, Covid preprints were rapidly published in peer-reviewed journals––at least according to a preprint study).^[21] Predictably, some media sources were unwilling to heed the prominent warnings on preprints. Medarxiv states: ‘This article is a preprint and has not been certified by peer review (what does this mean?). It reports new medical research that has yet to be evaluated and so should not be used to guide clinical practice.’ However, careful, discrete, deliberate reporting is unlikely when the public hangs on every Covid word while the 24x7 news media compete for clicks.

Covid research has had many successes. Vaccines were produced and published in record time; treatments were researched, preprinted and published and understanding of the new illness evolved quickly.^[22] But what are the lessons for medical publishing? For starters, requiring the use of existing reporting guidelines^[23] could help reviewers and editors conduct better and more efficient reviews. Authors and journals should have tools to identify retracted articles to prevent continued inaccurate citations. Preprints help provide rapid access for researchers and clinicians, yet the media persist in reporting unreviewed results to the public: better solutions are needed. When extraordinary events occur and rapid results are essential, relying on a voluntary system of peer review may not be enough. Healthcare systems, governments and/or funders could underwrite emergency peer review teams to enable experts to thoroughly review important submitted research and/or review preprints in real time. What could be more important than getting the best possible carefully evaluated research out to clinicians, public health professionals and the public as quickly as humanly possible?

Perhaps research needs its own Operation Warp Speed to coordinate design of comprehensive, thorough, efficient observational and interventional research across medical systems through an infrastructure designed to support it; efficient journals and well-qualified peer reviewers to enable expeditious publications and timely follow-up research on studies that generate wasted effort and expense (e.g. emphasizing deep cleaning of surfaces while ignoring masks). Study facilitation and rapid reviewers alone are not enough; losing experienced researchers in a pandemic because they cannot get childcare is surely a social crisis in need of a solution.^[24]

Covid has exposed existing weaknesses in both societies and journals. Journals need to learn from this moment and use the tools that are available such as requiring reporting guidelines. Transparent peer review can help expose its strengths and weaknesses (from unaddressed comments or from poor quality reviews). Preprints have their place, but only for researchers and clinicians with the capacity to evaluate them. Moreover, preparation for the next pandemic may need to include comprehensive research planning, from design to publication. Covid is having devastating effects throughout the world, but if we learn from this moment, its legacy could have some benefits as well.

Conflicts of interest. None declared