Translate this page into:
What Can COVID Teach Us about Research Integrity?
Trustee, World Association of Medical Editors (WAME)
|How to cite this article:|
Winker M. What Can COVID Teach Us about Research Integrity?. Natl Med J India 2021;34:65-67
The pre-Covid world faced many issues including extreme politics, nationalism and racism; huge wealth disparities were widespread. The Covid pandemic, with more than 160 million cases and 3.3 million deaths reported worldwide to date, amplified these problems through disinformation and conspiracy theories and affecting impoverished and essential workers most of all. Research and publishing likewise had their weaknesses highlighted by the Covid pandemic.
Peer review, the lynchpin of scholarly publishing, depends on experts willing to devote volunteer time to carefully evaluate what is and is not reported, often while juggling their own research, a busy clinical schedule and other responsibilities. Editors must identify red flags and missing elements while similarly juggling multiple responsibilities and the many deadlines publishing a journal involves. With the onset of the Covid pandemic, however, time was a luxury no one had—not researchers, who could not instantly plan and conduct well-designed multicentre studies, enrol sufficient numbers of patients and check and recheck data; not editors and reviewers, who had less time to carefully review research and ask for additional information as necessary; not clinicians, who sought answers in the myriad but inadequate observational studies (freely available, at least) to decide which best applied to their patients; and least of all, patients, who were getting ill and dying while their families and society in general were distraught and clamouring for information.
The Covid crisis had several consequences for research. First, researchers studied what information was readily available to them, inundating journals with submissions. Second, the volume of research submitted to journals overtaxed editors and reviewers, and the latter used different review criteria to avoid slowing time to publication. At least one major journal accepted some research without external peer review. Some journals inadvertently published duplicate submissions, as they quickly went through peer review to publication simultaneously. Third, the volume of research published overwhelmed clinicians and others trying to find the best available evidence, which was not up to usual standards.
What were the consequences of a rushed and overloaded system? To date, 109 articles have been retracted, plus an additional 12 retracted due to journal error, 4 retracted and reinstated and 6 Expressions of Concern issued. Two of the most notable retractions were published in the Lancet and N Engl J Med, journals generally not lacking in resources. In brief, vascular surgeon Sapan Desai and co-authors published two observational studies ostensibly of the Surgisphere database, one concluding that hydroxychloroquine for Covid was associated with an increased risk of death and another that patients with cardiovascular disease were more likely to die from Covid. When asked for more information, as stated in the retraction notices, all authors other than company founder and CEO Sapan Desai were ‘unable to complete an independent audit of the data’ and ‘all the authors were not granted access to the raw data and the raw data could not be made available to a third-party auditor’.
Could these issues have been identified before acceptance? A spokesperson for N Engl J Med said, ‘We have limited experience with reviewing or publishing studies like this one, which used a large database based on electronic medical records. The reviewers and editors asked the authors questions about the data sources and data validity. The editors accepted the authors' responses, rather than asking for help from reviewers with expertise in this type of data. In the future, our review process of big data research will include reviewers with such specific expertise.’ However, other researchers found that had existing reporting guidelines for research using medical databases been adhered to, the study limitations would have been apparent. Given these issues, should peer review and reporting be more transparent? Suggested guidelines describing the specific peer review conducted are rarely used. Publishing the peer reviews themselves could make their strengths and limitations more apparent.
Furthermore, retraction is not the end of the story: research frequently continues to be cited after it has been retracted, without indicating that the studies were retracted. According to the N Engl J Med website, the Surgisphere study has been cited 382 times to date.
Preprints are one way to enable sufficient time for peer review while not delaying access, and they have served as both a lifeline and a source of misinformation during Covid. As a lifeline, some badly-needed descriptive information about the clinical characteristics of Covid was quickly made available. At the same time, some research, notably the use of hydroxychloroquine and azithromycin for Covid, was widely disseminated before the study was peer-reviewed and found to be of poor quality (fortunately, Covid preprints were rapidly published in peer-reviewed journals––at least according to a preprint study). Predictably, some media sources were unwilling to heed the prominent warnings on preprints. Medarxiv states: ‘This article is a preprint and has not been certified by peer review (what does this mean?). It reports new medical research that has yet to be evaluated and so should not be used to guide clinical practice.’ However, careful, discrete, deliberate reporting is unlikely when the public hangs on every Covid word while the 24x7 news media compete for clicks.
Covid research has had many successes. Vaccines were produced and published in record time; treatments were researched, preprinted and published and understanding of the new illness evolved quickly. But what are the lessons for medical publishing? For starters, requiring the use of existing reporting guidelines could help reviewers and editors conduct better and more efficient reviews. Authors and journals should have tools to identify retracted articles to prevent continued inaccurate citations. Preprints help provide rapid access for researchers and clinicians, yet the media persist in reporting unreviewed results to the public: better solutions are needed. When extraordinary events occur and rapid results are essential, relying on a voluntary system of peer review may not be enough. Healthcare systems, governments and/or funders could underwrite emergency peer review teams to enable experts to thoroughly review important submitted research and/or review preprints in real time. What could be more important than getting the best possible carefully evaluated research out to clinicians, public health professionals and the public as quickly as humanly possible?
Perhaps research needs its own Operation Warp Speed to coordinate design of comprehensive, thorough, efficient observational and interventional research across medical systems through an infrastructure designed to support it; efficient journals and well-qualified peer reviewers to enable expeditious publications and timely follow-up research on studies that generate wasted effort and expense (e.g. emphasizing deep cleaning of surfaces while ignoring masks). Study facilitation and rapid reviewers alone are not enough; losing experienced researchers in a pandemic because they cannot get childcare is surely a social crisis in need of a solution.
Covid has exposed existing weaknesses in both societies and journals. Journals need to learn from this moment and use the tools that are available such as requiring reporting guidelines. Transparent peer review can help expose its strengths and weaknesses (from unaddressed comments or from poor quality reviews). Preprints have their place, but only for researchers and clinicians with the capacity to evaluate them. Moreover, preparation for the next pandemic may need to include comprehensive research planning, from design to publication. Covid is having devastating effects throughout the world, but if we learn from this moment, its legacy could have some benefits as well.
Conflicts of interest. None declared
COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU), Johns Hopkins School of Medicine Covid Resource Center. Available at https://coronavirus.jhu.edu/ map.html (accessed on 13 May 2021).[Google Scholar]
Finley K. Global officials call for free access to Covid-19 research. Wired; 13 Mar 2020. Available at www.wired.com/ story/global-officials-call-free-access-covid-19-research/ (accessed on 13 May 2021).[Google Scholar]
Else H. How a torrent of COVID science changed research publishing—In seven charts. Nature 2020;588:553.[Google Scholar]
Horbach SP. No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic. Res Eval 2021;rvaa037. doi: 10.1093/reseval/rvaa037.[Google Scholar]
Bauchner H, Fontanarosa PB, Golub RM. Editorial evaluation and peer review during a pandemic: How journals maintain standards. JAMA 2020;324:453–4.[Google Scholar]
Marcus A. Researchers publish the same COVID-19 paper three times. Retraction Watch; 29 Oct 2020. Available at https://retractionwatch.com/2020/10/29/researchers-publish-the-same-covid-19-paper-three-times (accessed on 13 May 2021).[Google Scholar]
Brainard J. Scientists are drowning in COVID-19 papers. Can new tools keep them afloat? Science 2020. Available at www.sciencemag.org/news/2020/05/scientists-are-drowning-covid-19-papers-can-new-tools-keep-them-afloat (accessed on 13 May 2021). doi: 10.1126/science.abc7839.[Google Scholar]
Whitmore KA, Laupland KB, Vincent CM, Edwards FA, Reade MC. Changes in medical scientific publication associated with the COVID-19 pandemic. Med J Aust 2020;213:496–9.e1.[Google Scholar]
Retracted coronavirus (COVID-19) papers. Retraction Watch. Available at https://retractionwatch.com/ retracted-coronavirus-covid-19-papers/ (accessed on 13 May 2021).[Google Scholar]
Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: A multinational registry analysis. Lancet 2020. Available at www.sciencedirect.com/science/article/pii/S0140673620311806?via%3Dihub (accessed on 13 May 2021).[Google Scholar]
Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med 2020;382:e102.[Google Scholar]
Mehra MR, Ruschitzka F, Patel AN. Retraction—hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: A multinational registry analysis. Lancet 2020;395:1820.[Google Scholar]
Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: Cardiovascular disease, drug therapy, and mortality in Covid-19. N Engl J Med 2020;382:2582.[Google Scholar]
Davey M. Covid-19 studies based on flawed Surgisphere data force medical journals to review processes. The Guardian; 12 Jun 2020. Available at www.theguardian.com/world/2020/jun/12/covid-19-studies-based-on-flawed-surgisphere-data-force-medical-journals-to-review-processes (accessed on 13 May 2021).[Google Scholar]
Benchimol EI, Moher D, Ehrenstein V, Langan SM. Retraction of COVID-19 pharmacoepidemiology research could have been avoided by effective use of reporting guidelines. Clin Epidemiol 2020;12:1403–20.[Google Scholar]
Black C, Kowalczuk M, Alam S, Moylan E, Jones L, Mehmani B, et al. Standard taxonomy for peer review (2.0). International Association of Scientific, Technical and Medical Publishers; 29 Sep 2020. Available at https://osf.io/aynr5/ (accessed on 13 May 2021).[Google Scholar]
Meadows A. What does transparent peer review mean and why is it important? Scholarly Kitchen. 3 Aug 2017. Available at https://scholarlykitchen.sspnet.org/2017/08/03/transparent-peer-review-mean-important/ (accessed on 13 May 2021).[Google Scholar]
Piller C. Many scientists citing two scandalous COVID-19 papers ignore their retractions. Science 2021. Available at www.sciencemag.org/news/2021/01/many-scientists-citing-two-scandalous-covid-19-papers-ignore-their-retractions (accessed on 13 May 2021).[Google Scholar]
Guterman EL, Braunstein LZ. Preprints during the COVID-19 pandemic: Public health emergencies and medical literature. J Hosp Med 2020;10:634–6.[Google Scholar]
Tufekci Z. 3 ways the pandemic has made the world better. The Atlantic; 18 Mar 2021. Available at www.theatlantic. com/health/archive/2021/03/three-ways-pandemic-has-bettered-world/618320/ (accessed on 13 May 2021).[Google Scholar]
Kodvanj I, Homolak J, Virag D, Trkulja V. Publishing of COVID-19 preprints in peer-reviewed journals, preprinting trends, public discussion and quality issues. bioRxiv 2020. doi: 10.1101/2020.11.23.394577.[Google Scholar]
Conroy G. How COVID-19 could make science kinder. Nature Index; 24 Feb 2021. Available at www.natureindex. com/news-blog/coronavirus-could-make-science-research-peer-review-more-constructive (accessed on 13 May 2021).[Google Scholar]
EQUATOR Network. Available at www.equator-network.org (accessed on 13 May 2021).[Google Scholar]
COVID is amplifying the inadequacy of research-evaluation processes. Nature 2021;591:7.[Google Scholar]