Retraction of COVID-19 Pharmacoepidemiology Research Could Have Been Avoided by Effective Use of Reporting Guidelines

Introduction

Routinely collected health data, such as registries, health administrative data, and electronic health record data, are increasingly being used for pharmacoepidemiology, clinical and health services research. As the COVID-19 pandemic rages in the world, we observed with great concern that two studies were published and subsequently retracted by The Lancet¹ and The New England Journal of Medicine.² These two pharmacoepidemiology studies used routinely collected health data in the form of the Surgical Outcomes Collaborative (Surgisphere), an international registry derived from data automatically extracted from electronic health records, health and financial administrative data, and a point-of-care data entry process for procedures. Both studies made a significant impact in COVID-19 research, public health, and health policy spheres, and received international attention from the media, as evidenced by Altmetric scores of 24,615 and 3829, respectively, at the time of this writing. The conduct of research that is subsequently retracted represents “research waste”, as initially defined in The Lancet.³ Publication of these manuscripts resulted in the temporary suspension of important COVID-19 clinical trials of hydroxychloroquine by the World Health Organization and others. More importantly, the publication and subsequent retraction of research erodes the public trust in science. While publication of these studies may have been symptomatic of the known limitations of the peer review and editorial process, lack of transparent and complete reporting may have also contributed to this failure. Both retracted studies would fall under the purview of the guidelines for the REporting of studies Conducted using Observational Routinely collected Data (RECORD),⁴ and more specifically the extension of RECORD for Pharmacoepidemiology (RECORD-PE).⁵ RECORD and RECORD-PE are extensions to the STengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines for non-interventional research,^6,7 the overall study design for the retracted studies. RECORD and RECORD-PE more directly address observational research using routinely collected health data, expanding on parts of the STROBE checklist to ensure readers, peer-reviewers and editors recognize the strengths and weaknesses of research using health data. We believe that, had these checklists been submitted for assessment by peer reviewers, inherent weaknesses in the manuscripts would have become apparent, potentially resulting in rejection by the journals. Unfortunately, journals and editors still do not routinely ask peer reviewers to use reporting guidelines as part of their assessment.⁸ In fact, the pressures from both the scientific community and journal publishing industry resulted in the strong drive to accept COVID-19 articles early in the pandemic, which may have compounded the problem of inadequate peer review and the need for retraction of published articles.⁹

Most research using routinely collected health data is observational in design. While NEJM endorses CONSORT for reporting of clinical trials, STROBE for non-interventional research is not suggested in the instruction for authors.¹⁰ The Lancet endorses use of STROBE,¹¹ but endorsement is insufficient to ensure the appropriate use of reporting guidelines by authors and peer reviewers. In addition, STROBE does not fully convey all intricacies in the methodology of research using routinely collected health data. We previously demonstrated that the reporting of research using routinely collected health data was gravely deficient, even in journals that endorsed STROBE.¹² Therefore, RECORD and RECORD-PE were created as extensions to STROBE for observational studies involving such data.

Methods

We evaluated both articles in question using the RECORD-PE checklist (Tables 1 and 2), and determined the proportion of adequately reported items on the STROBE, RECORD, and RECORD-PE checklists. We reported the proportion of items on each checklist that were adequately reported, allowing for half-points when an item was partially reported. Chi-square statistic was used to determine differences in the proportion of adequately reported items for each checklist.

Table 1 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)5 Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, The Lancet, 20201. Red Font Indicates Areas That Were Not Adequately Reported

Table 2 The RECORD Statement for Pharmacoepidemiology (RECORD-PE)5 Checklist of Items, Extended from the STROBE and RECORD Statements, for the Study Mehra et al, N Engl J Med, 20202. Red Font Indicates Areas That Were Not Adequately Reported

Results

In the article published by The Lancet, 29 of 34 STROBE items (85.3%) were adequately reported, compared with 3.5 of 13 RECORD items (26.9%), and 9.5 of 15 RECORD-PE items (63.3%) (χ² = 14.839, P <0.001). Similarly, the article published in NEJM reported 24 of 34 STROBE items (70.6%), two of 13 RECORD items (15.4%), and 7.5 of 15 RECORD-PE items (50.0%)(χ² = 11.668, P = 0.003). These results indicate that while STROBE items were generally adequately reported, RECORD and RECORD-PE items were not. Important aspects of the methods unique to research using routinely collected health data were not reported, including variables used to identify exposure, outcome and confounders, and validation of the coding or algorithms. RECORD also requires a detailed description of the database population (ie, how data were collected in the population from which the study database is created), and the description and accuracy of linkage of databases. The lack of description of the underlying Surgisphere database should have raised concerns amongst peer reviewers and editors.

Discussion

Non-transparent and incomplete reporting may have prevented peer reviewers from recognizing significant weaknesses in the research methods, especially if peer reviewers were unfamiliar with research using large health datasets. We found significant differences in the proportion of adequately reported items on the STROBE, RECORD and RECORD-PE checklists, with STROBE items being better reported than the items on the checklists designed for studies using routinely collected health data.

The reason given by three of the authors for the retraction of The Lancet manuscript was the inability of independent auditors to obtain the original data to verify the analyses.^1,2 While we understand that legal circumstances may be in place that prevent the data holder from sharing the original data, this runs counter to the movement toward open science and data sharing being adopted by many journals, especially those that endorsed the Wellcome COVID-19 statement on Gold Open Access and reusability,¹³ indicating underlying support for data availability. The availability of data and programming code allows for the replication of research and verification of results, paramount to the process of scientific discovery.¹⁴ The RECORD reporting guidelines include an item (RECORD 22.1) that requests authors “provide information on how to access any supplemental information such as the study protocol, raw data, or programming code”, and the availability of these data was not adequately reported in the two retracted manuscripts. As detailed in the RECORD explanatory document,⁴ we recognize that some circumstances may restrict the free sharing of raw data, such as privacy laws or data sharing agreements. The RECORD statement is a reporting guideline and does not mandate the open sharing of data. However, RECORD mandates reporting of whether data are available for sharing, and how to access the data, programming code, and study protocol. Had this information been reported in the two manuscripts, the editors and peer reviewers could have made informed decisions about whether to accept the manuscripts for publication.

Reporting guidelines improve the quality of research publications. There is both direct evidence and indirect evidence to support this statement. A randomised controlled trial (RCT) demonstrated a small benefit of including CONSORT and STROBE in the journal peer review process.¹⁵ Similarly, a systematic review and meta-analysis of reports assessing the completeness of RCTs reported that journal endorsement of the CONSORT guidelines were associated with improved completeness of reporting.¹⁶ Compared to manuscripts based on RCTs, observational research may be more at risk for hidden sources of bias. Therefore, it is essential to encourage transparent and complete reporting of research using routinely collected health data. Journals play a key role in raising awareness amongst authors and peer reviewers about the importance of reporting guidelines to aid with transparent reporting. However, journal editors may be concerned with the rigidity posed by reporting guidelines, and the risk that authors may bypass submission to journals requiring them as part of the submission process.¹⁷ Nevertheless, researchers have a generally positive attitude toward reporting guidelines.¹⁸ The Equator Network’s catalogue of reporting guidelines by main study type (https://www.equator-network.org/) and interactive walk-through of choice of reporting guideline (https://www.goodreports.org/) has made their use easier for both authors and peer-reviewers.

The design, conduct, and submission of research that is not transparent, open, and adequately reported is a danger to the public trust in science and is wasteful. We have demonstrated that the two retracted studies were inadequately reported, as defined by RECORD and RECORD-PE. We have written this article as an appeal to authors, publishers, editors, and peer reviewers to endorse and effectively implement the correct reporting guidelines in their submission and evaluation of manuscripts. Scientists, journals and the public can no longer accept research that is not transparent and open, especially in the time of a public health emergency such as the COVID-19 pandemic.