ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Research Article

A review of the Cochrane COVID-19 Study Register reveals inconsistency in the choice and measurement of SARS-CoV-2 infection outcomes in prevention trials

[version 1; peer review: 1 approved]
PUBLISHED 28 May 2021
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

This article is included in the Emerging Diseases and Outbreaks gateway.

This article is included in the Coronavirus collection.

Abstract

Background: Multiple studies are evaluating how to prevent SARS-CoV-2 infection. Interventions are wide ranging and include vaccines, prophylactic drugs, public health safety measures, and behavioural interventions. Heterogeneity in the outcomes measured and reported is leading to research waste and inefficiency, slowing worldwide identification and implementation of effective methods to prevent infection. A core outcome set (COS) for studies of interventions to prevent SARS-CoV-2 infection has recently been developed, identifying infection as a critical outcome to measure. This paper examines how SARS-CoV-2 infection outcomes are measured in registered COVID-19 prevention trials and considers how this can be improved.

Methods: We searched the Cochrane COVID-19 Study Register to identify and review SARS-CoV-2 infection outcomes in prevention trials, including the rationale for choice of outcome measurement. We included phase 3 and 4 trials of COVID-19 prevention interventions. Early phase trials and studies relating to the transmission, treatment or management of COVID-19 were excluded.

Results: We identified 430 entries in the register, of which 199 unique prevention trials were included across eight settings and 12 intervention types. Fifteen (8%) trials did not include any SARS-CoV-2 infection outcomes. The remaining 184 (92%) studies included a total of 268 SARS-CoV-2 infection outcomes, of which 32 (17%) did not specify how infection would be measured. Testing (i.e. formal diagnostic test) as a standalone method for determining infection was used in 57 (31%) trials, whereas defining infection by symptoms alone was used in 16 (9%) trials. All other trials (n=79, 43%) included multiple infection outcomes, defined in different ways.

Discussion: There is considerable variation in how SARS-CoV-2 infection is measured within and across different interventions and settings. Furthermore, few studies report the rationale for outcome selection and measurement. Better transparency and standardisation of SARS-CoV-2 infection measurement is needed for the findings from prevention trials to inform decision-making.

Keywords

SARS-CoV-2, infection, outcomes, outcome measurement, COVID-19, prevention, trials

Introduction

Coronavirus disease 2019 (COVID-19), resulting from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, has significant morbidity and mortality. Addressing human-to-human transmission of SARS-CoV-2 is a global research priority.1 Research to fully understand the modes of transmission of SARS-CoV-2 is ongoing. Studies to prevent the spread of SARS-CoV-2 are increasing, with more than 1000 studies registered between December 2020 and February 2021.2 Interventions are wide ranging, including public health measures such as lockdown of countries/cities, closure of schools and public places, social distancing measures and handwashing, interventions in primary and secondary care practice such as the use of personal protective equipment (PPE) and increased time for appointments/interventions, alongside the development of effective vaccines and drugs to prevent disease.

Evidence is accumulating to evaluate the effectiveness of prevention interventions, but limitations in evidence synthesis have been identified because of inconsistent selection, measurement and reporting of outcomes in the studies.35 Whilst studies primarily focus on mitigating the risk of contracting COVID-19, they may measure other outcomes relevant to the population of interest and type of intervention, for example, educational outcomes in studies of school closures6 and tolerance/acceptability in studies of PPE use in secondary care.5 However, unnecessary heterogeneity in the outcomes measured and reported in COVID-19 prevention trials leads to research waste and inefficiency. For example, a recent commentary highlighted the rarity of randomised trials of public health interventions for COVID-19, which is likely due to the need for unattainably large sample sizes.7 Fretheim argues that underpowered trials in public health should be regarded as contributions to the larger body of evidence on COVID-19. However, for underpowered trials to be collectively assessed and synthesised in systematic reviews they need to be measuring the same outcomes.

A solution to this issue is to develop a core outcome set (COS), which is defined as an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical research studies in a specific area of health care to address inconsistencies in outcome measurement.8 The COS-COVID-P study, in collaboration with an international and multidisciplinary committee of experts and public contributors, has developed a COS for studies of interventions to prevent SARS-CoV-2 infection. During the COS development process, online workshops were held to consider the evidence on COVID-19 prevention and the issue of outcome heterogeneity. More than 70 key international stakeholders, including clinicians, researchers, public representatives, methodologists, systematic reviewers, guideline developers, and regulatory agency representatives shared their expert opinions. Agreement on the final COS was reached in August 2020. Two outcome domains were identified as being essential to all COVID-19 prevention studies: infection and intervention-specific harms.9 How to measure these outcomes now requires consideration and led to this paper.

SARS-CoV-2 infection can be measured through laboratory testing, symptom assessment or a combination of both.2 However, there is no single standalone diagnostic test nor agreement on diagnostic symptoms.10,11 Thus, there is a pressing need to consider how to assess SARS-CoV-2 infection uniformly across studies to optimise data reporting and synthesis. The aim of this study is to review SARS-CoV-2 infection outcomes in registered COVID-19 prevention trials and examine the reported rationales for choice of outcome measurement. Our intention is that this will inform optimal methods for the design of future studies.

Methods

Search strategy

For the purposes of the review presented here, we searched the Cochrane COVID-19 Study Register https://covid-19.cochrane.org/ on 24 November 2020. The Cochrane COVID-19 Study Register is a freely available, annotated reference collection of primary research studies on COVID-19, including interventional, observational, diagnostic, prognostic, epidemiological and qualitative designs. The register is continually updated to support rapid evidence synthesis, and includes studies from multiple data sources including Clinicaltrials.gov, the Cochrane Central Register of Controlled Trials (CENTRAL), Embase, medRxiv, PubMed and the WHO International Clinical Trials Registry Platform (ICTRP). We restricted our Cochrane COVID-19 Study Register search results by study type (“interventional”), study design (“parallel/crossover”), intervention assignment (“randomised”) and study aim (“prevention”). The Cochrane COVID-19 Study Register defines prevention studies as those aiming to assess one or more interventions for preventing the development of a specific disease or health condition.

Eligibility criteria

We included phase three and four trials of prevention interventions for COVID-19. We excluded phase one and two trials, including those that used other terms to imply that they were early phase (e.g. “explorative”), because these trials focus on safety, rather than the effectiveness of an intervention. Studies relating to the transmission, treatment or management of COVID-19 were also excluded. Studies identified from the Cochrane COVID-19 Study Register search were split equally between five experienced systematic reviewers (KA, SD, SG, NH, RM), who assessed the eligibility of their cohort of studies according to these eligibility criteria.

Data extraction

Data from eligible trials were extracted using a piloted, standardised data extraction spreadsheet, with the work split equally between five experienced systematic reviewers (KA, SD, SG, NH, RM). In cases of any ambiguity or doubt, queries were assessed by SD. Any queries were discussed with the whole team who collectively reached agreement on how to progress. The following data were extracted for each study: intervention type (e.g. drug, vaccine, behavioural), intervention (as described verbatim), setting (i.e. population of interest), SARS-CoV-2 infection outcomes (as described verbatim), SARS-CoV-2 infection outcome definition (i.e. how SARS-CoV-2 infection would be measured), whether or not the SARS-CoV-2 infection outcome was specified as being a primary or secondary outcome, and the rationale provided for SARS-CoV-2 infection outcome measurement selection. In addition, the reviewers recorded whether the study record related to a registry entry, trial protocol or the main report. If the registry entry record provided a link to a study protocol, this protocol was also reviewed, and the relevant data extracted.

Data analysis

Extracted data were summarised according to the number of SARS-CoV-2 infection outcomes per trial. For trials that included one SARS-CoV-2 infection outcome, data were summarised according to the single outcome definition specified in each trial (i.e. testing, symptoms, testing or symptoms, testing and symptoms). For trials that included multiple SARS-CoV-2 infection outcomes, data were summarised according to each outcome definition combination. For example, some trials may have measured one SARS-CoV-2 infection outcome using testing alone, a second SARS-CoV-2 infection outcome by symptoms alone and a third SARS-CoV-2 infection outcome by testing and symptoms. In addition to analysing the data by the overarching outcome definition (e.g. testing), we also summarised the data according to the specific underlying definition. For example, a trial would be summarised as having two versions of testing if they specified that one SARS-CoV-2 infection outcome was determined by reverse-transcriptase polymerase chain reaction (RT-PCR), and another SARS-CoV-2 infection outcome was determined by serology (antibodies, seroconversion, seroprevalence). The categorisation of outcome definitions was based on the verbatim text available in the registry records. Analysis was carried out using Stata version 16.1.

Results

Search results

Our searching of the Cochrane COVID-19 Study Register returned 430 records that were screened for eligibility, of which 231 were excluded. A total of 199 unique prevention trials were identified (Figure 1).

031b935b-3c51-4562-8f09-bcb02b22e49e_figure1.gif

Figure 1. Flow diagram of identified trials.

Trial characteristics

Of the 199 eligible trials, 176 (88%) had a trial registry record only. The remaining 23 trials had registry records plus a protocol (n = 15, 8%) or journal article (n = 6, 3%) or both a protocol and journal article (n = 2, 1%). Most trials examined drug prophylaxis (n = 92, 46%) or vaccine (n = 54, 27%) interventions. Other interventions included dietary supplements (n=20, 10%), behavioural/social distancing (n = 12, 6%) and PPE or other physical barriers (n = 7, 4%). Fourteen (7%) trials evaluated other interventions that included Ayurveda, homeopathy, traditional Chinese medicine, a medicinal plant, Unani, telehealth and financial support.

The trials were undertaken across eight combinations of setting and population of interest, with almost half (n = 98, 49%) in healthcare workers. Other settings included adults in the community (n = 45, 23%), high risk groups (e.g. elderly, hospitalised, or specific disease; n = 21, 11%), people in close contact with COVID-19 cases (e.g. family members; n = 16, 8%), nursing home residents and/or staff (n = 10, 5%), people with potential exposure to COVID-19 (n = 5, 3%), police (n = 3, 2%) and schools (n = 1, <1%). The characteristics of included trials are described in Table 1.

Table 1. Trial characteristics (n = 199).

Number (%)
Publication sources
Trial registry record only176 (88.4)
Trial registry record + protocol15 (7.5)
Trial registry record + journal article6 (3)
Trial registry record + protocol + journal article2 (1)
Intervention types
Drug prophylaxis92 (46.2)
Vaccine54 (27.1)
Dietary supplement20 (10.1)
Behavioural/social distancing12 (6.0)
Personal protective equipment or other physical barrier7 (3.5)
Other14 (7.0)
Ayurveda5 (2.5)
Homeopathy4 (2.0)
Chinese medicine1 (0.5)
Medicinal plant1 (0.5)
Unani1 (0.5)
Telehealth1 (0.5)
Financial support1 (0.5)
Setting
Health care workers198 (49.3)
Community/adults45 (22.6)
High risk groups (e.g. elderly, hospitalised, or specific disease)21 (10.6)
In close contact with COVID case (e.g. family members)16 (8)
Nursing home210 (5)
Potentially exposed to SARS-CoV-25 (2.5)
Police3 (1.5)
Schools1 (0.5)
SARS-CoV-2 infection outcomes measured/reported in the trial
No SARS-CoV-2 infection outcomes15 (7.5)
Single SARS-CoV-2 infection outcome3118 (59.3)
Two SARS-CoV-2 infection outcomes50 (25.1)
Three SARS-CoV-2 infection outcomes14 (7.0)
Four SARS-CoV-2 infection outcomes2 (1.0)

1 Seven of these studies also included others in close contact with COVID cases (two of which also included high risk groups); a further two of these studies included others with potential exposure to COVID (one of which also included police).

2 Or equivalent term.

3 In 32 (16.1%) studies, the single COVID outcome was a non-specific term.

SARS-CoV-2 infection outcomes

Of the 199 trials, 15 (8%) did not include any SARS-CoV-2 infection outcomes. These 15 studies predominantly evaluated behavioural (n = 8, where interventions were ‘STOP touching your face' mindfulness online training programme; Brief informational infographic; E-E video about COVID hygiene; Mental Imagery; Digital health literacy intervention; e-learning module; Telepsychoeducation with personalized videos; combined video display and live demonstration as training methods to healthcare providers for donning and doffing PPE) or PPE/physical barrier (n = 3) interventions, with the remaining four trials evaluating social distancing, dietary supplements, a medicinal plant and homeopathy. Examples of outcomes recorded in trials that did not include a SARS-CoV-2 infection outcome included frequency of face-touching behaviour in a behavioural intervention trial12 and contamination of any part of the base clothing or exposed skin of the upper body in a PPE trial.13

A total of 268 SARS-CoV-2 infection outcomes were identified across 184 (92%) trials, with infection measured as a primary outcome in 149 (81%) studies. Table 2 provides the outcome definitions recorded. We categorised outcome definitions using the verbatim text available in the registry records. In 118 (64%) trials, a single infection outcome was included. Fifty (27%) trials included two infection outcomes, 14 (8%) included three infection outcomes and two (1%) included four infection outcomes. Examples of multiple infection outcomes within the same trial are provided in Table 2. Table 3 lists the specific outcome definitions across the 184 trials.

Table 2. SARS-CoV-2 infection outcome definitions, by number of infection outcomes per trial.

Number of trials
(% of 184)
One SARS-CoV-2 infection outcome per trial1118 (64.1)
Testing alone49 (26.6)
Non-specific term for infection32 (17.4)
Symptoms alone16 (8.7)
Positive on both testing AND symptoms15 (8.2)
Positive on either testing OR symptoms6 (3.3)
Two SARS-CoV-2 infection outcomes per trial250 (27.2)
Testing alone (2 versions)7 (3.8)
Testing alone; Non-specific term for infection3 (1.6)
Testing alone; Symptoms alone9 (4.9)
Testing alone; Testing AND symptoms12 (6.5)
Testing alone; Testing OR symptoms3 (1.6)
Symptoms alone; Non-specific term for infection7 (3.8)
Testing AND symptoms; Non-specific term for infection2 (1.1)
Symptoms alone; Testing AND symptoms3 (1.6)
Symptoms alone; Testing OR symptoms2 (1.1)
Testing AND symptoms (2 versions)2 (1.1)
Three SARS-CoV-2 infection outcomes per trial314 (7.6)
Testing alone (3 versions)1 (0.5)
Testing alone (2 versions); Symptoms alone2 (1.1)
Testing alone (2 versions); Testing AND symptoms3 (1.6)
Testing alone; Symptoms alone; Testing AND symptoms2 (1.1)
Testing alone; Symptoms alone; Non-specific term for infection1 (0.5)
Testing alone; Testing AND symptoms; Testing OR symptoms1 (0.5)
Testing alone; Testing AND symptoms (2 versions)2 (1.1)
Symptoms alone; Testing AND symptoms (2 versions)2 (1.1)
Four SARS-CoV-2 infection outcomes per trial2 (1.1)
Testing alone; Symptoms alone; Testing AND symptoms (2 versions)1 (0.5)
Testing AND symptoms (3 versions); Testing OR symptoms1 (0.5)

1 Note that when the same COVID outcome definition was measured at multiple time points or analysed in multiple ways (e.g. as a binary outcome AND a time to event outcome), these repetitions were considered as a single COVID outcome for the purpose of this review.

2 Example of two infection outcomes: ‘Number of people turning symptomatic’ and ‘Number of people turning Covid19 positive’ (see: http://www.ctri.nic.in/Clinicaltrials/pmaindet2.php?trialid=44065).

3 Example of three infection outcomes: ‘positive serology or RT-PCR for COVID-19 up until day 28’, ‘positive serology at day 28’ and ‘symptoms of COVID-19’ (see: https://clinicaltrials.gov/ct2/show/NCT04342156).

4 Example of four infection outcomes: ‘Number of symptomatic infections confirmed by SARS-CoV-2 (COVID-19)’, ‘Number of asymptomatic infections confirmed by SARS-CoV-2 (COVID-19)’, ‘Severity of SARS-CoV-2 (COVID-19) infection’ and ‘Duration of symptoms of COVID-19 coronavirus infection’ (see: https://www.clinicaltrialsregister.eu/ctr-search/trial/2020-001530-35/ES).

Table 3. SARS-CoV-2 infection outcome definitions.

Number of trials with ≥1 outcome (% out of 184)Number of outcomes (% out of 268)
Non-specific term for infection (no explicit test/symptoms detailed)45 (24.5)45 (16.8)

  • COVID

  • SARS-Cov-2 infection

  • infected

29 (15.8)29 (10.8)

  • confirmed

  • documented

16 (8.7)16 (6.0)
Confirmed with testing alone96 (52.2)110 (41.0)

  • positivity (positive disease/test)

13 (7.1)13 (4.9)

  • antigen test

1 (0.5)1 (0.4)

  • swab

2 (1.1)2 (0.7)

  • lab confirmation

9 (4.9)9 (3.4)

  • microbiologically confirmed

1 (0.5)1 (0.4)

  • molecular test

1 (0.5)1 (0.4)

  • RT-PCR

36 (19.6)36 (13.4)

  • serology (antibodies, seroconversion, seroprevalence)

22 (12.0)23 (8.6)

  • RT-PCR OR serology

16 (8.7)16 (6.0)

  • test OR seroconversion

2 (1.1)2 (0.7)

  • virology OR serology

3 (1.6)3 (1.1)

  • molecular OR serological test

2 (1.1)2 (0.7)

  • RT-PCR AND SARS-Cov-2 test

1 (0.5)1 (0.4)
Symptoms alone45 (24.5)45 (16.8)

  • symptomatic1,2

40 (21.7)40 (14.9)

  • clinical evidence

2 (1.1)2 (0.7)

  • clinical disease without laboratory confirmation

1 (0.5)1 (0.4)

  • confirmed via online questionnaire

1 (0.5)1 (0.4)

  • confirmed OR probable

1 (0.5)1 (0.4)
Testing OR symptoms13 (7.1)13 (4.9)

  • lab confirmation OR symptomatic

  • positive test OR symptomatic

2 (1.1)2 (0.7)

  • asymptomatic OR symptomatic

4 (2.2)4 (1.5)

  • microbiological test OR clinical features

1 (0.5)1 (0.4)

  • RT-PCR OR symptomatic

4 (2.2)4 (1.5)

  • RT-PCR OR serology OR symptomatic

1 (0.5)1 (0.4)

  • molecular confirmation OR symptomatic

1 (0.5)1 (0.4)
Testing AND symptoms46 (25.0)55 (20.5)

  • asymptomatic

2 (1.1)2 (0.7)

  • lab confirmation AND asymptomatic

3 (1.6)3 (1.1)

  • lab confirmation AND symptomatic

8 (4.3)8 (3.0)

  • microbiologically confirmed AND symptomatic

1 (0.5)1 (0.4)

  • microbiologically AND lab confirmed AND symptomatic

1 (0.5)1 (0.4)

  • RT-PCR AND symptomatic

21 (11.4)21 (7.8)

  • (RT-PCR OR serology) AND asymptomatic

1 (0.5)1 (0.4)

  • (RT-PCR OR serology) AND symptomatic

3 (1.6)3 (1.1)

  • (RT-PCR OR molecular test) AND symptomatic

2 (1.1)2 (0.7)

  • (serology OR lab confirmed) AND symptomatic

2 (1.1)2 (0.7)

  • serology AND asymptomatic

1 (0.5)1 (0.4)

  • serology AND symptomatic

2 (1.1)2 (0.7)

  • (serology AND symptomatic) OR RT-PCR

1 (0.5)1 (0.4)

  • virologically-confirmed AND symptomatic

2 (1.1)2 (0.7)

  • test AND symptomatic

4 (2.2)4 (1.5)

  • molecular confirmation AND symptomatic

1 (0.5)1 (0.4)

1 Includes one study that specified individual symptoms only.

2 Note that “symptomatic” includes references to clinical confirmation.

Forty-five (25%) trials defined infection in non-specific terms (e.g. “COVID”, “SARS-CoV-2 infection”, “infected”, “confirmed”), without specifying how infection would be/was determined (i.e. via a specific test to be undertaken or symptoms to be recorded). In 32 of these trials (17% of all included trials), the non-specific infection term was the only infection outcome listed. Figure 2 displays the number of trials with non-specific COVID outcomes by month of registration. Of the 45 trials with non-specific COVID outcomes, 23 (51%) trials were registered within the first three months (March to May 2020) of the COVID-19 outbreak being declared a global pandemic.

031b935b-3c51-4562-8f09-bcb02b22e49e_figure2.gif

Figure 2. Trials with ≥1 non-specific SARS-CoV-2 infection outcome by month of registration (out of 184 trials with at least one SARS-CoV-2 infection outcome).

Left-hand axis - number of trials. Right-hand axis - % of total number of trials.

Testing was the only means employed for determining infection in 57 (31%) trials, whereas relying on symptoms alone to define infection occurred in 16 (9%). All other trials included multiple infection outcomes, defined in different ways (Table 2). Where one of the infection outcomes was to be determined by testing alone (n = 96), this was most commonly by RT-PCR (n = 36, 20%) and serology (n = 22, 12%) tests. Where infection was defined on the basis of symptoms alone (e.g. symptomatic COVID-19) (n = 45), the number of symptoms and definitions varied (Tables 4 and 5). Some trials included outcomes defined by combinations of testing and/or symptoms: 13 (7%) using either testing or symptoms to determine infection; 46 (25%) specifying both testing and symptoms must be positive to define infection (e.g. RT-PCR and symptomatic).

Table 4. “Symptomatic” definitions provided for SARS-CoV-2 infection outcomes.

“Symptomatic” definitions
https://clinicaltrials.gov/ct2/show/NCT04318015 “Symptomatic infection rate by COVID-19 defined as cough, dyspnea, fever, myalgia, arthralgias or rhinorrhea along with a positive COVID-19 real-time polymerase chain reaction test”
https://pubmed.ncbi.nlm.nih.gov/33068425 “The definition of Covid-19-compatible symptoms was based on guidance from the US Council for State and Territorial Epidemiologists (Appendix)13. Specifically, probable disease was defined as having cough, shortness of breath, or difficulty breathing, OR two or more of the following symptoms: fevers, chills, rigors, myalgia, headache, sore throat, new olfactory and taste disorders. Possible disease was defined as one or more COVID-19-compatible symptoms. Three blinded infectious diseases physicians independently adjudicated cases of symptomatic participants based on the above criteria.”
http://www.ctri.nic.in/Clinicaltrials/pmaindet2.php?trialid=43105 “defined as self-reported fever or cough, or shortness of breath, or respiratory distress, or runny or blocked nose plus a positive PCR test and/or antibody test”
https://clinicaltrials.gov/ct2/show/NCT04321174 “fever, cough or other respiratory/systemic symptoms (including but not limited to fatigue, myalgias, arthralgias, shortness of breath, sore throat, headache, chills, coryza, nausea, vomiting, diarrhea) by day 14 in a patient with laboratory confirmed infection, combined with microbiologic confirmation of COVID-19 infection in the participant”
https://clinicaltrials.gov/show/NCT04377646 “Any COVID-19 related symptoms (cough, fever, headache, vomiting, nausea, dyspnea, diarrhea, smell disorder, conjunctivitis, dizziness)”
https://clinicaltrials.gov/show/NCT04360122 “any symptoms suggestive of COVID19: fever, plus at least one sign or symptom of respiratory disease including cough, shortness of breath, respiratory distress/failure, runny/blocked nose, plus Laboratory: Sampling of the following will be withdrawn from all participants at beginning and end of the study: COVID 19 IgM (for recent infection) or IgG or (old infection)”
https://clinicaltrials.gov/show/NCT04445428 “Episodes with self-reported infectious disease morbidity suspected to be caused by COVID (three or more of the following: fever, cough, sore thought, extreme fatigue, loss of smell/taste).”
https://clinicaltrials.gov/show/NCT04452643 “Incidence of subjects with COVID-19, defined by the presence of: Fever, Any of the respiratory signs and/or symptoms: cough, dyspnea, respiratory failure, runny nose/nasal obstruction, Positive test for SARS-COV-2 (PCR or serology)”
https://clinicaltrials.gov/show/NCT04461379 “disease defined as positive SARS-Cov-2 test (serology), plus fever (using self-reported questionnaire), or at least one sign or symptom of respiratory disease including cough, shortness of breath, respiratory distress/failure (using self-reported questionnaire)”
https://clinicaltrials.gov/show/NCT04470427 “The participant must have experienced at least TWO of the following systemic symptoms: Fever (≥ 38°C), chills, myalgia, headache, sore throat, new olfactory and taste disorder(s)”
https://clinicaltrials.gov/show/NCT04471766 “Self-reported main symptoms of COVID-19 (three or more - fever, cough, fatigue, shortness of breath, loss of smell/taste)”
https://www.clinicaltrialsregister.eu/ctr-search/trial/2020-002287-31/ES: “Examine the incidence of typical symptoms of SARS-CoV-2 infection: fever, cough, headache, arthromyalgia, pharyngeal pain, dyspnea, diarrhea, vomiting, abdominal pain, anosmia”
https://clinicaltrials.gov/show/NCT04437693 “COVID-19 related symptoms, that may include: Temperature over 37.8 degrees Celsius, Shortness of Breath, Cough”
https://clinicaltrials.gov/show/NCT04614948 “Mild COVID-19 includes: Fever, sore throat, malaise, headache, muscle pain, gastrointestinal symptoms, cough, chest congestion, runny nose, wheezing, skin rash, eye irritation or discharge, or chills, without shortness of breath or dyspnea.”
https://clinicaltrials.gov/show/NCT04614948 “COVID-19 symptoms consistent with those defined by the US FDA harmonized case definition at the time of finalization of this protocol: fever or chills, cough, shortness of breath or difficulty breathing, fatigue, muscle or body aches, headache, new loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, diarrhea.”
https://clinicaltrials.gov/show/NCT04350931 “COVID 19 infection in the form of: fever, dry cough, fatigue, & dyspnea”
https://pactr.samrc.ac.za/TrialDisplay.aspx?TrialID=12175 “defined as I. fever (using a self-reported questionnaire), plus II. at least one sign or symptom of respiratory disease including cough, runny/blocked nose (using self-reported questionnaire)”
https://clinicaltrials.gov/show/NCT04534803 “fever (as documented in EHR) or at least one sign or symptom of respiratory disease including cough, shortness of breath, respiratory distress/failure (as documented in EHR)”
https://clinicaltrials.gov/show/NCT04349228 “The rate of COVID19 infections is defined by the occurrence of the clinical signs below: Cough, Dyspnea, Fever, Myalgia, Arthralgia, Rhinorrhea, Anosmia, Asthenia, Fatigability”
https://clinicaltrials.gov/show/NCT04452643 “WHO definition”
https://clinicaltrials.gov/show/NCT04526821 “Severity of SARS-CoV-2 infection (asymptomatic, mild, moderate, severe infection), assessed by the World Health Organization definitions”

Table 5. Symptoms included in definitions of SARS-CoV-2 infection outcomes.

Trial reference #
Symptom12345678910111213141516171819Frequency of inclusion in any definition (n = 19)
FeverYYYYYYYYYYYYYYYYYYY19
CoughYYYYYYYYYYYYYYYYYY18
Dyspnoea/shortness of breath/Difficulty breathing/ respiratory distress/respiratory failureYYYYYYYYYYYYYYYY16
Rhinorrhoea/coryza/runny or blocked noseYYYYYYYYY9
Loss of smell/anosmia/new olfactory/smell disorderYYYYYYYY8
Fatigue/extreme fatigue/malaiseYYYYYYY7
HeadacheYYYYYYY7
Myalgia/muscle painYYYYYYY7
Sore throatYYYYYY6
ChillsYYYYY5
Loss of taste/new taste disorderYYYYY5
Gastrointestinal symptoms/nausea/abdominal painYYYYY5
VomitingYYYY4
DiarrhoeaYYYY4
ArthralgiasYYY3
Conjunctivitis/eye irritation/dischargeYY2
RigorsY1
DizzinessY1
ArthromyalgiaY1
Pharyngeal painY1
Chest congestionY1
WheezingY1
Skin rashY1
AstheniaY1

SARS-CoV-2 infection outcomes by setting and intervention type

Tables 6 and 7 detail the use of various SARS-CoV-2 infection outcome definitions by setting and intervention type, respectively.

Table 6. SARS-CoV-2 infection outcome definitions by setting: Values are number of trials in each setting with at least one COVID outcome with this definition (% out of total number of trials in this setting).

SARS-CoV-2 infection outcome definition
Non-specific termTest onlySymptoms onlyTest AND symptomsTest OR symptoms
Close contact with COVID (n = 15)4 (27)6 (40)5 (33)3 (20)0 (0)
Community/adults (n = 37)12 (32)18 (49)7 (19)12 (32)1 (3)
Health care workers (n = 92)16 (17)53 (58)24 (26)21 (23)9 (10)
High risk groups (n = 21)8 (38)9 (43)2 (10)4 (19)2 (10)
Nursing home (n = 10)4 (40)5 (50)4 (40)5 (50)1 (10)
Police (n = 3)0 (0)1 (33)2 (67)0 (0)0 (0)
Potential exposure (n = 5)1 (20)3 (60)1 (20)1 (20)0 (0)
Schools (n = 1)0 (0)1 (100)0 (0)0 (0)0 (0)

Table 7. SARS-CoV-2 infection outcome definitions by intervention group: Values are number of trials in each intervention group with at least one COVID outcome with this definition (% out of total number of trials with this intervention).

SARS-CoV-2 infection outcome definition
Non-specific termTest onlySymptoms onlyTest AND symptomsTest OR symptoms
Behavioural/distancing (n = 3)1 (33)1 (33)1 (33)1 (33)0 (0)
Dietary supplement (n = 19)4 (21)15 (79)3 (16)2 (11)1 (5)
Drug prophylaxis (n = 92)16 (17)51 (55)23 (25)24 (26)9 (10)
Other (n = 12)5 (42)4 (33)6 (50)0 (0)0 (0)
PPE/physical barrier (n = 4)1 (25)2 (50)2 (50)1 (25)0 (0)
Vaccine (n = 54)18 (33)23 (43)10 (19)18 (33)3 (6)

Rationale provided for SARS-CoV-2 infection outcome selection/definition

Of the 184 trials that included SARS-CoV-2 infection outcomes, seven provided a rationale for their choice of outcome selection and/or measurement (see Table 8). Of these seven studies, this was often to justify their specific choice of testing, with one drug prophylaxis trial explaining that “testing and criteria for diagnosis COVID-19 is likely to rapidly evolve over the course of this trial and will vary internationally”14 and a school closure trial basing outcome measures on routinely collected testing data to ensure “feasibility, facilitate speed, and limit costs”.15 A drug prophylaxis trial in healthcare workers argued that relying on symptoms consistent with infection without laboratory confirmation raises “concerns of type II error from asymptomatic participants”.16 Conversely, another drug prophylaxis trial justified the decision to use symptoms given issues of access to testing, “Given limited availability of outpatient PCR testing in many jurisdictions during our study period, particularly in April 2020, probable Covid 19 based on Covid-19-compatible symptoms was included in the composite primary endpoint”.17

Table 8. Rationale provided for definition of SARS-CoV-2 infection outcomes.

Rationale
https://clinicaltrials.gov/show/NCT04346329 “For this, once the informed consent form is signed, the molecular test for the diagnosis of SARS-CoV-2 infection by RT-PCR will be carried out every 4 days in order to determine as closely as possible the moment the participant become positive.”
https://clinicaltrials.gov/ct2/show/NCT04519125 “To determine the occurrence of infection with the virus the study will use both molecular tests that detect the presence of viral genes in respiratory secretions, and serological tests that detect the response of the immune system to the virus.”
https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2020-001448-24 “Testing and criteria for diagnosis COVID-19 is likely to rapidly evolve over the course of this trial and will vary internationally. Therefore, the following pragmatic definitions will be applied:

  • o Confirmed positive test for COVID-19: either a laboratory test (performed according to each participating hospital’s local protocols) or a computed tomography (CT) thorax scan (based on individual radiologist interpretation and diagnosis) that confirms COVID-19 diagnosis.

  • o Confirmed negative test for COVID-19: a laboratory test (performed according to each participating hospital’s local protocols) that is negative for COVID-19 diagnosis.”

https://pubmed.ncbi.nlm.nih.gov/33001138 “A recent randomized trial for postexposure COVID-19 prophylaxis with a 5-day course of hydroxychloroquine did not demonstrate clinical benefit. However, the composite primary outcome measure for this study included symptoms consistent with infection without laboratory confirmation; most patients did not have assessment of SARS-CoV-2 infection by reverse-transcriptase polymerase chain reaction (RT-PCR), raising concerns of type II error from asymptomatic participants. We sought to test the hypothesis that administering daily hydroxychloroquine would prevent SARS-CoV-2 infection in hospital-based HCWs over 8 weeks of exposure via RT-PCR testing of nasopharyngeal (NP) swabs and serologic antibody testing from participants at baseline, 4 weeks, and 8 weeks of treatment.”
https://clinicaltrials.gov/show/NCT04328441 “The aim of our trial is to ensure the continuity of health care, and this is the reason for this pragmatic primary endpoint. Moreover, BCG vaccine is expected to reduce all-cause infections and not only COVID-19. Moreover, this was a non-funded study, precluding intensive personal follow-up, and the measurement of absenteeism as this was considered relevant, and immediately obvious.... One of the secondary endpoints in the HCW trial is documented COVID-19 infection. It is important to keep in mind that health workers in the Netherlands are structurally examined for the presence of COVID-19 in case of any respiratory symptoms or in case of fever. Therefore, we expect a high detection rate.”
https://pubmed.ncbi.nlm.nih.gov/33068425 “Given limited availability of outpatient PCR testing in many jurisdictions during our study period, particularly in April 2020, probable Covid 19 based on Covid-19-compatible symptoms was included in the composite primary endpoint.”
https://pubmed.ncbi.nlm.nih.gov/33068425 “To ensure feasibility, facilitate speed, and limit costs, we aim to base our outcome measures on routinely collected data, to the extent possible. Testing for COVID-19 takes place continuously and consistently across the country. All suspect cases have access to testing and laboratories use valid tests meeting quality assurance requirements.”

Discussion

We have reviewed COVID-19 prevention trials that were registered in the first nine months of the pandemic and examined in detail their rationale, selection and method of measuring SARS-CoV-2 infection. Of the 199 included trials, 268 SARS-CoV-2 infection outcomes were identified, across 184 studies. Of these, almost one fifth did not define how infection would be measured in the register entry, with the remaining studies specifying a combination of testing alone, symptoms alone, or testing and/or symptoms. Very few trials provided a rationale for the choice of how infection would be measured. The observed heterogeneity of outcome measurement of SARS-CoV-2 infection in COVID-19 prevention studies limits effective evidence synthesis and scientific comparison of interventions, which has enormous consequences for decision-making at an individual and public health level. We recommend that studies provide a rationale for choice of infection measurement and, if possible, that uniformity is agreed internationally on how to measure SARS-CoV-2 infection in trials of prevention interventions.

The heterogeneity of outcome measurement identified in the current study is not uncommon. A recent methodological systematic review revealed significant heterogeneity in the outcome measurement instruments used in trials evaluating therapeutic interventions for COVID-19, thus highlighting the need for greater consistency.18 However, such heterogeneity is not restricted to COVID-19 research studies. A multiplicity of outcome measurement instruments has recently been reported in areas such as delirium, hearing loss, chronic pelvic pain, and breast reconstruction, with studies concluding the need for consensus on outcomes and their associated measures for use in future clinical trials in these disease areas.1922

To our knowledge, this is the first study to examine how SARS-CoV-2 infection outcomes are measured in registered COVID-19 prevention trials. However, there are some methodological limitations. Only the Cochrane COVID-19 Study Register https://covid-19.cochrane.org/ was searched to identify COVID-19 prevention trials, which may limit the number of relevant studies identified. The Cochrane register, however, is updated regularly and includes studies identified from six data sources, including ClinicalTrials.gov. Furthermore, although we did not specifically search each of the data sources, we used the trial URL provided in the Cochrane Study Register entry to access the original trial registry entry. Another limitation is the lack of detail about infection outcome definitions reported in the trial registry entries. Where readily available via links within the registry, protocols and published papers were also examined. However, due to resource limitations we did not manually search for protocols and full text papers corresponding to the trials included in this review, meaning some detail on outcome definitions may have been missed and may be readily available in trial protocols thus potentially changing the results of this study. Trialists may have a rationale for their choice of outcome measurements, but this was not included in the registry entry.

The COS-COVID-P study recommended SARS-CoV-2 infection be measured in all COVID-19 prevention trials, regardless of the intervention and setting.9 We found 92% of trials included SARS-CoV-2 infection as an outcome; however, very few reported the rationale for SARS-CoV-2 infection outcome selection and measurement. From the limited details we have regarding rationale, decisions often appear to be based on accessibility of testing and feasibility. This is understandable and will likely be due to circumstances specific to the setting and country where the trial is being conducted. Going forward, we advise trialists to be transparent about their rationale for the definition of SARS-CoV-2 infection outcomes used when registering and reporting their trials, to improve understanding of the context of their research. The trial registration should include both the rationale for outcome measurement, along with details on how infection will be analysed. In addition, we understand trialists will collect as much data as they consider relevant, but we recommend they develop a coherent statistical analysis plan (SAP) to ensure transparency and reproducibility.23

In the current review, we found two thirds of the studies evaluating behaviour interventions, such as those intended to increase handwashing or SMS text prompts to wear masks, did not include any SARS-CoV-2 infection outcomes. Trials of simple behavioural interventions may be unlikely to measure infection, as their purpose is to modify an individual’s behaviour rather than to directly prevent an individual from acquiring COVID-19. This is in contrast with those of complex interventions that include a behavioural component (e.g. donning standardised PPE versus specialised PPE; or hand washing with soap versus using alcohol wipes) where the overarching aim is still likely to be prevention of SARS-CoV-2 infection.

Alongside SARS-CoV-2 infection, the COS-COVID-P study also led to the inclusion of intervention-specific harms in the COS for all COVID-19 prevention research studies.9 The current review did not examine this aspect. Determining these harm outcomes requires agreement amongst those with expertise in the specific interventions being tested. It may be that the harms are similar within certain intervention types, e.g. drug prophylaxis and vaccines. However, expert opinion will be necessary to determine the relevance of such an approach, but consistency will be important to avoid research waste and to facilitate evidence synthesis in which the results of studies of similar interventions are compared, contrasted and, where appropriate, combined. There may also be a need for COS for studies in specific settings. For example, a COS evaluating interventions for the prevention of COVID-19 transmission in care homes (COS-COVID-PCARE) is currently under development. Building on the COS-COVID-P study, the COS-COVID-PCARE study is proposing to develop a supplementary specific module for COVID-19 prevention in care homes.24

Conclusion

We have described the variation in the selection and measurement of SARS-CoV-2 infection outcomes, with illustrated examples from on-going trials across various settings and interventions. As knowledge about COVID-19 increases over time, this is leading to improvements in the design of studies and reporting. Nevertheless, our findings highlight the need for clarity, transparency and standardisation of measures of SARS-CoV-2 infection. It is now timely to highlight these issues, to improve the collection of infection data to inform key public health decisions.

Data availability

Underlying data

University of Liverpool Data Catalogue: SARS-CoV-2 infection outcomes. https://doi.org/10.17638/datacat.liverpool.ac.uk/1279.24

This project contains the following underlying data:

  • - The dataset consists of COVID-19 prevention trials, with the following data extracted for each trial: intervention type (e.g. drug, vaccine, behavioural), intervention (as described verbatim), setting (i.e. population of interest), SARS-CoV-2 infection outcomes (as described verbatim), SARS-CoV-2 infection outcome definition (i.e. how SARS-CoV-2 infection would be measured), whether or not the SARS-CoV-2 infection outcome was specified as being a primary or secondary outcome, and the rationale provided for SARS-CoV-2 infection outcome measurement selection.

Data are available under the terms of the Creative Commons: Attribution 3.0 license.

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 28 May 2021
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Dodd S, Gorst S, Avery K et al. A review of the Cochrane COVID-19 Study Register reveals inconsistency in the choice and measurement of SARS-CoV-2 infection outcomes in prevention trials [version 1; peer review: 1 approved] F1000Research 2021, 10:426 (https://doi.org/10.12688/f1000research.52491.1)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 28 May 2021
Views
15
Cite
Reviewer Report 13 Jul 2021
Natasha Tyler, Faculty of Biology, Medicine and Health, NIHR Greater Manchester Patient Safety Translational Research Centre (PSTRC), University of Manchester, Manchester, UK 
Approved
VIEWS 15
Summary:

The authors present a systematic review to assess outcome reporting in Covid-19 infection outcome trials. They review the Cochrane COVID-19 Study Register. They found heterogeneity in the SARS-CoV-2 infection outcome measurement choices in prevention trials.
... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Tyler N. Reviewer Report For: A review of the Cochrane COVID-19 Study Register reveals inconsistency in the choice and measurement of SARS-CoV-2 infection outcomes in prevention trials [version 1; peer review: 1 approved]. F1000Research 2021, 10:426 (https://doi.org/10.5256/f1000research.55776.r86394)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 28 May 2021
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.