Computerized monitoring of COVID-19 trials, studies and registries in ClinicalTrials.gov registry

Number of studies over time

Using the date when the study was first registered on CTG, we counted the number of total studies for each study type on a given date. We created a plot showing the temporal trend over time.

Study sites and country

Despite the existence of national registries, CTG registry contains studies from many countries. For example, as of 2 June 2020, 61% of recruiting studies were located solely outside the US (according to CTG’s overview page (“Trends, Charts, and Maps—ClinicalTrials.gov”, https://clinicaltrials.gov/ct2/resources/trends)). Additional incentive for registration on CTG are FDA rules that require studies submitted in support of new drug applications to FDA to be registered at CTG. Similar requirements of CTG registration also exist from many study funders such as NIH, and journals such as International Committee of Medical Journal Editors (ICMJE) member journals. We analyzed the geographic data for each study by reviewing the location fields in the CTG study record and counted the number of sites and identified the country of their locations. Each study could consequently include one or multiple countries.

Study update activity

The clinical trial registry allows principal investigators to update the public about study completion, the final number of enrolled participants and basic summary results. High public interest in updates about COVID-19 studies was the main motivation for measuring update activity and study record recency.

We quantified the level of update activity for a study by looking at the number of updates and what fields are updated for a given study after its initial registration. We attempted to classify type of updates into technical updates (e.g., study sites changes) and updates of significant public interest (actual primary completion date or deposition of study results). We also evaluated the recency of the CTG study record by evaluating the number of days between the last update and the current date. Study update data is not available via AACT or the CTG Application Protocol Interface (API) (CTG Team, 2020b). However, information about study updates is available via the CTG website. To obtain this information we wrote an R script to scrape the data into a computable form.

We also evaluated the level of update activity for studies based in each country and found studies from which countries were more active in updating CTG study records.

Study design

We analyzed CTG metadata pertaining to study design to classify studies. One feature analyzed for all study types was study enrollment (number of participants). CTG allows the reporting of estimated and actual enrollment into the trial. Study record managers can use this mechanism to publicly post updates about the number of enrolled participants.

Interventional trials

Observational studies and registries

Observational studies and registries have metadata features that are not recorded for interventional trials. Such analyzed features were: time perspective and observational model for the set of observational studies and registries. In addition, one feature recorded and analyzed for only registries was follow up time.

Search strategy

We found that the first search method, using the official title of the study, was the most comprehensive and included the most COVID-19 studies. The numbers listed below reflect only the search strategy and not applying the criteria based on study status. As of 11 May 2020, the first search method returned a total of 1,302 studies. The second search method, based on the free text condition field, found fewer records (1,165 studies). The third method based on the MeSH term, returned 328 studies. The significant difference in the studies captured in the third search strategy is likely due to the fact, that there is no specific MeSH term for COVID-19 at this point and the MeSH condition field is not required and is left blank for many studies (38.2% of studies captured in the first search method left MeSH condition term blank).

We then applied metadata inclusion criteria (studies that are active, recruiting or completed and are not expanded access). This reduced the set for the first search method to 752 studies, the set from the second search method to 680 studies, and the set from the third search method to 210 studies.

This led us to select the set of COVID-19 studies generated from the first, most comprehensive search method, based on study title.

Final study set

In terms of completion and presence of results, 48 studies in the final set were completed at the time of this analysis. None have provided summary results to this point. It is important to note that studies are typically required to submit results within 1 year after the primary completion date (“FDAAA 801 and the Final Rule—ClinicalTrials.gov”, https://clinicaltrials.gov/ct2/manage-recs/fdaaa). Also, at the time of the analysis, 106 studies have past their primary completion date (12 studies when using primary completion day + 30 days) declared in the latest study record and have a status that indicates the study is still ongoing. This indicates that the record is possibly not kept current. Administrators do typically have 30 days after a status change to update the record (see 42 Code of Federal Regulation [CFR] 11.64(a)(1)(ii)) (“Frequently Asked Questions—ClinicalTrials.gov”, https://clinicaltrials.gov/ct2/manage-recs/faq#updatesToCT). In an extreme case, 20 studies of those 106 studies have a status of “not yet recruiting” and are past their primary completion date.

To understand how our metadata study inclusion criteria affects the final set, we briefly analyzed the set of studies excluded due to our study metadata criteria. The studies removed due to metadata included 516 studies that were not yet recruiting, 10 that were withdrawn, five that were suspended and two that were terminated. The reasons for termination of the two studies were “We cannot meet number of subjects as recently published similar studies” for NCT04357535 and “The epidemic of COVID-19 has been controlled well in China, no eligible patients can be enrolled at present” for NCT04257656. The interventions of the terminated studies were ACE-I (angiotensin-converting enzyme inhibitors) and ARB (angiotensin receptor blocker) for the former and Remdesivir for the latter. Our study type criteria also excluded 17 studies with a study type of “Expanded access”.

Interventional trials

We identified a total of 401 COVID-19 interventional trials from CTG. These 401 studies included a total of 1,666 interventions.

Interventions

There were a total of 449 distinct interventions listed prior to the implementation of our normalization and mapping process. Once the interventions were mapped the amount of normalized interventions was reduced to 403. The full mapping is available at the study repository (file: intervention_map2.xlsx) (“regCOVID Project Repository,” 2020; https://github.com/lhncbc/r-snippets-bmi/tree/master/regCOVID). Table 7 shows the most common interventions used in COVID-19 interventional trials. Given our counting methodology for interventions, each study can be counted multiple times in Table 7 because combined interventions are expanded into their components as well as kept as a combination. The most common drug intervention was Hydroxychloroquine with 92 studies, followed by Azithromycin with 24 studies. The two (Hydroxychloroquine and azithromycin) appeared together four times. The most common combination intervention was Lopinavir/Ritonavir with 16 studies. We also found the presence of interventions most likely listed as a comparator or a non-intervention group, rather than a specific intervention. This is seen as 99 studies have placebo listed as an intervention while another 40 studies have standard care listed.

Table 7:

Most frequent interventions by study count (with a minimum study count of 13).

Intervention type	Intervention	Study count	Date when first appeared
Placebo	Placebo	99	20-Februay-2020
Drug	Hydroxychloroquine	92	06-February-2020
Other	Standard care	40	23-January-2020
Drug	Azithromycin	24	23-March-2020
Drug	Ritonavir	24	28-January-2020
Drug	Tocilizumab	21	09-March-2020
Biological	Convalescent plasma	20	23-March-2020
Drug	Lopinavir	20	28-January-2020
Drug	Lopinavir/Ritonavir	16	30-January-2020
Drug	Chloroquine	13	19-March-2020

DOI: 10.7717/peerj.10261/table-7

As for non-drug interventions, the most common biological is convalescent plasma with 20 studies. Other leading interventions for different types (not shown in Table 7) include oxygen supplying equipment for device with six studies and Vitamin C for dietary supplements with four studies. We also found that the same intervention can be listed as different intervention types. For example, convalescent plasma was listed for 14 studies as the intervention type biological, three times as other and three times as drug. We combined each intervention to count as the most commonly used intervention type when counting the intervention. For this case of convalescent plasma, that would count as 20 studies and categorize convalescent plasma as having the type biological.

Interventions over time: We also evaluated the temporal change for the most common interventions by analyzing the amount of new studies weekly for the most common interventions as seen in Fig. 2.

The plot shows how most interventions, including the most common intervention of hydroxychloroquine, peak in new weekly studies in early April. The plot also shows the later emergence of other interventions, such as convalescent plasma (shown in green).

Observational studies

We found a total of 287 observational studies. Similarly, to interventional trials, the vast majority of observational studies had just one site (238 studies, 82.9%). The country with the most observational studies was France with 75 (26.1%), followed by the United States with 47 (16.4%). Observational studies are updated less frequently than interventional trials as only 52.6% (151 studies) of the COVID-19 observational studies have been updated since first being submitted to CTG (compared to the 71.1% of interventional trials that have been updated at least once). The observational study with the most updates was NCT04334954 “SARS-COV2 Pandemic Serosurvey and Blood Sampling” with 25 updates since registration on 6 April 2020. The most commonly updated public interest feature for observational studies was the “Study Status” which was updated 270 times by 151 studies and the most common technical feature updated was “Contacts/Locations” with 99 updates from 83 studies.

The median enrollment for observational studies was 353 participants. One feature of observational study design is the time perspective. A majority of the observational studies analyzed were prospective (180 studies, 62.7%), as opposed to 58 studies (20.2%) which were retrospective. For observational model, 167 of the observational studies (58.2%) use a cohort model. The second most commonly used model for the analyzed observational studies was case (45 studies, 15.7%).

Contrary to our expectation, we found observational studies that included interventions in their CTG record. Of the 287 observational studies, 179 (62.4%) listed something in the free-text intervention field. However, this number is misleading as in many cases the listed intervention was something that stated that there was no intervention (such as “no intervention”, “observation”, “non-interventional”, etc.). Of the listed interventions most are listed as intervention type “Other” (86 studies, 30.0%) or “Diagnostic Test” (34, 11.9%).

Registries

We analyzed a total of 64 COVID-19 registries (shorter term for registry-based studies). Of these registries 52 (81.3%) were limited to one site. The largest number of sites was 53. The countries with the most COVID-19 registries were France and the United States with nine studies each. Similar to observational studies, just over half of the analyzed registries, 51.6% (33 registries), have been updated at least once since their first registration. Also similar to observational studies, the most common public interest update for registries is to the study status, which has been updated 56 times by all 33 registries with an update, and the most common technical update is to the contacts and locations with 28 updates from 18 studies.

The median enrollment for the set of registries was 388 participants. Registries have many specific design features that differentiate them from other study types. One is the presence of a targeted follow-up time. The most common follow-up time for the analyzed registries was 1 year for 15 studies (23.4%), which was listed as either “1 year” or “12 months” and was combined to get the accurate value. The shortest follow-up time was 1 day for NCT04331171, “Epidemiological Observation From a Smartphone Self-monitoring Application for Suspected COVID-19 Patients’ Triage”, while the longest targeted follow-up duration for a registry was 20 years, for NCT04359602, “COVID-19 Recovered Volunteer Research Participant Pool Registry”. For registries, CTG collects their observational model (similar to observational studies). The majority of registries, 48 (75.0%), use a cohort model. Also similar to observational studies, registries can include a time perspective. However, unlike observational studies, no registries are retrospective. Instead the time perspective is usually either prospective (50 studies, 78.1%), or cross-sectional (six studies, 9.4%). A cross-sectional perspective means that the observation or intervention is made at a single point in time rather than on a continuous or recurring basis (CTG Team, 2020a).

Like observational studies, more than half (53.1%, 34 of 64 registries) included an intervention in the free text field. These interventions also include many that are not representative of an actual intervention and rather state the absence of an intervention just like with the previously mentioned observational studies. This is also shown in the intervention type as 19 of the 34 registries (55.9%) have an intervention type of “other”.

Study limitations

Our study has several limitations. First, we only used CTG registry to look for COVID-19 studies. Within this registry, we evaluated three search strategies, but some relevant COVID-19 studies may possibly be missed. Without a benchmark gold standard of all COVID-19 studies, the recall of our search strategy cannot be evaluated. It was out of scope of this study to establish the precision of our search. Second, our semantic harmonization of interventions is based on manual mapping by a single expert. Third, there are significant limitations of the informatics-based approach compared to manual review.

Related studies

For example, COVID-19 Evidence Service from Centre for Evidence-Based Medicine at University of Oxford offers more comprehensive reviews. It was out of scope of our project to offer results comparable to human review. Fragkou et al. (2020) used a search and manual review methodology to compile and analyze a set of COVID-19 interventional trials and their interventions. Checcucci et al. (2020) did a literature and clinical trial registries search based on built-in search criteria to review COVID-19 vaccine trials. Rosa & Santos (2020) did a manual search of CTG to analyze COVID-19 trials using repurposed interventions. Considering the existing published studies, we conclude that our study is the first study to rely solely on computerized data science methods to compile and analyze a set of COVID-19 interventional trials, observational studies and registries. Our approach of using computerized data science methods allows for the continuous monitoring of the current state of COVID-19 research with minimal additional effort compared to a resource intensive manual review methodology. During a continuously changing public health emergency, this ability for any researcher to quickly and efficiently monitor changes and trends in clinical research is invaluable in informing the direction of their research efforts.

Generalizing report to other diseases

Our emphasis on fully computerized analysis of a COVID-19 set of studies was motivated by our larger vision to apply the R scripted report for all MeSH encoded diseases found within the CTG registry. We refer to this result as the regCTG project and report repository. regCTG allows analysis of research by MeSH keyword for all clinical domains. We generated reports for all MeSH terms with at least 100 registered studies. A collection of nearly 1,000 disease-based reports is available at https://github.com/lhncbc/CRI/tree/master/regCTG.

We consider this generalization from a COVID-19 research report to a research report for nearly 1,000 diseases an important result of our project.

In another follow-up research project for this COVID-19 case study, we have also built a disease-intervention snapshot knowledge base (called D-SHOT) that lists all interventions appearing in interventional trials for a given condition (“Project Repository for Disease Snapshot”, https://github.com/lhncbc/r-snippets-bmi/tree/master/D-SHOT). This knowledge base of disease-intervention pairs has many parameters for each intervention, such as date when first introduced, count of regularly completed studies or count of unusually completed studies (“terminated”, “suspended”, or “withdrawn”) studying that intervention. Experience from semantic harmonization of CTG’s free text field into terminology concepts gained during this COVID-19 project was crucial in these two follow-up projects by our team. A related, non-open source project called Sherlock, proprietary to Johnson and Johnson is similarly parsing CTG’s terms into formal concepts (Cepeda, Lobanov & Berlin, 2013).