Introduction

While COVID-19 outbreaks continue to emerge, accurate diagnosis in real-time by office-based clinicians is limited by a number of factors. Patients with COVID-19 present along with cases of seasonal respiratory viruses including influenza, which have overlapping symptoms. SARS-CoV-19 PCR testing has a turn-around time that makes it impractical for immediate decision-making in the clinic. Finally, rapid antigen point-of-care (POC) tests, when available, have limited sensitivity, with frequent false negatives [1].

Accurate interpretation of a POC rapid test result in clinical practice relies upon accurate pretest probability estimation [2]. Clinical prediction rules have previously been used to aid in interpretation of POC diagnostic test results in clinical diagnosis, for example in acute pharyngitis [3]. We anticipated the need in outpatient clinics of an evidence-based clinical prediction rule to use in combination with an imperfect SARS-CoV-2 POC antigen test for real-time decision-making.

Our hypothesis was that patients with COVID-19 would present with a spectrum of symptoms that differed from those with influenza and seasonal respiratory viral infections.

Methods

Participants

We studied the outpatient population of a Connecticut Health Maintenance Organization (HMO) with 45,300 members and an electronic medical record, for presenting symptoms in SARS-CoV-2-positive cases and in contemporaneous symptomatic SARS-CoV-2-negative cases. This study was approved by the Yale University Institutional Review Board. Patients calling a COVID-19 hotline or clinic triage were interviewed by trained nurses using a structured COVID-19 questionnaire, and decisions to order SARS-CoV-2 PCR testing were made upon consultation with a clinician. Study participants were consecutive adults with symptoms in the spectrum of COVID-19, predominantly with fever (subjective or self-measured) and cough, who were tested for SARS-CoV-2 by PCR between March 10, 2020, and May 26, 2020. Patients with SARS-CoV-2 PCR positive results (N = 120) were paired with symptomatic SARS-CoV-2 PCR negative patients (N = 120) matched by age, gender, and date of testing. We predetermined a minimum sample size of 100 cases and 100 controls, allowing for development of a rule containing up to 10 candidate predictor variables, to minimize risk of model overfitting [4]. Subjects were excluded if a structured interview had not been performed prior to testing, or if their SARS-CoV-2 test was resulted as inconclusive (Fig. 1). We also selected an independent, temporally distinct (May 27, 2020 to September 23, 2020) cohort of 40 consecutive SARS-CoV-2 PCR positive symptomatic outpatients for validation of the rule.

Fig. 1
figure 1

Flow of data collection

Symptom Ascertainment

Prior to tests being ordered, patients were interviewed by triage nurses using a structured symptom questionnaire, which had been developed based on early reports in hospitalized COVID-19 patients [5]. Data elements obtained from the structured questionnaire included fever, T max, cough, sore throat, shortness of breath, wheezing, fatigue, diarrhea, loss of taste, loss of smell, myalgia, duration of symptoms, healthcare worker status, and contact with a possible COVID-19 case. Additional symptoms that might help differentiate COVID-19 from competing influenza and respiratory viruses, but not included in the standardized triage questionnaire, were captured in unstructured clinical notes written by the same nurse or during a telemedicine visit with a clinician (MD, APRN, or PA). Symptoms extracted from clinical notes through patient-level chart review included chills/sweats, dry (nonproductive) cough, productive cough, chest pain, chest tightness, headache, red/irritated eye, nausea/vomiting, anorexia, nasal congestion/rhinorrhea, sinus pressure/pain, eye/retroorbital pain, ear pain/ache, and painful/red toes. Physical examinations were not performed for the safety of patients and staff.

Test Methods

Patients were referred for SARS-CoV-2 PCR testing based on clinical judgment and local guidelines. At the onset of the pandemic, testing supplies were scarce and criteria were strict, limiting testing to patients with fever, cough, and/or suspected COVID-19 contact. In later stages of data-gathering, criteria for testing had relaxed and included patients with a broad spectrum of viral syndromes. Nasopharyngeal samples were obtained at drive-through testing sites by trained clinical personnel. Patients were determined to have COVID-19 or a non-COVID-19 respiratory viral infection based on the result of SARS-CoV-2 PCR testing. We did not find any evidence of false-negative SARS-CoV-2 tests in this cohort, i.e., a positive result within the subsequent 2 weeks. Testing for alternative viral pathogens, such as influenza virus, was not performed.

Data Extraction

Patient-level data contained within the structured questionnaire and chart notes written prior the SARS-CoV-2 test being ordered were copied from the electronic medical record by a physician (DSS) who removed identifying data. Cases and controls were randomized, and prespecified data elements were extracted by a physician (EAR) who was blinded to the patient identity and test results. The variable fever included subjective fever or a patient-measured temperature of 100.4 or greater. Chest tightness was abstracted if the word “tightness” was specifically noted; other descriptions of discomfort including “chest pressure” and “heavy feeling in chest” were captured as “chest pain.” COVID-19 exposure included documented close contact with someone who had a positive test for SARS-CoV-2. Possible contact included both close contact with a person suspected of having COVID-19 and possible contact or contact of unknown or limited duration with a person with a known positive test result.

Data Analysis

All analyses were performed using IBM SPSS Statistics v 27.0. Individual symptoms were compared with the SARS-CoV-2 PCR result for conditional independence, and the odds ratio with 95% confidence interval was determined using the Cochran-Mantel-Haenszel test. We systematically combined individual symptoms into 326 clinical phenotypes and assessed each as a predictor of the SARS-CoV-2 result. The most common significant features (individual and combined) were used as predictors in a multivariable logistic regression model, with a binary outcome (presence or absence of COVID-19 by SARS-CoV-2 PCR). Stepwise variable selection, both forward and backward, was performed using multiple logistic regression, with a P value of 0.05 for addition or removal of variables.

The resulting logistic regression diagnostic model was analyzed for discrimination using the area under the receiver-operating curve (AUC). Calibration of agreement between the predicted and observed probabilities was assessed using the Hosmer-Lemeshow goodness-of-fit test, indicating a poor fit for P < 0.05.

We assessed internal validity of the model using a bootstrapping procedure. The modeling process was repeated with 1000 samples using a stratified method of case resampling with replacement with a 95% confidence interval. We validated the rule performance by applying it to an independent, temporally distinct cohort of 40 consecutive SARS-CoV-2 PCR positive symptomatic outpatients.

Results

The age range of subjects was 20–88 (median 43), and they were 62% female and 38% male. These factors were balanced between cases and controls by design. Duration of symptoms reported by patients at the time of the patient interview was 4.0 days (SD 4.63, range 1–30) for COVID-19 cases and 4.5 days (SD 6.65, range 1–35) for controls.

Of the 326 unique clinical phenotypes assessed, 18 were significantly associated with a SARS-CoV-2-positive result, and 10 were significantly associated with a SARS-CoV-2-negative result. Univariate association identified symptoms, phenotypes, and risk factors as significantly predictive of COVID-19 in symptomatic outpatients (Table 1), including loss of taste or smell, fever and cough, chills or sweats, and anorexia, as well as exposure to a known COVID-19 individual. The phenotype of wheezing or chest tightness was negatively associated with COVID-19, that is, more likely to be triggered by a virus other than SARS-CoV-2. Significance of phenotype combinations was driven by a few key symptoms. Either fever or cough was present in 87% of the 240 subjects. Symptoms of fever, T max, and cough alone were not significant individual predictors of COVID-19.

Table 1 Association between clinical characteristics and COVID-19 diagnosis among patients from a Connecticut HMO population triaged for SARS-CoV-2 testing from March to May 2020

When the significant symptoms, phenotypes and risk factors were analyzed using multivariable logistic regression; the resulting model found the phenotypes of loss of smell or taste (olfactory phenotype), fever and cough (febrile respiratory phenotype), and wheeze and chest tightness (reactive airways phenotype) to independently contribute to diagnostic prediction (Table 2). Significant individual factors that failed to independently contribute in the model included chills or sweats, anorexia, and exposure to a known COVID-19 patient. Discrimination of the model, as assessed by the area under the receiver-operating curve (AUC), was 0.736 (95% CI 0.674–0.798); P = 0.000 (Fig. 2). The Hosmer and Lemeshow test indicated a good fit of the model with P = 0.870.

Table 2 COVID-19 clinical decision rule derived from logistic regression modeling
Fig. 2
figure 2

Receiver-operating characteristic (ROC) curve for C19 logistic regression model. Area under receiver-operating characteristic curve (AUC) 0.736 (95% CI 0.674–0.798); P = 0.000

To develop the clinical prediction rule, we rounded the β-coefficients of independent predictors in the logistic regression model to proportional whole integers, to make an additive “fast-and-frugal” [6] rule with good face validity that is easy to recall and apply. The C19 rule consists of three clinical phenotypes that differentiate COVID-19 infection from influenza and viral respiratory infections: loss of smell or taste (2 points), fever and cough (1 point), and wheezing or chest tightness (− 1 point). When the C19 rule was applied to the entire dataset of 240 cases and controls (Table 3), patients with a C19 rule score of 3 had a LR of 15.0 (95% CI 2.0–112) for COVID-19 infection, present in 12.5% of cases and in only 0.8% of controls. Patients with a score of 2 had a LR 4.2 (CI 1.9–9.1), found in 46.7% of cases and 23.3% of controls; a score of 1, LR 1.2 (CI 0.7–1.9) in 39.2% of cases and 58.3% of controls; and a score of 0, LR 0.7 (CI 0.5–0.7) in 1.7% of cases but 17.5% of controls. A C19 score of − 1 was found more often in SARS-CoV-2-negative patients (17.5%), with a LR 0.1 (CI 0.02–0.4).

Table 3 Likelihood ratios stratified by C19 rule scorea

In the validation cohort, ages ranged from 18 to 67, and there were 21 males and 19 females. Loss of smell or taste was present in 70% of cases. The C-19 rule performed comparably in this validation set: score 3 LR 25 (20% of cases), score 2 LR 3.6 (50% of cases), score 1 LR 0.5 (10% of cases), score 0 LR 0.4 (20% of cases), and score − 1 (0 cases).

Discussion

Our aim was to develop a pragmatic evidence-based clinical prediction rule, focusing on clinical data readily available during the outpatient visit, to use in combination with a SARS-CoV-2 rapid antigen test with limited sensitivity. We felt that this would augment the ability of clinicians to differentiate symptomatic infection with SARS-CoV-2 from other background respiratory viruses.

The US Center for Disease Control (CDC) [8] notes that, since the symptoms of COVID-19 and influenza overlap, new loss of smell or taste is the main finding to help differentiate between the two. A meta-analysis of studies in 3563 patients with COVID-19 found loss of smell or taste in 47% [9]. In our study, loss of taste or smell is the finding most strongly associated with COVID-19 infection (LR 4.2), present in 38% of patients with COVID-19, and only 9% of those with another respiratory virus. Since the symptoms of fever or cough were often used to decide when to order testing, one or the other symptom was present in most (88%) of individuals tested but individually was not useful as predictors of COVID-19 infection. On the other hand, the combination of fever and cough, possibly a marker of more severe illness, appeared more often in SARS-CoV-2-positive patients. Wheezing or chest tightness is a negative predictor, that is, more strongly associated with alternate respiratory viruses rather than COVID-19, possibly a reflection of the triggering of reactive airways disease commonly seen in viral respiratory infections.

During the course of our data collection, the COVID-19 epidemic was rapidly evolving. In the early phase, test resources were scarce and testing was only recommended for patients with what were thought to be “classic” symptoms—fever and/or cough, or with history of COVID-19 exposure. Our initial phase of testing yielded a 20–25% SARS-CoV-2 positivity rate (Fig. 3). That phase also coincided with the peak of the influenza and respiratory virus season. By the later phase of data collection, a wider spectrum of symptoms was being more widely recognized, including loss of taste and smell, as well as dermatologic manifestations. Testing supplies became more readily available, and the criteria for testing, while still being recommended only for symptomatic patients, were relaxed. The positivity rate fell progressively to less than 10%, then finally below 5%. Influenza and respiratory viruses also receded, with seasonal allergy symptoms and tick-borne infections becoming more common as competing diagnoses. By summer 2020 (our validation cohort), new loss of smell or taste was the main presenting symptom of COVID-19, in 70% of cases. With widespread testing, the positivity rate had decreased below 1%.

Fig. 3
figure 3

SARS-CoV-2 prevalence and testing over time

The symptoms found in our study to be associated with COVID-19 were also identified as predictors of COVID-19 in prepublication reports from international cohorts using differing methodologies [10,11,12,13]. These studies included subjects from the UK, US, China, Australia, and Israel and found significant correlations of SARS-CoV-19 with symptoms of loss of taste and smell, fever, severe and persistent cough, and shortness of breath. These reports from external data sources strengthen the case for use of the specific independent variables included within our C19 rule.

Recommendations for Using the C19 Rule in Clinical Practice

A Bayesian approach to diagnosis in an ambulatory setting begins with prior probability (prevalence), adjusts the probability estimate using an evidence-based clinical prediction rule, then applies a point of care test, usually with limited sensitivity. To illustrate, we will use the prevalence spectrum seen in our population (about 5–20% during different phases of the pandemic in Connecticut), the stratified likelihood ratios of the C19 rule, and SARS-CoV-2 rapid antigen test characteristics derived from a meta-analysis of 10 studies reporting a pooled sensitivity of 64.8% (54.5–74.0) and specificity of 98.0% (95.8–99.0), or a LR + 32.4 and LR − 0.36 [14]. The C19 rule score can be combined with the SARS-CoV-2 rapid antigen test result using a smartphone calculator or a nomogram [7], when the local prevalence of SARS-CoV-2 test positivity is known.

A symptomatic patient in the setting of a widespread community outbreak of COVID-19 would have a prior probability of about 20%. If the patient has symptoms highly suggestive of COVID-19 such as cough or fever, combined with new loss of taste or smell, and no wheeze or chest tightness to suggest a viral upper respiratory infection (URI), the clinical (pretest) probability of COVID-19 would increase from 20% to about 51%, based on a C19 rule score of 2 (LR 4.2). If a rapid test is positive, the likelihood of COVID-19 is 97%, almost certain. If the rapid test result is negative, however, the probability is 27%, likely a false negative. The patient should be isolated, and a PCR test should be obtained. Another patient reports cough and chest tightness, but no fever or loss of smell or taste, giving a C19 rule score of − 1 (LR 0.1), so the clinical probability is adjusted from 20 to 2.4%. A positive rapid test would give a likelihood of COVID-19 of only 44%, an uncertain outcome, possibly a false-positive result but too high to ignore, so a PCR test should be obtained. A negative rapid test however would indicate a probability of COVID-19 of 0.9%, so it can be believed even though the rapid test is imperfect.

Recommendations for interpretation of the C19 rule combined with SARS-CoV-2 rapid antigen test vary with prevalence. When COVID-19 prevalence is high (20% or higher), patients with a C19 rule score of 0 or more should be tested with PCR, ideally prior to their visit. If presenting in person to the clinic, both a POC and a PCR swab could be simultaneously obtained while using PPE. When COVID-19 prevalence is low (5% or lower), patients scheduled for in-person visits can be screened for symptoms with the C19 rule. Typical viral URI symptoms are not suggestive of COVID-19 in a low prevalence scenario unless they are accompanied by the “plus factors” in the C19 rule. While the spectrum of symptoms caused by SARS-CoV-2 is wide, the less-typical manifestations are not helpful in differential diagnosis. If a patient has C19 rule symptoms suggestive of COVID-19 infection, however, they should be managed by telemedicine, or if they must be seen, they should first have a rapid POC or PCR test done. During seasonal peaks of influenza, concurrent POC testing for both COVID-19 and influenza could be a useful strategy, since influenza diagnosis would provide an alternate explanation for the symptoms, lowering the probability of COVID-19 [15], and would also suggest antiviral treatment with a neuraminidase inhibitor.

Limitations

The C19 rule should be applied in symptomatic mildly or moderately ill adult outpatients. It was not developed for use in a severely ill emergency department or hospitalized patients, or for screening asymptomatic individuals. It has not been tested in children or adolescents.

We did not collect tests for other viral pathogens, because of concern about increasing the possibility of airborne exposures. Since this is not a standard practice in outpatient clinic settings, however, this study represents pragmatic real-world practice.

We did not include every SARS-CoV-2-positive patient within our analysis. There were several alternate pathways to have the test ordered, with varying degrees of clinical data documented in the electronic medical record, so patients without the standardized screening and documentation were excluded from analysis. We did not include all patients with negative tests as controls, but instead selected symptomatic SARS CoV-2 negatives matched to a SARS-CoV-2-positive patient by age, gender, and date of testing who also had the standardized COVID-19 questionnaire administered prior to test ordering. We used this methodology to maintain a consistent method of symptom ascertainment across all subjects.

The main limitations of our study are that the C19 rule has yet to be verified by testing in an external validation cohort, or in combination with a SARS-CoV-2 rapid antigen test. In support of the findings in our study, however, the symptoms identified within our C19 rule were also reported to be associated with COVID-19 across four additional international ambulatory cohorts, and we were able to validate the performance of the C-19 rule in an independent, temporally distinct cohort of patients.

Conclusion and Relevance

We have developed a clinical prediction rule using symptom phenotypes to help differentiate between COVID-19 infection and seasonal respiratory viral infections. When applied in a Bayesian approach to diagnosis in an ambulatory setting—beginning with community prevalence (prior probability), adjusting the probability estimate using the evidence-based C19 rule, then applying a point of care SARS-CoV-2 rapid antigen test with limited sensitivity—clinicians should be able to manage patients across a range of clinical scenarios. The need for timely and accurate diagnosis of COVID-19 in clinical practice is important now with resurgent COVID-19 and will continue to be so as we enter another season of respiratory viral infections, whether COVID-19 is in the foreground or the background of causes.