Introduction

Coronavirus disease 2019 (COVID-19) is a highly contagious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), firstly reported in Wuhan, Hubei Province, China [1, 2], that has rapidly spread into other countries leading to a current world-wide pandemic [3].

If a definite diagnosis of COVID-19 infection requires real-time reverse transcription polymerase chain reaction (RT-PCR) of viral nucleic acids, chest CT scan has proved to be of clinical importance and the main tool for screening. The Fleischner Society recently validated the use of imaging in patients suspected of having COVID-19 presenting with mild clinical features and at risk for disease progression, and as a help for medical triage of patients suspected of having COVID-19 in a resource-constrained environment, in case of moderate-to-severe clinical features and high pre-test probability of disease [4].

A recent meta-analysis by Kim et al showed that the diagnostic value of chest CT depends on the prevalence of COVID-19 infection in the studied population [5]. In areas where the prevalence is low, chest CT screening of patients with suspected disease has a low positive predictive value [5]. On the other hand, in the case of epidemic surge of patients at the emergency department, the clinicians will face a difficult challenge of rapid triage depending on disease presentation and severity. Patients with typical clinical symptoms and bilateral radiographic opacities may be hospitalized without a diagnostic CT scan. Patients presenting with fever and mild or no respiratory symptoms and a normal chest radiograph may leave the hospital after RT-PCR testing without chest CT scan and with surveillance by phone call during the following days. The patients for whom the diagnosis is unclear represent the group with intermediate probability and may benefit from a chest CT scan, looking for evidence of COVID versus other pathologies.

The objective of this study was to evaluate the inter-observer agreement and diagnostic accuracy including positive and negative predictive values of chest CT to identify COVID-19 pneumonia in patients with intermediate clinical probability during an acute disease outbreak in a European country.

Materials and methods

This retrospective monocentric study was approved by the Ethics Committee of Foch Hospital affiliated to Paris Ouest University, and the requirement for informed consent was waived.

Patients

Facing a sudden and huge increase in the number of patients arriving at the hospital for suspicion of COVID-19 pneumonia, physicians of the Respiratory Department, Internal Medicine Department, and Intensive Care Unit established recommendations by consensus to help physicians in charge of triage of these patients in the Emergency Medicine Department.

These recommendations were as follows:

  • Patients presenting with fever and moderate or severe respiratory symptoms (oxygen saturation less than 92%, or 92–96% + comorbidity) and bilateral crackles at auscultation should be hospitalized after a chest X-ray, without undergoing a CT scan.

  • Patients presenting with fever, mild respiratory symptoms (oxygen saturation more than 96%), and no comorbidity should be discharged at home with telephonic surveillance and usual recommendations.

  • Patients presenting with fever and/or respiratory symptoms, atypical findings at auscultation (no crackles or unilateral crackles or diminished breath sounds), and normal or equivocal chest radiograph (including unilateral opacities) should be referred to the department of radiology for a chest CT scan before the decision of hospitalization, as they are considered to have an intermediate clinical probability of COVID-19 pneumonia.

Three-hundred nineteen consecutive patients with intermediate clinical probability and who underwent a chest CT scan the day of their arrival between March 20 and April 8, 2020, were included in the study. In the same time interval, 225 consecutive patients with a high clinical probability of COVID-19 pneumonia were hospitalized without an initial chest CT scan. These patients were not included in the study, nor those who underwent a chest CT scan the following days for worsening of symptoms or to exclude thromboembolic disease.

CT scans

A CT room was fully dedicated to patients suspected of having COVID-19 accessible from the emergency department through a well-delineated route. Unenhanced low-dose volumetric acquisition was performed over the entire chest at full inspiration in all patients. Contrast injection was used in 53 patients because of suspicion of pulmonary embolism associated with COVID-19. Contiguous 0.6-mm axial images were reconstructed with high-resolution and soft tissue kernels.

Radiologist readings

Two independent chest radiologists (A.L.B. and P.A.G.) with 15 and 30 years of experience reviewed CT examinations being blinded for RT-PCR results and any clinical information except patient sex and age. They read the images using lung window and mediastinal settings on the PACS system (GE).

The radiologists were asked to classify the CT scans as highly probable, probable, and less probable of COVID-19 pneumonia, alternative diagnosis, or normal. They established their diagnosis on the basis of the recent publications from China illustrating typical and atypical patterns observed in patients with COVID-19 pneumonia [6,7,8,9,10,11,12,13,14,15,16] and according to the Radiological Society of North America expert consensus statement [17]. A highly probable diagnosis was based on a typical CT appearance of COVID-19 pneumonia with either peripheral, bilateral ground glass opacity (GGO) with or without consolidation or visible intralobular lines (crazy paving), or multifocal GGO of rounded shape with or without consolidation or visible intralobular lines. A probable diagnosis was proposed when multifocal, diffuse, perihilar, or unilateral GGO with or without consolidation were lacking a specific distribution and were non-rounded or non-peripheral, or when few very small GGO were present with a non-rounded or non-peripheral distribution. An alternative diagnosis was proposed when an obvious differential diagnosis was present on CT, e.g., lobar pneumonia, bronchopneumonia (including bronchial wall thickening, endobronchial secretions, fluffy centrilobular, and coalescent nodules), lung cancer, and lung abscess. A less probable diagnosis was proposed in the absence of high probable, probable, or alternative diagnosis and when the CT scan was not normal, e.g., marked bronchial wall thickening or bronchiectasis without evidence of active infection, isolated consolidation without GGO, focal GGO associated with discrete small nodules, or smooth interlobular septal thickening with pleural effusion. A normal chest CT scan was the last possible option, keeping in mind it does not exclude early COVID-19 pneumonia.

RT-PCR

The real-time reverse transcription polymerase chain reaction (RT-PCR) of viral nucleic acid was used as the reference standard for the diagnosis of COVID-19 infection. In 296 patients, RT-PCR assay was performed in our hospital by nasopharyngeal swab with a technique based on a reagent containing three viral targets (PCR Seegene Allplex, Eurobio). In ten patients, RT-PCR was performed in a different laboratory before patients arrived on site. In 13 patients, RT-PCR was not available because of an alternative diagnosis suggested by CT scan or because the patient was transferred to another hospital.

Statistical analysis

Concordances and discordances between the two radiologists for the diagnosis of COVID-19 pneumonia were assessed using kappa statistics. The values of kappa can range from + 1 (almost perfect agreement) to ‑ 1, where 0 represents the amount of agreement than can be expected from random chance [18].

The result of RT-PCR was used as the ground truth for measuring CT diagnostic accuracy. Because of the limited sensitivity of RT-PCR reported in the literature, clinical management and outcomes of patients with a negative RT-PCR and a CT scan classified by both radiologists as “highly probable or probable” for COVID-19 pneumonia were reviewed. On the basis of this analysis, patients were considered definite COVID-negative or reclassified COVID-positive.

Receiver operating characteristics (ROC) was plotted and the area under the curve (AUC) was calculated with 95% confidence intervals for each radiologist.

Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for both radiologists.

PPV and NPV were calculated with the following formula [19]:

$$ \mathrm{PPV}=\mathrm{sensitivity}\times \mathrm{prevalence}/\mathrm{sensitivity}\times \mathrm{prevalence}+\left(1-\mathrm{specificity}\right)\times \left(1-\mathrm{prevalence}\right) $$
$$ \mathrm{NPV}=\mathrm{specificity}\times \left(1-\mathrm{prevalence}\right)/\mathrm{specificity}\times \left(1-\mathrm{prevalence}\right)+\left(1-\mathrm{sensitivity}\right)\times \mathrm{prevalence} $$

Results

The studied population was made of 319 consecutive patients including 54.9% (175/319) of men and 45.1% (144/319) of women. The mean age of patients was 62.3 years old (± 17.7).

The concordances and discordances between radiologists to classify the patients using the 5 confidence level scale for diagnosing COVID-19 pneumonia are reported in Table 1 with a kappa value of 0.61 (p < 0.01). The concordances between radiologists to consider CT patterns as highly probable of COVID-19 pneumonia were observed in 127 of 319 (39.8%) patients with a kappa value of 0.83 (p < 0.001). The concordances between radiologists to consider CT pattern as highly probable or probable of COVID-19 pneumonia were observed in 159 (49.8%) patients with a kappa value of 0.82 (p < 0.001).

Table 1 Concordances between radiologists’ ranking (1–5 scale) in 319 patients (%). Kappa value: 0.61 (p < .001). R1, radiologist 1; R2, radiologist 2
Full size table

The diagnostic accuracy was measured in 307 patients having available RT-PCR test results that were positive in 174 patients and negative in 133, leading to a disease prevalence in the population of 56.7% (174/ 307). The radiologists’ rankings and RT-PCR results are reported in Table 2. The ROC curves for both radiologists were plotted (Fig. 1). The measured AUCs were 0.89 (IC 95%: 0.86–0.93) for the first radiologist and 0.87 (IC 95%: [0.83–0.91]) for the second (Fig. 1).

Table 2 Radiologist’ rankings (1–5 scale) for the diagnosis of COVID-19 pneumonia and reverse transcription polymerase chain (RT-PCR) results in 307 patients
Full size table
Fig. 1
figure 1

Receiver operating characteristics (ROC) curves for the diagnosis of COVID-19 pneumonia by radiologist 1 (a) and radiologist 2 (b) computed against the results of reverse transcription polymerase chain reaction (disease prevalence: 56.4%)

Full size image

Clinical data and outcomes of 16/307 (5.2%) patients classified as highly probable or probable by both radiologists despite a negative RT-PCR result were reviewed by two physicians, a biologist and a radiologist, not involved in the reading sessions. This retrospective analysis (supplemental material) allowed to reclassify 14 of these patients as COVID-19 positive (one patient had a second PCR testing positive during hospitalization and one patient was discharged at home without sufficient data on the clinical evolution). This led to increase the disease prevalence to 61.2% (188/307) and the AUC values for radiologists were 0.94 (IC95%: [0.91–0.97]) and 0.92 (IC95%: [0.89–0.95]) respectively (Fig. 2).

Fig. 2
figure 2

Receiver operating characteristics (ROC) curves for the diagnosis of COVID-19 pneumonia by radiologist 1 (a) and radiologist 2 (b) computed against results of reverse transcription polymerase chain reaction after integration of 16 cases of probable diagnosis of COVID-19 pneumonia by both radiologists among COVID-19-positive patients (disease prevalence: 61.6%)

Full size image

A discrepancy between a positive PCR test and a CT scan classified as less probable was found in 9 to 16 patients depending on the reader (2.9 to 5.2%).

Sensitivity, specificity, PPV, and NPV for both radiologists are reported in Table 3. With a 61.2% prevalence of COVID-19 infection in the studied population, the PPV and NPV of chest CT for diagnosing COVID-19 pneumonia were 95.9% and 84.4% respectively for the first radiologist and 94.3% and 77.1% for the second radiologist.

Table 3 Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of chest CT for diagnosing COVID-19 pneumonia compared to reverse transcription polymerase chain (RT-PCR) and patient outcomes
Full size table

Figures 3 and 4 illustrate concordant cases assessed by radiologists as highly probable of COVID-19 pneumonia with positive (Fig. 3) or negative (Fig. 4) RT-PCR results. Figure 5 illustrates a discordant case between radiologists.

Fig. 3
figure 3

CT scan of a 51-year-old male patient with positive RT-PCR test for COVID-19 and classified as a highly probable diagnosis by both radiologists. Bilateral and peripheral areas of ground glass opacities in the upper and right middle lobes (a, b) and band-like opacities and consolidations in lung bases (c, d)

Full size image
Fig. 4
figure 4

CT scan of a 46-year-old male patient with a negative RT-PCR test but classified as a highly probable diagnosis of COVID-19 pneumonia by both radiologists. Bilateral and peripheral areas of ground glass opacities within the upper lobes (a, b) and in the right lower lobe (c, d)

Full size image
Fig. 5
figure 5

CT scan of a 34-year-old female patient with a positive RT-PCR test classified as probable diagnosis of COVID pneumonia by one radiologist and as alternative diagnosis (bronchopneumonia) by the other. Note the presence of two rounded areas of ground glass opacities (yellow arrows) within the right upper lobe (a) and multiple small nodular opacities of ground glass attenuation disseminated in the right lower lobe (a, b, c) and to a lesser extent the left upper lobe (a). Bronchial wall thickening, tree-in-bud sign, and subpleural focal area of consolidation are also visible in the left lower lobe (c)

Full size image

Discussion

Our study showed a very good inter-observer agreement and diagnostic accuracy of chest CT for the diagnosis of COVID-19 pneumonia in patients with intermediate clinical probability during an acute COVID-19 outbreak. This confirms the results by Prokop et al who obtained substantial inter-observer agreement and very good performance for predicting COVID-19 in patients with moderate to severe symptoms using a categorical assessment scheme (CO-RADS) [20].

One limitation of our study could have been to use strictly the result of RT-PCR as ground truth with the risk of underestimation of CT diagnostic accuracy. The retrospective analysis of outcomes observed in our patients classified by both radiologists with a highly probable or probable diagnosis at CT despite negative RT-PCR result permitted to reclassify them as COVID-19 positive. Actually, RT-PCR has demonstrated a limited sensitivity for diagnosing COVID-19 infection [21,22,23]. In a series by Fang et al including 51 patients with chest CT and RT-PCR assay performed within 3 days, the sensitivity of CT for COVID-19 infection was 98% compared with RT-PCR sensitivity of 71% [23]. In a larger series by Ai et al, 59% of 1014 patients had positive RT-PCR results, and 88% had a positive (highly probable or probable) chest CT scan, and 30% of patients had negative RT-PCR and positive CT [22]. In our series, the percentage of patients presenting with a highly suggestive or suggestive CT scan according to both radiologists despite a negative RT-PCR was lower than that previously reported. This difference might be related to the use of differences in the quality of reagent used for the RT-PCR technique, given the great majority of our patients were tested using a reagent able to assess three viral targets. In addition, according to Zou et al [24], viral load in our study was detected in nasopharyngeal swabs rather than in throat swabs like in the studies by Xie et al [21] and Ai et al [22].

On the other hand, a negative result of RT-PCR could be associated with a non-SARS-CoV-2 viral pneumonia. In a study by Bai et al, radiologists had high specificity and moderate sensitivity in differentiating COVID-19 from other viral cases of pneumonia on chest CT [25]. In their study, the most discriminating features for COVID-19 pneumonia included a peripheral distribution, ground glass opacity, and vascular thickening, findings that fit very well with the criteria we used to classify our patients as a highly probable diagnosis of COVID-19 pneumonia.

Some of our patients had a positive PCR test with CT scans that were classified as less probable by the readers. This finding can be explained by the presence of SARS-cov2 in the nasopharynx but not in the lungs, and/or by the presence of preexisting or unrelated abnormalities in the lung parenchyma. These patients should be followed and referred again for a chest CT scan in case of a worsening of respiratory symptoms.

The main limitation of our study was to be retrospective on data collected during the ascending phase of the outbreak. The results depend totally on the prevalence of the COVID-19 in our population and consequently are not generalizable. Actually, the results of our study would have been very different if performed during the descending phase when the prevalence of disease dropped down. In a recent report, Eng and Bluemke emphasized in the context of the COVID-19 pandemic the need to move beyond sensitivity and specificity to predictive values according to the differences in disease prevalence between studied populations [19]. The Fleischner Society COVID-19 consensus statement acknowledged the differences in patient populations and provided guidelines stratified according to disease risk and severity [4].

Another limitation of our study is that the triage recommendations and the definition of intermediate probability were informal in the absence of international multidisciplinary consensus at the time it started.

Despite recent recommendations, there is no consensus yet upon the integrated use of chest radiography or CT in case of the massive arrival of patients suspected of COVID-19 pneumonia [26]. Our strategy was based on the principle that chest radiography remains helpful at the first-line evaluation of patients with a high clinical probability of overt COVID-19 pneumonia. Because it allows detecting subtle diffuse ground glass opacities that are variably detected by radiography, CT scan is helpful for the management of patients with equivocal clinical symptoms and chest radiographs. Interestingly, our results allow us to emphasize that a normal chest X-ray in a patient presenting with fever and respiratory symptoms should not lower the clinical probability of COVID-19 pneumonia. According to Wong et al, abnormalities on chest radiography may develop in the following days [27]. Hence, in a clinical context of the COVID-19 outbreak, patients who manifest respiratory symptoms and fever without visible radiographic abnormalities should be referred for a CT scan.

To our knowledge, this is the first study to evaluate agreements between radiologists and diagnostic accuracy of chest CT in patients selected on the basis of an intermediate clinical probability during an acute phase of the outbreak.

In conclusion, during an acute COVID-19 outbreak, CT scan may be used in the context of first-line dedicated triage of patients presenting an intermediate clinical probability with very good inter-observer concordances and diagnostic accuracy.