Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

COVID-19 pneumonia on chest X-rays: Performance of a deep learning-based computer-aided detection system

  • Eui Jin Hwang,

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea

  • Ki Beom Kim,

    Roles Data curation, Resources, Writing – review & editing

    Affiliation Department of Radiology, Daegu Fatima Hospital, Daegu, Korea

  • Jin Young Kim,

    Roles Data curation, Resources, Writing – review & editing

    Affiliation Department of Radiology, Keimyung University School of Medicine, Daegu, Korea

  • Jae-Kwang Lim,

    Roles Data curation, Resources, Writing – review & editing

    Affiliation Department of Radiology, School of Medicine, Kyungpook National University, Daegu, Korea

  • Ju Gang Nam,

    Roles Validation, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea

  • Hyewon Choi,

    Roles Validation, Writing – review & editing

    Current address: Department of Radiology, Chung-ang University Hospital, Seoul, Korea

    Affiliation Department of Radiology, Seoul National University College of Medicine, Seoul, Korea

  • Hyungjin Kim,

    Roles Validation, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea

  • Soon Ho Yoon,

    Roles Validation, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea

  • Jin Mo Goo,

    Roles Validation, Writing – review & editing

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea, Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Korea

  • Chang Min Park

    Roles Conceptualization, Formal analysis, Funding acquisition, Methodology, Resources, Supervision, Writing – original draft

    cmpark.morphius@gmail.com

    Affiliations Department of Radiology, Seoul National University College of Medicine, Seoul, Korea, Department of Radiology, Seoul National University Hospital, Seoul, Korea, Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Korea

Abstract

Chest X-rays (CXRs) can help triage for Coronavirus disease (COVID-19) patients in resource-constrained environments, and a computer-aided detection system (CAD) that can identify pneumonia on CXR may help the triage of patients in those environment where expert radiologists are not available. However, the performance of existing CAD for identifying COVID-19 and associated pneumonia on CXRs has been scarcely investigated. In this study, CXRs of patients with and without COVID-19 confirmed by reverse transcriptase polymerase chain reaction (RT-PCR) were retrospectively collected from four and one institution, respectively, and a commercialized, regulatory-approved CAD that can identify various abnormalities including pneumonia was used to analyze each CXR. Performance of the CAD was evaluated using area under the receiver operating characteristic curves (AUCs), with reference standards of the RT-PCR results and the presence of findings of pneumonia on chest CTs obtained within 24 hours from the CXR. For comparison, 5 thoracic radiologists and 5 non-radiologist physicians independently interpreted the CXRs. Afterward, they re-interpreted the CXRs with corresponding CAD results. The performance of CAD (AUCs, 0.714 and 0.790 against RT-PCR and chest CT, respectively hereinafter) were similar with those of thoracic radiologists (AUCs, 0.701 and 0.784), and higher than those of non-radiologist physicians (AUCs, 0.584 and 0.650). Non-radiologist physicians showed significantly improved performance when assisted with the CAD (AUCs, 0.584 to 0.664 and 0.650 to 0.738). In addition, inter-reader agreement among physicians was also improved in the CAD-assisted interpretation (Fleiss’ kappa coefficient, 0.209 to 0.322). In conclusion, radiologist-level performance of the CAD in identifying COVID-19 and associated pneumonia on CXR and enhanced performance of non-radiologist physicians with the CAD assistance suggest that the CAD can support physicians in interpreting CXRs and helping image-based triage of COVID-19 patients in resource-constrained environment.

Introduction

The Coronavirus disease (COVID-19) pandemic has posed a major threat to health care systems worldwide. As of April, 2021, COVID-19 has affected more than 126 million people worldwide, causing more than 2.8 million deaths [1]. Although COVID-19 patient may show various symptoms including fever, cough, and dyspnea [2], more than half of infected patients can be asymptomatic [3], which may lead to delayed diagnosis and important cause of disease transmission [4]. Therefore, early identification of infected patients and adequate isolation is extremely important to interrupt human-to-human transmission and prevent morbidity and mortality caused by COVID-19 [5].

The primary diagnostic test for COVID-19 is a reverse transcriptase-polymerase chain reaction (RT-PCR) on nasopharyngeal swabs for virus detection [6, 7]. Radiologic examinations including chest X-ray (CXR) and chest CT may show findings of pneumonia, including consolidations and ground-glass opacities [810], however, since considerable proportion of patients may show normal radiologic finding (up to 40% in CXR) especially in early phase of the disease, CXRs and chest CTs are currently not recommended for screening or diagnosis of COVID-19 [1114]. However, the massive COVID-19 outbreak has resulted in a shortage of medical resources [1517], limiting timely testing of RT-PCR. Recent international consensus statements suggested that radiologic examinations can be used as a triage tool in resource-constrained environments [11, 12].

CXR has been the primary radiologic examination for pneumonia. Especially for highly contagious diseases such as COVID-19, CXR has an advantage over chest CT in preventing transmission, since transportation of patients can be minimized with portable radiography units, and disinfection of the scanners and environment is relatively easy [11, 13]. However, CXRs have low sensitivity and high inter-reader variability for COVID-19 pneumonia compared to chest CTs [8, 1820]. In a previous study, CXRs showed a median sensitivity of 25% for opacities identified on chest CTs, and the sensitivity ranged from 20% to 50% across radiologists [19].

Recent studies have reported that artificial intelligence systems using deep learning techniques can detect various diseases on CXRs, showing comparable performance to expert radiologists [2124]. These computer-aided detection systems (CADs) may help imaging-based triage of patients in resource-constrained environments where expert radiologists are not available, and improve accuracy and inter-reader variability of physicians’ CXR interpretations [25, 26]. Several recent studies have demonstrated that deep learning-based CAD can identify COVID-19 on CXRs with radiologist-level performance [27, 28] and can be implemented for clinical practice [29]. However, the performance of a commercialized, clinically-available CAD in identifying COVID-19 and associated pneumonia has been rarely investigated and whether the CAD can enhance the performance of physicians and their inter-reader agreement

Therefore, in this study, we evaluated the performance of a commercialized, clinically-available CAD in identifying COVID-19 and associated pneumonia. We compared the performance of the CAD with physicians’ interpretation, and investigated whether the CAD can enhance the performance of physicians’ interpretation and their inter-reader agreement.

Materials and methods

The present study was approved by the institutional review boards of all participating institutions, with a waiver of patients’ informed consent.

Patients

Patients with COVID-19 were retrospectively included from four institutions (Daegu Fatima Hospital, Daegu, Korea; Keimyung University Dongsan Medical Center, Daegu, Korea; Kyungpook National University Hospital, Daegu, Korea; and Seoul National University Hospital, Seoul, Korea) with following inclusion criteria: a) COVID-19 patients confirmed by RT-PCR between January 20th and March 20th, 2020; and b) patients underwent CXR and chest CT within 24 hours (Fig 1).

thumbnail
Fig 1. Flow diagram for patients’ inclusion.

A total of 80 patients with COVID-19 were retrospectively included from four institutions, while 92 patients without COVID-19 were included from single institution. Among patients with COVID-19, 67 exhibited findings of pneumonia on chest CT obtained within 24 hours from CXR, while the other 13 patients did not exhibited any findings of pneumonia on chest CT.

https://doi.org/10.1371/journal.pone.0252440.g001

Patients without COVID-19 were also retrospectively included from a single institution (Seoul National University Hospital) with following inclusion criteria: a) patients with negative RT-PCR result for COVID-19; b) patients underwent CXR and chest CT within 24 hours; c) patients without any abnormality suggesting pneumonia on chest CT (Fig 1).

Deep learning-based CAD

We used a commercialized deep learning-based CAD (Lunit INSIGHT CXR 2, Lunit Inc., Seoul, Korea) for evaluating the CXRs. The CAD was designed to detect pulmonary nodules or masses, pulmonary infiltrates, and pneumothoraxes on CXRs. The CAD was initially trained using 54,221 normal CXRs and 35,613 abnormal CXRs (including 6,903 CXRs with pneumonia) [21], and was not specifically trained with CXRs from COVID-19 patients.

The CAD provided a probability score between 0 and 100% for the presence of abnormality on each CXR, with a heat map overlaid on the CXR for the localization of abnormality when the probability score was 15% or greater (Figs 2 and 3).

thumbnail
Fig 2. Representative case with COVID-19 pneumonia.

A CXR (A) of a patient with confirmed COVID-19 shows patchy infiltrates in both lower lung fields (arrows). The corresponding chest CT image (B) obtained in the same day with the CXR shows multifocal patchy ground-glass opacities in both lower lobes of the lung. The CT severity score of the patient was 13. The CAD system correctly detected pulmonary infiltrates with a probability score of 56% (C). In the reader-alone interpretation, four thoracic radiologists correctly identified the abnormality while none of the non-radiologist physicians identified the abnormality. In the CAD-assisted interpretation, all five thoracic radiologists and four non-radiologist physicians identified the abnormality.

https://doi.org/10.1371/journal.pone.0252440.g002

thumbnail
Fig 3. Representative case without COVID-19 pneumonia.

A CXR (A) and corresponding chest CT (B) of a patient with fever and dyspnea but negative RT-PCR result for COVID-19 show no pulmonary abnormality suggestive of pneumonia. The CAD system did not detect any abnormalities in the CXR and the probability score was 13% (C). In the reader-alone interpretation, four thoracic radiologists and four non-radiologist physicians misclassified the CXR as having findings of pneumonia. In the CAD-assisted interpretation, only one thoracic radiologist and two non-radiologist physicians made false-positive classification of the CXR. Mediastinal window CT image (D) show pulmonary embolism in the right descending pulmonary artery (arrow), presumed cause of patients’ symptom.

https://doi.org/10.1371/journal.pone.0252440.g003

Definition of reference standards

For evaluation of CXR interpretation, we defined two different classes of reference standard: a) the diagnosis of COVID-19 by RT-PCR; and b) the presence of pneumonia on chest CT.

To define the reference standard for the presence of pneumonia, two thoracic radiologists (E.J.H and C.M.P, 9 and 21 years of experience in CXR and chest CT interpretation respectively) who were blinded to the CAD results reviewed the chest CT images, which were obtained within 24 hours from the CXRs. The two radiologists determined whether there was any finding of pneumonia on CT, in consensus.

Features of pneumonia on chest CT (the presence of consolidations, ground-glass opacities, and pleural effusion, and the bilaterality of pneumonia) were evaluated by one thoracic radiologist (E.J.H). For evaluation of the extent of pneumonia, a previously reported scoring system [30] based on the segmental involvement of the infiltrates (CT severity score, range 0–40) was adopted. For evaluation of CT severity score, bilateral lungs were divided into 20 regions based on lobar and segmental anatomy. Opacities of all 20 lung regions were visually evaluated by the thoracic radiologist on CT, and score 0, 1, and 2 were assigned if parenchymal opacities involved 0%, <50%, and ≥50% of each region, respectively. Finally, the CT severity score was defined as the sum of the individual scores of the 20 lung regions [30].

Reader test

A total of 10 readers (5 thoracic radiologists [5–29 years of experience in CXR interpretation] and 5 non-radiologist physicians) participated in a reader test. All physicians independently interpreted the CXRs, to address whether there was any abnormality suggestive of pneumonia. Physicians were informed to provide a five-point scale score for each CXR regarding the presence and absence of the abnormality suggestive of pneumonia: a) score 1, definitely absent; b) score 2, probably absent; c) score 3, equivocal; d) score 4, probably present; and e) score 5, definitely present. All readers were informed that CXRs were obtained from patients suspected for COVID-19, however, the readers were blinded to other clinical information.

At first, each physician interpreted the CXRs at their discretion (reader-alone interpretation). Subsequently, physicians were provided with the CAD results and asked to modify their original decision as needed (CAD-assisted interpretation).

Subgroup analyses

To evaluate performances of the CAD and readers for detection of pneumonia in patients with different clinical and radiologic findings, we compared sensitivities of the CAD and readers for identification of findings of pneumonia, in following subgroups: a) patients with symptom duration >5 days versus ≤5 days; b) patients with versus without consolidation on their CTs; and c) patients with CT severity score >10 versus ≤10.

Statistical analyses

We evaluated the performances of positive interpretation results by the CAD and physicians for the prediction of positive reference standards: The diagnosis of COVID-19 by RT-PCR, and the presence of pneumonia on CT. Area under the receiver operating characteristic curves (AUCs), sensitivities, and specificities were used for performance evaluations. For evaluating sensitivity and specificity, the CAD results with a probability score ≥15% were considered as positive results, while physicians’ scores ≥3 were considered as positive interpretations. Average AUCs of multiple readers were obtained and compared using multiple reader multiple cases receiver operating characteristic analyses, as suggested by Obuchowski and Rockette [31]. Average sensitivities and specificities of multiple readers were estimated using generalized estimating equations. Inter-reader agreements among physicians in five-point scale scores and in binary classifications were evaluated with Fleiss’ kappa coefficient.

All statistical analyses were done with R (version 3.6.3, R project for statistical computing, Vienna, Austria). A P-value <0.05 was considered to indicate a statistically significant difference.

Results

Demographic and radiologic information

A total of 172 CXRs from 172 patients (87 men; median age, 66 years [interquartile range (IQR), 57–75 years]) were included in the study. Among them, 80 (46.5%) were COVID-19 patients confirmed by RT-PCR. 94.7% (163/172) of patients had symptoms suggestive of acute respiratory illness, and the median time interval between symptom onset and CXR acquisition was 3 days (IQR, 1–6 days) (Table 1).

thumbnail
Table 1. Clinical and radiological information of patients.

https://doi.org/10.1371/journal.pone.0252440.t001

Based on chest CTs, 83.8% (67/80) of COVID-19 patients had findings of pneumonia, while the other 16.2% (13/80) of COVID-19 patients and all the 92 non-COVID-19 patients did not exhibit findings of pneumonia. Among 67 patients with pneumonia on CTs, ground-glass opacities, consolidations, and pleural effusion were observed in 100% (67/67), 47.8% (32/67), and 6.0% (4/67) of patients. 85.1% (57/67) of patients showed bilateral involvement. The median CT severity score of patients with pneumonia was 13 (IQR, 7–21).

CAD versus reader-alone interpretations

For identification of RT-PCR-positive COVID-19 patients, the CAD exhibited an AUC, sensitivity, and specificity of 0.714 (95% confidence interval [CI], 0.641–0.781), 71.3% (95% CI, 61.3–81.2%), and 52.2% (95% CI, 42.0–62.4%), respectively. The AUC of the CAD did not significantly differ from that of average thoracic radiologist (0.701 [95% CI, 0.619–0.783]; P = .712), while significantly higher than that of average non-radiologist physician (0.584 [95% CI, 0.469–0.699]; P = .031). The sensitivity of the CAD was not significantly different form that of average thoracic radiologist (64.5% [95% CI, 54.7–73.2%]; P = .228), while significantly higher than that of average non-radiologist physician (54.3% [95% CI, 44.9–63.4%]; P = .003). The specificity of the CAD was significantly lower than that of average thoracic radiologist (64.3% [95% CI, 59.9–68.6]; P = .032), and did not significantly differ from that of average non-radiologist physician (58.9% [95% CI, 54.4–63.3%]; P = .236). Regarding individual readers, the CAD exhibited significantly higher AUC than four non-radiologist physicians, significantly higher sensitivity than one thoracic radiologist and four non-radiologist physicians. However, the specificity of the CAD was significantly lower than those of three thoracic radiologists and two non-radiologist physicians (Table 2, Fig 4).

thumbnail
Fig 4. Performance of the CAD versus reader-alone interpretations.

For identification of RT-PCR-positive COVID-19 patients (A), the CAD exhibited AUC of 0.714 (black line), which did not significantly differ from that of thoracic radiologists (0.701, blue line) but significantly higher than that of non-radiologist physicians (0.584, red line). For identification of pneumonia defined on chest CT (B), the CAD exhibited AUC of 0.790 (black line), which was not significantly different from that of thoracic radiologists (0.784, blue line), but significantly higher than that of non-radiologist physicians (0.650, red line).

https://doi.org/10.1371/journal.pone.0252440.g004

thumbnail
Table 2. The CAD and physician’s performances with reference standard of diagnosis of COVID-19 by RT-PCR.

https://doi.org/10.1371/journal.pone.0252440.t002

For identification of pneumonia with the reference standard of CT, the CAD exhibited an AUC, sensitivity, and specificity of 0.790 (95% CI, 0.722–0.849), 80.6% (95% CI, 71.1–90.0%), and 55.2% (95% CI, 45.7–64.7%), respectively. The AUC of the CAD did not significantly differ from that of average thoracic radiologist (0.784 [95% CI, 0.713–0.856]; P = .839), while significantly higher than that of average non-radiologist physician (0.650 [95% CI, 0.544–0.756]; P = .016). The sensitivity of the CAD was not significantly different from that of average thoracic radiologist (74.6% [95% CI, 69.7–79.0%]; P = .268), while significantly higher than that of average non-radiologist physician (61.8% [95% CI, 52.5–66.8%]; P = .001). The specificity of the CAD was significantly lower than that of average thoracic radiologist (67.2% [95% CI, 63.1–71.1]; P = .023), and did not significantly differ from that of average non-radiologist physician (62.1% [95% CI, 57.9–66.2%]; P = .195). Regarding individual readers, the CAD exhibited significantly higher AUC than four non-radiologist physicians, significantly higher sensitivity than one thoracic radiologist and five non-radiologist physicians. However, the CAD exhibited significantly lower specificity than three thoracic radiologists and two non-radiologist physicians (Table 3, Fig 4).

thumbnail
Table 3. The CAD and physician’s performances with reference standard of pneumonia on chest CT.

https://doi.org/10.1371/journal.pone.0252440.t003

Reader-alone interpretations versus CAD-assisted interpretations

In the CAD-assisted interpretation, AUC (0.699 [95% CI, 0.621–0.777]; P = .815), sensitivity (63.0% [95% CI, 53.2–71.9%]; P = .659), and specificity (69.3% [95% CI, 65.0–73.4%]; P = .107) of average thoracic radiologist did not significantly differ from those in the reader-alone interpretation, for identification of COVID-19 patients with the reference standard of RT-PCR result. Meanwhile, all the AUC (0.664 [95% CI, 0.580–0.748]; P = .006), sensitivity (61.8% [95% CI, 52.3–70.4%]; P = .031), and specificity (65.2% [95% CI, 60.8–69.4%]; P = .048) of average non-radiologist physician were significantly improved in the CAD-assisted interpretation, with reference standard of RT-PCR result. Regarding individual readers, significant improvement of readers’ AUCs, sensitivities, and specificities were observed in three (three non-radiologist physicians), two (two non-radiologist physicians), and five (three thoracic radiologists and two non-radiologist physicians) readers, respectively in the CAD-assisted interpretation (Table 2, Fig 5)

thumbnail
Fig 5. Performance of physician alone versus CAD-assisted interpretations.

For identification of RT-PCR positive COVID-19 patients, the AUCs of thoracic radiologists did not significantly differ between reader-alone (red line) and CAD-assisted interpretations (blue line) (0.701 vs. 0.699; P = .815) (A), while the AUC non-radiologist physicians was significantly improved in the CAD-assisted interpretation (blue line) compared to the reader-alone interpretation (red line) (0.584 vs. 0.664; P = .006) (B). For identification of pneumonia defined on chest CT, the AUCs of thoracic radiologists also did not significantly differ between reader-alone (red line) and CAD-assisted interpretations (blue line) (0.784 vs. 0.789; P = .524) (C), while the AUC non-radiologist physicians was significantly improved in the CAD-assisted interpretation (blue line) compared to the reader-alone interpretation (red line) (0.650 vs. 0.738; P = .003) (D).

https://doi.org/10.1371/journal.pone.0252440.g005

As for the identification of pneumonia with the reference standard of CT, AUC (0.789 [95% CI, 0.721–0.858]; P = .524), sensitivity (74.0% [95% CI, 69.1–78.4%]; P = .860), and specificity (72.4% [95% CI, 68.4–76.0%]; P = .069) of average thoracic radiologist in the CAD-assisted interpretation did not significantly differ from those in the reader-alone interpretation. Meanwhile, AUC (AUC (0.738 [95% CI, 0.660–0.815]; P = .003), sensitivity (71.0% [95% CI, 57.0–75.7%]; P = .011), and specificity (67.8% [95% CI, 63.7–71.7%]; P = .048) of average non-radiologist physicians exhibited significant improvement in the CAD-assisted interpretation. Regarding individual readers, significant improvement of readers’ AUCs, sensitivities, and specificities were observed in four (one thoracic radiologist and three non-radiologist physicians), three (three non-radiologist physicians), and five (three thoracic radiologists and two non-radiologist physicians) readers, respectively in the CAD-assisted interpretation (Table 3, Fig 5).

As for inter-reader agreement, the CAD-assisted interpretation exhibited better agreement (Fleiss’ kappa coefficient, 0.322 [95% CI, 0.320–0.344] for the five-point-scale scores, and 0.688 [95% CI, 0.665–0.710] for the binary classification) compared to the reader-alone interpretations (Fleiss’ kappa coefficient, 0.209 [95% CI, 0.197–0.220] for the five-point-scale scores, and 0.510 [95% CI, 0.488–0.533] for the binary classification), without overlapping 95% CIs.

Sensitivities in different subgroups

In the reader-alone interpretations, the readers exhibited significantly higher sensitivities for patients with a symptom duration >5 days (77.7%), for patients with consolidations on their CTs (73.4%), and for patients with CT severity scores >10 (84.5%), compared to those whose symptom duration was ≤5 days (59.4%; P < .001), those who did not have consolidations on their CTs (63.4%; P = .005), and those with CT severity scores ≤10 (46.9%; P < .001), respectively (Table 4).

thumbnail
Table 4. Sensitivities varied by clinical and radiological findings.

https://doi.org/10.1371/journal.pone.0252440.t004

The CAD exhibited similar trends with physicians, and exhibited a higher sensitivity for patients with a symptom duration >5 days (90.3%), for patients with consolidations on their CTs (90.6%), and for patients with CT severity scores >10 (89.5%), compared to those whose symptom duration was ≤5 days (71.4%; P = .068), those who did not have consolidations on their CTs (71.4%; P = .065), and those with CT severity scores ≤10 (69.0%; P = .059), although the differences did not reach statistical significance.

Discussion

Herein, we evaluated the performance of a commercialized deep learning-based CAD for identification of CXRs from RT-PCR positive COVID-19 patients and those with associated pneumonia proven by chest CT and compare it with those of thoracic radiologists and non-radiologist physicians. The performance of CAD (AUC, 0.714 and 0.790 for RT-PCR positive COVID-19 and associated pneumonia, respectively) was similar with those of thoracic radiologists (AUC, 0.712 and 0.784 for RT-PCR positive COVID-19 and associated pneumonia, respectively), and higher than those of non-radiologist physicians (AUC, 0.584 and 0.650 for RT-PCR positive COVID-19 and associated pneumonia, respectively).

Since CXRs and chest CTs of COVID-19 patients may appear normal, especially in the early stages of the disease [9, 32, 33], diagnosing COVID-19 using CXRs or CTs may be inappropriate [1114]. In our study population, 16.3% of COVID-19 patients did not exhibit any findings of pneumonia on CTs. Not surprisingly, the performances of the CAD and readers against RT-PCR results were unsatisfactory. Sensitivities of the CAD (71.3%) and thoracic radiologists (56.3–68.8%) were comparable to previously reported sensitivity (69%) of baseline CXRs by Wong et al [18].

In spite of limited diagnostic performance compared to RT-PCR testing, identification of radiologic findings of pneumonia is still clinically important for the following reasons: first, in situations with limited medical resources due to the outbreak, a timely diagnosis with an RT-PCR test can be limited. Since the results of radiologic examinations can be obtained faster than RT-PCR results, it can aid timely clinical decision-making in resource-constrained environments. Second, radiological findings of pneumonia may precede positive RT-PCR results [18, 3436]. In situations where there is a high pre-test probability, identification of pneumonia via radiologic examinations can help early diagnosis and allow for isolation in order to prevent further transmission. Third, since the extent of radiological findings of pneumonia can mirror the clinical severity of COVID-19 [30, 37, 38], radiological findings of pneumonia may aid hospitalization and intensive care decision-making, and may be utilized for monitoring the severity of the disease. The radiologist-level performance of the CAD for identification of findings of pneumonia suggests that it may facilitate the triage of CXRs with COVID-pneumonia, especially when interpretations from expert radiologists are limited or unavailable.

The CAD evaluated in the present study was not specifically trained for findings of COVID-19. Instead, it was trained for various types of abnormalities including pulmonary nodules and infiltrates. The reasonably high performance of the CAD indicates that an existing versatile CAD can be utilized for the detection of COVID-19 pneumonia. Indeed, radiographic findings of COVID-19 pneumonia include bilateral ground-glass opacities and consolidation [810, 39], which has substantial overlap with pneumonia from other etiologies. Although it is difficult to directly compare the performance across different studies because of difference in test datasets, recent studies where deep learning-based CADs that were specifically trained for COVID-19 pneumonia reported higher AUCs compared to our results (AUCs, 0.81–0.99, Table 5) for the identification of CXRs from COVID-19 patients [27, 28, 40, 41]. Additional training of the CAD with COVID-19 CXRs may improve the performance.

thumbnail
Table 5. Comparison of performance of CAD for identifying COVID-19 with results from different studies.

https://doi.org/10.1371/journal.pone.0252440.t005

In addition to stand-alone performance, we also observed that the CAD may enhance the performances of readers. Although thoracic radiologists did not exhibit significant improvement in performance in the CAD-assisted interpretation, less-experienced non-radiologist physicians exhibited significantly improved AUCs, sensitivities, and specificities in the CAD-assisted interpretations. Our results suggest that the CAD may help less-experienced readers to identify subtle findings of pneumonia on CXRs. In addition, the substantial improvement of inter-reader agreement using the CAD suggests that CAD can also help reduce variability among physicians in terms of CXR interpretations.

Regarding sensitivities in different subgroups, the CAD exhibited similar trends to physicians. Higher sensitivity in patients with consolidations and higher CT severity scores may be related to better visibility of pulmonary infiltrates on CXRs. A previous study reported that the entire burden of pneumonia reflecting both the extent and density of pneumonia on CTs was a significant factor for the visibility of pneumonia on CXRs [19]. Higher sensitivities in patients with longer time intervals since their symptom onset (>5days) may also be associated with the extent of pneumonia. Previous studies reported that the extent of pneumonia on CXR tended to increase until 2 weeks after symptom onset [30, 39]. Considering that imaging-based triage is indicated in patients with moderate to severe clinical symptoms [11], and disease extents in CTs reflect the clinical severity of diseases [30], we believe the CAD can help triage COVID-19 patients with CXRs.

Limitations exist in the present study. Despite being a multi-center study, our study included a limited number of patients. In addition, we included patients with available chest CT, as a reference standard for findings of pneumonia. Therefore, patients in our study did not fully reflect actual clinical situation, and the generalizability of our result may be limited. Additionally, the reader tests performed in our study also did not fully reflect actual clinical practice, where radiologists and clinicians may interpret CXRs in different environments from the reader test and can use the additional clinical and laboratory information. Therefore, the performances of the physicians in our study may not be fully reproducible in actual practice.

Conclusion

A commercialized, clinically-available deep learning-based CAD exhibited similar performances CAD (AUC, 0.714 and 0.790 for RT-PCR positive COVID-19 and associated pneumonia, respectively) with thoracic radiologists (AUC, 0.712 and 0.784 for RT-PCR positive COVID-19 and associated pneumonia, respectively) in identifying COVID-19 and associated pneumonia on CXRs, and outperformed non-radiologist physicians (AUC, 0.584 and 0.650 for RT-PCR positive COVID-19 and associated pneumonia, respectively). It also enhanced the detection performances of non-radiologist physicians (AUC, 0.584 to 0.664 and 0.650 to 0.738 for RT-PCR positive COVID-19 and associated pneumonia, respectively) and inter-reader agreement (Fleiss’ kappa coefficient, 0.510 to 0.688). We believe that the CAD system can help less-experienced physicians to identify findings of pneumonia associated with COVID-19 on CXRs and to reduce variability among physicians in CXR interpretations. The CAD may facilitate imaging-based triage of COVID-19 patients in resource-constrained environment.

References

  1. 1. World Health Organization. Weekly epidemiological update on COVID-19–30 March 2021 March 30, 2021 [cited 2021 April 12]. Available from: https://www.who.int/publications/m/item/weekly-epidemiological-update-on-covid-19—31-march-2021.
  2. 2. Guan WJ, Ni ZY, Hu Y, Liang WH, Ou CQ, He JX, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. N Engl J Med. 2020;382(18):1708–20. Epub 2020/02/29. pmid:32109013; PubMed Central PMCID: PMC7092819.
  3. 3. Sakurai A, Sasaki T, Kato S, Hayashi M, Tsuzuki SI, Ishihara T, et al. Natural History of Asymptomatic SARS-CoV-2 Infection. N Engl J Med. 2020;383(9):885–6. Epub 2020/06/13. pmid:32530584; PubMed Central PMCID: PMC7304419.
  4. 4. Rothe C, Schunk M, Sothmann P, Bretzel G, Froeschl G, Wallrauch C, et al. Transmission of 2019-nCoV Infection from an Asymptomatic Contact in Germany. N Engl J Med. 2020;382(10):970–1. Epub 2020/02/01. pmid:32003551; PubMed Central PMCID: PMC7120970.
  5. 5. Hellewell J, Abbott S, Gimma A, Bosse NI, Jarvis CI, Russell TW, et al. Feasibility of controlling COVID-19 outbreaks by isolation of cases and contacts. Lancet Glob Health. 2020;8(4):e488–e96. Epub 2020/03/03. pmid:32119825; PubMed Central PMCID: PMC7097845.
  6. 6. World Health Organization. Laboratory testing for coronavirus disease 2019 (COVID-19) in suspected human cases March 2, 2020 [cited 2020 May 15]. Available from: https://apps.who.int/iris/bitstream/handle/10665/331329/WHO-COVID-19-laboratory-2020.4-eng.pdf.
  7. 7. Unites States Centers for Disease Control and Prevention USCfDCa. Interim Guidelines for Collecting, Handling, and Testing Clinical Specimens from Persons for Coronavirus Disease 2019 (COVID-19) February 14, 2020 [cited 2020 March 18]. Available from: https://www.cdc.gov/coronavirus/2019-ncov/lab/guidelines-clinical-specimens.html.
  8. 8. Yoon SH, Lee KH, Kim JY, Lee YK, Ko H, Kim KH, et al. Chest Radiographic and CT Findings of the 2019 Novel Coronavirus Disease (COVID-19): Analysis of Nine Patients Treated in Korea. Korean J Radiol. 2020. Epub 2020/02/27. pmid:32100485.
  9. 9. Bernheim A, Mei X, Huang M, Yang Y, Fayad ZA, Zhang N, et al. Chest CT Findings in Coronavirus Disease-19 (COVID-19): Relationship to Duration of Infection. Radiology. 2020:200463. Epub 2020/02/23. pmid:32077789.
  10. 10. Chung M, Bernheim A, Mei X, Zhang N, Huang M, Zeng X, et al. CT Imaging Features of 2019 Novel Coronavirus (2019-nCoV). Radiology. 2020:200230. Epub 2020/02/06. pmid:32017661.
  11. 11. Rubin GD, Ryerson CJ, Haramati LB, Sverzellati N, Kanne JP, Raoof S, et al. The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society. Radiology. 2020:201365. Epub 2020/04/08. pmid:32255413.
  12. 12. Foust AM, Phillips GS, Chu WC, Daltro P, Das KM, Garcia-Peña P, et al. International Expert Consensus Statement on Chest Imaging in Pediatric COVID-19 Patient Management: Imaging Findings, Imaging Study Reporting and Imaging Study Recommendations. Radiology: Cardiothoracic Imaging. 2020;2(2):e200214. pmid:33778577
  13. 13. American College of Radiology. ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection March 11, 2020 [cited 2020 March 18]. Available from: https://www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection.
  14. 14. Society of Thoracic Radiology. STR/ASER COVID-19 Position Statement March 11, 2020 [cited 2020 Marh 20]. Available from: https://thoracicrad.org/wp-content/uploads/2020/03/STR-ASER-Position-Statement-1.pdf.
  15. 15. Emanuel EJ, Persad G, Upshur R, Thome B, Parker M, Glickman A, et al. Fair Allocation of Scarce Medical Resources in the Time of Covid-19. N Engl J Med. 2020;382(21):2049–55. Epub 2020/03/24. pmid:32202722.
  16. 16. American Society for Microbiology. Supply shortages impacting COVID-19 and non-COVID testing. 2020. Available from: https://asm.org/Articles/2020/September/Clinical-Microbiology-Supply-Shortage-Collecti-1.
  17. 17. Ranney ML, Griffeth V, Jha AK. Critical Supply Shortages—The Need for Ventilators and Personal Protective Equipment during the Covid-19 Pandemic. N Engl J Med. 2020;382(18):e41. Epub 2020/03/27. pmid:32212516.
  18. 18. Wong HYF, Lam HYS, Fong AH, Leung ST, Chin TW, Lo CSY, et al. Frequency and Distribution of Chest Radiographic Findings in COVID-19 Positive Patients. Radiology. 2019:201160. Epub 2020/03/29. pmid:32216717.
  19. 19. Choi H, Qi X, Yoon SH, Park SJ, Lee KH, Kim JY, et al. Extension of coronavirus disease 2019 (COVID-19) on chest CT and implications for chest radiograph interpretation. Radiology: Cardiothoracic Imaging. 2020;2(2):e200107. pmid:33778565
  20. 20. Orsi MA, Oliva AG, Cellina M. Radiology Department Preparedness for COVID-19: Facing an Unexpected Outbreak of the Disease. Radiology. 2020:201214. Epub 2020/04/02. pmid:32228364.
  21. 21. Hwang EJ, Park S, Jin KN, Kim JI, Choi SY, Lee JH, et al. Development and Validation of a Deep Learning-Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs. JAMA Netw Open. 2019;2(3):e191095. Epub 2019/03/23. pmid:30901052; PubMed Central PMCID: PMC6583308.
  22. 22. Park S, Lee SM, Lee KH, Jung KH, Bae W, Choe J, et al. Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings. Eur Radiol. 2020;30(3):1359–68. Epub 2019/11/22. pmid:31748854.
  23. 23. Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, et al. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. Radiology. 2019;290(1):218–28. Epub 2018/09/27. pmid:30251934.
  24. 24. Majkowska A, Mittal S, Steiner DF, Reicher JJ, McKinney SM, Duggan GE, et al. Chest Radiograph Interpretation with Deep Learning Models: Assessment with Radiologist-adjudicated Reference Standards and Population-adjusted Evaluation. Radiology. 2020;294(2):421–31. Epub 2019/12/04. pmid:31793848.
  25. 25. Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019;293(3):573–80. Epub 2019/10/23. pmid:31638490.
  26. 26. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G. Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology. 2019;291(1):196–202. Epub 2019/01/23. pmid:30667333; PubMed Central PMCID: PMC6438359.
  27. 27. Murphy K, Smits H, Knoops AJG, Korst M, Samson T, Scholten ET, et al. COVID-19 on the Chest Radiograph: A Multi-Reader Evaluation of an AI System. Radiology. 2020:201874. Epub 2020/05/10. pmid:32384019.
  28. 28. Wehbe RM, Sheng J, Dutta S, Chai S, Dravid A, Barutcu S, et al. DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large US Clinical Dataset. Radiology. 2020:203511. Epub 2020/11/25. pmid:33231531.
  29. 29. Hwang EJ, Kim H, Yoon SH, Goo JM, Park CM. Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19. Korean J Radiol. 2020;21(10):1150–60. Epub 2020/07/31. pmid:32729263; PubMed Central PMCID: PMC7458860.
  30. 30. Yang R, Li X, Liu H, Zhen Y, Zhang X, Xiong Q, et al. Chest ct severity score: An imaging tool for assessing severe covid-19. Radiology: Cardiothoracic Imaging. 2020;2(2):e200047. pmid:33778560
  31. 31. Obuchowski NA Jr, Rockette HE Jr. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests an anova approach with dependent observations. Communications in Statistics-simulation and Computation. 1995;24(2):285–308.
  32. 32. He JL, Luo L, Luo ZD, Lyu JX, Ng MY, Shen XP, et al. Diagnostic performance between CT and initial real-time RT-PCR for clinically suspected 2019 coronavirus disease (COVID-19) patients outside Wuhan, China. Respir Med. 2020;168:105980. Epub 2020/05/05. pmid:32364959; PubMed Central PMCID: PMC7172864.
  33. 33. Yang W, Yan F. Patients with RT-PCR-confirmed COVID-19 and Normal Chest CT. Radiology. 2020;295(2):E3. Epub 2020/03/07. pmid:32142398.
  34. 34. Xie X, Zhong Z, Zhao W, Zheng C, Wang F, Liu J. Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing. Radiology. 2020:200343. Epub 2020/02/13. pmid:32049601.
  35. 35. Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P, et al. Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR. Radiology. 2020:200432. Epub 2020/02/20. pmid:32073353.
  36. 36. Ai T, Yang Z, Hou H, Zhan C, Chen C, Lv W, et al. Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology. 2020:200642. Epub 2020/02/27. pmid:32101510.
  37. 37. Zhao W, Zhong Z, Xie X, Yu Q, Liu J. Relation Between Chest CT Findings and Clinical Conditions of Coronavirus Disease (COVID-19) Pneumonia: A Multicenter Study. AJR Am J Roentgenol. 2020:1–6. Epub 2020/03/04. pmid:32125873.
  38. 38. Toussie D, Voutsinas N, Finkelstein M, Cedillo MA, Manna S, Maron SZ, et al. Clinical and Chest Radiography Features Determine Patient Outcomes In Young and Middle Age Adults with COVID-19. Radiology. 2020:201754. Epub 2020/05/15. pmid:32407255.
  39. 39. Shi H, Han X, Jiang N, Cao Y, Alwalid O, Gu J, et al. Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study. Lancet Infect Dis. 2020. Epub 2020/02/28. pmid:32105637.
  40. 40. Castiglioni I, Ippolito D, Interlenghi M, Monti CB, Salvatore C, Schiaffino S, et al. Machine learning applied on chest x-ray can aid in the diagnosis of COVID-19: a first experience from Lombardy, Italy. Eur Radiol Exp. 2021;5(1):7. Epub 2021/02/03. pmid:33527198; PubMed Central PMCID: PMC7850902.
  41. 41. Minaee S, Kafieh R, Sonka M, Yazdani S, Jamalipour Soufi G. Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning. Med Image Anal. 2020;65:101794. Epub 2020/08/12. pmid:32781377; PubMed Central PMCID: PMC7372265.