Motivation of LUS utilization for COVID-19 screening

In the last few years, Lung Ultrasound (LUS) imaging has been proposed as an alternative to the use of CT or X-ray for screening and follow-up of lung diseases [19–23]. For instance, it has been suggested that lung visualization through ultrasound imaging effectively replaces physical auscultation with stethoscopes [35]. Moreover, when used correctly, LUS imaging could even help to reduce infections between patients and medical staff [35].

Recent medical correspondence has pointed out the advantages of using LUS imaging as a tool for early diagnosis and follow-up of COVID-19 patients [36–41]. Some works highlight the benefits of using LUS in the context of the COVID-19 pandemic, especially considering its portability, accessibility, no radiation, ease of disinfection (e.g., using disposable caps of the ultrasound probes), and low cost [37, 38].

A short review of LUS findings in COVID-19 patients is presented in [36]. Some of these findings are consistent with CT results, including multiple fused bilateral B lines, subpleural pulmonary consolidations, irregular pleural line, and poor blood flow. An important finding reported by Fiala [36] is that subpleural lesions in COVID-19 patients differ signficantly from pulmonary diseases, including bacterial pneumonia, tuberculosis, and cardiogenic pulmonary edema, among others. Based on these observations, the author suggests adopting lung ultrasound for early detection of pulmonary alterations as a triage tool, particularly in environments with limited resources [36]. Additionally, other authors [38–40] reported that LUS imaging findings of COVID-19 progression in diagnosed patients are related to the observed patterns on CT images. Particularly, ground-glass opacity, consolidation shadow, and thickened pleura observed in CT images had special manifestations in LUS images as B-lines, consolidations, irregular or fragmented pleural line, pleural effusion, and absence of lung sliding. Furthermore, Soldati et al. [41] suggested that LUS findings in superficial pulmonary tissue are correlated with histopathological findings revealed in CT scanning.

Based on the previous discussion, it can be asserted that LUS imaging is a promising option for the screening, diagnosis, and follow-up of pulmonary diseases. Thus, over the past decade, interest in developing computational tools for computer-assisted analysis of LUS imaging has increased [24–30]. A brief description of the results reported in these works will be provided in the next section. We will first review works related to the identification of general lung conditions by means of computer-assisted analysis of LUS imaging. We will then describe previous work on computer-assisted screening of COVID-19.

Related work on computer-assisted analysis of LUS

Earliest studies involving computer-assisted LUS imaging analysis were based on image processing techniques [42–44]. These works were aimed at segmentation (identification) of artifacts in LUS images. Most of the identified artifacts are the so-called B-lines, which are associated with the disease presence. Subsequent works applied the typical training-testing approach of classical Machine Learning algorithms (ML) and Deep Learning (DL) networks [24–31]. The study presented in [24–26] focuses on identifying pneumonia in children by analyzing brightness patterns in LUS frames. Authors [27–29] proposed pre-trained Convolutional Neural Networks (CNNs) to detect pulmonary conditions such as pleural effusion, consolidation, and pneumothorax utilizing images obtained from swine models. Their work is based on detecting the presence of lung sliding and quantification of A- and B-lines. In addition [31], proposes an automated quantification of B-lines in LUS frames using CNN. Meanwhile, a weakly-supervised strategy (i.e., needs only frame-level annotation) for localization of B-lines by means of Class Activation Maps was introduced in [30]. The previously mentioned studies are aimed at detecting artifacts in LUS frames to aid in the identification of different lung pathologies.

Taking into account the strong correlation between LUS and CT imaging findings in COVID-19 patients and the advantages of LUS equipment [38–40], it makes sense to develop tools for COVID-19 screening and diagnosis by means of computer-assisted analysis of LUS imaging. To the best of our knowledge, at the time of writing this paper, only three works have proposed the use of DL models for computer-assisted analysis of COVID-19 LUS imaging [32–34]. In [32], the authors addressed COVID-19 disease severity prediction considering the disease scoring scheme proposed in [45]. The deep learning architecture introduced in [34] produces a disease severity score from LUS imaging with the aim of assisting practitioners in the diagnosis of lung pathologies related to COVID-19. The work, first presented in [33] and extended in [34], aims to classify LUS frames obtained from healthy, bacterial pneumonia, and COVID-19 patients. These works introduce and use the Point-Of-Care Ultrasound (POCUS) dataset of LUS imaging. The POCUS dataset was collected by the authors of [33, 34] from different sources and was made publicly available in their GitHub repository [46]. The proposed model consists of a modified convolutional VGG16 network, called POCOVID-net, which provided an accuracy of 89% and balanced accuracy of 82% [33]. In the extended work [34], the POCUS database was enlarged, and four new models were evaluated: a NasNetMobile architecture; a VGG network combined with class activation maps (compute class-specific heatmaps); and two VGG-segment models using a version of segmentation ensemble introduced in [32]. In this work, the POCOVID-net model achieved the best 5-fold cross-validation results (accuracy and balanced accuracy of 87%) compared to other models.

Considering the results reported in [33, 34], there is still room for improvement in COVID-19 screening by means of computer-assisted analysis of LUS imaging based on artificial intelligence techniques.

Artifacts and related diseases in LUS

Lung ultrasound imagery is generated from the relative amounts of air and fluid in lungs according to the physical phenomenon of acoustic impedance: a measure of particle resistance to mechanical vibrations in a medium [21]. The resistance increases proportionally to the density of the medium and propagation velocity of ultrasound in that medium. At a basic level, an ultrasound probe transmits high-frequency sound signals and listens to the echo caused by the reflection of such signals when they reach a boundary between tissues, known as interface. Ultrasound waves propagate across different tissues (mediums) in the organism. When ultrasound waves cross different interfaces, they experience different physical phenomena like attenuation, reflection, and refraction. In the examination of patients, these effects produce the so-called artifacts. These are shapes visualized in the echographic image generated from intensity and trajectory alterations of the beam when crossing structures with different acoustic properties (i.e., echogenicity). Among these artifacts, the main ones are [47]:

• Pleural line. A thin and echogenic line is seen just at the bottom of the intercostal musculature between ribs. A normal pleural line measures 0.2 to 0.3 mm.
• A-lines. These are a form of reverberation artifact produced by the reflection of the pleura. A-lines can be spotted as parallel lines to the pleural sliding (see Fig 1a), appearing uniformly spaced and descending to the bottom of the image. These lines allow us to recognize normally ventilated lungs, although they may also be present in pneumothorax.
• B-lines. Another form of reverberation artifact is seen as bright lines (also called comet-tails or comets) extending from the pleural line to the bottom of the image (see Fig 1b), obliterating A-lines. They move synchronously with respiration, and healthy lungs may show up to three lines per window/intercostal space.
• Lung sliding. Is described as a shimmering appearance of the pleura, that seems to slide back and forth as the patient breathes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Artifacts.

(a) normal lung with a regular pleural line, A-lines (green), and lung sliding; (b) infected lung with pleural line broken (red) and fused B-lines (green), (c) subpleural consolidation. Images were taken from the POCUS database [46].

https://doi.org/10.1371/journal.pone.0255886.g001

The appearance of different artifacts on LUS is related to particular diseases or lung conditions. Fig 2 shows a flowchart for systematic diagnosis of lung diseases by means of LUS proposed by [23]. The combination of the pleural line, A-lines, and bilateral lung sliding indicates regular lung aeration (see Fig 1a). However, the presence of these artifacts in LUS images is not exclusive to healthy lung conditions. For example, some of these artifacts, in addition to other patterns, can be observed in patients with Chronic Obstructive Pulmonary Disease (COPD) [22, 23]. Therefore, it is necessary to carefully analyze which artifacts appear in the LUS image altogether with its frequency (e.g., the number of B-lines), length, separation, morphology, etc., for the diagnosis of different lung conditions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Flowchart for systematic diagnosis of lung diseases by means of LUS.

(DVT—Deep Venous Thrombosis).

https://doi.org/10.1371/journal.pone.0255886.g002

The appearance of B-lines (see Fig 1b) in LUS often indicate the presence of fibrous tissue, inflammatory cells, or extravascular lung water. These features are typical of Acute Respiratory Distress Syndrome (ARDS), interstitial lung diseases, pneumonia, pulmonary edema, among others. It is important to mention that isolated B-lines could have no pathological significance (less than three). However, depending on the number and distance between them, its appearance may be related to a particular lung condition. In general, the shorter the distance between B-lines (or even coalesced B lines), the less lung aeration [19]. Pleural effusion refers to an abnormal accumulation of fluid and no air in the pleural cavity. Ultrasound not only detects the effusion but also provides information on its nature and indicates the area for thoracentesis (i.e., removing fluid or air from the pleural space) [20]. Pleural effusion is related to conditions such as pneumonia or pulmonary edema. In contrast to pleural effusion, pneumothorax echo sonograms contain only air and no fluid. Besides, lung points (the alternation of normal and abolished lung sliding) with exclusive A-lines are a strong indicator of pneumothorax diagnosis [23]. Lung consolidation (see Fig 1c) appears as a dense tissue structure and the presence of white points characterized by an inspiratory reinforcement [19]. This indicates a complete loss of lung aeration because air spaces are filled with neoplastic tissue or pleural effusion. Lung consolidations are detected in pneumonia, lung atelectasis, lung contusion, ARDS, among others [22].

Regarding COVID-19 LUS images, multiple B-lines patterns (focal, multifocal, confluent), pleural thickening with line irregularity, pleural effusion, and different consolidation patterns have been detected in patients with COVID-19 diagnosis [40]. The previously mentioned artifacts and patterns appearing in LUS images are identifiable by trained clinicians and can be used to diagnose different lung conditions. Thus, as mentioned in the Introduction, this has led to the proposal of several computer-assisted approaches based on different artificial intelligence techniques [24–31, 42–44] to aid in the screening, diagnosis and follow-up of patients with different lung conditions.

Deep learning architectures considered in the evaluation

This work evaluates different DL-based architectures to classify COVID-19 in LUS frames. As mentioned in the Introduction, three works have proposed the use of DL models for computer-assisted analysis of COVID-19 LUS imaging [32–34]. In particular, [33, 34] propose a DL-based model called POCOVID-net, which is based on a convolutional VGG16 network. CNNs are capable of extracting more complex features with less effort, but at the cost of large volumes of data [ref]. Besides the VGG16 network, different CNNs have been used successfully for computer-assisted analysis of medical images [48–50], including: VGG16 and VGG19 [51], InceptionV3 [52], Xception [53], and ResNet50 [54].

Deep-learning based models for LUS COVID-19 classification

This work evaluates various DL-based models to classify COVID-19 in LUS frames. Considering that lung ultrasound is a portable, easy to disinfect, low cost and non-invasive tool, our goal is to lay the foundations needed to develop a tool for computer-assisted analysis of LUS imaging that can be effectively used for screening of COVID-19 patients in places where other testing alternatives are not readily available. With this in mind, in the following discussion, the classification performance achievable with different CNN architectures and classical machine learning techniques is provided. To perform the evaluation we used the dataset collected by Born et al. [34], which is publicly available at the GitHub repository provided by the authors [46].

Baseline model: POCOVID-net

In this work, we adopted the POCOVID-net architecture [33, 34] as our baseline model for performance comparison of COVID-19 identificacion from LUS imagery. The goal of POCOVID-net is to correctly classify LUS frames obtained from healthy, bacterial pneumonia, and COVID-19 patients. In this regard, an accuracy of 89% and balanced accuracy of 82% for POCOVID-net was reported in [33], whereas an accuracy and balanced accuracy of 87% was reported in the extended work [34].

The POCOVID-net architecture consists of the convolutional part of the VGG16 architecture, pre-trained on ImageNet [55]. A hidden layer of 64 neurons replaced the VGG16 fully connected layer with ReLU activation, batch normalization, and dropout of 0.5, followed by an output layer of three nodes with a softmax activation function. For the training stage, fine-tuning was carried out only on the last three layers of the model (freezing the above layers) using cross-entropy loss function and the Adam optimization model [56] with a learning rate of 0.0001. To prevent overfitting, data augmentation was realized, allowing transformations such as rotations (10 degrees), horizontal and vertical flips, and shifts (10%). POCOVID-net implementation is provided by their authors in the GitHub repository [46].

Deep-learning based models

We compare different CNN architectures and the baseline POCOVID-net model. In particular, we used the following CNN architectures: VGG19 [51], InceptionV3 [52], Xception [53], and ResNet50 [54]. We replaced the VGG16 base model of POCOVID-net with one of these CNN networks, preserving the last fully connected layers (hidden layer, batch normalization, dropout, and output layer). The number of trainable layers varied according to each model. Table 1 shows CNN configurations, where “Cropped layer” refers to the last layer used from the original architecture, and “First trainable layer” refers to the first unfrozen layer of the network for fine-tuning. All architectures were implemented in Python using Keras library [57].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. CNN configurations considered for COVID-19 identification in LUS imagery.

https://doi.org/10.1371/journal.pone.0255886.t001

Through numerous experiments, we found that the configurations shown in Table 1 provide very good network performance. To further improve the network performance, more extensive and time-consuming research should be done by varying the number of hidden network layers, the first trainable layer, batch size, epoch number, dropout, optimizer, learning rate, etc. To optimize the model hyper-parameters, it is additionally necessary to perform an exhaustive search such as gridSearchCV [58] or Hyperas [59]. Note that the number of hyper-parameter combinations can be quite large. A possible solution to reduce the number of combinations is to use estimates of hyper-parameters that have already been used to successfully solve similar problems. This will be performed in a future contribution.

Dataset

For all experiments reported in this paper, we used the POCUS dataset collected by Born et al. [34], which is publicly available on the GitHub repository provided by the authors [46]. Fig 3 shows a few examples including healthy, pneumonia, and COVID-19 images. The POCUS dataset includes 202 videos and 59 images recorded with either convex or linear probes comprising samples of 216 patients with COVID-19, bacterial and (non-COVID19) viral pneumonia, and healthy controls. The POCUS dataset was recollected from 41 different sources, including clinical data donated from hospitals or academic ultrasound course instructors, LUS recordings published in other scientific literature, community platforms, open medical repositories, and health-tech companies et al. [34]. Thus, the videos have different length and frame rate (160±144 frames, 2510Hz). Furthermore, due to the variety of data sources, not all videos include patient metadata. Only 42% of data include age (average age is 41.3 years) and gender (57% were male) information, and 30% include symptoms descriptions. Nevertheless, all samples of the POCUS database were reviewed and approved by two medical experts, and the COVID-19 diagnosis was usually made by RT-PCR.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Examples of images obtained from LUS videos after pre-processing.

Healthy (first row), pneumonia (second row), and COVID-19 (third row) LUS image classes.

https://doi.org/10.1371/journal.pone.0255886.g003

For our experiments, we employed 185 convex videos and 58 single images. The videos and images were pre-processed as described in [34] by using the tools available at [54], hence:

• LUS videos were split into images at a frame rate of 3 Hz.
• Images were cropped to a quadratic window.
• Finally, images were resized to 224 × 224 pixels.

After video sampling and image pre-processing, our data set comprised 3326 images (including the single images) of which 1283 correspond to COVID-19, 731 correspond to bacterial pneumonia, and 1312 correspond to healthy labeled images. To prevent overfitting, data augmentation transformations such as rotations (10 degrees), horizontal and vertical flips, shifts (10%), and zoom (zoom range of 20%) were added.

In order to increase the robustness of the results, we carried out repeated 5-fold cross-validation [60]. For each 5-fold cross-validation, the dataset was randomly partitioned into five folds, with considerations that the folds are class balanced and disjoint (i.e., all frames sampled from the same video are assigned to only one fold). From the five folds, four of them were chosen and merged as the training set while the remaining fold was used as the validation set. Then, the 5-fold cross-validation was repeated 5 times modifying the folds division (5 × 5-fold cross-validation). Detailed per fold image information is included in S1 File. For each 5-fold cross-validation, all CNN architectures were fine-tuned using the corresponding training set under 30 epochs with a batch size of 32 (with early stopping enabled). Then, the resulting trained models were evaluated under different performance metrics utilizing the corresponding validation set. Detailed learning curves, confusion matrices, and ROC curves of models for each 5 × 5-fold cross-validation are included in supporting information (S2–S6 Files).

Statistical analysis

Performance results obtained by the different architectures are presented by averaging a total of 25 independent runs of the repeated 5 × 5-fold cross-validation. These results are introduced using the most common utilized performance metrics for evaluating machine learning methods [61, 62]:

• Precision, Recall, and F1-score metrics for per class classification performance.
• Accuracy (ACC), Balanced Accuracy (BACC), and Area Under the Receiver Operating Characteristic Curve (AUC-ROC) metrics for overall method performance evaluation.

Both Precision and Recall metrics indicate how many model predictions are actually correct. The Precision metric considers false positives, whereas the Recall metric is associated with false negatives. Note that for COVID19 classification it is more important to reduce the number of false negatives (high Recall), since a false positive can be discarded in the second test. The F1-score represents the Harmonic mean between Precision and Recall metrics.

On the other hand, Accuracy and Balanced Accuracy (for imbalanced class data) indicate the number of correct predictions out of the total data samples, evaluating the overall method performance. Finally, the ROC curve represents a trade-off between true and false positive rates under different probability thresholds for the predictive model.

Additionally, the ANOVA test was applied to analyze the statistical difference in the overall performance of the evaluated models. However, normal distribution and variance homogeneity are assumed for the ANOVA test. This can not always be guaranteed for the performance analysis of machine learning algorithms [62, 63]. Thus, we also applied the non-parametric Friedman test [62, 63]. Then, post-hoc analysis was carried out as suggested in [64] using the Wilcoxon signed-rank test with Holm’s step down correction [63, 65], for pair-wise comparison between the evaluated classifiers. Detailed results are presented in suplemental information S7 File. This enabled us to further assess statistical significant differences in the performance results provided by the different CNNs evaluated in this work. All average results are presented with a 95% confidence interval (95% C.I.).

Principal results

In this work, we evaluated the use of different deep learning architectures to classify LUS frames acquired from COVID-19, bacterial pneumonia, and healthy patients. The modified VGG19, InceptionV3, Xception, and ResNet50 networks were trained, evaluated, and compared with the baseline POCOVID-net model on the POCUS dataset under a 5 × 5-fold cross-validation scheme. After carrying out the statistical analysis of the evaluated DL-based models (summarized in Table 2 and the corresponding discussion), we found out that the fine-tuned InceptionV3-based network achieved a statistically significantly higher mean ACC of 89.1%, BACC of 89.3% and AUC-ROC of 97% than the baseline POCOVID-net, VGG19-, and ResNet50-based models. No statistically significant difference was found between InceptionV3- and Xception-based models performances. Thus, it can be concluded that the InceptionV3-based network provides the most promising predictive results of all AI-based techniques evaluated in this work. Furthermore, it could be expected that with more data, the performance of this model will continue to increase. In this regard, we consider that the Xception-based model is also worthy of consideration for further evaluation as more data becomes available.

Limitations

The experiments were carried out on an open access dataset of LUS images, including COVID-19, bacterial pneumonia, and examples from healthy people. A larger dataset would be required to develop a fine-tuned version of the evaluated techniques. To the best of our knowledge, this is a limitation of most modern medical imaging machine learning tools. In the future, a classification tool trained on a huge dataset of hundreds or even thousands of patients (rather than a few images taken from a few patients) will be needed to validate the generalization of any automated diagnostic assistant tool.

Comparison with prior work

While previous work focused on the use of a VGG16 convolutional architecture for the classification of LUS frames acquired from COVID-19, bacterial pneumonia, and healthy patients, the present work took a step further. Particularly we studied the use of other CNN configurations to address this task.

Conclusions

In the context of the current COVID-19 pandemic, where health services are often saturated, the use of automated image diagnosing tools could importantly help to alleviate the burden of health systems with a limited number of specialized clinicians. In this paper, we do not compete with AI-based solutions previously proposed for computer-assisted COVID-19 screening based on CT or X-ray imaging. Instead, we seek to lay the foundations to develop an AI-based COVID-19 screening tool that uses LUS imaging as an alternative when there is limited or no access to CT or X-ray equipment. In this sense, the advantages offered by LUS (e.g., portability, cost, ease of disinfection, etc.) combined with the feasibility of readily implementing and deploying already trained AI-based solutions in a wide variety of portable devices (e.g., laptops, smartphones, etc.), has the potential of providing an accessible and mobile COVID-19 screening tool for medical staff. This fact becomes crucial in rural areas or developing countries, where wide access to ultrasound devices might be more feasible because of its portability and low-cost features compared to medical imaging devices like CT and X-ray. In addition to the benefits for screening and diagnosis, trained algorithms applied to the recognition of LUS artifacts would allow to follow the progress of COVID-19 in diagnosed patients, assisting decisions in medical treatments. Thus, besides improving the performance of the classifiers as more LUS images become available, future research will address the use of CNN-models for computer-assisted analysis of LUS imaging with the aim of helping with the screening and follow-up of other lung related pathologies.