Next Article in Journal
Periodontal Behavior and Patient Satisfaction of Anterior Teeth Restored with Single Zirconia Crowns Using a Biologically Oriented Preparation Technique: A 6-Year Prospective Clinical Study
Next Article in Special Issue
Weighted Gene Co-Expression Network Analysis Combined with Machine Learning Validation to Identify Key Modules and Hub Genes Associated with SARS-CoV-2 Infection
Previous Article in Journal
Extensive Ossification of the Achilles Tendon with and without Acute Fracture: A Scoping Review
Previous Article in Special Issue
SARS-CoV-2 and Acute Cerebrovascular Events: An Overview
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Ruling Out Coronavirus Disease 2019 in Patients with Pneumonia: The Role of Blood Cell Count and Lung Ultrasound

1
Clinica Medica, Department of Medical Surgical and Health Sciences, University of Trieste, Strada di Fiume 447, 34149 Trieste, Italy
2
Emergency Department, ASUGI, Ospedale di Cattinara, Strada di Fiume 447, 34149 Trieste, Italy
3
Pharmacy Department, ASUGI, Ospedale di Cattinara, Strada di Fiume 447, 34149 Trieste, Italy
4
Department of Advanced Medical and Surgical Sciences, University of Campania, L. Vanvitelli, 80131 Naples, Italy
5
Biostatistics Unit, Department of Medical Surgical and Health Sciences, University of Trieste, Strada di Fiume 447, 34149 Trieste, Italy
*
Author to whom correspondence should be addressed.
Equal contribution.
J. Clin. Med. 2021, 10(16), 3481; https://doi.org/10.3390/jcm10163481
Submission received: 31 May 2021 / Revised: 28 July 2021 / Accepted: 30 July 2021 / Published: 6 August 2021

Abstract

:
Coronavirus disease 2019 (COVID-19) is characterized by a distinctive blood leucocyte pattern and B-lines on lung ultrasound (LUS) as marker of alveolar-interstitial syndrome. We aimed to evaluate the accuracy of blood leucocyte count alone or in combination with LUS for COVID-19 diagnosis. We retrospectively enrolled consecutive patients diagnosed with community acquired pneumonia (CAP) at hospital admission to derive and validate cutoff values for blood cell count that could be predictive of COVID-19 before confirmation by the nucleic acid amplification test (NAAT). Cutoff values, generated and confirmed in inception (41/115, positive/negative patients) and validation (100/180, positive/negative patients) cohorts, were ≤17 and ≤10 cells/mm3 for basophils and eosinophils, respectively. Basophils and/or eosinophils below cutoff were associated with sensitivity of 98% (95%CI, 94–100) and negative likelihood ratio of 0.04 (95%CI, 0.01–0.11). In a subgroup of 265 subjects, the sensitivity of B-line on LUS was 15% lower (p < 0.001) than that of basophils and/or eosinophils below cutoff. The combination of B-lines with basophils and eosinophils below cutoff was associated with a moderate increase of the positive likelihood ratio: 5.0 (95%CI, 3.2–7.7). In conclusion, basophil and eosinophil counts above the generated cutoff virtually rule out COVID-19 in patients with CAP. Our findings can help optimize patient triage pending the NAAT results.

1. Introduction

Community-acquired pneumonia (CAP) is one of the most common acute infections requiring hospital admission [1]. The diagnosis of CAP generally requires the demonstration of an infiltrate on the chest radiograph in a patient with a clinically compatible syndrome, i.e., fever, dyspnea, and cough. During the coronavirus disease 2019 (COVID-19) pandemic, all patients with possible CAP are required to be tested for SARS-CoV-2, since CAP and COVID-19 pneumonia can show similar clinical presentations [2,3]. Early COVID-19 recognition is crucial for timely patient isolation to prevent the spread of the disease. A positive nucleic acid amplification test (NAAT) for SARS-CoV-2 generally confirms the diagnosis and no additional diagnostic testing is necessary [4,5]. The time to complete NAATs varies from less than an hour (rapid NAAT) to more than a day [6]. While NAATs for SARS-CoV-2 are expected to be very specific (>99%), reported false-negative NAAT from upper respiratory specimens have ranged between less than 5% up to 40% [7,8]. The pooled sensitivity of initial NAAT in 11 studies was 84% [9]. If the initial test is negative but suspicion for COVID-19 remains high, the test is usually repeated, 24 to 48 hours later, from a specimen obtained by nasopharyngeal swab (NPS) or bronchoalveolar lavage (BAL). Patients awaiting definitive diagnosis pose a major triage challenge in the Emergency Department (ED), due to the contagiousness of COVID-19 and frequent space limitation. Accurate and rapid diagnostic tools would be needed to stratify suspected COVID-19 patients until the definitive confirmation by NAAT.
Thoracic imaging is an important complement to NAAT for diagnosing COVID-19 pneumonia [10]. Computed tomography (CT) is the gold standard method for detection of alveolar-interstitial syndrome, which is the distinctive feature of COVID-19 pneumonia [10]. CT is more sensitive than NAAT for COVID-19 diagnosis but also much less specific [9]. Low-dose non-contrast CT is reliable in detecting COVID-19 pneumonia in daily practice, with a significant reduction in radiation dose [11]. Rapid NAAT and low-dose chest CT can now rapidly and accurately confirm or rule out COVID-19 infection [6,11]. Lung ultrasound (LUS) is also of great interest for rapid COVID-19 screening due to its low cost, widespread availability, ease of use, and reduced risk of spreading the infection [10,11,12,13,14,15]. Single or confluent interstitial artifactual signs, defined as B-lines, represent the alveolar-interstitial syndrome on LUS evaluation [13,14,15]. Although LUS and CT may have comparable diagnostic accuracy [10,14], no chest imaging technique can safely rule out COVID-19.
Immune dysregulation plays a pivotal role in SARS-CoV-2 infection leading to a characteristic pattern of blood leukocytes [16]. Lymphocyte, monocyte, basophil, and eosinophil counts are reduced in proportion to the severity of the disease [17,18,19]. Severe lymphopenia is associate with delayed viral elimination and uncontrolled cytokine production, leading to influx of myeloid cells into the lungs [20,21]. In particular, lung influx of blood eosinophils, mediated by interferon-γ secretion, is the key mechanism of alveolar inflammation [16,22]. Eosinophil counts are related to the outcome of COVID-19 patients [23,24]. Basophil depletion in the acute phase of COVID-19 is associated with downregulation of the antibody response to the virus [16]. Several lines of evidence indicate that such leukocyte pattern represents a distinct cellular signature that could allow COVID-19 pneumonia to be identified among patients with CAP of different etiologies [25].
The main goal of this study was to evaluate the accuracy of performing blood leucocyte count to predict SARS-CoV-2 diagnosis in patients hospitalized for pneumonia not requiring either invasive or non-invasive ventilator support. In addition, the diagnostic accuracy of blood leucocyte count in combination with LUS was evaluated. The results were analyzed according to a retrospective, observational, case-control design during the first (inception cohort) and second (validation cohort) pandemic waves.

2. Materials and Methods

We retrospectively reviewed all consecutive patients hospitalized from 14 March to 30 April 2020 (n = 201) and from 1 September to 31 December 2020 (n = 469) in the Academic Unit of Internal Medicine of the University Hospital of Trieste, Italy, admitting patients with or without COVID-19. After a first evaluation at the ED, patients with fever or symptoms and signs of respiratory tract infection, not requiring either invasive or non-invasive ventilator support, were admitted to the internal medicine ward. We enrolled in the study all patients who had evidence of community-acquired, acute, low-respiratory tract infection during the first (inception cohort, n = 156) and second (validation cohort, n = 280) evaluation periods. Clinical diagnosis of pneumonia was defined as cough or sputum production, chest pain, dyspnea, tachypnea, abnormal lung examination, or respiratory failure, in combination with evidence of pneumonia at chest radiography obtained within 48 hours after emergency department admission. All laboratory data were available within one day from admission. Patients were excluded if they had a clear alternative diagnosis. Two physicians (G.B. and P.V.) retrospectively and independently evaluated all available records from clinical workups and established the diagnosis of CAP or non-pneumonia disease. When the two physicians were in agreement the diagnosis was considered definitive. In cases of disagreement, a third physician (F.M.) established independently the final diagnosis. At ED admission, LUS was performed with a multiprobe machine (MyLab25 Gold-Esaote, Genoa, Italy) whenever an attending physician, who had completed a LUS training course accredited by the Italian Society of Emergency Medicine, was available. LUS was performed in 88 out of 156 and in 177 out of 280 patients with pneumonia in the inception and validation cohort, respectively. Both the anterior and posterior chest were studied with a 4- to 8-MHz linear probe and a 2.5- to 3.5-MHz curved array probe [26]. In agreement with previously published descriptions [12,15], the presence of focal or diffuse interstitial syndrome and/or parenchymal consolidations was considered suggestive of SARS-CoV-2-related pneumonia. Alveolar-interstitial syndrome was defined as the presence of three or more B-lines in a longitudinal plane between two ribs in a given lung region [13]. Anterolateral and posterior chest scans were performed with both convex and linear probes. LUS was performed on the whole thorax by placing the probe in the intercostal spaces. The probe was positioned both perpendicular to the ribs and along the intercostal spaces. Each hemithorax was divided into five areas: two anterior, two lateral, one posterior. The anterior chest-wall, bounded by the parasternal line and the anterior axillary line, was divided into an upper region (from the clavicle to the third intercostal space) and a lower region (from the fourth intercostal space to the diaphragm). The lateral area, between the anterior and posterior axillary lines, was divided into upper and lower halves. Finally, the posterior area from the posterior axillary line to the paravertebral line was identified. For all patients, SARS-CoV-2 detection was at first performed on NPS samples. All NAAT positive cases were considered true positive. Patients with an initially negative SARS-CoV-2 NAAT were classified as COVID-19 positive if a second NAAT (performed within 72 h from the initial assessment), either by NPS or BAL, resulted in being positive. Conversely, patients with at least two SARS-CoV-2 NAAT concordantly negative were classified as true COVID-19 negative cases. The patients epidemiological, clinical, laboratory, and radiological findings were collected and analyzed. Comorbid conditions were evaluated with use of the Charlson Comorbidity Index [27]. Severity of CAP hospital presentation was defined according to the five-point score CURB-65, based on confusion, blood urea nitrogen, respiratory rate, blood pressure, and age [28]. The study complies with the Helsinki declaration and was approved by the institutional review board of the Friuli-Venezia Giulia region, Italy (CEUR).

Statistics

The primary end point of the study was the overall diagnostic accuracy, i.e., the percentage of correctly diagnosed patients, from blood cell counts and alveolar-interstitial syndrome by LUS in the differentiation of SARS-CoV-2 positive and negative patients with pneumonia. Results were statistically analyzed according to a retrospective, observational, case-control design in inception and validation cohorts. Mortality was recorded at 10 and 30 days after hospital admission. Continuous variables were expressed as medians and interquartile ranges (IQR). Categorical variables were summarized as counts and percentages. Results of COVID-19 and non-COVID-19 patients were compared by the Mann-Whitney test or the chi-square test, when appropriate. In order to observe the association between blood cell count and COVID-19 pneumonia, binary logistic regression analysis and Receiver Operator Characteristic (ROC) curve were carried out, then the Youden index was used to dichotomize the continuous variables. Cutoff values were generated in the inception cohort and validated in the validation and combined cohorts. To assess the diagnostic performance of dichotomous variables, i.e., blood cell count below cutoff and B-lines in LUS, the results were tabulated in 2 × 2 contingency tables with groups of subjects divided according to positive or negative SARS-CoV-2 NAAT in columns and categories according to test results in rows. Sensitivity, negative predictive values (NPV), and negative likelihood ratios (NLR) were calculated to assess the ability of blood cell count to rule out COVID-19 infection. Specificity, positive predictive values (PPV) and positive likelihood ratios (PLR) were calculated to assess the ability of blood cell count to rule in COVID-19 infection. The diagnostic performance of eosinophil or basophil count below the cutoff and of B-lines with LUS were assessed both individually and in various combinations. The diagnostic performance of the different tests was compared in combined cohort. The McNemar test was used to compare sensitivity and specificity. To compare the NPV and PPV values we used the “N-1” Chi-squared test. All the statistics were obtained by SPSS (Version 21-0, SPSS Inc, Chicago, IL, USA) and MedCalc statistical software (available at https://www.medcalc.org/, last access 4 August 2021).

3. Results

A flow chart of enrollment and adjudication in the inception and validation cohorts is shown in Figure 1. Considering the whole population (i.e., inception and validation cohorts), 78% and 60% had clinical and radiological signs of pneumonia, respectively. The causes of hospitalization of the 234 patients who were excluded from the analysis in the inception and validation cohorts were as follows: heart failure, 22%; urinary tract infection and acute kidney injury, 16%; gastrointestinal bleeding, 7%; decompensated liver cirrhosis, 7%; acute exacerbation of chronic obstructive pulmonary disease, 6%; lung cancer, 5%; abdominal infection, 4%; anemia, 5%; trauma, 4%; acute coronary syndrome, 3%; stroke, 3%; atrial fibrillation, 3%; pulmonary embolism, 3%; connective tissue disease, 2%; pancreatitis, 2%; other diagnoses, 7%. Twenty-eight of the 234 excluded patients (12%) were SARS-CoV-2 NAAT positive but they did not have clinical and radiological signs of pneumonia.
Table 1 shows clinical characteristics and outcomes of patients with pneumonia in the inception and validation cohorts with or without positive SARS-CoV-2 NAAT. The Charlson Comorbidity Index and CURB-65 scores were significantly greater in the inception as compared to the validation cohort while 30-day mortality was significantly lower in the validation cohort. Prevalence of COVID-19 positive subjects was lower (p < 0.05) in the inception (26%) than in the validation (36%) cohort. In the validation cohort, median age of COVID-19 patients was lower than that of non-COVID-19 patients while 30-day mortality of COVID-19 patients was higher than that of non-COVID-19 cases in the validation but not in the inception cohort. Cell blood count analysis showed significantly lower lymphocyte, monocyte, eosinophil, and basophil counts in the COVID-19 patients. Among patients with COVID-19, 82% had zero eosinophils at presentation, compared to 29% of non COVID-19 patients (p < 0.05). Median basophil count was 68% lower in the COVID-19 as compared to that of non COVID-19 patients (p < 0.05).
Results of binary logistic regression analysis indicated (coefficient ± SE) that only eosinophils (−0.013 ± 0.005, p = 0.009) and basophils (−0.026 ± 0.011, p = 0.016) were independently associated with COVID-19 diagnosis. Therefore, we restricted the analysis on susceptibility to COVID-19 pneumonia to eosinophils and basophils only. ROCs for basophils and eosinophils in diagnosis of COVID-19 pneumonia in inception, validation, and combined cohorts are shown in Figure 2, and their AUC in Table 2. The AUCs were highly significant for all cell types in all cohorts. Optimal cutoff values for basophil and eosinophil count generated in the inception cohort were confirmed in the validation cohort (Table 2).
Table 3 shows the results of diagnostic performance of eosinophil or basophil count below the generated cutoff values, both individually and in various combinations, in inception, validation, and combined cohorts. The diagnostic performance of the different tests was compared in combined cohorts (All). Regarding COVID-19 ruling out, NPV and sensitivity of eosinophils below the cutoff were significantly greater than those of basophils. In the combined cohort, the finding of at least one or both values of basophils and eosinophils below the cutoff, tested as dichotomous variables for COVID-19 diagnosis, was associated with 36% and 7% greater (p < 0.001) sensitivity than those of basophils and eosinophils, respectively. According to NPV and NLR, a negative test, i.e., basophils and/or eosinophils above cutoff, was associated with very low probability of SARS-CoV-2 infection and allowed negative from positive COVID-19 patients with pneumonia to be distinguished in about 98% of cases. Regarding COVID-19 ruling in, the finding of both values of basophils and eosinophils below the cutoff, tested as dichotomous variables for COVID-19 diagnosis, was associated with 8% and 36% greater specificity than those of basophils and eosinophils, respectively. According to PPV and PLR, a positive test, i.e., basophils and eosinophils below cutoff, was associated with moderate probability of SARS-CoV-2 infection and allowed positive from negative COVID-19 patients with pneumonia to be distinguished in about 64% of cases.
The prevalence of COVID-19 in the subgroup of patients in whom LUS was performed was 34%. Table 4 shows the results of diagnostic performance of detection of B-lines with LUS alone and in combination with eosinophil or basophil count below cutoff. Sensitivity of LUS was 15% lower than that of basophils and/or eosinophils below cutoff. The NPV of B-lines was significantly lower than that of basophils and/or eosinophils below cutoff. Positive LUS and/or basophils and/or eosinophils below cutoff were associated with 100% NPV and 100% (95%CI, 96–100) sensitivity. Specificity, PPV, and PLR of positive LUS indicate the poor ability of ultrasound to distinguish positive from negative COVID-19 patients with pneumonia. Nonetheless, the combination of B-lines in LUS with basophils and eosinophils below cutoff was associated with about 88% specificity and the ability to distinguish positive from negative COVID-19 patients with pneumonia in about 72% of cases of positive test. Such PPV of positive LUS in combination with basophils and eosinophils below cutoff was not significantly greater than that of basophils and eosinophils below cutoff alone.

4. Discussion

Using retrospective data from 436 CAP patients with suspected COVID-19, enrolled during the first and the second waves of the COVID-19 pandemic, we derived and validated optimal cutoff values of basophil and eosinophil count to discriminate SARS-CoV-2 positive from negative patients. COVID-19 was confirmed in 32% of patients by NAAT. The identified cutoffs were ≤17 cells/mm3 for basophils and ≤10 cells/mm3 for eosinophils. The diagnostic performance of observing basophils and/or eosinophils below the cutoff indicated NPV and NLR of about 98% and 0.04, respectively. Thus, in the 40% of our CAP patients, who had both basophil and eosinophil values higher than the cutoff, SARS-CoV-2 infection could be virtually ruled out before obtaining the NAAT results.
We observed severe eosinopenia and basopenia in our patients with SARS-CoV-2 pneumonia at hospital admission. Our results were in agreement with previous observations in COVID-19 patients requiring hospitalization in acute care settings of different intensity, i.e., from internal medicine wards to intensive care units [17,18,19,20,25]. Regarding eosinopenia, as a simple biological marker of SARS-CoV-2 infection, our results and those of a recent published study in hospitalized patients with pneumonia indicate the same cutoff of 10 eosinophils/mm3 and the same sensitivity of 89% for COVID-19 diagnosis [29].
In a subgroup of 265 subjects, with COVID-19 prevalence of 34%, point-of-care LUS was performed to detect focal or diffuse B-lines with or without consolidation as a characteristic pattern of COVID-19 interstitial pneumonia. In agreement with a recent meta-analysis [9], we found that detection of B-lines in LUS is sensitive (86%) but not specific (44%) for the diagnosis of COVID-19. Detection of parenchymal consolidation is an expression of a more advanced stage of the disease [15]. When our analysis included in addition those patients who presented consolidations without B-lines, the diagnostic accuracy of LUS decreased with a slight change in sensitivity (88%; 95%CI, 79–94) but with significant reduction in specificity (24%; 95%CI, 18–31). In our study the diagnostic accuracy of basophils and eosinophils to rule out COVID-19 was much greater than that of LUS. In fact, basophils and/or eosinophils below thresholds as compared to B-line detection with LUS were associated with about 15% greater sensitivity and much lower NLR. Nonetheless, the integration of clinical evaluation with LUS can be used as pre-test probability of suspected COVID-19 pending the results of blood leukocyte count [30].
While the detection of high values of basophils and eosinophils virtually excludes COVID-19, this approach based on circulating white blood cells yields mediocre results in identifying positive patients. Detection of basophils and eosinophils below the cutoff was associated with small increase in the likelihood of disease. Nonetheless, despite the fact that LUS alone has poor specificity for COVID-19 diagnosis, B-lines on LUS in combination with basophils and eosinophils below the threshold are associated with a moderate increase in the likelihood of disease with specificity of 88%.
This study has several limitations. First, we investigated the diagnostic performance of the combination of basophils and eosinophils to identify SARS-CoV-2 infection in hospitalized patients with severe CAP within 48 h of admission to the ED. It would be necessary to confirm whether the diagnostic criteria we have highlighted are also valid in asymptomatic patients or in those with mild disease [31]. Second, the etiology of non-COVID-19 pneumonia was not systematically determined by microbiological and serological evaluations, unless required by clinical needs, e.g., when empiric antibiotic therapy was not successful. We avoided induced sputum sampling to minimize the generation of potentially infectious aerosols. Evidence shows that basopenia and eosinopenia are distinctive features of severe COVID-19 [16,17,18,19,20,21,22,23,24,25,29]. In bacterial pneumonia the counts of these cells are generally not reduced [32], while pneumonia due to influenza virus is characterized by a tendency to reduce the levels of basophils and eosinophils but not as markedly as in SARS-Cov-19 infection [33,34]. Third, the mortality rate of COVID-19 patients was about 2-times greater in the inception than that in the validation cohort. These differences in mortality were mainly due to a higher Charlson Comorbidity Index and a greater CURB-65 pneumonia severity score in the inception cohort. Inception cohort patients were enrolled during the first COVID-19 pandemic wave, while validation cohort patients were enrolled during the second. Similar changes in mortality between the first and second wave have been described in different geographical settings [35,36,37]. Possible additional explanations include the increased ability to perform an early etiological diagnosis and the availability of randomized clinical trials showing the effectiveness of pharmacological interventions in COVID-19 pneumonia [38,39]. Nonetheless, despite differences in patient severity in the two cohorts, the diagnostic cutoff generated in the inception cohort and the observed diagnostic accuracy of eosinophils and basophils were fully confirmed in the validation cohort.
In conclusion, assessment of blood leukocyte counts at hospital admission in patients with CAP virtually rule out COVID-19 in those cases with basophils and eosinophils greater than 17 and 10 cells/mm3, respectively. Our findings will help to optimize the triage of patients with pneumonia pending the results of NAAT and they will also help to improve the management of false negative results on the first NPS.

Author Contributions

Conceptualization, G.B., M.G. and N.F.; methodology, G.B., N.F., F.M. and P.V.; software, N.F. and L.T.; validation, F.C., L.T. and F.M.; formal analysis, G.B. and N.F.; investigation, G.B., F.C., F.G.D.G., E.P., N.A., C.C., A.P., F.M. and P.V.; resources, G.B. and F.C.; data curation, G.B., N.F. and F.G.D.G.; writing—original draft preparation, G.B.; writing—review and editing, N.F., F.G.D.G., F.C. and F.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of Azienda Regionale di Coordinamento per la Salute (ARCS), Regione Friuli Venezia Giulia, Comitato Etico Unico Regionale (CEUR), protocol code: CEUR-2020-Os-140, date of ethical committee approval: 4 June 2020.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy and ethical reasons.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Musher, D.M.; Thorner, A.R. Community-Acquired Pneumonia. N. Engl. J. Med. 2014, 371, 1619–1628. [Google Scholar] [CrossRef] [PubMed]
  2. Wang, D.; Hu, B.; Hu, C.; Zhu, F.; Liu, X.; Zhang, J.; Wang, B.; Xiang, H.; Cheng, Z.; Xiong, Y.; et al. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus–Infected Pneumonia in Wuhan, China. JAMA 2020, 323, 1061. [Google Scholar] [CrossRef]
  3. Corradini, E.; Ventura, P.; Ageno, W.; Cogliati, C.B.; Muiesan, M.L.; Girelli, D.; Pirisi, M.; Gasbarrini, A.; Angeli, P.; Querini, P.R.; et al. Clinical factors associated with death in 3044 COVID-19 patients managed in internal medicine wards in Italy: Results from the SIMI-COVID-19 study of the Italian Society of Internal Medicine (SIMI). Intern. Emerg. Med. 2021, 16, 1005–1015. [Google Scholar] [CrossRef]
  4. Lieberman, J.A.; Pepper, G.; Naccache, S.N.; Huang, M.-L.; Jerome, K.R.; Greninger, A.L. Comparison of Commercially Available and Laboratory-Developed Assays for In Vitro Detection of SARS-CoV-2 in Clinical Laboratories. J. Clin. Microbiol. 2020, 58, e00821-20. [Google Scholar] [CrossRef] [PubMed]
  5. Tali, S.H.S.; LeBlanc, J.J.; Sadiq, Z.; Oyewunmi, O.D.; Camargo, C.; Nikpour, B.; Armanfard, N.; Sagan, S.M.; Jahanshahi-Anbuhi, S. Tools and Techniques for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)/COVID-19 Detection. Clin. Microbiol. Rev. 2021, 34, e00228-20. [Google Scholar] [CrossRef]
  6. Hanson, K.E.; Altayar, O.; Caliendo, A.M.; Arias, C.A.; Englund, J.A.; Hayden, M.K.; Lee, M.J.; Loeb, M.; Patel, R.; El Alayli, A.; et al. The Infectious Diseases Society of America Guidelines on the Diagnosis of COVID-19: Antigen Testing. Clin. Infect. Dis. 2021. [Google Scholar] [CrossRef]
  7. Weissleder, R.; Lee, H.; Ko, J.; Pittet, M.J. COVID-19 diagnostics in context. Sci. Transl. Med. 2020, 12, eabc1931. [Google Scholar] [CrossRef] [PubMed]
  8. Long, D.R.; Gombar, S.; Hogan, C.A.; Greninger, A.L.; O’Reilly-Shah, V.; Bryson-Cahn, C.; Stevens, B.; Rustagi, A.; Jerome, K.R.; Kong, C.S.; et al. Occurrence and Timing of Subsequent Severe Acute Respiratory Syndrome Coronavirus 2 Reverse-transcription Polymerase Chain Reaction Positivity Among Initially Negative Patients. Clin. Infect. Dis. 2020, 72, 323–326. [Google Scholar] [CrossRef]
  9. Mair, M.D.D.; Hussain, M.; Siddiqui, S.; Das, S.; Baker, A.; Conboy, P.; Valsamakis, T.; Uddin, J.; Rea, P. A systematic review and meta-analysis comparing the diagnostic accuracy of initial RT-PCR and CT scan in suspected COVID-19 patients. Br. J. Radiol. 2021, 94, 20201039. [Google Scholar] [CrossRef]
  10. Islam, N.; Salameh, J.-P.; Leeflang, M.M.; Hooft, L.; McGrath, T.A.; Van Der Pol, C.B.; Frank, R.A.; Kazi, S.; Prager, R.; Hare, S.S.; et al. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst. Rev. 2020, 11, CD013639. [Google Scholar] [CrossRef]
  11. Finance, J.; Zieleskewicz, L.; Habert, P.; Jacquier, A.; Parola, P.; Boussuges, A.; Bregeon, F.; Eldin, C. Low Dose Chest CT and Lung Ultrasound for the Diagnosis and Management of COVID-19. J. Clin. Med. 2021, 10, 2196. [Google Scholar] [CrossRef]
  12. Bar, S.; Lecourtois, A.; Diouf, M.; Goldberg, E.; Bourbon, C.; Arnaud, E.; Domisse, L.; Dupont, H.; Gosset, P. The association of lung ultrasound images with COVID-19 infection in an emergency room cohort. Anaesthesia 2020, 75, 1620–1625. [Google Scholar] [CrossRef] [PubMed]
  13. Brahier, T.; Meuwly, J.-Y.; Pantet, O.; Vez, M.-J.B.; Donnet, H.G.; Hartley, M.-A.; Hugli, O.; Boillat-Blanco, N. Lung Ultrasonography for Risk Stratification in Patients with Coronavirus Disease 2019 (COVID-19): A Prospective Observational Cohort Study. Clin. Infect. Dis. 2020. [Google Scholar] [CrossRef]
  14. Kirschner, J.; Hunter, B. In the ED, LUS and CT did not differ for sensitivity or specificity for diagnosing COVID-19 pneumonia. Ann. Intern. Med. 2021, 174, JC57. [Google Scholar] [CrossRef]
  15. Volpicelli, G.; Gargani, L.; Perlini, S.; Spinelli, S.; Barbieri, G.; Lanotte, A.; Casasola, G.G.; Nogué-Bou, R.; Lamorte, A.; Agricola, E.; et al. Lung ultrasound for the early diagnosis of COVID-19 pneumonia: An international multicenter study. Intensiv. Care Med. 2021, 47, 444–454. [Google Scholar] [CrossRef]
  16. Rodriguez, L.; Pekkarinen, P.T.; Lakshmikanth, T.; Tan, Z.; Consiglio, C.R.; Pou, C.; Chen, Y.; Mugabo, C.H.; Nguyen, N.A.; Nowlan, K.; et al. Systems-Level Immunomonitoring from Acute to Recovery Phase of Severe COVID-19. Cell Rep. Med. 2020, 1, 100078. [Google Scholar] [CrossRef] [PubMed]
  17. Martens, R.J.H.; van Adrichem, A.J.; Mattheij, N.J.A.; Brouwer, C.G.; van Twist, D.J.L.; Broerse, J.J.C.R.; Magro-Checa, C.; van Dongen, C.M.P.; Mostard, R.L.M.; Ramiro, S.; et al. Hemocytometric characteristics of COVID-19 patients with and without cytokine storm syndrome on the sysmex XN-10 hematology analyzer. Clin. Chem. Lab. Med. 2020, 59, 783–793. [Google Scholar] [CrossRef]
  18. Thell, R.; Zimmermann, J.; Szell, M.; Tomez, S.; Eisenburger, P.; Haugk, M.; Kreil, A.; Spiel, A.; Blaschke, A.; Klicpera, A.; et al. Standard blood laboratory values as a clinical support tool to distinguish between SARS-CoV-2 positive and negative patients. Sci. Rep. 2021, 11, 1–9. [Google Scholar] [CrossRef] [PubMed]
  19. Perlini, S.; Ciprandi, G.; Castagnoli, R.; Licari, A.; Marseglia, G.L. Eosinopenia could be a relevant prognostic biomarker in patients with coronavirus disease 2019. Allergy Asthma Proc. 2020, 41, e80–e82. [Google Scholar] [CrossRef] [PubMed]
  20. Lin, A.; He, Z.-B.; Zhang, S.; Zhang, J.-G.; Zhang, X.; Yan, W.-H. Early Risk Factors for the Duration of Severe Acute Respiratory Syndrome Coronavirus 2 Viral Positivity in Patients with Coronavirus Disease 2019. Clin. Infect. Dis. 2020, 71, 2061–2065. [Google Scholar] [CrossRef]
  21. Qin, C.; Zhou, L.; Hu, Z.; Zhang, S.; Yang, S.; Tao, Y.; Xie, C.; Ma, K.; Shang, K.; Wang, W.; et al. Dysregulation of Immune Response in Patients with Coronavirus 2019 (COVID-19) in Wuhan, China. Clin. Infect. Dis. 2020, 71, 762–768. [Google Scholar] [CrossRef] [PubMed]
  22. Rosenberg, H.F.; Foster, P.S. Eosinophils and COVID-19: Diagnosis, prognosis, and vaccination strategies. Semin. Immunopathol. 2021, 43, 383–392. [Google Scholar] [CrossRef] [PubMed]
  23. González, M.M.; Gonzalo, E.S.; Lopez, I.C.; Fernández, F.A.; Pérez, J.L.B.; Monge, D.M.; Núñez, J.A.V.; Fenoll, R.G.; Fernández, C.S.; Castro, S.J.F.; et al. The Prognostic Value of Eosinophil Recovery in COVID-19: A Multicentre, Retrospective Cohort Study on Patients Hospitalised in Spanish Hospitals. J. Clin. Med. 2021, 10, 305. [Google Scholar] [CrossRef] [PubMed]
  24. Huang, R.; Xie, L.; He, J.; Dong, H.; Liu, T. Association between the peripheral blood eosinophil counts and COVID-19. Med. 2021, 100, e26047. [Google Scholar] [CrossRef] [PubMed]
  25. Sun, S.; Cai, X.; Wang, H.; He, G.; Lin, Y.; Lu, B.; Chen, C.; Pan, Y.; Hu, X. Abnormalities of peripheral blood system in patients with COVID-19 in Wenzhou, China. Clin. Chim. Acta 2020, 507, 174–180. [Google Scholar] [CrossRef]
  26. Volpicelli, G.; Elbarbary, M.; Blaivas, M.; Lichtenstein, D.A.; Mathis, G.; Kirkpatrick, A.W.; Melniker, L.; Gargani, L.; Noble, V.E.; et al.; International Liaison Committee on Lung Ultrasound (ILC-LUS) for the International Consensus Conference on Lung Ultrasound (ICC-LUS) International evidence-based recommendations for point-of-care lung ultrasound. Intensiv. Care Med. 2012, 38, 577–591. [Google Scholar] [CrossRef] [Green Version]
  27. Quan, H.; Li, B.; Couris, C.M.; Fushimi, K.; Graham, P.; Hider, P.; Januel, J.-M.; Sundararajan, V. Updating and Validating the Charlson Comorbidity Index and Score for Risk Adjustment in Hospital Discharge Abstracts Using Data From 6 Countries. Am. J. Epidemiol. 2011, 173, 676–682. [Google Scholar] [CrossRef] [Green Version]
  28. Lim, W.S.; Van Der Eerden, M.M.; Laing, R.; Boersma, W.G.; Karalus, N.; Town, G.I.; Lewis, S.A.; Macfarlane, J.T. Defining community acquired pneumonia severity on presentation to hospital: An international derivation and validation study. Thorax 2003, 58, 377–382. [Google Scholar] [CrossRef] [Green Version]
  29. Outh, R.; Boutin, C.; Gueudet, P.; Suzuki, M.; Saada, M.; Aumaître, H. Eosinopenia <100/μL as a marker of active COVID-19: An observational prospective study. J. Microbiol. Immunol. Infect. 2021, 54. [Google Scholar] [CrossRef]
  30. Pivetta, E.; Goffi, A.; Tizzani, M.; Locatelli, S.M.; Porrino, G.; Losano, I.; Leone, D.; Calzolari, G.; Vesan, M.; Steri, F.; et al. Lung Ultrasonography for the Diagnosis of SARS-CoV-2 Pneumonia in the Emergency Department. Ann. Emerg. Med. 2021, 77, 385–394. [Google Scholar] [CrossRef]
  31. Chen, J.; Han, T.; Huang, M.; Yang, Y.; Shang, F.; Zheng, Y.; Zhao, W.; Luo, L.; Han, X.; Lin, A.; et al. Clinical characteristics of asymptomatic carriers of novel coronavirus disease 2019: A multi-center study in Jiangsu Province. Virulence 2020, 11, 1557–1568. [Google Scholar] [CrossRef]
  32. Song, L.; Liang, E.-Y.; Wang, H.-M.; Shen, Y.; Kang, C.-M.; Xiong, Y.-J.; He, M.; Fu, W.-J.; Ke, P.-F.; Huang, X.-Z. Differential diagnosis and prospective grading of COVID-19 at the early stage with simple hematological and biochemical variables. Diagn. Microbiol. Infect. Dis. 2021, 99, 115169. [Google Scholar] [CrossRef] [PubMed]
  33. Luo, Y.; Yuan, X.; Xue, Y.; Mao, L.; Lin, Q.; Tang, G.; Song, H.; Liu, W.; Hou, H.; Wang, F.; et al. Using a diagnostic model based on routine laboratory tests to distinguish patients infected with SARS-CoV-2 from those infected with influenza virus. Int. J. Infect. Dis. 2020, 95, 436–440. [Google Scholar] [CrossRef] [PubMed]
  34. Chen, J.; Pan, Y.; Li, G.; Xu, W.; Zhang, L.; Yuan, S.; Xia, Y.; Lu, P.; Zhang, J. Distinguishing between COVID-19 and influenza during the early stages by measurement of peripheral blood parameters. J. Med. Virol. 2020, 93, 1029–1037. [Google Scholar] [CrossRef]
  35. Asch, D.A.; Sheils, N.E.; Islam, N.; Chen, Y.; Werner, R.M.; Buresh, J.; Doshi, J.A. Variation in US Hospital Mortality Rates for Patients Admitted With COVID-19 During the First 6 Months of the Pandemic. JAMA Intern. Med. 2021, 181, 471. [Google Scholar] [CrossRef] [PubMed]
  36. Fan, G.; Yang, Z.; Lin, Q.; Zhao, S.; Yang, L.; He, D. Decreased Case Fatality Rate of COVID-19 in the Second Wave: A study in 53 countries or regions. Transbound. Emerg. Dis. 2020, 68, 213–215. [Google Scholar] [CrossRef] [PubMed]
  37. Domingo, P.; Pomar, V.; Mur, I.; Castellví, I.; Corominas, H.; de Benito, N. Not all COVID-19 pandemic waves are alike. Clin. Microbiol. Infect. 2021, 27, 1040.e7–1040.e10. [Google Scholar] [CrossRef]
  38. Berlin, D.; Gulick, R.; Martinez, F.J. Severe Covid-19. N. Engl. J. Med. 2020, 383, 2451–2460. [Google Scholar] [CrossRef]
  39. Meyerowitz, E.A.; Sen, P.; Schoenfeld, S.R.; Neilan, T.G.; Frigault, M.J.; Stone, J.H.; Kim, A.Y.; Mansour, M.K.; CIG (COVID-19 Immunomodulatory Group). Immunomodulation as Treatment for Severe Coronavirus Disease 2019: A Systematic Review of Current Modalities and Future Directions. Clin. Infect. Dis. 2020, 72, e1130–e1143. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Enrolment and adjudication. LUS, lung ultrasound.
Figure 1. Enrolment and adjudication. LUS, lung ultrasound.
Jcm 10 03481 g001
Figure 2. Receiver-Operating-Characteristic Curves for low eosinophil or basophil counts predicting COVID-19 diagnosis in patients with pneumonia in the inception (41 and 115 COVID-19 positive and negative patients, respectively) validation (100 and 180 COVID-19 positive and negative patients, respectively) and combined (141 and 295 COVID-19 positive and negative patients, respectively) cohorts. AUCs and cutoff are shown in Table 2.
Figure 2. Receiver-Operating-Characteristic Curves for low eosinophil or basophil counts predicting COVID-19 diagnosis in patients with pneumonia in the inception (41 and 115 COVID-19 positive and negative patients, respectively) validation (100 and 180 COVID-19 positive and negative patients, respectively) and combined (141 and 295 COVID-19 positive and negative patients, respectively) cohorts. AUCs and cutoff are shown in Table 2.
Jcm 10 03481 g002
Table 1. Clinical characteristics and laboratory findings and outcomes of COVID-19 and non-COVID-19 patients with pneumonia in the inception and validation cohorts.
Table 1. Clinical characteristics and laboratory findings and outcomes of COVID-19 and non-COVID-19 patients with pneumonia in the inception and validation cohorts.
Inception CohortAll
(n = 156)
non-COVID-19
(n = 115)
COVID-19
(n = 41)
Age, median (IQR)—years81 (73–88)82 (73–88)80 (74–86)
Female sex—no. (%)69 (44)49 (43)20 (49)
Charlson Comorbidity Index3 (1–4)3 (1–4)3 (2–5)
CURB-653 (2–3)3 (2–4)3 (2–3)
10-day mortality, no. (%)26 (17)20 (17)6 (15)
30-day mortality, no. (%)49 (31)30 (26)19 (46) *
C-reactive protein, mg/L79 (34–165)78 (37–163)79 (20–167)
Creatinine0.98 (0.79–1.45)0.99 (0.79–1.45)0.98 (0.80–1.34)
Bilirubine0.84 (0.57–1.23)0.93 (0.60–1.29)0.73 (0.46–1.06)
Hemoglobin, g/dL11.7 (10.4–13.2)11.7 (10.5–13.0)12.3 (10.5–13.9)
Platelet count, per mm3224,000 (178,000–304,000)218,000 (172,000–302,000)242,000 (184,000–321,000)
White-cell count, per mm39485 (7155–12,978)9550 (7340–12,770)9120 (6260–14,310)
Neutrophil count, per mm37159 (5078–10,167)7146 (5197–10212)7366 (4919–9673)
Lymphocyte count, per mm31044 (659–1656)1138 (682–1848)914 (615–1216) *
Monocyte count, per mm3683 (465–916)762 (509–1035)524 (299–652) *
Eosinophil count, per mm319 (0–108)43 (0–135)0 (0–2) *
Basophil count, per mm331 (12–48)38 (21–51)12 (7–29) *
Validation CohortAll
(n= 280)
non-COVID-19
(n= 180)
COVID-19
(n= 100)
Age, median (IQR)—years80 (71–87)81 (74–89)76 (63–85) *
Female sex—no. (%)124 (44)84 (47)40 (40)
Charlson Comorbidity Index2 (1–3) 2 (1–3)2 (1–3)
CURB-652 (1–3) 2 (1–3)2 (1–3)
10-day mortality, no. (%)24 (9)17 (9)7 (7)
30-day mortality, no. (%)62 (22) 41 (23)21 (21)
C-reactive protein, mg/L72 (27–140)87 (26–176)60 (31–116)
Creatinine, mg/dL0.99 (0.76–1.36)0.98 (0.74–1.34)1.01 (0.80–1.37)
Bilirubine, mg/dL0.69 (0.52–1.08) 0.78 (0.55–1.18)0.63 (0.49–0.81) *
Hemoglobin, g/dL12.3 (10.5–13.9)11.8 (10.2–13.2)13.5 (11.6–14.9) *
Platelet count, per mm3217,000 (163,000–286,000)230,000 (167,000–286,000)201,000 (158,000–284,000)
White-cell count, per mm38780 (6795–12,635)9225 (6875–13,155)8540 (5390–11,410)
Neutrophil count, per mm37169 (4796–10,651)7107 (4988–11,145)6918 (4192–9457)
Lymphocyte count, per mm3957 (569–1359) 1084 (670–1477)740 (490–1036) *
Monocyte count, per mm3592 (352–836) 681 (450–940)474 (250–628) *
Eosinophil count, per mm30 (0–70) 23 (0–133)0 (0–0) *
Basophil count, per mm320 (10–39) 31 (14–51)11 (6–19) *
, p < 0.05; validation vs. inception cohort. *, p < 0.05; COVID-19 vs. non-COVID-19.
Table 2. Generation of optimal cutoff in inception, validation, and combined cohorts.
Table 2. Generation of optimal cutoff in inception, validation, and combined cohorts.
Positive/NegativeAUCCI 95%pCuoffs
BasophilsInception41/1150.770.68–0.86<0.000117.5
Validation100/1800.750.70–0.81<0.000117.5
All141/2950.760.72–0.81<0.000117.5
EosinophilsInception41/1150.770.68–0.84<0.000110.5
Validation100/1800.770.72–0.83<0.000110.5
All141/2950.770.73–0.82<0.000110.5
Table 3. Diagnostic performance of eosinophils and basophils below cutoff in the diagnosis of COVID-19 pneumonia.
Table 3. Diagnostic performance of eosinophils and basophils below cutoff in the diagnosis of COVID-19 pneumonia.
SensitivitySpecificityPPVNPVPLRNLR
Basophils ≤ 17 cells/mm3Inception68 (52–82)84 (76–90)61 (49–71)88 (83–92)4.36 (2.72–7.00)0.38 (0.24–0.59)
Validation73 (63–81)72 (65–78)59 (52–65)83 (77–87)2.58 (1.98–3.35)0.38 (0.27–0.53)
All72 (63–79)77 (71–81)59 (54–65)85 (81–88) a,b3.06 (2.43–3.86)0.37 (0.28–0.48)
Eosinophils ≤ 10 cells/mm3Inception81 (65–91)70 (60–78)49 (41–56)91 (84–95)2.72 (2.00–3.71)0.25 (0.12–0.49)
Validation92(85–97)61 (54–68)57 (52–61)93 (88–96)2.37 (1.95–2.87)0.13 (0.07–0.26)
All89(83–94)64 (59–70)55 (51–59)93 (89–95) a,c2.51(2.13–2.96)0.37(0.28–0.48)
Basophils ≤ 17 cells/mm3
and
Eosinophils ≤ 10 cells/mm3
Inception56 (40–72)87 (79–93)61 (47–73)85 (80–89)4.30 (2.50–4.71)0.50 (0.35–0.72)
Validation66 (56–75)80 (73–86)65 (57–72)81 (76–85)3.30 (2.39–4.56)0.42 (0.32–0.56)
All63 (55–71)83 (78–87)64 (57–70) a82 (79–85) d3.65 (2.76–4.83)0.45 (0.36–0.56)
Basophils ≤ 17 cells/mm3
and/or
Eosinophils ≤ 10 cells/mm3
Inception95 (84–99)67 (58–75)51 (44–57)98 (91–99)2.88 (2.20–3.77)0.07 (0.02–0.28)
Validation99 (95–100)53 (45–60)54 (50–58)99 (93–100)2.10 (1.79–2.45)0.02 (0.00–0.13)
All98 (94–100)58 (53–64)53 (50–56) a98 (94–99) b,c,d2.35 (2.05–2.69)0.04 (0.01–0.11)
PPV, positive predictive value; NPV, negative predictive value; PLR, positive likelihood ratio value; NLR, negative likelihood ratio. The diagnostic performance of the different tests was compared in the combined cohort (All). Sensitivity and specificity of all tests are significantly different (p ≤ 0.001) according the McNemar test. PPV and NPV were compared using the “N-1” Chi-squared test; a,b,c or d, values sharing the same superscript letter were significantly different (p < 0.05).
Table 4. Diagnostic Performance of B-lines on lung ultrasound alone or combined with eosinophils and basophils below cutoff in the diagnosis of COVID-19 pneumonia.
Table 4. Diagnostic Performance of B-lines on lung ultrasound alone or combined with eosinophils and basophils below cutoff in the diagnosis of COVID-19 pneumonia.
SensitivitySpecificityPPVNPVPLRNLR
B-lines86(77–92) a,b,c,d44(36–51) a,b,c,d44(41–48) a,b,c,d85(78–91) a1.52(1.30–1.78)0.33(0.19–0.56)
Basophils ≤ 17 cells/mm3 and/or Eosinophils ≤ 10 cells/mm399(94–100) a,e,f58(50–65) a,e55(51–59) a,e99(93–100) a,b,c2.33(1.95–2.77)0.02(0.00–0.13)
Basophils ≤ 17 cells/mm3 and Eosinophils ≤ 10 cells/mm369(59–79) b,e,g79(73–85) b,f64(56–71) b83(78–87) b3.35(2.43–4.62)0.39(0.28–0.53)
B-lines and Basophils ≤ 17 cells/mm3 and Eosinophils ≤ 10 cells/mm360(50–71) c,f,h88(82–92) c72(63–80) c,e,f81(77–85) c5.01(3.24–7.73)0.45(0.35–0.58)
B-lines and/or Basophils ≤ 17 cells/mm3 and/or Eosinophils ≤ 10 cells/mm3100(96–100) d,g,h57(50–65) d,e,f55(51–59) d,f1002.35(1.98–2.79)0
Data of patients from the inception and validation cohorts who underwent lung ultrasound (91/174, positive/negative patients). Sensitivity and specificity were compared using the McNemar test. PPV and NPV were compared using the “N-1” Chi-squared test; a,b,c,d,e,f,g or h, values sharing the same superscript letter were significantly different (p < 0.05).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Biolo, G.; Fiotti, N.; Cominotto, F.; Di Girolamo, F.G.; Panizon, E.; Altamura, N.; Casarsa, C.; Pipoli, A.; Giordano, M.; Torelli, L.; et al. Ruling Out Coronavirus Disease 2019 in Patients with Pneumonia: The Role of Blood Cell Count and Lung Ultrasound. J. Clin. Med. 2021, 10, 3481. https://doi.org/10.3390/jcm10163481

AMA Style

Biolo G, Fiotti N, Cominotto F, Di Girolamo FG, Panizon E, Altamura N, Casarsa C, Pipoli A, Giordano M, Torelli L, et al. Ruling Out Coronavirus Disease 2019 in Patients with Pneumonia: The Role of Blood Cell Count and Lung Ultrasound. Journal of Clinical Medicine. 2021; 10(16):3481. https://doi.org/10.3390/jcm10163481

Chicago/Turabian Style

Biolo, Gianni, Nicola Fiotti, Franco Cominotto, Filippo Giorgio Di Girolamo, Emiliano Panizon, Nicola Altamura, Chiara Casarsa, Alessandro Pipoli, Mauro Giordano, Lucio Torelli, and et al. 2021. "Ruling Out Coronavirus Disease 2019 in Patients with Pneumonia: The Role of Blood Cell Count and Lung Ultrasound" Journal of Clinical Medicine 10, no. 16: 3481. https://doi.org/10.3390/jcm10163481

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop