Abstract
Rationale Prognostic accuracy of the quick sequential organ failure assessment (qSOFA) and CRB-65 (confusion, respiratory rate, blood pressure and age (≥65 years)) risk scores have not been widely evaluated in patients with SARS-CoV-2-positive compared to SARS-CoV-2-negative community-acquired pneumonia (CAP). The aim of the present study was to validate the qSOFA(-65) and CRB-65 scores in a large cohort of SARS-CoV-2-positive and SARS-CoV-2-negative CAP patients.
Methods We included all cases with CAP hospitalised in 2020 from the German nationwide mandatory quality assurance programme and compared cases with SARS-CoV-2 infection to cases without. We excluded cases with unclear SARS-CoV-2 infection state, transferred to another hospital or on mechanical ventilation during admission. Predefined outcomes were hospital mortality and need for mechanical ventilation.
Results Among 68 594 SARS-CoV-2-positive patients, hospital mortality (22.7%) and mechanical ventilation (14.9%) were significantly higher when compared to 167 880 SARS-CoV-2-negative patients (15.7% and 9.2%, respectively). All CRB-65 and qSOFA criteria were associated with both outcomes, and age dominated mortality prediction in SARS-CoV-2 (risk ratio >9). Scores including the age criterion had higher area under the curve (AUCs) for mortality in SARS-CoV-2-positive patients (e.g. CRB-65 AUC 0.76) compared to SARS-CoV-2 negative patients (AUC 0.68), and negative predictive value was highest for qSOFA-65=0 (98.2%). Sensitivity for mechanical ventilation prediction was poor with all scores (AUCs 0.59–0.62), and negative predictive values were insufficient (qSOFA-65=0 missed 1490 out of 10 198 patients (∼15%) with mechanical ventilation). Results were similar when excluding frail and palliative patients.
Conclusions Hospital mortality and mechanical ventilation rates were higher in SARS-CoV-2-positive than SARS-CoV-2-negative CAP. For SARS-CoV-2-positive CAP, the CRB-65 and qSOFA-65 scores showed adequate prediction of mortality but not of mechanical ventilation.
Abstract
Mortality and mechanical ventilation rates are higher in SARS-CoV-2-positive compared to SARS-CoV-2-negative CAP. Only scores with an age criterion (CRB-65 and qSOFA-65) provide adequate mortality prediction but sensitivity is low for mechanical ventilation. https://bit.ly/41lnByb
Introduction
Since 2019, the COVID-19 pandemic has caused an extraordinary strain on global health systems. Despite changing epidemiology, SARS-CoV-2 remains one potential pathogen among others causing community-acquired pneumonia (CAP). As with bacterial CAP, SARS-CoV-2 pneumonia severity is highly variable, with potential disease courses from mild to critical disease. Thus, adequate risk scoring is essential for initial assessment.
Although risk stratification in SARS-CoV-2-positive CAP has been studied recently, it remains poorly defined, and most guidelines do not recommend specific risk scores for COVID-19 [1–3]. A systematic review updated in July 2022 found 265 prognostic models for mortality and 84 for progression to severe or critical course. However, many of these studies were at high or unclear risk of bias [4]. The best predictive performance had been shown with specific, but often complicated, risk scores, like the International Severe Acute Respiratory Infection Consortium Clinical Characterisation Protocol (ISARIC4C) scores involving 8–11 variables [4–6]. Because CAP aetiology is often not established at first presentation, most patients presenting with CAP in busy emergency departments will initially be evaluated according to CAP algorithms. The current German CAP guideline [7] recommends initial risk screening including the CRB-65 (confusion, respiratory rate, blood pressure and age ≥65 years) score and evaluation of potential sepsis with the quick sequential organ failure assessment (qSOFA). Both scores have only been partially validated for initial assessment of SARS-CoV-2-positive CAP, with conflicting results in recent studies [5, 8–12]. Additionally, prediction models show heterogeneous results according to the specified outcome (mortality versus severe disease/mechanical ventilation).
The aim of the present study was to evaluate the accuracy of the risk scores qSOFA, qSOFA-65 and CRB-65 for predicting hospital mortality or mechanical ventilation in a large, representative population of hospitalised SARS-CoV-2-positive and SARS-CoV-2-negative CAP patients.
Methods
Database
We used the database of the German nationwide mandatory quality assurance programme. All hospitals in Germany are required to document clinical routine data from all hospitalised CAP cases in a prespecified electronic data sheet, as described previously [13]. CAP cases are identified by International Classification of Diseases (ICD) codes as listed elsewhere [13]; patients with nosocomial pneumonia and with immunosuppression are excluded. Cases admitted to the emergency department but not hospitalised are not recorded. The study was approved by the ethics committee of TU Dresden (EK 609122017).
Study population
As a reference, we included all hospitalised patients with CAP with or without SARS-CoV-2 infection from the study period of 1 January 2020 to 31 December 2020. SARS-CoV-2 infection was defined as documented code U07.1 (SARS-CoV-2 confirmed by laboratory test) together with the qualifying pneumonia code. Patients with unclear SARS-CoV-2 status, defined as SARS-CoV-2 infection with missing laboratory confirmation (U07.2) or post-COVID conditions (U07.3–U07.5) were excluded from all analyses.
For our main cohorts, we further excluded all patients who had been transferred to another hospital (no outcome parameter) or were on invasive ventilation on admission (no meaningful risk scoring). Applying those eligibility criteria, we formed two main CAP cohorts from SARS-CoV-2-positive and SARS-CoV-2-negative patients.
For sensitivity analyses, we further excluded patients who were chronically bedridden, admitted from a nursing home or those with a documented palliative treatment goal in order to avoid potential confounding by treatment restrictions (reduced cohort 1). A second sensitivity cohort was based on excluding patients with either missing documented respiratory rate on admission or admission from another hospital (reduced cohort 2).
Score definition and outcome parameters
The CRB-65 and qSOFA scores were calculated from the first set of parameter measurements after hospital admission as described previously [14, 15]. For better comparison, we additionally calculated the CRB to the CRB-65 score and the qSOFA-65 to the qSOFA score by eliminating/involving the age criterion ≥65 years.
Predefined outcomes were all-cause hospital mortality, mechanical ventilation (noninvasive or invasive) during the hospital stay and the combination of hospital mortality and/or mechanical ventilation.
Statistical analysis
The characteristics of the patients are represented as absolute and relative frequencies or as median with first and third quartiles. For univariable analyses, continues variables were compared using the Mann–Whitney U-test and categorical variables using the Fisher's exact test. Multivariable results are based on Poisson regressions with a log link function. In the primary multivariable model, we included the CRB-65 and qSOFA criteria as independent variables. The results are represented as risk ratios with 95% confidence intervals. To evaluate diagnostic qualities, we used area under the curve (AUC), sensitivity, specificity, positive/negative predictive values (PPV/NPVs) and positive/negative likelihood ratios with their 95% confidence intervals. For sensitivity analyses we added an extended multivariable model that included the variables sex, admission from nursing home, chronically bedridden and palliative treatment goal.
We applied a two-sided significance level of 0.05. For the statistical analyses, we used R (version 3.6.2) and the R package pROC (https://www.r-project.org/).
Results
Patient characteristics and outcomes
Overall, 280 100 patients with CAP were included in the database in 2020. After application of the predefined exclusion criteria, 167 880 SARS-CoV-2-negative CAP cases (71%) and 68 594 SARS-CoV-2-positive CAP cases (29%) were left in the main cohort (figure 1). For sensitivity analyses, we excluded a further 74 422 patients who were chronically bedridden, living in a nursing home or who had a documented palliative treatment goal (reduced cohort 1) and 15 428 patients with no documented respiratory rate or admission from another hospital (reduced cohort 2).
When comparing SARS-CoV-2-positive and SARS-CoV-2-negative CAP in the main cohort, hospital mortality was 22.7% (n=15 540 out of 68 594) in SARS-CoV-2-positive patients and 15.7% (n=26 309 out of 167 880) in SARS-CoV-2-negative patients (p<0.001) (supplementary table S1). The mechanical ventilation rate was 14.9% (n=10 198 out of 68 594) in SARS-CoV-2-positive patients versus 9.2% (n=15 587 out of 167 880) in SARS-CoV-2 negative patients (p<0.001). Of the SARS-CoV-2-positive patients with mechanical ventilation, 5336 (52%) received only noninvasive, 1974 (19%) only invasive and 2888 (28%) both noninvasive and invasive ventilation. After excluding frail or palliative patients (reduced cohort 1), hospital mortality and ventilation rate decreased in both groups but remained significantly higher in the SARS-CoV-2 group (hospital mortality 8.8% versus 6.1%, p<0.001; ventilation rate 14% versus 8.3%, p<0.001) (supplementary tables S2, S3). Demographic data and score parameters of the main cohort of SARS-CoV-2 cases are provided in table 1 and those of CAP without SARS-CoV-2 infection in supplementary table S4.
Score prediction in the main cohort
All criteria of the CRB-65 and qSOFA scores were associated with the prognosis of SARS-CoV-2-positive CAP in both univariable and multivariable analyses (tables 1 and 2). The corresponding values are shown in supplementary figure S1. With a risk ratio of 9.3 (95% CI 8.7–10), age was the strongest predictor of hospital mortality, while respiratory rate was the strongest predictor for mechanical ventilation (risk ratio 1.7, 95% CI 1.6–1.8) after multivariable analyses.
Prognostic performance measures of the scores to predict hospital mortality or mechanical ventilation in SARS-CoV-2-positive CAP are depicted in tables 3 and 4, respectively. While the CRB-65 provided better prognostic accuracy in predicting mortality than the qSOFA (AUC 0.76, 95% CI 0.76–0.77, versus AUC 0.65, 95% CI 0.64–0.65), after including the age criterion in the qSOFA, the resulting qSOFA-65 score provided a similar accuracy (AUC 0.76, 95% CI 0.75–0.76). For scores without the age criterion, NPVs for qSOFA/CRB=0 were only 83.5% and 84.7% for mortality prediction, respectively. Only the inclusion of age enabled high negative prediction concerning mortality (NPVs >97%).
When comparing mortality prediction of the SARS-CoV-2-positive with the SARS-CoV-2-negative cohort, prognostic accuracies of the scores including the age criterion were higher in SARS-CoV-2-positive CAP (AUCs: CRB-65 0.76, qSOFA 0.65, qSOFA-65 0.76) than in SARS-CoV-2-negative CAP (AUCs: CRB-65 0.68, qSOFA 0.65, qSOFA-65 0.69; table 3 and supplementary table S5). This was associated with a higher predictive dominance of the age criterion in the SARS-CoV-2-positive cohort (risk ratio 9.3, table 2) compared to the SARS-CoV-2-negative cohort (risk ratio 2.99, supplementary table S6) in multivariable analyses.
For prediction of mechanical ventilation, while all single score parameters were again associated with the outcome, prognostic accuracies of all scores were lower compared to mortality prediction. Among all evaluated scores, qSOFA-65 provided the best prediction with an AUC of 0.62 (table 4). No score achieved sufficient NPVs because even with a qSOFA-65 of 0, the requirement for mechanical ventilation would have been missed in 8.4% or 1490 out of 17 761 patients, corresponding to 15% of all patients with mechanical ventilation in the disease course. By contrast, patients with at least one score criterion except age (CRB or qSOFA >0) had a high risk for mechanical ventilation with positive predictive values of >20%.
Performance measures of all scores for predicting mechanical ventilation in the SARS-CoV-2-negative cohort are provided in supplementary table S7. When comparing mechanical ventilation prediction for SARS-CoV-2-positive with SARS-CoV-2-negative CAP, prognostic accuracies of scores including the age criterion were similar (AUCs: qSOFA-65 0.62 versus 0.61; CRB-65 0.60 versus 0.59, respectively). Higher NPVs for scores of 0 were achieved in the SARS-CoV-2-negative cohort (NPVs 93–94% versus 88–92% in the SARS-CoV-2-positive cohort).
The corresponding values for the combined end-point mortality and/or mechanical ventilation in the SARS-CoV-2-positive and SARS-CoV-2-negative CAP cohorts are shown in supplementary tables S8 and S9.
Sensitivity analyses (score prediction in the reduced cohorts)
After exclusion of frail and palliative patients, the mortality rate in the SARS-CoV-2-positive reduced cohort 1 decreased considerably (4512 out of 51 015, 8.8%) while rates of mechanical ventilation remained high (7119 out of 51 015, 14.0%; supplementary table S2). After multivariable analyses, the risk ratio of age for mortality was considerably decreased compared to the main cohort (risk ratio 6.9, 95% CI 6.2–7.5, versus risk ratio 9.3, 95% CI 8.7–10; supplementary table S10). In contrast, there was no major change in the risk ratio of the age criterion for predicting mechanical ventilation in reduced cohort 1 compared to the main cohort (risk ratio 1.3, 95% CI 1.2–1.3, versus risk ratio 1.2, 95% CI 1.1–1.2).
Overall, AUCs for all scores regarding mortality and mechanical ventilation were similar to the results from the main cohort (e.g. CRB-65 for mortality 0.75 versus 0.76, qSOFA 0.63 versus 0.65; supplementary tables S11–S13). This also applied to the SARS-CoV-2-negative cohort (supplementary tables S14–S16).
As expected, NPVs of all scores for mortality prediction improved after excluding patients with potential treatment limitations in reduced cohort 1 (e.g. qSOFA score=0, NPV 94.1% versus 84.7%). Nonetheless, 5.9% of patients (1971 out of 33 643) died without any positive criterion detected by the qSOFA. However, after adding the age criterion, a qSOFA-65 of 0 achieved the highest NPV of all scores with 98.8% (200 out of 17 309 deceased patients missed).
For mechanical ventilation prediction, similarly no risk score in reduced cohort 1 was able to achieve high NPVs (risk scores=0 with NPVs between 89% and 92%).
For reduced cohort 2, after excluding all cases with missing respiratory rate on admission or transfer from another hospital, predictive accuracies of all scores showed nearly identical values to the main cohort (e.g. AUC for mortality prediction: CRB-65 0.76; qSOFA 0.65; data not shown).
Discussion
The current evaluation of established and commonly used risk scores including the CRB-65 and the qSOFA in our population-based representative cohort of all 2020 hospitalised cases with SARS-CoV-2-positive and SARS-CoV-2-negative CAP in Germany generated the following main results: 1) Both hospital mortality and mechanical ventilation rates were higher in SARS-CoV-2-positive than in SARS-CoV-2-negative CAP patients; 2) for mortality, especially in SARS-CoV-2-positive CAP, prediction was largely dominated by the age criterion and only scores including age showed sufficient screening sensitivity; 3) sensitivity of the qSOFA criteria was slightly superior than the CRB criteria; and 4) sensitivity for predicting mechanical ventilation was poor with all evaluated scores, because even within the highest sensitivity constellation (qSOFA-65=0), 8% of patients required mechanical ventilation and 15% of all patients requiring mechanical ventilation would have been missed.
The study presents real-world data of virtually all hospitalised patients with an ICD-based diagnosis of CAP with or without documented SARS-CoV-2 infection in Germany in 2020. Thus, the main strength of our analysis lies in its population-based large sample avoiding the bias inherent in selected study populations. Therefore, we believe that our data are of high clinical relevance and external validity.
Patients with CAP and SARS-CoV-2 infection had a significantly higher risk of dying (22.7% versus 15.7%) or mechanical ventilation (14.9% versus 9.2%) than those with CAP without SARS-CoV-2 infection. Such differences have also been described by others [16, 17]; however, we found surprisingly few data directly comparing CAP patients with and without SARS-CoV-2 infection hospitalised within the same period. Mortality rate in our cohort of patients hospitalised with CAP and COVID-19 was high compared to previous prospective studies [8, 16, 18], but closely resembled that of other European population-based cohorts for patients hospitalised with CAP [19–21] and COVID-19 [9, 10, 12, 22]. Additionally, the median age of our cohorts was high at 72 years in the SARS-CoV-2-positive cohort and 78 years in the SARS-CoV-2-negative cohort. Of note, older age defined by the CRB-65 cut-off of 65 years was associated with a much higher relative mortality risk in SARS-CoV-2-positive CAP compared to SARS-CoV-2-negative CAP in multivariable analyses, underscoring the dominating prognostic relevance of age described in COVID-19 [22, 23]. Pneumonia-related confusion showed a stronger association with mechanical ventilation and mortality in SARS-CoV-2-negative patients (risk ratio 2.2 and 1.9) compared to SARS-CoV-2-positive patients (risk ratio 1.5 and 1.6), probably reflecting the lower rate of extrapulmonary sepsis in patients with COVID-19 compared to bacterial CAP [16].
In the current epidemiologic situation, SARS-CoV-2 needs to be considered as one possible aetiological pathogen among many others during clinical evaluation of patients presenting with CAP. Thus, a universally applicable risk stratification approach for CAP patients is essential. It seems questionable whether complex COVID-19-specific risk scores like the ISARIC4C models [5, 6] represent feasible screening tools for initial all-cause CAP evaluation in busy emergency departments. By contrast, the evaluated risk scores CRB-65 and qSOFA have been widely implemented for initial CAP evaluation [7, 24] and are clinically relevant screening tools to be validated in SARS-CoV-2-positive CAP.
As described previously in SARS-CoV-2-negative CAP [20, 25], the age criterion is also necessary to provide a sensitive mortality prediction with both the CRB and the qSOFA criteria in SARS-CoV-2-positive CAP. The resulting CRB-65 and qSOFA-65 scores provided similar prognostic accuracies with an adequate NPV for hospital mortality in our SARS-CoV-2-positive cohort (AUC 0.76 and NPV 98% each). Moreover, they showed superior accuracy for predicting hospital mortality compared to the SARS-CoV-2-negative cohort (AUC 0.68 and 0.69). When compared to previously published cohorts including >2000 patients with SARS-CoV-2-positive CAP, our mortality predictions by the qSOFA (AUC 0.65) and the CRB-65 (AUC 0.76) were comparable (published AUC range qSOFA 0.55–0.63; CRB-65 0.68–0.75) [5, 8–11], with the exception of one cohort showing a higher AUC for the qSOFA of 0.73 [12]. We found no published data for the qSOFA-65 in SARS-CoV-2-positive CAP. The initially recommended qSOFA score cut-off of ≥2 [15] did not show sufficient sensitivity (15%) for hospital mortality in our cohort, as previously also demonstrated for SARS-CoV-2-negative CAP [20, 26] and sepsis [27].
For predicting mechanical ventilation in the SARS-CoV-2-positive CAP cohort, there was no sufficient sensitivity for any evaluated risk score. With an AUC of 0.62, qSOFA-65 provided the best prediction, but still would have missed 15% of all patients requiring mechanical ventilation with the most sensitive cut-off of 0. Compared to the SARS-CoV-2-negative cohort, prognostic accuracies, especially of scores without the age criterion, were even lower in SARS-CoV-2-positive patients (e.g. qSOFA AUC for SARS-CoV-2-positive 0.61, for SARS-CoV-2-negative 0.64). This supplements previous data showing poor prediction of mechanical ventilation and organ support by the qSOFA or the CRB-65 in COVID-19 [8, 10, 12]. Therefore, similar to SARS-CoV-2-negative CAP [7, 24, 28], additional risk assessment tools seem necessary for organ failure prediction in SARS-CoV-2-positive CAP. Oxygenation measurement repeatedly has been shown to provide independent prediction of mechanical ventilation in COVID-19 [8, 23, 29, 30]. For example, a respiratory score including respiratory rate, respiratory saturation and oxygen flow rate in one study accurately predicted early mechanical ventilation for COVID-19 when compared to the qSOFA (AUC 0.81 versus 0.59) [30]. Additionally, current CAP guidelines recommend the American Thoracic Society/Infectious Diseases Society for America minor criteria for organ failure evaluation [7, 28], which recently have also been validated for SARS-CoV-2-positive patients [29].
The following limitations of our study should be mentioned. The interpretation of our findings is limited by the lack of data from other important outcomes such as intensive care unit admission, septic shock or follow-up data to calculate 30-day mortality. Additionally, we cannot provide data on microbiology including co-infections, comorbidities, treatment and cause of death. Patients with a primary hospital discharge diagnosis of acute respiratory distress syndrome are not included in the database because the ICD-based filter for the quality assurance programme does not identify them. 3.3% of cases had no documented respiratory rate on admission, which might have biased the prognostic accuracies of the scores. However, sensitivity analysis after excluding these cases did not relevantly alter the results. Because measurements are performed during routine clinical work and collection of the data is mandatory, data quality may not be comparable with trial data. Furthermore, the quality assurance programme lacks data for calculating additional risk scores like CURB-65, the pneumonia severity index, minor criteria, or data on oxygenation and COVID-specific risk scores.
We evaluated data of SARS-CoV-2-positive patients hospitalised with CAP in Germany during 2020. Whether our findings are applicable for SARS-CoV-2-positive CAP in later stages of the pandemic affected by the continuous evolution of the virus, host immunity and clinical management of COVID-19 is unknown and subject to further study.
However, the coverage of virtually all hospitalised patients with SARS-CoV-2-positive and SARS-CoV-2-negative CAP in Germany reflecting real-world management and minimising any selection bias and the completeness of the outcome parameters as well as the other parameters of the qSOFA and CRB-65 scores make the evaluated database unique for evaluating SARS-CoV-2-associated hospitalised CAP.
Conclusion
Hospital mortality and mechanical ventilation rate in Germany in 2020 was higher for patients with SARS-CoV-2-positive compared to SARS-CoV-2-negative hospitalised CAP. For SARS-CoV-2-positive CAP, the qSOFA had low sensitivity for hospital mortality prediction, but scores including the age criterion like the CRB-65 and the qSOFA-65 showed an acceptable accuracy, which was even higher than that for SARS-CoV-2-negative CAP. For predicting mechanical ventilation, all evaluated scores showed insufficient sensitivity.
Supplementary material
Supplementary Material
Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.
Supplementary material 00168-2023.SUPPLEMENT
Acknowledgements
We used data from the quality assurance programme under section 136 SGB V of the Federal Joint Committee (G-BA) of Germany. The authors thank all physicians and technicians working for and with the quality control initiative for community-acquired pneumonia and the Institute for Quality Assurance and Transparency in Healthcare (Institut für Qualitätssicherung und Transparenz im Gesundheitswesen (IQTiG)).
Footnotes
Provenance: Submitted article, peer reviewed.
Author contributions: Conception, hypothesis and design of the study: T. Richter, F. Tesch, J. Schmitt, D. Koschel and M. Kolditz; acquisition of data: T. Richter, F. Tesch and M. Kolditz; analysis and interpretation: T. Richter, F. Tesch and M. Kolditz; substantial involvement in the writing and/or revision of the article: T. Richter, F. Tesch, J. Schmitt, D. Koschel and M. Kolditz; final approval of the version to be published: T. Richter, F. Tesch, J. Schmitt, D. Koschel and M. Kolditz.
Conflict of interest: The authors have nothing to disclose.
- Received March 15, 2023.
- Accepted April 5, 2023.
- Copyright ©The authors 2023
This version is distributed under the terms of the Creative Commons Attribution Non-Commercial Licence 4.0. For commercial reproduction rights and permissions contact permissions{at}ersnet.org