In this retrospective study, all patients with a positive COVID-19 diagnosis according to any one of the following diagnostic criteria were included in the present study: (1) Respiratory tract or blood specimens were positive for severe acute respiratory syndrome coronavirus 2 nucleic acids by real-time fluorescence reverse transcription-polymerase chain reaction; (2) Genetic sequencing of respiratory tract or blood specimens revealed that the material had high homology with severe acute respiratory syndrome coronavirus 2; and (3) Suspected cases with imaging features of pneumonia consistent with that described in the “Diagnosis and treatment plan for pneumonia infected with new coronavirus [trial version 5]” issued by the National Health Commission of China (this standard was limited to Hubei Province).

Consecutive patients diagnosed with COVID-19 in the Wuhan Third Hospital between December 2019 and March 2020 were included in the training cohort. There were 681 patients in the training cohort. The predictive model was built and internally validated using the above data. The features evaluated for the predictive model included baseline demographics and laboratory data of each patient obtained at their first examination after admission. All blood and urinary samples were processed within 2 h of collection. Figure 1 presents a flowchart illustrating the patients in the training and validation cohorts. The data for each cohort was obtained and analyzed retrospectively. Cases in need of ICU admission were defined according to the following criteria[19]: (1) Respiratory rate ≥ 30 times/min; (2) Pulse oximeter oxygen saturation ≤ 93% at rest; and (3) Partial pressure of arterial oxygen/fraction of inspired oxygen ≤ 300 mmHg.

Open in New Tab   Full Size Figure   Download Figure

Figure 1 The flowchart illustrating the patients in the training and validation cohorts. The data for each cohort was obtained and analyzed retrospectively. A: Patients in the training cohort; B: Patients in the validation cohort. COVID-19: Coronavirus disease 2019; ICU: Intensive care unit.

The proposed algorithm used in this study was built on the basis of the random forest (RF) algorithm[20], with modifications made to improve the selection of features (Figure 2). The number of trees was set to 480, and the number of variables selected at each split was set to 4.

Open in New Tab   Full Size Figure   Download Figure

Figure 2 Feature selection. Thirteen predictors were selected in the information gain algorithm and were to train the random forest (RF) model. Six predictors with P < 0.05 were selected in the multivariate logistic regression (LR) analysis and were used to train the LR model. COVID-19: Coronavirus disease 2019; LASSO: Least absolute shrinkage and selection operator.

Feature selection consisted of the following two steps. First, the least absolute shrinkage and selection operator logistic regression (LR) and univariate LR were used to determine which variables were associated with disease prognosis. We performed a tenfold cross-validation of the training set to calculate the weight of least absolute shrinkage and selection operator penalty. Furthermore, physicians’ knowledge, together with previously published predictors significantly associated with COVID-19 severity[21] were also used to guide feature selection. Then, since each feature’s relative rank could reflect its relative significance[22-24], the information gain algorithm based on entropy and out-of-bag error assessment were used to screen the selected variables for training the RF model. Furthermore, we carried out stepwise multivariate regression analysis to screen the selected variables for training the LR model. We used variables with a P < 0.05 to build the model.

A tree-based ensembled machine learning algorithm, RF, was used to build a risk prediction model based on 13 selected variables. GridsearchCV was performed to search the best parameter for the optimal model. A nomogram for the LR model was built based on six selected variables, which were then screened by multivariate logistic regression analysis. The models were developed in Python version 3.6.5. The reporting followed the TRIPOD statements[25].

Here, we comparatively assessed the predictive performances of scores yielded by the present and conventionally used models, as described below:

Discrimination: To evaluate discrimination, we used the area under the receiver operating characteristic curve, accuracy, specificity, sensitivity, box plots of predicted probabilities of ICU admission and corresponding discrimination slopes, defined as the differences between the mean predicted risks for ICU admission.

Calibration: The conventional Hosmer-Lemeshow statistic was avoided due to its shortcomings[26,27]. Calibration of the predictive model was assessed by the visual representation of the relationship between the predicted and observed values[28]. We constructed calibration curves by plotting the predicted risk of ICU admission divided into 20 groups based on the model risk score against the observed ICU admission.

The net reclassification index (NRI), which has been devised to overcome the limitations of usual discrimination and calibration measures, was computed to compare our proposed algorithm to the other scores[29]. The NRI comparing risk score A of ICU admission to score B was defined as two times the difference between the proportion of no ICU admission and ICU admission groups, respectively, which deemed the risk of ICU admission to be higher according to score A than according to score B[30]. Positive values of the NRI indicated that score A had better discriminative ability than score B, whereas negative values indicated the opposite.

A decision curve analysis (DCA) was used to estimate the clinical usefulness and net benefit of the intervention[31]. The decision curve[32] is a novel and clever graphical device used to assess the potential population impact of adopting a risk prediction instrument into clinical practice. It is grounded in a decision-theoretical framework that accounts for both the benefits of intervention and the costs of intervention to a patient who cannot benefit from the intervention.

A completely independent dataset was then used to externally validate the predictive performance of the algorithm developed herein. To this end, we randomly collected patients with COVID-19 that had been clinically confirmed by reverse transcription-polymerase chain reaction between January 19, 2020 and March 14, 2020, in Shenzhen Third People’s Hospital, which is a tertiary-care teaching hospital. Informed consents were obtained from all patients or from their families by telephone before their data were used in this study. All patient privacy data were protected under the confidentiality policy. Data were analyzed using the statistical software package R, version 3.4.3 (R Core Team, 2017) and EmpowerStats (X&Y solutions, Inc. Boston, Massachusetts).

The performance of the proposed predictive model was compared to other recently published models on the same external validation data. For convenience, the two published models were designated model A[33] and model B[34].

All participants’ baseline demographic and clinical characteristics were obtained at admission. Continuous variables were presented as means ± SD or medians (interquartile ranges), whereas categorical variables were presented using frequencies (percentages). Intergroup differences were analyzed with the χ² test, one-way analysis of variance and Kruskal-Wallis test for categorical variables, normal variables and continuous variables with skewed distribution, respectively. A P < 0.05 was considered statistically significant.

This study describes the development of an algorithm for the early prediction of ICU admission among COVID-19 patients at hospital admission. For developing the prediction model, we first used a training cohort consisting of 681 patients from Wuhan Third Hospital and analyzed their basic baseline demographic and laboratory data obtained at the first admission. We then calibrated the performance of this prediction tool using an entirely different sample set of 296 patients from Shenzhen Third People’s Hospital.

Table 1 presents a comparison of the baseline clinical and laboratory characteristics between the training and validation cohorts. The patients in the training cohort were older than those in the validation cohort (median age: 63.0 vs 49.0 years, P < 0.001), and the percentages of male gender were similar (49.6% vs 49.3%, P = 0.958). There was also some heterogeneity in laboratory results among the different patient groups. Table 1 presents all the patient characteristics.

We selected a total of 65 baseline clinical features for use in our prediction tool. Those with missing values were deleted (n = 18), and the remaining 47 features with complete data were used as potential predictors of critical illness requiring ICU admission. Twenty-three predictors with non-zero coefficients were selected in the least absolute shrinkage and selection operator LR model. Of these, those with P > 0.05 were excluded, and 19 predictors with P < 0.05 were selected for the univariate LR analysis (Table 2). After adjusting the model based on expert opinion, a total of 17 predictors were decided.

The information gain of each variable and its importance were calculated and ranked based on entropy and out-of-bag error. Figure 3 depicts the relative importance of each of the features. Variables were dropped from the bottom of the list, starting from the variable that was deemed least important and progressing in ascending order. The least classification error was obtained when gender, alanine transaminase, aspartate aminotransferase and white blood cell were dropped from the prediction model. Hence, these four features were removed. Figure 4 shows the relationship between the number of discarded variables and classification error. After all, a final of 13 predictors were selected. SHapley Additive exPlanations value was calculated to explain the output of the RF model (Figure 5). It connected optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions[35]. In the next step, six predictors [neutrophil-to-lymphocyte ratio (NLR), age, lactate dehydrogenase, creatinine, glucose and albumin] with P < 0.05 were selected to build the multivariate LR model (Table 2). The personalized nomogram was then used to show the probability of ICU admission (Figure 6).

Open in New Tab   Full Size Figure   Download Figure

Figure 3 Importance of the variables included in the predictive model for coronavirus disease 2019 events based on the random forest algorithm. ALT: Alanine transaminase; AST: Aspartate aminotransferase; CK: Creatine kinase; Cr: Creatinine; CRP: C-reactive protein; GLU: Glucose; LAC: Lactate; LDH: Lactate dehydrogenase; NLR: Neutrophil-to-lymphocyte ratio; PCT: Procalcitonin; PLT: Platelet; TBil: Total bilirubin; WBC: White blood cell.

Open in New Tab   Full Size Figure   Download Figure

Figure 4 Relationship between the number of discarded variables and classification error.

Open in New Tab   Full Size Figure   Download Figure

Figure 5 SHapley Additive exPlanations values of every feature used to train the random forest model for every sample. Each dot corresponds to an individual person in the study. The dot’s position on the X axis shows the impact that feature has on the model’s prediction for that person. The color represents the feature value (red high, blue low). This reveals for example that an older age increases the predicted intensive care unit admission probability. CK: Creatine kinase; Cr: Creatinine; CRP: C-reactive protein; GLU: Glucose; LAC: Lactate; LDH: Lactate dehydrogenase; NLR: Neutrophil-to-lymphocyte ratio; PCT: Procalcitonin; PLT: Platelet; TBil: Total bilirubin.

Open in New Tab   Full Size Figure   Download Figure

Figure 6 Nomogram of the logistic regression model to triage coronavirus disease 2019 patients. One patient had a total nomogram score of 155 points, and the probability of intensive care unit admission was 0.689. ¹Shows significance between 0.01 and 0.05; ²Shows significance between 0.001 and 0.01; ³Shows significance at a value of < 0.001. Cr: Creatinine; GLU: Glucose; LDH: Lactate dehydrogenase; LR: Logistic regression; NLR: Neutrophil-to-lymphocyte ratio.

After feature selection, two predictive models, LR and RF, were built on the basis of the selected variables. For internal validation, the area under the receiver operating characteristic curve for the RF model was found to be 0.94, which was higher than that for the LR model at 0.91 (P = 0.111, DeLong’s test). The accuracy, sensitivity and specificity for the RF model were 91%, 88% and 93%, respectively, higher than those for the LR model (87%, 82%, and 89%, respectively) (Figure 7).

Open in New Tab   Full Size Figure   Download Figure

Figure 7 Performances of the newly developed prediction models and traditional scoring systems for internal and external validation. A: The receiver operating characteristic (ROC) curve for the random forest (RF) model and the logistic regression (LR) model; B: Receiver operating characteristic curve for models RF, LR, A and B; C: The performance matrix comparison for the RF model and the LR model; D: The performance matrix for models RF, LR and A. Internal validation: A and C. External validation: B and D. The performance matrix of RF, LR and model A models are shown in blue, orange and green, respectively. AUC: Area under the receiver operating characteristic curve.

Moreover, we compared our results with those of previously published methods in the external validation dataset as well (Figure 7). The area under the receiver operating characteristic curves for the RF, LR, model A and model B models were 0.90, 0.86, 0.82 and 0.75, respectively. The DeLong’s test between these models was: P = 0.135 (RF vs LR), P = 0.004 (RF vs model A), P = 0.006 (RF vs model B), P = 0.333 (LR vs model A), P = 0.045 (LR vs model B) and P = 0.185 (model A vs model B). The accuracy for the RF, LR and model A models were 86%, 83% and 76%, respectively. The sensitivity was 82%, 79% and 82%, respectively. The specificity was 86%, 83% and 75%, respectively. Since model B showed the lowest performance in this validation set, only the predictive performance of LR, RF and model A were compared for the following experiments.

Figure 8A, 8C and 8E shows the observed risk of ICU admission vs model-predicted risk in groups based on the calculated model risk score. The overestimation and underestimation in the probability range of ICU admission risk were evident from the plots. Model A underestimated risk in patients with a predicted risk less than 30%. The RF and LR models show a perfect calibration. The RF risk score demonstrated an excellent ability to categorize patients in separate risk strata. Figure 8B, 8D and 8F shows the differences in the predicted probability values between ICU admission and no ICU admission using each of the prediction models. The discrimination slope for the RF, LR and model A models were 0.281, 0.246 and 0.143, respectively. The plots indicated a lack of fit for the model A.

Open in New Tab   Full Size Figure   Download Figure

Figure 8 The calibration and discrimination of model random forest, logistic regression and A in external validation dataset. A, C and E: The graph represents the relationship between observed (data markers represent the mean and the error bars represent the 95% confidence interval) and predicted risk of intensive care unit (ICU) admission using the models (orange line); B, D and F: The discrimination potentials of the random forest (RF), logistic regression (LR) and model A models. The values of the discrimination slope were 0.281, 0.246 and 0.143, respectively.

We further calculated the risk of each individual in the entire testing cohort and divided all patients into three groups based on the risk cut-off at 95% sensitivity and 95% specificity[36]. Then we computed the NRI with RF as the second model, and LR and model A as the first models. In this scenario, a positive NRI value would indicate that the RF model has better discriminative ability compared to the other models, while a negative value would indicate the converse. Table 3 summarizes the results of this analysis.

Finally, DCA was used to facilitate the comparison between different prediction models. As shown in Figure 9, the DCA graphically shows the clinical usefulness of each model based on a continuum of potential thresholds for major high risk and the standardized net benefit of using the model to stratify patients relative to assuming that there were no ICU admission patients. As shown from our results, the standardized net benefit yielded by the models developed in this study was larger across the major high-risk range compared with model A.

Open in New Tab   Full Size Figure   Download Figure

Figure 9 Decision curve analysis for the model A, logistic regression and random forest risk prediction models. The vertical axis displays standardized net benefit. The two horizontal axes show the correspondence between high-risk threshold and cost: benefit ratio. The thin gray line is the standardized net benefit of allocating intensive care unit resources to all patients; the thick black line is the standardized net benefit of no intensive care unit admission. Decision curve analysis shows that our models had more significant standardized net benefits in the major threshold probabilities interval than model A, demonstrating that our models have better clinical benefit. LR: Logistic regression; RF: Random forest.

The novel coronavirus disease 2019 (COVID-19) pandemic is a global threat caused by the severe acute respiratory syndrome coronavirus-2.

The development of a prognostic model is crucial to address the problem of micro-allocation of scarce healthcare resources in the face of a pandemic.

To develop and validate a risk stratification tool for the early prediction of intensive care unit admission among COVID-19 patients at hospital admission.

We selected 13 of 65 baseline laboratory results and developed a risk prediction model with the random forest algorithm. A nomogram for the logistic regression model was built based on six selected variables.

The accuracy, sensitivity and specificity for the random forest model were 91%, 88% and 93%, respectively, higher than those for the logistic regression model. The area under the receiver operating characteristic curve of our model was much better than those of two other published methods (0.90 vs 0.82 and 0.75).

Our model can identify intensive care unit admission risk in COVID-19 patients at admission, who can then receive prompt care, thus improving medical resource allocation.

In the future, research should include repeated measures data to identify whether temporal changes in clinical indicators are better able to predict disease prognosis in COVID-19.