Evaluating the Effect of a COVID-19 Predictive Model to Facilitate Discharge: A Randomized Controlled Trial

 

 Permissions and Reprints

Abstract

Background We previously developed and validated a predictive model to help clinicians identify hospitalized adults with coronavirus disease 2019 (COVID-19) who may be ready for discharge given their low risk of adverse events. Whether this algorithm can prompt more timely discharge for stable patients in practice is unknown.

Objectives The aim of the study is to estimate the effect of displaying risk scores on length of stay (LOS).

Methods We integrated model output into the electronic health record (EHR) at four hospitals in one health system by displaying a green/orange/red score indicating low/moderate/high-risk in a patient list column and a larger COVID-19 summary report visible for each patient. Display of the score was pseudo-randomized 1:1 into intervention and control arms using a patient identifier passed to the model execution code. Intervention effect was assessed by comparing LOS between intervention and control groups. Adverse safety outcomes of death, hospice, and re-presentation were tested separately and as a composite indicator. We tracked adoption and sustained use through daily counts of score displays.

Results Enrolling 1,010 patients from May 15, 2020 to December 7, 2020, the trial found no detectable difference in LOS. The intervention had no impact on safety indicators of death, hospice or re-presentation after discharge. The scores were displayed consistently throughout the study period but the study lacks a causally linked process measure of provider actions based on the score. Secondary analysis revealed complex dynamics in LOS temporally, by primary symptom, and hospital location.

Conclusion An AI-based COVID-19 risk score displayed passively to clinicians during routine care of hospitalized adults with COVID-19 was safe but had no detectable impact on LOS. Health technology challenges such as insufficient adoption, nonuniform use, and provider trust compounded with temporal factors of the COVID-19 pandemic may have contributed to the null result.

Trial registration ClinicalTrials.gov identifier: NCT04570488.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the Institutional Review Board of NYU Grossman School of Medicine.

Publication History

Received: 07 December 2021

Accepted: 17 April 2022

Article published online:
27 July 2022

© 2022. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

• References

• 1 Liu J, Li S, Liu J. et al. Longitudinal characteristics of lymphocyte responses and cytokine profiles in the peripheral blood of SARS-CoV-2 infected patients. EBioMedicine 2020; 55: 102763
  CrossrefPubMedGoogle Scholar
• 2 Guan WJ, Ni ZY, Hu Y. et al; China Medical Treatment Expert Group for Covid-19. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 2020; 382 (18) 1708-1720
  Google Scholar
• 3 Chen R, Liang W, Jiang M. et al; Medical Treatment Expert Group for COVID-19. Risk factors of fatal outcome in hospitalized subjects with coronavirus disease 2019 from a nationwide analysis in China. Chest 2020; 158 (01) 97-105
  CrossrefPubMedGoogle Scholar
• 4 Huang C, Wang Y, Li X. et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020; 395 (10223): 497-506
  Google Scholar
• 5 Galloway JB, Norton S, Barker RD. et al. A clinical risk score to identify patients with COVID-19 at high risk of critical care admission or death: an observational cohort study. J Infect 2020; 81 (02) 282-288
  CrossrefPubMedGoogle Scholar
• 6 Ji M, Yuan L, Shen W. et al. A predictive model for disease progression in non-severely ill patients with coronavirus disease 2019. Eur Respir J 2020; 56 (01) 2001234
  CrossrefPubMedGoogle Scholar
• 7 Li Q, Zhang J, Ling Y. et al. A simple algorithm helps early identification of SARS-CoV-2 infection patients with severe progression tendency. Infection 2020; 48 (04) 577-584
  CrossrefPubMedGoogle Scholar
• 8 Liang W, Liang H, Ou L. et al; China Medical Treatment Expert Group for COVID-19. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med 2020; 180 (08) 1081-1089
  CrossrefPubMedGoogle Scholar
• 9 Yan L, Zhang HT, Goncalves J. et al. An interpretable mortality prediction model for COVID-19 patients. Nat Mach Intell 2020; 2 (05) 283-288
  CrossrefPubMedGoogle Scholar
• 10 Singh K, Valley TS, Tang S. et al. Evaluating a widely implemented proprietary deterioration index model among hospitalized patients with COVID-19. Ann Am Thorac Soc 2021; 18 (07) 1129-1137
  CrossrefPubMedGoogle Scholar
• 11 Wynants L, Van Calster B, Collins GS. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328
  CrossrefPubMedGoogle Scholar
• 12 Harish K, Zhang B, Stella P. et al. Validation of parsimonious prognostic models for patients infected with COVID-19. BMJ Health Care Inform 2021; 28 (01) e100267
  CrossrefPubMedGoogle Scholar
• 13 Razavian N, Major VJ, Sudarshan M. et al. A validated, real-time prediction model for favorable outcomes in hospitalized COVID-19 patients. NPJ Digit Med 2020; 3: 130
  CrossrefPubMedGoogle Scholar
• 14 Wiens J, Saria S, Sendak M. et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med 2019; 25 (09) 1337-1340
  CrossrefPubMedGoogle Scholar
• 15 Drysdale E, Dolatabadi E, Chivers C. , et al. Implementing AI in Healthcare; 2019 DOI: 10.13140/RG.2.2.30793.70241
  PubMed
• 16 Begg C, Cho M, Eastwood S. et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276 (08) 637-639
  CrossrefPubMedGoogle Scholar
• 17 Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. Squire 2.0 (Standards for Quality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. Am J Crit Care 2015; 24 (06) 466-473
  CrossrefPubMedGoogle Scholar
• 18 Pinnock H, Barwick M, Carpenter CR. et al; StaRI Group. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ 2017; 356: i6795
  CrossrefPubMedGoogle Scholar
• 19 Haynes RB, Del Fiol G, Michelson M, Iorio A. Context and approach in reporting evaluations of electronic health record-based implementation projects. Ann Intern Med 2020; 172 (Suppl. 11) S73-S78
  CrossrefPubMedGoogle Scholar
• 20 R Core Team. R: A Language and Environment for Statistical Computing. Published online 2018. Accessed August 31, 2021 at: https://www.R-project.org/
  PubMedGoogle Scholar
• 21 Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen 2015; 144 (01) 114-126
  CrossrefPubMedGoogle Scholar
• 22 Shcherbatykh I, Holbrook A, Thabane L, Dolovich L. COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J Am Med Inform Assoc 2008; 15 (05) 575-580
  CrossrefPubMedGoogle Scholar
• 23 Horwitz LI, Kuznetsova M, Jones SA. Creating a learning health system through rapid-cycle, randomized testing. N Engl J Med 2019; 381 (12) 1175-1179
  CrossrefPubMedGoogle Scholar
• 24 Rosenfeld LS, Goldmann F, Kaprio LA. Reasons for prolonged hospital stay; a study of need for hospital care. J Chronic Dis 1957; 6 (02) 141-152
  CrossrefPubMedGoogle Scholar
• 25 Brasel KJ, Lim HJ, Nirula R, Weigelt JA. Length of stay: an appropriate quality measure?. Arch Surg 2007; 142 (05) 461-465
  Google Scholar
• 26 Garibaldi BT, Fiksel J, Muschelli J. et al. Patient trajectories among persons hospitalized for COVID-19: a cohort study. Ann Intern Med 2021; 174 (01) 33-41
  CrossrefPubMedGoogle Scholar
• 27 Pereira NL, Ahmad F, Byku M. et al. COVID-19: understanding inter-individual variability and implications for precision medicine. Mayo Clin Proc 2021; 96 (02) 446-463
  CrossrefPubMedGoogle Scholar
• 28 Guan WJ, Liang WH, Zhao Y. et al; China Medical Treatment Expert Group for COVID-19. Comorbidity and its impact on 1590 patients with COVID-19 in China: a nationwide analysis. Eur Respir J 2020; 55 (05) 2000547
  CrossrefPubMedGoogle Scholar
• 29 Sanyaolu A, Okorie C, Marinkovic A. et al. Comorbidity and its impact on patients with COVID-19. SN Compr Clin Med 2020; 2 (08) 1069-1076
  Google Scholar

• Supplementary Material