Intended for healthcare professionals

  1. Anand R Habib, resident physician1,
  2. Nathan C Lo, fellow2
  1. 1Department of Medicine, University of California San Francisco, San Francisco, CA 94143-0119, USA
  2. 2Division of HIV, Infectious Diseases, and Global Medicine, University of California San Francisco, San Francisco, CA, USA
  1. Correspondence to: A R Habib anand.habib{at}ucsf.edu

A new prediction model showcases new ways to collaborate

As of mid-February 2021, more than 405 million people have received a diagnosis of covid-19 and 5.8 million have died globally.1 Despite public health measures, vaccines, antiviral treatments, and monoclonal antibodies, covid-19 continues to overrun hospital wards and strain health systems. Clinical prediction models that help clinicians accurately identify those patients admitted to hospital with covid-19 at greatest risk of clinical deterioration may help to reduce morbidity and mortality.

In a linked paper, Kamran and colleagues (doi:10.1136/bmj-2021-068576) report on the development and validation of a novel, clinical deterioration prediction model for covid-19, the Michigan Critical Care Utilization and Risk Evaluation System (M-CURES).2 The authors used a statistical learning algorithm to narrow 2686 potentially predictive variables from the electronic health record to a parsimonious nine variable model—age, respiratory rate, oxygen saturation, oxygen flow rate, pulse oximetry type (eg, intermittent, continuous), head-of-bed position, position of patient during blood pressure measurement (eg, standing, sitting, lying), venous blood gas pH, and partial pressure of carbon dioxide in arterial blood.

M-CURES automatically collected these variables from the electronic health record every four hours and dynamically recalculated the risk (through logistic regression) of a composite primary outcome of either in-hospital mortality or decompensation requiring heated high flow nasal cannula, intravenous vasopressors, or mechanical ventilation. The prediction model secondarily sought to identify patients at low risk of deterioration 48 hours after admission who could potentially be discharged early.

When internally validated among a cohort of 956 hospital admissions for covid-19 at the University of Michigan from 1 March 2020 to 28 February 2021, M-CURES had a c statistic (or area under the receiver operating characteristics curve) of 0.80 (95% confidence interval 0.77 to 0.84), indicating good discrimination.3 Impressively, the authors then worked with colleagues at 12 geographically distinct and demographically diverse medical centers across the US to externally validate their model in 8335 patients with covid-19. Here too, M-CURES showed good discrimination, with c statistics of 0.77 to 0.84, which were robust across time and subgroup analyses of age, sex, and race or ethnicity.

Although their prediction model was not prospectively evaluated, Kamran and colleagues estimated that it could identify 95% of patients with covid-19 at low risk of deterioration, potentially saving up to 7.8 bed days for each low risk patient. These results are comparable to those for the widely cited 4C deterioration model that uses 11 variables manually collected by the clinician on admission alone to identify hospital patients with covid-19 at risk of death or transfer to an intensive care unit; the c statistic for this model was 0.76 in both internal and external validations.45

Kamran and colleagues’ analysis has limitations. First, we can only speculate on the real world effect of M-CURES in reducing mortality and transfers to intensive care. Although the model retrospectively predicted the primary outcome with a median lead time of 7-18 hours, it is unknown whether and how healthcare providers would act prospectively on alerts from the electronic health record, especially in the context of staffing shortages and alert fatigue.6 The true clinical benefit can only be ascertained from a randomized trial. Second, it is unclear why some of the model’s variables were selected, including pulse oximetry type, head-of-bed position, and venous rather than arterial blood gas pH. Third, although M-CURES’ algorithm is laudably open sourced, its use is limited to hospitals with the same proprietary electronic health record used in this study and to hospitals resource rich enough to employ bioinformaticians.

The most compelling contribution of this study is its model of a multicenter research collaboration for the rapid validation of a clinical prediction tool; it exemplifies the higher standard for research quality that a pandemic such as the covid-19 one demands. Nearly all clinical deterioration models published early in the pandemic were plagued by small, non-representative samples with risk for bias, limited or no external validation, and modest clinical value.78 Kamran and colleagues externally validated their prediction model at 12 hospitals without time consuming data sharing agreements that can discourage and delay external validation at several institutions. The authors did this by creating autonomous research teams who simultaneously employed common data dictionaries and data extraction approaches that, when paired with a plug-and-play machine learning algorithm, enabled not only external validation for M-CURES but also showed its durable discriminative capacity and calibration across time, geography, and patient populations.

As with transnational covid-19 clinical trials using adaptive designs that efficiently test multiple treatments in parallel, informatics efforts must find new ways to work together that leverage collective resources in the pursuit of common public health goals.9 Although creating such shared research infrastructure is labor intensive upfront, it prioritizes collaboration over competition in delivering robust, “living” clinical prediction models that can be recalibrated easily as new covid-19 variants and treatments emerge and lays the groundwork for prospective analyses evaluating the real world utility of prediction models such as M-CURES.10

Footnotes

  • Research, doi: 10.1136/bmj-2021-068576
  • Competing interests: ARH has no disqualifying financial ties to commercial companies. NCL has received personal fees from the World Health Organization unrelated to the present work.

  • Provenance and peer review: Commissioned; not externally peer reviewed.

This article is made freely available for personal use in accordance with BMJ's website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.

https://bmj.com/coronavirus/usage

References