Background:
The covid-19 pandemic has overwhelmed hospitals worldwide and clinical prediction models may assist in timely identification of covid-19 patients at risk for clinical deterioration, i.e. `early warning'.
In this article, we report on the development and validation of
...
Background:
The covid-19 pandemic has overwhelmed hospitals worldwide and clinical prediction models may assist in timely identification of covid-19 patients at risk for clinical deterioration, i.e. `early warning'.
In this article, we report on the development and validation of a new early warning model that predicts unplanned ICU admission or unexpected death within 24 hours from the moment of prediction, specifically
for covid-19 patients. We compared the performance with two well-known and widely used early warning scores (EWSs), i.e. the Modified Early Warning Score (MEWS) [2] and National Early Warning Score (NEWS) [3].
Methods:
We collected electronic medical record (EMR) data from covid-19 patients admitted to six Dutch hospitals between February 2020 and May 2021. We defined the clinical endpoint as a surrogate of unplanned
ICU admission or unexpected death. To examine the added value of including non-linear predictor-outcome relations, we trained both a (linear) logistic regression (LR) and a (non-linear) random forest (RF) model. We
included predictors based on patient demographics, vital signs and laboratory test results. We validated the models retrospectively in a `leave-one-hospital-out' cross-validation (LOHO-CV) procedure. Furthermore, we simulated a prospective validation by splitting all included patients admitted before and after August 1 2020 and simulated as if the models would have been developed based on the data collected until August 2020 and implemented during the remaining study period. Additionally, we examined different strategies
for monthly model updating. We evaluated model discrimination and calibration for the proposed models as well as the traditional EWSs, and performed a decision curve analysis [22]. Importance of individual
predictors was quantified using SHAP values [13].
Findings:
In the retrospective validation, the LR model yielded a significant improvement in partial area under the receiver operating curve (pAUC) compared to the traditional EWSs in four of the six included hospitals, and in all hospitals by the RF model. In the simulated prospective validation, significant improve-
ment was shown in two and four hospitals by the LR and RF models, respectively. Without any model updating, both model showed risk overestimation. We proposed a combination of monthly model retraining and hospital-specific re-calibration that could correct for this miscalibration effectively. In the decision curve analysis, the proposed models outperformed the traditional EWS in terms of net benefit (NB) over a wide range of clinically relevant model thresholds.
Interpretation We have derived and validated a new early warning model specifically for covid-19 patients that outperformed traditional EWSs and showed good generalizability over different Duch hospitals. Also, we introduced SpO2-to-O2 ratio as an important marker for disease severity in covid-19 patients. Finally, we showed the importance of repeated model updating when developing medical prediction models in the midst of the covid-19 pandemic and proposed an effective model updating strategy. Future research is needed to
validate the model outside the Netherlands.
Background:
The covid-19 pandemic has overwhelmed intensive care units (ICUs) worldwide. Improved prediction of a covid-19 patient's risk of dying may assist decision making in the intensive care unit (ICU) setting. In contrast to traditional mortality models like APACHE II [1] and SAPS II [2], dynamic mortality models allow for repeated risk stratication of patients throughout the ICU stay. Earlier works [3, 4, 5] in dynamic mortality modelling show promising results, although most of these works propose models for the general (non-covid) ICU population and use relatively long or unspecied prediction horizons. In this study, we report on the development and retrospective validation of a model for dynamic, near-term mortality for critically ill covid-19 patients.
Methods:
We collected EMR data from 3 481 ICU admissions with a covid-19 infection from the Dutch Data Warehouse (DDW) [6], coming from 25 dierent ICUs in the Netherlands. We extracted daily samples of each patient and trained both a linear (logistic regression) and non-linear (random forest) model to
predict in-ICU mortality within 24 hours from the moment of prediction. Isotonic regression was used to re-calibrate the predictions of the trained models. We evaluated the models in a leave-one-ICU-out (LOIO) cross-validation procedure.
Findings Validation in 21 out of 25 and 18 out of 25 ICUs yielded an area under the receiver operating characteristic curve (AUROC) >0.80 for the logistic regression and random forest model, respectively. In the four hospitals that yielded an AUROC<0.8, local differences in protocols concerning discontinuation
of treatment may have played a role. The re-calibrated model estimations showed good calibration for both models (calibration intercept = −0.12, slope = 0.87 for logistic regression and intercept = −0.05, slope = 0.82 for random forest).
Interpretation:
This study is different from previous works on dynamic mortality prediction in the ICU as we presented a model speciffcally for covid-19 patients and introduced near-term mortality predictions (compared to long-term or in-hospital mortality). The predictions were calculated based on a mixture of
static information (e.g. age and sex) and dynamic information (e.g., vital signs and laboratory values) and the importance of individual predictors was quantied using SHAP values [7], where we found FiO2, oxygen saturation and pH to be important predictors. The potential clinical utility of dynamic mortality models such as guidance in resource allocation and real-time patient benchmarking could be topics for future research.