SCCM Resource Library-27-Machine Learning Prediction of Intensive Care Unit Delirium

Video Transcription

Hello, everyone. I'm Kirby Gong, a researcher at Johns Hopkins University, here today to talk about using machine learning to predict delirium in ICUs.  I've been working in collaboration with Ryan Liu, Taya Bergamaschi, Akash Sanyal, Joanna McGuill, Han Kim, and Dr. Robert Stevens. And we have no financial disclosures to report.  Today, we're going to start with some background on the problem of delirium, then go into what data sets we used and what we extracted from them.  From there, we'll talk about our inclusion and exclusion criteria for our patient selection.  And finally, we'll talk about two different predictive models that we've built for predicting delirium in ICUs. Delirium is important for a host of reasons.  It is quite common in intensive care settings, in addition to being independently associated with poor patient outcomes and high costs.  It is also estimated that a large proportion of delirium is preventable, which is why an early prediction model for delirium might be especially helpful.  We believe that a classifier trained on an extensive data set, including physiological time series data, will outperform existing models and could be clinically useful.  For our work, we used two publicly available data sets, eICU from Philips and MIMIC-3. eICU was used for initial development since it spanned a larger number of hospitals.  We ended up using about 25,000 ICU stays, with about 15% of them being positive for delirium.  We have been using MIMIC-3 for external validation, with about 6,000 relevant unit stays and 22% of those positive for delirium. We extracted a large number of features and labels.  To determine whether patients had delirium or not, we used CAMICU or ICDSC scores, as well as listings in the problem list.  For features to input into our machine learning models, we looked at demographics, medications, labs, charting, comorbidities, and physiological time series data.  To get our patient cohort, we began with the eICU database, which spans about 200,000 unit stays across about 200 hospitals.  The first thing we needed was a ground truth label for delirium, for which we used diagnoses listed in the problem list or CAMICU or ICDSC tests.  Patients without such information or that were not in the ICU for at least 12 hours were excluded.  We then also excluded patients that came into the ICU with delirium, defined as being positive for delirium within the first 12 hours of ICU stay.  Ultimately, we ended up with about 23,000 patient stays, which were then subject to a few more minor criteria depending on the model being trained.  So with patient cohort in hand, the first predictive model we've built we call the first 24-hour model, which uses data from the first 24 hours of ICU stay to predict  whether a patient will develop delirium at any time during their ICU stay later on. For comparison, we recreated the existing gold standard for delirium prediction, known  as predeliric, in our dataset. Predeliric uses the same framework, using the first 24 hours of ICU data to predict delirium at any point in the stay later on.  Predeliric uses 10 features listed here, which we recreated to the best of our ability and used to train a generalized linear model.  The performance of our recreated predeliric was comparable to what we've seen in the literature, with an area under the receiver-operator curve, or AUROC for short, of about 0.73.  We then built our own predictive model, using our expanded feature space and various algorithms. Our best-performing model used around 100 features plugged into an algorithm called  CapBoost, with performance and top features shown here. It had statistically significant better AUROC than the predeliric feature space, achieving  an AUROC of about 0.79 in our nested cross-validation outer loop. Our area under the precision-recall curve was about 0.39, and the Briar score on average  was about 0.101, with less consistency at higher predicted risks. Our most relevant features were obtained using Shapley values, and are all fairly expected,  including GCS, RAS, and Apache scores, age, and labs like BUN, RDW, MCHC, and creatinine.  However, we felt that the first 24-hour framework was rather lacking, and so built an additional model.  The goal of our so-called dynamic predictive models is to create a predictive model that could be re-evaluated on patients in real-time, and tries to predict onset of delirium a set  amount of time in the future, with delirium onset being defined as the first positive CAM-ICU or IC-DSC test.  We experimented with lead times ranging from 1 to 6 hours, and with various lengths of observation windows, also between 1 to 6 hours.  Here we have performance metrics of our model, using random forests across different lead times, observation window lengths, and with or without physiological time series data,  or PTS for short. Overall, we found that decreasing the lead time of prediction tended to slightly increase  performance of our predictive models, regardless of the algorithm being used, or regardless of the metric, although here we've shown AUROC.  This is to be expected, as it should be easier to predict something that is closer in the future.  Here the AUROC increased beyond a 95% confidence interval, as the lead time was decreased from 6 to 1 hour. Observation window length did not seem to have clear effects on AUROC.  The addition of physiological time series features did not improve AUROC in our models.  We also looked at precision recall as another metric, and the effects of lead time, observation window length, and physiological time series features.  Similar to AUROC, as lead time decreased, precision recall increased, and observation window length did not have any clear trend.  In contrast, though, to AUROC, precision recall actually slightly improved with the addition of physiological time series features, although not beyond a 95% confidence interval.  Finally, we looked at Breyer score, which is a measurement of calibration. The Breyer scores had similar trends to AUROC, again improving with shorter lead times, having  no clear relationship with observation window lengths, and with the addition of physiological  time series features degrading the performance of the model beyond a 95% confidence interval. Now that you've seen the big picture, let's take a deeper dive into our best performing  model, with a lead time of just 1 hour. As you can see, the performance is pretty good, with AUROC of about 0.857, and a precision recall of about 0.59.  The calibration curve is also fairly good, with a Breyer score of about 0.104. The top 20 most important features have a lot of overlap with our first 24-hour model,  including GCS, RAS, and Apache scores, as well as a few labs, like sodium, chloride, BUN, and creatinine.  Highly relevant medications included general anesthetics, andronergic, bronchodilators, and vasopressors. Mechanical ventilation and age were also highly important features.  Per our clinical advisor, these features are clinically plausible, or even already linked with delirium in studies.  In summary, we have created a novel algorithm to predict a patient's risk of delirium any time later in the stay, that outperforms the existing gold standard, predeliric.  Our model performs similarly to other recently created models, but was developed from a multi-center database.  Surprisingly, our best performing first 24-hour model thus far did not use any physiological time series data.  Besides that model, we have also created a dynamic model that could be used to predict delirium onset several hours in advance, with best AUROC of about 0.85.  Surprisingly, physiological time series features don't clearly improve performance there either, although we've seen some preliminary success with certain measures.  So far, these features slightly improve precision recall, but not beyond a 95% confidence interval.  However, physiological time series features worsen AUROC and Breyer scores, beyond 95% confidence intervals.  Going forward, we plan to investigate more ways to improve performance, like new features, new algorithms, or other imputation methods.  We also plan to externally validate our models on the MIMIC3 database. As this work is developed further, this model could be used in the future to predict which  patients will develop delirium hours in advance and take preventative action. Thank you for your time, and have a good Congress!

Video Summary

Kirby Gong, a researcher at Johns Hopkins University, discusses using machine learning to predict delirium in intensive care units (ICUs). Delirium is common in ICUs and is associated with poor outcomes and high costs. Gong and his team used two publicly available datasets, eICU and MIMIC-3, to develop predictive models. The first model, the "first 24-hour model," predicts delirium using data from the first 24 hours of ICU stay. The team also created a dynamic model that predicts delirium onset a few hours in advance. The models performed well, outperforming existing models, and could be used to take preventative action in the future.

Asset Subtitle

Neuroscience, Quality and Patient Safety, 2021

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2021 Critical Care Congress held virtually from January 31-February 12, 2021.

Kirby D. Gong, BS

Introduction/Hypothesis: ICU delirium is frequent, associated with unfavorable outcomes, increased health expenditures, and may be largely preventable. Accurate delirium prediction could allow early intervention strategies in high risk patients. The aim of this study was to leverage machine (ML) learning applied to early physiological and clinical data to predict delirium onset at any time during an ICU stay.

Methods: Data were obtained from the multicenter eICU dataset. Delirium labels were determined by clinical documentation of either the Confusion Assessment Method in the ICU or Intensive Care Delirium Screening Checklist scoring systems at any time during the ICU stay. Predictive features were extracted data recorded in the first 24h after ICU admission. A statistically pruned feature space of 116 derived variables was used to train three different ML algorithms (logistic regression [LR], random forest [RF], and gradient boosting [CatBoost]). Model performance was evaluated by area under the receiver operating characteristic curve (AUROC) discrimination analysis and compared with the PRE-DELIRIC score which has been previously validated for prediction of delirium in the ICU.

Results: A total of 24,695 patient stays associated with delirium were identified. The AUROC of logistic regression, random forest, and CatBoost were 0.780Â±0.005, 0.797Â±0.005, and 0.792Â±0.005, respectively. For comparison, AUROC of the PRE-DELIRIC model was 0.731Â±0.006. The ML models calibrated well, with Brier scores of 0.125Â±0.001, 0.127Â±0.003, and 0.152Â±0.002. The top ten predictive features according to the LR model included mean verbal GCS score, mean eyes GCS score, APACHE IV score, age, coma, and minimum mean corpuscular volume. Physiological time series data recorded in the first 24h did not add to prediction accuracy.

Conclusions: Machine learning applied to features from the first 24h after admission predicted ICU delirium onset with higher accuracy than the reference PRE-DELIRIC score. Results suggest that high-resolution data contain predictive information on delirium risk in the ICU which is overlooked in current prediction systems. Results warrant prospective validation and additional studies integrating other data modalities such as brain-specific molecular biomarkers and neuroimaging.

Meta Tag

Content Type Presentation

Knowledge Area Neuroscience

Knowledge Area Quality and Patient Safety

Knowledge Level Advanced

Membership Level Select

Tag Scoring Systems

Tag Delirium

Year 2021

Keywords

Kirby Gong

machine learning

predicting delirium

intensive care units

ICUs

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.