SCCM Resource Library-Predicting Intensive Care Transfers and Unforeseen Events in Pediatric Inpatients (PICTURE-Peds)

Video Transcription

Hello, and thank you for attending this session. My name is Michael Stevens. I'm a second year Pediatric Critical Care Fellow at the University of Michigan in Ann Arbor.  I'll be presenting preliminary data on our machine learning algorithm known as PICTURE, which stands for Predicting ICU Transfers and Other Unforeseen Events.  I would like to thank the other members of the team for their contribution to this project, the abstract, and this presentation.  The implementation of the pilot for this project is partially funded by the company Airstrip. However, the data I am about to present were obtained prior to this partnership.  A potential problem that exists in pediatric inpatient medicine is the unrecognized deterioration of patients that could lead to increased morbidity and mortality.  To help predict which patients are at risk for decompensation, there exists different early warning scoring systems. We postulate that a computer-based machine learning model  that can incorporate more patient data and be continuously running would be better at predicting which patients are at risk.  Examples of current early warning scoring systems include PAWS, the Pediatric Advanced Warning Score on the left, and PEWS, the Pediatric Early Warning Score on the right.  Both utilize the degree of vital sign derangement along with subjective behavioral and neurostatus to obtain a score.  PICTURE Pediatrics is a machine learning suite that uses data generated by patients that is stored in the electronic medical record to predict general deterioration.  The aim of PICTURE Pediatrics is to implement a system that better discriminates between which patients will have a decompensation and which will  not compared to other early warning scores.  In order to develop a new system, we came up with inclusion criteria for the patients that would be used to train, validate, and ultimately test the model.  Patients included were aged one month to 18 years and were admitted to the general care floor. They also had to have at least one CBC or BMP during their admission.  PICTURE Pediatrics was compared to PEWS, PAWS, and Pediatric SOFA in its ability to predict ICU transfer, death, or vasoactive administration  or endotracheal intubation on the general care floor.  This is an infographic reviewing the development of PICTURE Pediatrics. It was trained and validated on over 8,000 patient encounters over a four-year period from 2014 to 2018.  Eighty percent of the encounters were used to train the model while 20 percent were used for validation. Data from 2019 were used as a test dataset.  The test dataset was compared to PEWS, PAWS, and Pediatric SIRS.  The training data utilized XGBoost technology, which uses both gradient boosting, shown in this picture, and data regularization.  In a basic sense, gradient boosting is a machine learning technique in which weak learners in the form of decision trees are turned into  better learners based off the errors of the previous decision tree. This is done by repeatedly taking the actual value subtracted from the predicted value,  and repeating this process until the difference gets smaller and smaller. Regularization is a process that desensitizes your model to the training data,  whereby variance of a best fit line is decreased at the expense of increasing bias in order to improve the model's predictive power over the long-term.  This table shows the area under the receiver operator curve, and the area under the precision recall curve for picture at an observational level,  which evaluates each alarm separately, and on an encounter level, evaluating the highest alert for a patient across his or her entire hospital encounter.  The area under the precision recall curve was included due to the small number of true positives. Picture outperformed these other scoring systems at the observational and encounter  level in both the area under the receiver operator curve, and the area under the precision recall curve.  Here, we see a graphical representation of the data that was shown in the previous table. On the left is a graph of the area under the receiver operator curve,  and on the right is a graph of the area under the precision recall curve. In both tables, picture is the blue line, pause is the orange line, bedside pews is the green line,  and pediatric SERS is the red line.  Many things make picture pediatrics different from other early warning scoring systems. As mentioned previously, picture incorporates data  generated by the patient and stored in the electronic medical record. The figure displayed shows the workflow of picture. Picture exists outside of the electronic medical record,  and can therefore be altered independent of the EMR.  The data incorporated by picture pediatrics includes vital signs, lab values, and interventions such as oxygen administration. Furthermore, as demonstrated by this table,  picture reveals which data points contributed the most for a particular patient's prediction. That information with the physician.  This figure shows an example patient's picture data trended over time. This 10-year-old male triggered the very high risk alert about 36 hours prior to ICU transfer.  This figure also demonstrates that picture can be fine-tuned by changing its alert threshold in response to physician feedback. Current stages include the picture program  running in the background and collecting data prospectively. It is currently not sending alerts. A pilot program for picture pediatrics to run live is currently under development,  and picture will continue to evolve after receiving feedback from clinicians. We want this system to be more than an academic exercise.  We want it to help both patients and physicians in their daily work.  Picture pediatrics has the potential to expand. The general deterioration model can be adjusted to predict a specific type of deterioration,  such as cardiogenic shock, respiratory failure, or sepsis. It can also be tailored for specific patient populations, such as COVID patients.  And it can also be used to predict and prevent different endpoints, such as rapid response calls.  In conclusion, picture pediatrics, compared to other early warning scoring systems, utilizes more data, assesses patients more frequently,  and outperforms in its ability to discriminate between at-risk and stable patients when compared to PAWS, PEWS, and pediatric SIRS.  Thank you for your attention and for choosing to spend your valuable time listening to my presentation.

Video Summary

The speaker, Michael Stevens, presents preliminary data on a machine learning algorithm called PICTURE Pediatrics. The algorithm aims to predict patient deterioration in pediatric inpatient medicine. It utilizes patient data from electronic medical records to predict general deterioration and outperforms other early warning scoring systems. The algorithm was trained and validated on over 8,000 patient encounters and showed better performance in predicting ICU transfers, death, and other critical events. PICTURE Pediatrics can be fine-tuned and tailored for specific patient populations or types of deterioration. The algorithm is currently in a pilot program and aims to assist physicians in improving patient care.

Asset Subtitle

Quality and Patient Safety, Pediatrics, 2022

Asset Caption

INTRODUCTION: Early detection of clinical instability and use of rapid response teams (RRTs) have been shown to reduce pediatric hospital mortality. Integral to these systems are pediatric specific early warning scores such as PEWS and PAWS, yet multiple studies have reported inadequate RRT activation in some deteriorating patients. Automated data collection and machine learning analytics could improve our ability to identify at-risk patients. PICTURE-Peds utilizes these tools for early identification of patient decompensation. We hypothesize that PICTURE-Peds will outperform PEWS, PAWS, and pediatric SIRS in identifying patients requiring ICU transfer. METHODS: PICTURE-Peds was developed utilizing 10,364 encounters from 2014-2018. Using outcomes of ICU transfer or death within 24 hrs, an XGBoost model with 30 round early stopping criterion was trained using vital signs, labs, and demographics from retrospective data (80% to train the model; 20% for validation). The model was then applied to a test set of 2,858 encounters from 2019. Inclusion criteria were general care patients 1 mo to 18 yrs with a CBC and/or BMP. AUROC and AUPRC for PICTURE-Peds were compared to bedside PEWS, PAWS, and pediatric SIRS. The comparison was made at an observational level, evaluating alarms separately, and at an encounter level, evaluating the maximum score for a patient during an admission. RESULTS: On the observation level, AUROC for PICTURE was 0.85 (95% CI 0.82–0.87) vs PAWS: 0.81 (0.78–0.84), PEWS: 0.77 (0.75–0.80) and SIRS: 0.53 (0.50–0.57). The AUPRC (observation) for PICTUE was 0.15 (0.11–0.19) vs PAWS: 0.12 (0.08–0.14), PEWS: 0.09 (0.06–0.11) and SIRS: 0.04 (0.02–0.04). At the encounter level AUROC for PICTURE was 0.85 (0.81–0.89) vs PAWS: 0.75 (0.72–0.79), PEWS: 0.66 (0.62–0.70) and SIRS: 0.49 (0.46–0.53). The AUPRC (encounter) for PICTURE was 0.43 (0.37–0.47) vs PAWS: 0.28 (0.23–0.32), PEWS: 0.20 (0.16–0.23) and SIRS: 0.10 (0.08–0.12). P < 0.001 comparing PICTURE to all others except observational AUPRC vs. PAWS (p=0.028). CONCLUSION: PICTURE-Peds outperformed traditional scoring systems in identifying patients requiring ICU transfer. This innovative early warning system may also allow for predictive analytics of patient deterioration and is being studied prospectively, deployed within the EMR at Mott Children’s Hospital.

Meta Tag

Content Type Presentation

Knowledge Area Quality and Patient Safety

Knowledge Area Pediatrics

Knowledge Level Advanced

Membership Level Select

Tag Scoring Systems

Tag Pediatrics

Year 2022

Keywords

PICTURE Pediatrics

machine learning algorithm

patient deterioration

electronic medical records

physicians

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.