SCCM Resource Library-25-Predicting Pulmonary Embolism Among Hospitalized Patients With a Machine Learning Algorithm

Video Transcription

Hello, my name is Anna Siefkis and I am a member of the research team at Decina Incorporated.  Today I am excited to share with you my team's research on the use of a machine learning algorithm for the prediction of pulmonary embolism among hospitalized patients.  My only disclosure for this presentation is that I am employed by Decina Incorporated, the company that developed the machine learning algorithm on which this presentation is based.  A pulmonary embolism or an obstruction of the pulmonary artery or its branching vessels is a life-threatening event. Pulmonary emboli contribute to somewhere between 60,000 and  100,000 deaths each year in the United States. Diagnosing a pulmonary embolism is complicated  by the fact that many key symptoms such as sudden shortness of breath, heart palpitations or chest pain have significant overlap with other serious acute conditions such as acute  coronary syndrome or heart failure. Several risk scoring systems such as the Geneva score  have been developed for patient risk stratification. These scores are designed to identify patients  for whom a pulmonary embolism can be definitively ruled out and patients for whom further diagnostic  testing is warranted. However, these scores have noted insufficiencies for accurately identifying the stratifying patients and are not validated to predict pulmonary emboli  before they occur. Despite these challenges, early identification of pulmonary embolism  is essential for optimizing patient outcomes by enabling early intervention. We have developed  a machine learning algorithm that is designed to predict the development of pulmonary emboli  before clinical onset in hospitalized patients. This algorithm was developed using gradient boosted decision trees and implemented using XGBoost in Python. Gradient boosting is a  method that allows for large sets of decision trees to iteratively classify patients and often enables more accurate predictions as compared to simpler models. This model was  initially trained and tested on a set of over 63,000 patient encounters from inpatient wards in the United States. The model was additionally externally validated on data  from the MIMIC3 database of intensive care unit patient encounters. This validation set represents a population of more critically ill patients as compared to the development  data and is therefore a robust test of the model's generalizability. The model was trained to make pulmonary embolism predictions early in the patient encounter using information  on patient vital signs, demographics, medical history, and laboratory measurements. We assessed  the model using a number of statistics, including area under the receiver operating characteristic  curve, which plots sensitivity versus 1-specificity at all possible operating points. Additionally,  we assessed sensitivity, specificity, positive and negative likelihood ratios, and diagnostic  odds ratios at fixed operating points. These metrics were assessed separately on the development  and validation data. On the development data, the model was tested on a holdout set that was not seen during the model training process. Model performance was additionally compared  to the commonly used Geneva score, which was calculated on the same sets of patients. On the development data, the algorithm obtained an area under the curve of 0.86. The Geneva  score obtained an area under the curve of 0.70. When sensitivity was fixed near 80% for both models, the algorithm obtained comparable or superior performance across all measured  statistics. On the MMIC3 validation data, the algorithm obtained an area under the curve of 0.76. The Geneva score obtained an area under the curve of 0.63. As on the holdout  test set from the development data, for fixed sensitivities, the algorithm outperformed the Geneva score across all measured metrics. On both the development and validation data,  the model was able to outperform the commonly used Geneva score for pulmonary embolism risk prediction. The development and validation data represent significantly different patient  populations, supporting the generalizability of this machine learning algorithm. This model was explicitly designed and assessed for its ability to predict the onset of a pulmonary  embolism, and while we compared it to the Geneva score as a best standard of care, this prediction task is one for which no standard of care currently exists. This algorithm,  which was designed to be easily integrated into existing workflows, may help improve patient outcomes through timely recognition of and intervention for pulmonary emboli.  Before closing, I would like to thank the rest of the Decina research team that helped to develop and assess this algorithm. Thank you for watching this presentation, and I  hope that you're enjoying the rest of the conference. Thank you.

Video Summary

Anna Siefkis from Decina Incorporated presents her team's research on using a machine learning algorithm to predict pulmonary embolism in hospitalized patients. Pulmonary embolism is a life-threatening condition that causes obstruction of the pulmonary artery and its branches, resulting in 60,000 to 100,000 deaths annually in the US. Current risk scoring systems have limitations in accurately identifying and predicting pulmonary emboli. The team developed a machine learning algorithm using gradient boosted decision trees, which was trained and tested on over 63,000 patient encounters. The algorithm demonstrated superior performance compared to the commonly used Geneva score for pulmonary embolism risk prediction. The algorithm can potentially improve patient outcomes through timely intervention.

Asset Subtitle

Pulmonary, Quality and Patient Safety, 2021

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2021 Critical Care Congress held virtually from January 31-February 12, 2021.

Anna Siefkas

Introduction/Hypothesis: Pulmonary embolism (PE) is a life-threatening condition that contributes to an estimated 60,000-100,000 annual deaths in the United States. Diagnosis of PE is made difficult due to significant overlap between symptoms of PE and of other conditions, including acute coronary syndrome and heart failure. However, rapid diagnosis of PE is essential to preventing PE-related morbidity and mortality. We hypothesized that a machine learning algorithm could be trained to predict PE before onset.

Methods: We developed the algorithm on 63,841 retrospective patient encounters from a large academic medical center. Patient data including vital signs, laboratory measurements, demographic information, and medical history were used to train a gradient boosted machine learning algorithm. The algorithm was implemented using the XGBoost method for fitting boosted decision trees in Python. The outcome of interest was PE at any point during the hospital stay. The algorithm was compared to the Geneva score for ability to identify patients likely to experience a PE. Algorithm performance was validated on external data from the Medical Information Mart for Intensive Care (MIMIC-III) dataset.

Results: On the development dataset, the algorithm obtained an area under the receiver operating characteristic (AUC) of 0.86, while the Geneva score obtained an AUC of 0.70. On the validation dataset, the machine learning algorithm obtained an AUC of 0.76, while the Geneva score obtained an AUC of 0.63. On both datasets, the algorithm improved sensitivity, specificity, and diagnostic odds ratio as compared to the Geneva score.

Conclusions: A machine learning algorithm demonstrated improved performance as compared to a commonly used scoring system for predicting the onset of PE. Improved risk stratification may lead to improved patient outcomes through earlier diagnosis and treatment of PE.

Meta Tag

Content Type Presentation

Knowledge Area Pulmonary

Knowledge Area Quality and Patient Safety

Knowledge Level Intermediate

Knowledge Level Advanced

Membership Level Select

Tag Pulmonary Embolism

Tag Scoring Systems

Year 2021

Keywords

Anna Siefkis

Decina Incorporated

machine learning algorithm

pulmonary embolism

risk prediction

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.