SCCM Resource Library-Assessment of a Combined Biomarker-EMR Data Machine Learning Model for Sepsis-3

Video Transcription

Hello, my name is Bobby Reddy. I'm the CEO and co-founder of Pernosis. Today, I'd like to talk to everyone about the sepsis immunoscore,  which is a combined biomarker EMR data machine learning model for the early diagnosis of sepsis in emergency departments and in hospital. Before we get into the sepsis immunoscore,  I want to first talk about the dilemma of sepsis in general. If you look down here at the x-axis, what we have is the onset of infection.  This is when a patient or a person first gets infected, typically out in the community, but this could also be in the hospital. After the person is initially infected,  it's actually very difficult to understand that a patient is infected until they show some symptoms. That's the left side of this graph. Anything to the left here,  where it's actually very difficult to detect an infection or difficult to detect if it's going to progress to sepsis. However, the good news is that if you do detect it early,  there are a variety of treatments that are effective. In most cases, the right broad-spectrum antibiotics can kill most of the bacterial infections,  and there are very effective antivirals. However, if you don't detect it and the patient continues to deteriorate, at some point on the right side of this line,  it becomes very easy to see that the patient is deteriorating. If a patient has a very serious infection that could be leading to sepsis at some point,  it was very obvious that the patient needs to be treated. However, the bad news is that after the patient crosses a certain line,  there's actually very limited effective treatment options. This is the fundamental dilemma. On the left side, hard to detect, effective treatments, on the right side,  very easy to detect, but limited effective treatments.  For Gnosis, overall approach is to combine a lot of pieces of data elements together to make up what we call the proprietary Gnosis dataset.  This consists of vitals, labs, outcomes, interventions. But on top of that, we add our own additional data, which we call the biomarker data,  which has actually been measured from samples that were drawn from patients for normal clinical care. When we say biomarkers,  what we mean is basically a whole set of signaling molecules, infection markers, organ dysfunction markers, all of which are not really ever measured in the hospital.  Finally, we do physician chart reviews on these patients that we recruit into our studies in order to get a better understanding of what's happening with the patient.  These are the elements that make up the proprietary Gnosis dataset. Then our vision here is to share this dataset with a whole ecosystem of partners,  including academics, diagnostic companies, hospitals, pharma companies, and health IT companies. The goal here is to make this dataset  available so that a whole host of products can be developed by us as well as by our partners. To date, we've recruited over 12,000 patients and  over 60,000 samples into the dataset in the biobank from six hospitals. You can see an overview here of the inclusion criteria for many of these patients.  The primary focus really has been on patients that have been suspected of a serious infection, as defined by the order of the blood culture in the hospital or emergency department.  This, again, doesn't need to be a positive blood culture, but just the order that is a surrogate for the suspicion of infection.  As you can see, we've also included many patients that were post-surgical, patients that were COVID-19 positive, and then patients with a variety of different comorbidities.  As far as we know, this is one of the largest dataset in biobanks that's ever been created that links very rich electronic medical record data or  the clinical data with the biological data.  When we talk about an ideal diagnostic for sepsis or ideal diagnostics in general, there's four main things that we can talk about. The first thing is performance.  How well can the tool differentiate between patients that are septic versus patients that are not septic? The second piece is timeliness.  Because it's well-known that patients with sepsis can deteriorate quite rapidly if they don't receive the appropriate treatment,  detecting and diagnosing sepsis on time is extremely important. The third piece is a prognostic capability. By this, what we mean is basically the ability of a tool to  identify which patients have the higher chance of deterioration, objectively speaking, even outside of just the diagnostic performance.  Then finally, this is related to the prognostic capability, but how severe is the response to an infection, so that a physician can make  a better and more informed decision about how aggressively the patient needs to be treated. These are the four main items that we keep in mind when we're  thinking about how should you grade or assess a sepsis diagnostic.  The interesting thing that if you go over past sepsis literature, there are a variety of different machine learning models that people have applied to this space,  typically in the clinical decision support space where you use a stream of EHR data to send sepsis alerts to a doctor or to a nurse.  On the other side, you have new biomarkers that are used essentially with sepsis diagnostics where you might test one or two molecules,  and then use those molecules to try to diagnose sepsis. There's not too much that combines the two approaches together,  which is really the land that the prognosis lives in with the sepsis in this world.  In order to develop the test, what we did is we actually assembled data from three different community hospitals in the United States. You can see OSF, Mercy Health, and Beaumont.  We recruited patients, again, only if they had a blood culture ordered for them in the ED or the hospital.  We had certain exclusion criteria based on essentially if data was missing, or if a sample is not available, we use those to exclude certain patients.  This ended up in roughly close to 2,400 total patients, 1,400 of which we use in the training process, and then 945 of which that we used as a standalone holdout validation.  There are many input features into the sepsis immunoscore. As we mentioned, it's a combination of clinical data from electronic medical record with biological data.  Here you can see about 20 different items that are either part of standard labs or standard vital measurements or demographic parameters, combined with three additional markers  that are not routinely measured for this patient population. Interleukin-6, procalcitonin, and C-reactive protein. The label that we used was basically  the sepsis 3 definition occurring within 24 hours. If any time after the blood culture order, the patient was positive for sepsis 3,  then that was considered a positive in the dataset. If they were positive for sepsis 3 after 24 hours or never had sepsis 3 at all, then they were considered a negative.  The way that we define sepsis 3 in this case was a modified definition, where infection was defined as more than four qualifying antimicrobial days.  Organ dysfunction was defined as a change in the SOFA score of two or greater. Then again, that event needed to happen within 24 hours of the blood culture work.  As we just talked about, the sepsis immunoscore basically takes these three biomarkers that we just discussed, eight vitals, 13 existing labs, and one demographic parameter.  This goes into machine learning model, which again is called the sepsis immunoscore, which then outputs a couple of things. First, it outputs a number between 0-100,  which is the probability of that patient developing sepsis 3 within 24 hours of the blood culture work. Second, it outputs a category, which can either be low, medium,  high, or very high. Depending on the category, the overall recommendation to healthcare providers is slightly different. If the patient is either at very high or high risk,  then sepsis is very probable. If they're at medium risk, sepsis is possible. If sepsis is either possible or probable, the overall recommendation is to be safe.  You should accelerate the administration of broad-spectrum antibiotics and overall, the sepsis bundle recommended by CMS.  However, if the probability of sepsis is low in the low category, then sepsis is unlikely, and we would prefer it if you considered alternate antimicrobials if necessary.  Again, two main outputs, the score itself from 0-100, as well as the risk category.  In order to set these thresholds between the low, medium, high, and very high categories, we thought a lot about what physicians actually want,  particularly in the emergency department. The threshold between low and medium was actually set in order to maintain a false positive rate of about 50 percent.  The reason why 50 percent was chosen was because we looked through our dataset and we observed that across many different hospitals,  we're seeing roughly a 50 percent false positive rate if you define identification of sepsis as the ordering of a qualifying sepsis antimicrobial.  Essentially, this threshold allows you to maintain the same level of over-prescription of antibiotics while hopefully increasing the sensitivity or increasing the number of  patients that you can catch at this level. The middle threshold here between medium and high was actually determined as the optimal threshold to get  the best combination of sensitivity and specificity on a receiver operating curve. The very high, high boundary was set to capture  the 95th percentile of sepsis 3 probabilities within 24 hours. This happens to also correlate to the incidence of septic shock in our datasets.  These are the four different categories and the three different thresholds that define those categories.  In order to evaluate this model, what we did is we took the independent test patient cohort of about 950 patients and we ran it through the machine learning model to create  a probability of sepsis 3 within 24 hours. In this case, of course, this model was locked after training and development, so it could no longer be changed.  We're only evaluating the performance of the model. The first thing we can evaluate is the diagnostic performance. The area under the receiver operating curve,  and then the four main diagnostic performance metrics, sensitivity, specificity, positive predictive value, negative predictive value.  Then the second piece that we did is we use that probability, as we just discussed, to place the patient to one of these four different categories, low, medium, high, or very high.  Then for each of these categories, we then observed in this validation holdout cohort, many different prognostic chances. In the very high category, for example,  we would figure out what is the length of stay for those patients, what is the 30-day mortality rate, ICU transfer rate, vasopressor administration rate.  We did that for each of these different categories to then show how each of these categories is related to the overall risk of the patient deteriorating.  First, we can talk about the diagnostic performance. You can see here that the overall model had an area under  the receiver operating curve of 0.855 for this population in the test cohort. Then you can see the diagnostic performance at each of those three thresholds.  Obviously, the low threshold is really about getting high sensitivity, so you catch as many patients as possible while maintaining as we talked about the false positive rate at 0.5.  Where the high threshold is really about making sure that you can rule in patients for having sepsis and it has a high specificity. The middle threshold is the optimal threshold,  which then has a sensitivity and specificity of about 0.79 and 0.78 respectively. One thing we do want to highlight is that this test at  the optimal threshold has a very high positive predictive value compared to other similar technologies in the field. This PPV of 0.64 is particularly  important when physicians suffer from a lot of alert fatigue where they can't trust positive alerts in the hospital or the ED.  For comparison's sake, we can also show you the diagnostic performance in this exact same test cohort of pro-calcitonin by itself and interleukin-6 by itself.  We're seeing about more than a 14 percent improvement in performance by using a machine learning model over PTP by itself. You can also with these models using SHAP techniques,  you can identify which features were the most important overall for calculating this probability for these patients. You can see up here the two biomarkers are  at the top of the list along with the Glasgow Coma Scale. But then you have a whole variety of the different labs and vitals that also contribute to generating this score.  Now, we'll talk briefly about the prognostic implications of the four categories we just mentioned. On this graph, you can basically see the time in  the hospital and the number of patients that were discharged by that time. The nice thing is that you see clear separation here. If you're in the very high group,  those patients tend to have a much longer length of stay compared to the high, medium, or low groups. You can also look at other things that are of interest,  for example, 30-day mortality, or the rate of ICU transfer within 24 hours, or the rate of vasopressor administration within 24 hours. Over and over again, you see that there is  this increasing risk or the increasing incidence of each of these different adverse events as you go from low to medium, to high, to very high risk categories.  As a summary of all of these prognostic implications, you can see some of these numbers. You can see the length of stay from low, medium, high, and very high.  You can see the 30-day mortality rate, the ICU transfer rate within 24 hours, and the vasopressor administration rate within 24 hours.  One thing that's important to note is that when we built this machine learning model, none of these parameters were fed to the model.  This just happens to come out of the data because of the increasing risk of substance. We feel that not just the diagnostic performance is important,  but also the prognostic performance, so that physicians can make very informed decisions about which patients they really need to ask for. There's a few conclusions.  Overall, earlier implementation of sepsis bundles, particularly broad-spectrum antibiotics, can reduce adverse events if you know the right patients to target for those activities.  Existing tests outpour both diagnostic and prognostic performance or non-existent prognostic performance.  Overall, the sepsis immuno score we demonstrated today has the potential to help assist an earlier diagnosis of sepsis, an accurate diagnosis of sepsis,  in addition to objectively determining which patients have a higher chance of deterioration. We're happy to take any questions during the live session. Thank you so much.

Video Summary

The CEO of Pernosis, Bobby Reddy, discusses the sepsis immunoscore, a biomarker EMR data machine learning model for early diagnosis of sepsis. Sepsis is difficult to detect until symptoms appear, at which point treatment becomes limited. Pernosis combines different data elements, including biomarker data, to create a dataset that can be used for developing sepsis diagnostics. They have recruited over 12,000 patients and 60,000 samples from six hospitals. The sepsis immunoscore uses clinical and biological data to calculate a probability of developing sepsis within 24 hours. It provides risk categories to guide treatment decisions and shows promise in diagnostic and prognostic performance.

Asset Subtitle

Sepsis, Pharmacology, 2022

Asset Caption

INTRODUCTION: Early administration of broad-spectrum antibiotics for patients at high risk of sepsis leads to reduced morbidity and mortality. However, timely broad-spectrum administration for these patients is often challenging due to the lack of information available to rapidly and accurately identify patients at high risk of sepsis. In this work, the diagnostic and prognostic capabilities of a machine learning model (MLM) that predicts the risk of sepsis within 12 hours after the first blood culture order (BCO), is explored. METHODOLOGY: Adult patients suspected of sepsis in an emergency department or hospital, as defined by a BCO were recruited from three U.S. hospitals (Nf2398). IL-6, PCT, and CRP measurements were measured from discards of samples drawn +/-3 hours from the first BCO using a magnetic bead assay. 23 other clinical features (results of a Complete Blood Count, a Complete Metabolic panel, vitals, and demographic information) were extracted from the Electronic Medical Records. A random forest model was trained (Nf1918) to provide a sepsis risk score and a prognostic risk category (low, medium, or high). Diagnostic performance for the MLM’s ability to predict a Sepsis-3 event within 12 hours of the BCO was quantified and compared to PCT and CRP in a test cohort (N = 480). The utility of the prognostic risk categories was explored by examining the differences in Length of Stay (LOS), 30-day-mortality, 30-day-readmission, and time to ICU transfer in the Test Cohort. RESULTS: The MLM exhibited an Area Under the Receiver Operating Curve (AUROC) of 0.87 compared to an AUROC of 0.765 and 0.581 for PCT and CRP, respectively. The MLM showed separability among the prognostic risk groups for LOS (p < 0.001), 30-day-mortality (p < 0.001), and time to ICU transfer (p < 0.0001), and 30-day-readmission (p = 0.018). CONCLUSION: In this work, a MLM which integrates 3 sepsis biomarkers (IL-6, PCT, and CRP) with 23 other routinely measured clinical parameters was shown to have excellent diagnostic performance and prognostic utility. Such a tool could help physicians quickly and accurately determine which patients should be prioritized for rapid administration of broad-spectrum antibiotics in hospital and emergency department environments.

Meta Tag

Content Type Presentation

Knowledge Area Sepsis

Knowledge Area Pharmacology

Knowledge Level Advanced

Learning Pathway Sepsis Resources

Membership Level Select

Tag Sepsis

Tag Antibiotics

Year 2022

Keywords

sepsis immunoscore

biomarker data

early diagnosis

treatment decisions

diagnostic performance

Sepsis

Pharmacology

Sepsis

Antibiotics

Presentation

Select

2022

Advanced

Sepsis Resources

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.