SCCM Resource Library-24-Performance Analytics of an Integrated Electronic Health Record Sepsis Trigger Tool

Video Transcription

Good evening and welcome everyone. My name is John Bersier, I'm a third-year internal medicine resident at Orlady of the Lake at LSU Baton Rouge.  The topic that we'll be discussing today is the performance analytics of an integrated BPA, electronic health sepsis trigger tool.  Before we begin, I have no financial disclosures.  Okay, so as an overview, machine learning algorithms have existed for some time and they have gained traction recently for their ability to be able to use  machine learning algorithms to implement raw data and put it into electronic health record.  It uses this information to be able to alert providers of this risk of severe sepsis. And obviously the performance characteristics of these risk scoring systems are necessary  to be able to appropriately integrate them into a clinical decision-making process.  The EPIC sepsis predictive analytics tool is one such machine learning algorithm and that relies on clinical variables, patient history, lab data, as well as vitals and diagnostics  to be able to assess the risk for severe sepsis.  And so the purpose of our study was to be able to  determine the diagnostic performance of this sepsis trigger tool.  We relied on data from the cognitive computing model brief. This was published previously by EPIC in 2016, in which they developed an algorithm  that we have at our facility currently. And that primarily was studied using physician coding  and billing data.  Primarily we looked at individuals that were enrolled in the emergency department and we utilize retrospective chart review as a means to be able to analyze them.  We enrolled individuals 18 years and older who met clinical criteria for SIRS with greater than or equal to two SIRS criteria. The window that we used was triage  to six hours following triage time. And we used a sepsis triggers tool of six or greater as being the cutoff value. Previously in EPIC in that 2016 analysis,  there was mentioned that any score between five and eight was more prognostically accurate.  We used a maximum score or six hour window as our window to be able to analyze individuals.  From this, we were able to input discharge disposition from discharge summaries, discharge to LTAC, SNF, home and discharge diagnosis.  In comparison to EPIC, we used adjudication by physician committees  to determine whether those enrolled met sepsis criteria.  And then from that information, we were able to determine the diagnostic performance and the receiver operating curves  used to characterize a sepsis predictive analytics tool diagnostic.  We use the Chi-square and Mann-Whitney U tests to analyze the non-parametric data.  A total of 226 individuals were enrolled. From that, we were able to gather the sensitivity and specificity as well as the negative predictive and positive predictive values.  A total of 188 individuals were evaluated for sepsis. And of those 11.1% were adjudicated as septic when there was evidence of infection and that was by physician committee.  So from the data we can see in the table,  the sensitivity of the test with a cutoff value of six was 76.1% and the specificity was 86%, slightly higher.  The true benefit of this test as we analyze it is really the negative predictive value with 96.6% that had a confidence interval of 93 to 98%.  From that information, we were able to develop a ROC  and the receiver operating curve is graphing the sensitivity over the one minus specificity or the false positive rate. We were able to see a good graphical interpretation  with an area under the curve of 0.85.  Then we evaluated the sepsis trigger tool and how it correlated with infection.  And what we see from that is that the sensitivity was significantly lower with a sensitivity of 33.7% and a specificity of 87.1%.  But the positive predictive and negative predictive values were sizably less.  Let's focus for a minute and determine kind of what that means. So we do know that individuals that have evidence of sepsis have both infection and organ failure.  And we know that this tool was designed to be able to accurately screen for those individuals that had evidence of sepsis.  So if the primary metrics that were inputted into this tool  were labs and vital signs, which were concerning for the development of sepsis, that hinged on the concept that there was organ failure or some widespread systemic inflammation.  And that didn't necessarily hold out as well when we tried to use it to be able to determine if there was infection or not. And so we see individuals  that had a score of greater than or equal to six that were slightly more infected when adjudicated,  but not significantly over.  And then in those that did have a score of less than six, we see that there was a greater majority of them that were not infected when evaluated by a physician committee.  So how to use this tool primarily as we kind of discussed previously was not in the analysis of either organ failure or infection but primarily in the combination of both.  It is to alert a physician of the early impending sepsis.  And so we use that information to be able to determine the ROC curve again, but this was for infection. And not surprisingly, we see that the curve is much flatter  and closer to this red line, which would indicate essentially random chance. And so we see that there is greater than a coin flip, but the area under the curve is approximately 67%.  And so this is sizably less than the ROC for the previous data, which was the sepsis data.  Then we were able to evaluate the length of stay for admitted patients. It was interesting that patients were admitted to both the medicine and the ICU floor  of which there was 149 patients admitted to the medicine floor and 14 to the ICU.  And it was noticed that individuals that had a score of greater than or equal to six had a longer length of stay on the medicine floor than on the score of less than six.  Score of less than six. This was not clinically significant as the P-value was 0.17.  And then when you look at the length of stay in the ICU, the average length of stay was actually shorter for those that had a larger sepsis score.  And when you kind of think about that, it makes sense.  Either individuals were sick enough to be able to expire in the ICU prior to discharge  or they were stepped down and ultimately had  a shorter length of stay as a result of that. And so again, this was not clinically significant with a P-value of 0.77.  And then we use this information to be able to compare organ dysfunction with the sepsis trigger score. We again used those meeting scores that were greater  than or equal to six and compared that to those less than six.  And we can see that the sepsis trigger score had sensitivity and specificity.  The greater than six, there was more likely to be organ dysfunction than no organ dysfunction. Again, that's not clinically surprising.  Given the parameters of what was inputted, vital signs and SOFA scores, including organ failure,  the BILI, the P to F ratio, the MAP, creatinine,  all of these parameters that we come to know are part of the SOFA constitute organ dysfunction. And if that trigger score is higher than it tended to correlate with the SOFA.  And so that is not surprising. And those individuals that scored less than six on this sepsis trigger tool, then we saw that there was more likely to be no organ dysfunction.  This correlated to a higher specificity than a sensitivity with the specificity being about 93% and the sensitivity being about 39%. This had a higher positive predictive value  but a less optimal negative predictive value.  And then we use the ROC curve and we can see slightly improved overall with an area under the curve of 0.74, but still not as high of a ROC curve  or as accurate, I guess I should say, as the first one with the organ dysfunction and evidence of infection.  And so then we looked at organ dysfunction without infection. And this was as determined by objective infections. This is information largely that we obtained from the CDC  to be able to more than just a gestalt have some objective basis to determine if someone met clinical criteria for an infection. So for instance, we use UTI.  And so for urinary tract infection,  you had to have documented evidence in the triage note  or in the ER physician notes or the HNP that had hematuria and dysuria. And if it clinically met evidence of objective infection  for a urinary tract infection with leukocyte esterase  or bacteria or all of these other parameters,  it was not just, you know, kind of mix and match. It had to be had to be thorough.  So we see that with organ dysfunction without infection,  those individuals that had a score of greater than or equal to six, there was more likely to be organ dysfunction. And those that had less than six,  there was more likely to be no organ dysfunction. This had a higher specificity with a 94.6% as compared to a sensitivity, which was significantly much lower at 22%.  Using that information, we were able to derive a ROC.  And this was not quite as good as previous. We see an AUC of 0.712.  Some of the information just to kind of touch on this, the sepsis trigger tool would fire for things  such as tachycardia or hypoxia or supplemental oxygen use or individual meds, which were started. And you can imagine that some of those may have been extemporaneous.  And some of those say the application of nasal cannula in those individuals that may have had dyspnea as a result who did not necessarily have hypoxia.  And so there may be those opportunities where individuals will have no evidence of infection and they will have documented organ dysfunction,  which would ultimately not be attributable indirectly to sepsis or to direct measures such as the SOFA.  And so then we looked the organ dysfunction and infection, and we were able to determine that the specificity was much higher than the sensitivity with 90%  as compared to the sensitivity of 58%. This had a higher positive predictive value.  And that ultimately contributed to a higher positive predictive value  as a result of the higher specificity than the sensitivity.  From that, we were able to implement the ROC curve, and we can see that this again had a much better  logistic regression with the area under the curve being 0.79.  That brings us to the end of our discussion. And I wanted to thank everyone for your patience and your attention and for your kind consideration  for listening to my talk. I hope that you gained something from it, and I really appreciate you.  Thank you. ♪♪♪

Video Summary

In this video, John Bersier discusses the performance analytics of an integrated electronic health sepsis trigger tool. The tool uses machine learning algorithms to analyze raw data from electronic health records and alert healthcare providers of the risk of severe sepsis. Bersier's study aimed to determine the diagnostic performance of this sepsis trigger tool by analyzing data from 226 individuals who met clinical criteria for systemic inflammatory response syndrome (SIRS). The study found that the tool had a sensitivity of 76.1% and a specificity of 86% in detecting sepsis. However, when it came to detecting infection, the sensitivity dropped to 33.7%. The tool was found to be more accurate in predicting organ dysfunction than infection. The study also analyzed the length of stay for admitted patients and found that those with higher sepsis trigger scores had longer stays on the medicine floor but shorter stays in the ICU. Overall, the sepsis trigger tool showed promise in identifying potential sepsis cases, but further research is needed to improve its ability to detect infection.

Asset Subtitle

Sepsis, Administration, 2021

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2021 Critical Care Congress held virtually from January 31-February 12, 2021.

John Bercier, MD

Introduction/Hypothesis: Machine-learning algorithms in the electronic health record are being used to alert providers of severe sepsis risk. Accurate performance characteristics of these risk scoring tools are necessary to appropriately integrate them into clinical decision making. The EpicÂ® sepsis predictive analytics (SPA) tool relies on clinical variables, patient history and diagnostics to assess risk for severe sepsis. This retrospective chart review evaluated the diagnostic performance of the proprietary EpicÂ® SPA Tool in the emergency department (ED).

Methods: Patients presenting to the ED with at least two of the Systemic Inflammatory Response Syndrome criteria were assessed. All SPA scores were recorded up to 6 hours after triage started. A score >6 was used to indicate a high likelihood of sepsis. The maximum score within the ED encounter or the 6hr window, whichever came first, was used for analysis. Discharge disposition, length of stay (LOS), and discharge diagnosis were recorded. Patients were adjudicated for sepsis by physician committee. The diagnostic performance and receiver operating curves were used to characterize the SPA Tool diagnostic performance. Chi-square and Mann-Whitney U tests were used to analyze non-parametric data.

Results: The SPA scores were obtained for 226 patients of which 188 were evaluated for sepsis and 11.1% were adjudicated as sepsis. The SPA Tool had a 76.2% sensitivity, 86.22% specificity, and a 96.64% NPV based on a cut off score of 6. The AUC was 0.859 (95%CI: 0.77â€“0.95). Of the 226, patients with a score <6 were more likely to have been discharged from the ED (41.7%) compared to those with a score â‰¥6 (7.8%, p<0.001) and more likely to have a non-infectious discharge diagnosis (<6: 65.1% v â‰¥6: 43.1%, p<0.001). 149 patients were admitted to the hospital. Those with a score <6 had a median (IQR) hospital LOS (3 (2,6)) comparable to those with a score â‰¥6 (4 (2,6), p=0.174). Of the 14 patients admitted to the ICU, there was no difference in the ICU LOS (<6: 3.5(1,6.5) v â‰¥6: 2.5 (1,6.25) p=0.775).

Conclusions: The benefit of the EpicÂ® SPA tool lies in its negative predictive value when used in a large hospital ED. With limitations of sensitivity, the scoring tool may function best when paired with other high sensitivity diagnostics for optimal patient categorization.

Meta Tag

Content Type Presentation

Knowledge Area Sepsis

Knowledge Area Administration

Knowledge Level Foundational

Knowledge Level Intermediate

Knowledge Level Advanced

Membership Level Select

Tag Scoring Systems

Tag Informatics

Year 2021

Keywords

performance analytics

sepsis trigger tool

machine learning algorithms

diagnostic performance

infection detection

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.