SCCM Resource Library-Star Research Presentations: Predictive Analytics

Video Transcription

Good afternoon, everybody. My name is Robert Stevens. I'm from Johns Hopkins University. It's my pleasure to open this special session. It's the STAR Research Award presentations,  and it's devoted to the topic of predictive analytics. Without further ado, I'd like to start with our first speaker. This is Dr. Rohit Gupta, and he's  going to be talking about Muse++, Predicting Clinical Deterioration in Admitted Patients Using a Novel Machine Learning Model. Dr. Gupta?  All right. Hi, good afternoon. My name is Rohit Gupta, and I'll be talking to you guys today about our experiences with machine learning models  and how we use it to predict clinical deterioration. So just coming to the timeline of deterioration, typically we have patients who have a period of clinical decline until they  reach a point of escalation. And at that point, a decision is made to upgrade to a higher level of care. And what the goal has been with prediction models in the past  has been to try and secure a lead time prior to the escalation point so that you have room to do some kind of intervention, whether it's diagnostic or therapeutic, in an attempt  to stabilize the patient. Just coming to the escalation point itself, it's the point at which care requirements necessitate escalation to a location  with more monitoring and support. And one thing that's important to realize is besides clinical criteria, which of course drive the escalation point, there  are also institutional criteria that are at play in defining the escalation point. So the same patient with rapid atrial fibrillation might be managed on the floor in one institute,  while at a different institution, they might require escalation to the ICU. So whenever we're dealing with these factors, it's important to incorporate that into your prediction  model so that the score is not just accurate, but it's also relevant to the practices. Coming to some of the traditional prediction models  that we've seen, one of the most commonly used is the modified early warning score. It's a pretty intuitive score. As you can see, it's based on four vital signs, urine output  and mentation. And the scores ranges from 0 to 17. And anyone who has a score of 3 or higher gets an ICU consult. And based on that, they get sent to the next level.  Some of the challenges with a score such as MUSE is that it utilizes a limited number of variables. There's equal weightage to all variables.  And as we all know, some of our patients are driven by a particular organ system dysfunction. So it can create challenging errors when you have equal weightage.  There is no defined time window of when the escalation point is going to arrive. And more importantly, the MUSE score is independent, again, of institutional practices.  Just before I go on to our machine learning model and how we've used it, I thought it might be prudent to start with just a little slides on the building blocks of the model.  And one of the building components of a machine learning model is the decision tree. And essentially, what it does is it classifies a particular value into one or two classes  based on a set of variables. So this is an example of a decision tree on whether or not to buy a car. And you go through a set of questions  where you're looking at is it all wheel drive, what's the engine capacity, and what's the ground clearance. And ultimately, you stratify the car into one of two categories.  So in this example, we have the questions which are the features which define the classification. And ultimately, you're classifying them in one of two groups.  So if you were to design a clinical triage decision tree, what would some of the features be? Well, you could look at white cell count. You could look at lactate.  You could look at mentation and blood pressure. And ultimately, you decide whether that patient, based on those features, needs to go to a higher level of care or not.  One of the challenges when you have a simplistic tree like this is that errors that come about from a weightage to each variable can get pronounced.  And it can skew the decision to where it should not be from where it should be. And this is where the power of machine learning models really comes into play.  What we can do, instead of having one decision tree, is having multiple decision trees that take from a set of features randomly different features,  each of which that we know predicts or contributes to escalation, and then generate independent predictions. And then using that, we can aggregate the results  of those multiple decision trees. So this would be a random forest model where the first decision tree uses lactate and mentation,  and the second tree uses white cell count and mentation, and so on and so forth. And you can generate hundreds of trees and then aggregate the results to minimize the error  and to avoid overfitting the prediction to the data points that you have. So what is our model? Our model is the Muse++ model. It's a random forest classifier similar to what  I just spoke of. And what it really does is it predicts clinical deterioration by defining deterioration as either an escalation  to a higher level of care or an unexpected cardiac arrest. So the first step that we did was we trained our model to identify these unexpected escalations.  And we did that using six years of patient data, almost 100,000 patients, and almost 250,000 bed movements to first identify bed movements that defined our clinical deterioration.  Once the model was able to identify, we then used a set of 36 features to predict the escalation event. And once the model was shown to be able to predict the escalation event,  we used a separate set of patient movement data to test its performance against the traditional Muse model to see how it performed in predicting escalation of care.  Some of the features or variables that the model uses to predict an escalation range from the location, whether they were admitted from the ER,  whether they were admitted from the clinic. It includes laboratory values, white cell, BUN, creatinine, lactate. Includes patient features like height, weight, vital signs.  And then it also incorporates some exam findings, all of which is pulled from the electronic health record system. Since we were training this on retrospective data,  we could generate predictions of the escalation multiple time points prior to the escalation. And so what we settled on was using a six-hour prediction  window so that you have a time window that is clinically actionable and relevant to our practice. So summarizing what we did with this model,  it provides a probability score between 0 and 1 for the likelihood of clinical deterioration as defined by escalation of care or death within a six-hour time period.  The model uses 36 different features to predict the likelihood of this clinical deterioration. And the scores get continuously updated in real time as new values become available.  This is a sample of a report that gets generated for a patient. So as you can see, it'll tell you the location of the patient. It'll give you a probability score.  And then out of those 36 features, the machine will list the top 10 features that it feels is contributing to the need for escalation of care.  What are the advantages of such a system? One of the strong points towards the Muse++ model is it gets updated in real time. As and when you get new values, the score  gets continuously generated and reported. It utilizes multiple distinct features and not just a list of four. And those features get assigned different weightages  depending on the patient that you have. It gives you a six-hour notice period. And because it's trained on bed movements, it is reflective of institutional practices.  So coming to the performance, compared to traditional Muse, the scores have significantly higher sensitivity specificity. And if you look at the sensitivity moving hours  prior to the escalation event, it remains stable as compared to the degradation in a traditional Muse model. This is, again, the specificity.  We can see that for Muse++, we have a superior specificity score compared to traditional Muse. Just really briefly, I'd like to talk about how the next step  we've gone with this model is to incorporate it into a rapid response team service. And what we're trying to see is whether using Muse++ as an independent trigger for a consult  leads to a reduction in the escalation of patients. So just coming as a summary to where we are, we now have a machine learning model that gives us a six-hour or greater lead time.  It gives us an intervention arm, which is the rapid response team, that allows us to try and stabilize the patients and minimize escalation.  Some of the challenges that we faced with this model is it is conceptually dense. And when something is conceptually dense, it creates issues for adoption.  It creates issues of trust. There are issues of liability that come up. If you feel someone can stay on the floor, but the model says it's high probability, that can be a challenge.  And then you're left with intervention limitations. Even if you get a score, what do you do with that? I think I'd like to end with just a slide on where we are going forward.  These models are becoming increasingly prevalent. We're using it for predicting faults, malnutrition. And the challenge is to do them effectively,  where we can use them in a relevant practice. And you don't have to be a data scientist to be involved in this process. But it's, I think, important that we are  there in the decision-making. I'd like to just end with our paper. And I'm happy to answer any questions, if you may have. Thank you.  Thank you very much, Dr. Gupta. We have a few minutes for questions.  While people are thinking about questions, I might just ask one. Have you been able to validate this model externally on a population outside of your medical center? No.  I think our first step was to try and see in-house what it does. We started out with just a validation set. And now we're working prospectively  with a randomized trial, looking at certain floors where we've implemented it versus other floors where we just use the traditional rapid response.  Well, the question has to do with the generalizability. Do you feel like this is a model that could be used in other places apart from your institution?  So I think one of the challenges is the training of the model to recognize bed movements in the institution. And some of that might require data from previous years  to try and train it to be relevant to an institution. But I think as we learn more, we're able to adapt it to our other institutes. OK, thank you. Sir? One question.  Thank you very much for this presentation. How do you deal with delta mus? Because that is something that has been traditionally used in predicting somebody getting worse.  Because if your patient stays statically at a certain mus level, your prediction model, how does that handle it? When you say delta mus, you're referring  to the change in the traditional mus score? Yeah, so changing the mus score from time to time. So I can speak to our experience with mus and with our machine learning model.  We've sort of moved past mus just because it's been a cumbersome practice to look at scores and look at changes. I can't speak specifically to whether a change in mus  can be used as a more accurate predictor. I think our experience has been that before our machine learning model, we had been relying more so  on the triggers for a rapid response consult and then using that as a trigger for escalation. Great. Can I? Excellent talk. I think one question I had for you  is Matt Sherbeck from University of Wisconsin. Can you mention a little bit more about your inclusion of lactate in the model? I guess there's two things I'd wonder about.  One would be how often it's missing and how that missing is handled in the model. And the second is the fact that that is something  that clinicians typically send when they're already worried. How that worry from checking the lactate alone may influence how your model performs. Right.  So I think one of the ways that the random forest model is at an advantage is when you have a particular feature that's missing, say the lactic acid value is not  there for a patient, what it does is it can incorporate the other features to sort of give you a more accurate prediction. If you don't have the lactic acid as a number,  it won't factor that into the prediction model. And it'll subtract that from the decision trees.  So just quickly, one last question. You mentioned real-time perspective prediction ability. So actually, can you tell us how long  it takes for your random forest to generate this probability?  Minutes, seconds? So as you get data, the machine populates it from the electronic health system and generates it real-time. So it's within seconds.  What we've done is to reduce alarm fatigue. We don't send the report out to the response team unless it's been a significant increase within four hours.  So that's how we try to balance out the alarms that are generated. Thank you very much. Thank you.  OK, so it's my great pleasure to introduce our second star award recipient and speaker today. This is Dr. Thomas Higgins. And he'll be talking about inpatient deterioration.  Can artificial intelligence predict who will be transferred to the ICU?  Thank you. Good afternoon, everyone. It's nice when synergy happens in terms of many people looking at the same issue. And one of the themes of this conference  is artificial intelligence. And I think this is all coming together and in just the right time, because we're going to need some help as manpower shortages get worse.  So I'm working on this project now on inpatient deterioration. And in a way, it's an expansion of what many of you would be familiar with as the Apache physiology.  We're not going to call this Apache 5. But in a way, it is, because it expands the Apache methodology to all inpatients.  So we know that early detection of deterioration is important and that subtle signs are challenging for us as humans to detect. But machines are much better at doing that.  And we also now have pretty much universal adoption of the electronic medical record, which will give us data like physiology, vital signs,  laboratory data, primary diagnosis, and comorbidities, and what support interventions are being provided to a patient. It's also easy for the machinery to calculate  for its elapsed length of stay and the current location of the patient, although that can sometimes be a challenge, because a lot of the EMRs  don't actually tell you when a patient's gone to the operating room and back to the floor again. But usually, the ADTR system will tell you exactly where the patient is located.  So these are all valuable pieces of information. So we took data on 56,000 patients at 11 hospitals that were contributing to the Cerner Health Facts database between 2014 and 2016.  And this was split into validation and development sets. And we considered a large number of variables, 40-plus variables, including the traditional Apache variables  of heart rate, respiratory rate, mean arterial pressure, temperature, hematocrit, platelet count, sodium, diagnoses from over 450 that were recorded,  more than 40 support interventions. And this would be giving the patient IPPB, or non-invasive positive pressure ventilation, or dialysis, for example, demographics,  and the minimum Glasgow Coma Scale. So if you look at Apache, or MPM, or Muse, or pretty much any system, these are the common things that we think, as clinicians, contribute  to a patient's success or failure, and ultimately their morbidity. But it's how they're combined and how they're interpreted in the light of other information that  becomes important. So we used logistic regression against the outcome of transfer to the ICU from a lower level of care, whether that was floor or a step-down unit.  And we created, with that logistic regression, a summary score, which we call a severity of illness score. We set the observation window at eight hours.  And this was primarily because that's the maximum interval at most hospitals that you'll go between getting vital signs. SAS was the software we used.  We dropped patients from analysis if more than one value was missing, and used the last prior data point if there was only one data point missing.  We excluded pediatric patients and those who expired in the hospital for this particular analysis, although it was used for different morbidity and mortality analyses.  Initially, we were getting fair areas under the area receiver operating curve, which is a measure that kind of summarizes sensitivity and specificity.  But we needed to do better than that. And I'm going to talk a little on that particular aspect of this project on Tuesday. But we did discrimination by AUROC, calibration by curves,  and looked at sensitivity, specificity, and false alarm rates at various levels. Now, one question you always have to ask is, are your patients like my patients?  Is this representative of what you see? And here's our population characteristics. About 5.5% of patients were transferred to a higher level of care in the next 24 hours.  Mean age was 63 and 1 half. Race was primarily white at 69%, but rather representative of what we see in much of the United States. And the hospital bed size were all over the map.  The one thing that may not be representative here is that these were all teaching hospitals. 99.1% of the patients were in a teaching hospital.  Now, the mortality was 10.2% for those patients who were transferred versus 0.14%. So that's a 70-fold relative risk if you have an unplanned transfer to the ICU.  And the predictive model incorporates the current severity of illness, the pattern of change in severity, and support intervention, some of which  actually decrease the risk of transfer. For example, pacemaker actually protects you against transfer. Elapsed hospital length of stay, the prior acuity level,  and selected diagnoses, which are listed there. Now, this is difficult to read from the back of the room, but the blue bars are those that increased  your risk of transfer, and the green were the ones that decreased the risk of transfer. And number one was the current severity of illness, followed by the number of support items  that that patient was receiving. If the patient was already in a step-down unit, that was actually protective, as was a permanent pacemaker.  Maximum measure of the minimum heart rate, so if the minimum respiratory rate didn't go down too far, that was actually protective. And female gender, for some reason,  was protective as well.  The contributors to the model are primarily the current severity of illness, the support items, and the change in the support index in the last 24 hours.  But there are other things that are important as well, including antibiotic use, comorbidities, gender, and the prior level of acuity.  One of the more important things that we did with this was we didn't use a static measurement. We used a rolling measurement and a sequence of three severity of illness scores,  and we were able to create 27 words using three letters, U for up, F for flat, and D for down. So if your sequence was UUU, in other words,  severity had gone up three times in a row, or up, up, and flat, there were four very high-risk sequences that had 2 and 1 half times the relative risk of a transfer.  And then there were a lot of sequences in the blue here where it actually was below the average transfer rate. So we categorized this as red alerts, yellow alerts,  and blue alerts, and we're testing this prospectively now in the background at a hospital to see how well it works. The model discriminates pretty well with an AUROC of 0.82.  With the alert threshold set at the top decile of risk predictions, it was 90% accurate, 51% sensitive, and 92% specific with a false alarm rate of about 7.5%.  That compares, and it's difficult to find this in the literature, but using the MUSE data with our data set, 0.698, or the 0.775 that's cited by Escobar and colleagues  in their deterioration results in 2012. It calibrates well, as you can see by this curve. The modified Briar score, which is a measure of calibration  that depends on the rate, was 19.7, and actually was 21.2 if you excluded the step-down patients. So the important points here are the patients  who transferred to the ICU unexpectedly had a 70-fold relative mortality. There's numerous studies out there documenting failure to rescue,  and early warning systems may not have sufficient impact. We think that the sequence of three captures important longitudinal changes. We recognize that nursing concern  is a very valuable indicator, and that's hard to capture electronically, but we also think that having the machine backing up the nurse in terms of a gut feeling  is a very valuable thing. Those of you who went to this morning's lecture noted that there is a penalty for predicting too early, and we found that as well.  So looking back at 88 hours before the patient was transferred, there's not much of a difference between the non-transfers and the transfers, but in the 24 hours prior to transfer,  you begin to get some spread between the two groups. So in conclusion, the score incorporates physiology much like Apache, ICD codes for diagnosis and support intervention,  plus age and minimum Glasgow Coma scale. Critical sequences in the change of the score in the prior three eight-hour intervals improves detection,  and it's about a 7.6% false positive rate. It would be possible for institutions to vary their threshold based on local conditions, and we're playing that now  with some of our development sites in the background just to figure out where that ideal threshold ought to be. And next steps will be prospective validation  on a much larger number of hospitals. It can be incorporated into the Cerner EMR. There's no reason it couldn't be incorporated in other EMRs as well.  And finally, as disclosure, my co-authors, Maureen Stark, Kathy Henson, and Laura Friesman-Freeman are all current employees of Cerner Corporation, and I'm a consultant for Cerner  and receive research support for this project. So I thank you for your attention. Thank you. Do we have any questions for Dr. Higgins?  Hi, Andrea Ryan from the Washington Hospital Center in D.C. Two questions, just a little bit about the methodology. One, how did you determine severity of illness?  And then second, I noticed Glasgow Coma Scale is pretty important in your methodology, and I'd be willing to bet not all general nursing units use the Glasgow Coma Scale.  They probably use some other kind of way to identify, probably mostly by nursing documentation, what the level of consciousness is. Are there other thoughts about other things  you could use besides the Glasgow Coma Scale? So severity of illness and GCI. That's a good question. So we were using the information that was documented in the record,  and we defaulted to 15 if the Glasgow Coma Scale was not charted on the assumption that the patient was fully conscious.  The techniques used to determine the weighting was very similar, and I know you were involved in mortality probability models way back and very similar to that methodology  for logistic regression to look at the weighting. I think the difference between this and Apache and MPM is it's driven entirely off the electronic medical record  rather than requiring a nurse to collect the data.  If I might ask, have you thought about other additional features that could be added or variables that could be added to your model to increase discrimination? Oh, yeah.  I mean, we were looking at all sorts of things. I mean, there were probably, if you include all of the diagnoses, which are 450, probably 600 or 700 things  that were looked at in terms of providing discrimination. But you have to be careful about overfitting the model, and it seems to be kind of a threshold of around 0.83,  0.82 in terms of the area under the ROC curve before you get to overfitting. It will be possible going forward, particularly  with more deep learning systems, to pick up additional data, but we didn't have the data to do that yet. Again, the beauty of incorporating it into an EMR  is that all of the data essentially becomes free and the analysis becomes easier. So I have to ask you the same question as for Dr. Gupta,  what is the timescale for generating this probability? Is it instantaneous? Well, it's real time, but it depends on how recently you've entered data.  So the maximum would be eight hours, because that would be the vital signs. Now, if the patient is on Q2 hour vitals or Q4 hour vitals, then it would be faster than that.  But eight hours is sort of the default to give you information of that sort. And it was a point I was going to make about that.  Oh, so you have to wait eight hours to get the result? So let's say the patient generates one of these, no, the result will flag immediately.  So let's say you document a respiratory rate of 26 right now and the patient had previously been 18. That's going to flag instantly if that's enough points  to drive them over the threshold. What we've also done with the sort of observe, monitor, and do something, act with the sequences of three,  if you're in that yellow middle sequence, one of the recommended indications will be to increase the frequency of vital signs to Q4 or Q2 hours rather than Q8.  Okay. Any other questions for Dr. Higgins? Okay. Thank you very much. Thank you. So our next speaker is from Johns Hopkins, Hanbin Kim.  His talk will be titled, A Machine Learning-Based Prediction of Cardiac Arrest Outcome Using a Large Multicenter Database. All right. Hello, everyone.  I'm a graduate student from biomedical, in the biomedical engineering department at Johns Hopkins University. And my talk will be about a recent study  to create a machine learning algorithm to predict post-cardiac arrest outcomes using a large multicenter database. So cardiac arrest is one of the leading cause of death  and neurological disabilities. And after resuscitation, a patient may receive many different types of outcomes, as you can see on the right. Green here representing good outcomes  where a patient may have full recovery. And red, meaning bad outcomes here, representing either severe disability or death. And one of the limitations currently is  that there are not many models or methods that may be able to predict earlier on a patient's neurological outcome or mortality in the ICU.  So the main goal of the study is to develop and validate a model that can make an early prediction of these outcomes within the ICU.  And for the study, we used the Philips EICU database. And it's a retrospective database that contains 200,000 ICU stays of 208 different hospitals.  And it contains a granular data of electronic health records and physiologic time series data. And here, physiologic time series data refers to the patient bedside vital monitors,  such as heart rate data, oxygen saturation, blood pressure, and respiratory rate, recorded at a five-minute interval. And so these physiological signals  and electronic health record data are available for a majority of the patients. And by using such a large database, our analysis will be very generalizable to a lot  of different populations. So the objectives of our study was to first predict the neurological outcome and mortality at discharge using data only  from the first 24 hours of ICU admission. And the second objective is to determine whether or not utilizing physiologic time series data, specifically just features derived  from the bedside monitoring data, are useful in terms of model performance. So to briefly go over our methods, we used the EICU database.  So the first step would be to isolate our patient cohort, mainly the post-cardiac arrest patient group, by filtering the 200,000 ICU stays through our inclusion  and exclusion criteria. Then we pre-processed the electronic health record and physiologic time series data so that we can extract relevant features from the data,  as well as dried features if needed. Then we applied machine learning algorithms, specifically logistic regression, neural network, XGBoost, random forest, yeah.  Those algorithms to make a prediction of a patient's neurological outcome and mortality. And here neurological outcome and mortality are both binary labels of zeros and ones.  So this is a classification task. So here is our plot for our results for predicting neurological outcome using only the first 24 hours of data from the ICU.  And here the Y label represents the area under the receiver operating curve, which all our models are evaluated on. And the X axis represents the different models that we evaluated.  And here on the left, the leftmost box plot represents the Apache logistic regression model, where we only utilize the Apache variables and train the logistic regression model.  And that serves as our reference model. And our model starting from neural network to XGBoost and our ensemble model, which is a stacked model created  by combining the four models, performs much better compared to the reference Apache model. And here are the results for mortality.  It is the same trend, except some models switch positions in the plot based on their area under the curve. And we again see that our model performs significantly better  compared to the Apache reference model. And to assess whether or not physiologic time series data does benefit model performance, we, so this plot shows you the different ROC  curves based on the different subsets of our feature space. And here we see that electronic health record and physiologic time series combined is best at prediction  for the neurological outcome. And for mortality, we see the same trend where the added information of physiologic time series, yeah,  added benefits are physiologic time series at about 5% when combined with electronic health record. And aside from trying to maximize the model performance  of our models, we also wanted to make sure that our models are interpretable and there are some interpretations as to which features are most predictive  and which features contribute most to the modeling. And here we have a plot here with a y-axis of relative importance, relative importance for neurological outcome.  And we see that motor Glasgow Coma Scale and its subscores as well as initial temperature, lactate value and mainly the heart rate variability and heart rate stability were some  of the most important variables at predicting neurological outcome. And as you can see in the rankings here, that a lot of the physiologic time series data categories are  ranked higher than some of the more static electronic health record features showing that the physiologic time series data does have an added benefit.  And we can see a similar plot for mortality. And since we're changing the outcome labels, we see there's a shift in some of the features except things  like motor, the Glasgow Coma Scale are still one of the top contributing features. And now that more lab features are contributing to the outcome as well as different demographics.  And to conclude, we were able to create a model that was able to predict post-cardiac arrest outcomes using, within the first 24 hours of ICU admission,  models utilizing both the physiologic time series and electronic health records, outperformed models of either alone. And models utilizing machine learning,  outperforms logistic regression in both cases. And for our future work, we mainly trained our models using the electronic EICU database.  There are other retrospective data sets out there and so we would like to do some external validation as well as some prospective validation work if possible.  And currently, I'm working on cardiac arrest treatment intervention modeling where we're trying to assess whether or not a patient will benefit from targeted temperature management.  So, yeah, I would like to acknowledge my colleagues, my engineering professors and Dr. Robert Stephens who's my PI and thank you.  Thank you very much, Han. So, any questions?  Very nice talk. I have two questions. The first one is can you elaborate a little bit on how you engineer a feature from the time series data?  And then the second question is can you comment a bit on why you choose neurological outcome as one of the end points for your cardiac arrest patients?  To start with the second point, for cardiac arrest and many of the patients in the ICU, from the expertise of a lot of clinicians, I was told that a lot of the main factors  of a good ICU stay comes from having a good neurological outcome where a patient can go back to normal life. So neurological outcome was one of the main factors  that we chose aside from mortality. And in terms of the feature space for time series, so given the periodic time series at five-minute increments,  we extracted not only the basic statistics from the entire 24-hour bin, but we also extracted some more granular statistics from different time windows within that 24 hours as well  as we took an approach of doing an exhaustive analysis by using a already created MATLAB package called HCTSA. And this was mainly created for genomic work  where there's a high dimension of periodic data involved. So, yeah. Yes. Any other questions? Yes. Thank you for the talk. I have a question. So in machine learning, the combination  of features plays a big role. So could you explain a little bit more how you were able to calculate the importance of each feature individually?  Oh, so the individual feature rankings were done in two ways. So first, there's the logistic regression or the GLM way to calculate the coefficients.  And based on the magnitude of the coefficients and the positive and negative of the coefficients, we can see which features are most important.  Another way that's more recent is the random forest ranking. So there are methods within random forests to rank based on the minimum depth of where a certain feature splits  on a decision tree and how frequently that feature appears within a bound of different nodes. So that's how we were able to assess which features were most important.  Okay. Thank you. Okay. Thank you very much. Thank you. Yeah. So our last speaker at this session is Laura Friesman-Freeman from Cerner Corporation.  And she's going to be talking about prospective validation of an automated electronic inpatient mortality prediction tool. I guess you didn't get the word.  So Laura couldn't make it. Okay. So I'm going to sub in for her today. It happened that I was here anyway. So here's a little story for you. It was 40 years ago at this meeting.  And the story I heard from the participants was that it was over a glass of wine after the end of the sessions that Apache was first developed.  It was a bunch of ICU docs getting together and saying, well, what really makes a mortality prediction? Yeah, old, that's not good. If you're in septic shock, that's not good.  And that was the beginnings of the Apache score. So in 40 years, there's been a lot of modeling going on, followed by mortality probability models,  SEPs from our European colleagues, ICNARC from the British for general use. And then many, many systems for specialized populations, cardiac, pediatric, and trauma.  And the reason is if you do logistic regression, it works if you have a homogeneous population, but you're never going to get good results unless you specify  the model for that population. So there are a lot of limits to existing systems. Among them, most had required manual data collection.  Apache, for example, used proprietary diagnostic groupings rather than ICD-10 codes, which meant you were doing things differently than everyone else.  And the models were optimized for the ICU, not what we're seeing today, which is the ICU without walls. So building on that Apache methodology,  we built new models using 100% EMR-extracted data. And now we're reporting on the prospective validation of this model, which we presented two years ago  at SCCM on an additional 83,000 patients. So this is the consort diagram. We used the Cerner Health Facts database. Importantly, we excluded pregnant patients  because the mortality rate is so low, you really can't develop a good model on them, and pediatric patients because they're physiologically so much more different than adults.  We had previously reported on this part of the study, and now we're reporting on the revalidation database at six hospitals, four of which overlapped  with the original series, but more recent and a larger number of patients. So there's 207,000 patients all told to go into this model development.  For this particular piece of the study, de-identified data was used. The exclusions I've already mentioned and variables were previously identified  by logistic regression against mortality, and these include the classic physiologic variables, diagnoses, support interventions, and demographics.  And performance was assessed by area under the receiver operating curve plus the modified Breyer score. The point values for the comorbidity index were also  determined based on logistic regression, and they vary from 41 points, which is high for a patient with a bleeding diathesis, down to the interaction terms.  For example, a COPD patient who also has a stroke gets additional points. And you can see the clinical rationale behind some of these things, because clearly a patient  who's had a stroke cannot cough or clear their secretions as well as someone who hasn't had a stroke. The support index was done the same way,  and interestingly enough, if you had a permanent pacemaker in place, that was protective. It gave you negative 31 points. The others would all be familiar to you  and probably make some clinical sense. Patients who are on IV antiarrhythmics, though, actually did better because they were less likely to have arrhythmias.  So I'm going to present some of the results of the revalidation here, including the hospital characteristics, the most frequent diagnoses, the distribution of the scores,  and the model performance. Again, this database was used for the other study that I reported earlier, but the numbers are slightly different because of exclusion criteria.  These patients were 61 years of age on average, 47% male. The racial distribution is a little bit different in this revalidation set than in the original set,  as is the hospital bedside and the proportion of teaching hospitals. The mortality rate is 3% in the development set and 3% in the revalidation,  although in the initial validation data, it was actually a little bit different. But that was not statistically significant than the developmental data.  But these are the things I want to point out as being a little bit different in the validation set, the revalidation set, than in the original study.  I think the teaching hospital issue is important. And if you think about it, the vast majority of hospitals in the United States are not teaching hospitals.  So I think it is important to have that in your mix with model development. I'd ask you to look at this list and see if it makes sense  to you in terms of the most frequent diagnoses. Now, understand that this was a retrospective study. So the diagnoses were coded off of when the patients were discharged.  If you're using this real time, you're using whatever the admission diagnosis is, and that can change. But the model allows for that by allowing the probabilities  to change as the patient's diagnosis changes as well. But septicemia was the most frequent diagnosis here  representing 14% in the validation set. Also common atrial fibrillation, heart failure, osteoarthritis, acute myocardial infarction, other chronic ischemic disease.  And this validation set is much larger than in the original developmental data set. Here's the distribution of the severity scores.  And as I mentioned earlier, you can think of this as akin to Apache 5, which doesn't exist. But it's called Care Predictor because it is  for all hospitalized patients, not just those in the ICU. And again, that reflects the fact that many hospitals now, we're delivering ICU level care out on the floor and certainly  in the step down units. The model does well. So we've mentioned AUROC several times here. But a perfect model, and there is not one in existence, would have an AUROC of 1.00.  And flipping a coin gives you an AUROC of 0.5. Generally, it's expected that models around 0.7 are good. Models of 0.8 and above are excellent.  And if you get up to the high reaches of the 0.8 range, those are really good models. And this is very similar to what Apache 4 was able to attain.  Calibration is not quite as good as we would like because if it were perfect, it would sit on that dotted line there. But it was pretty close to the original validation set  and pretty close with the revalidation. And the Breyer score is 18.0, which is a fairly good score. So 39 years after publication of the original Apache score,  there's still a minority of ICUs across the country that are reporting out standardized mortality rate. And that's your observed over expected where 1.0 means  as expected and less than one is desirable. And the barriers to implementation have included personnel costs for data collection, difficulties in assigning diagnoses.  And we have an opportunity now with EMRs and automation to allow all hospitalized patients to be assessed from the ED through discharge.  So this expands the Apache methodology to all inpatients. We're working on identifying risk of ICU transfer, which I spoke about earlier, but also complications and length of stay.  And as Dr. Kim mentioned in his talk, there's a lot of useful outcomes that you can model with this. The mean severity of illness score was 20 for all inpatients.  It was 32 for ICU patients. And I think that represents the fact that there's some pretty sick patients on the floor these days. And the mortality in the lowest decile was 0.002%.  And in the highest decile, 18.8%. And that's a much wider range than we see in most ICUs, which is why it's essential to capture the diagnoses.  I mentioned it's hard to develop high-performing models with low mortality rates. So that's why obstetrical patients were excluded from this study. The contribution of the variables  to the mortality prediction look a little like the slide I showed you earlier, but it's about a third on physiology, 26% on diagnosis. And as other people have mentioned,  that Glasgow Coma Scale is incredibly important for predicting mortality outcome in patients. Age plays a part, as does the support index and comorbidity.  And then other includes things like interaction terms. Now, there are some limitations to this study. The different contributing hospitals means  that the statistical, there are differences between the two datasets. But I think you do have to validate this prospectively on a different dataset to make sure it is transferable.  The diagnosis was determined at discharge, and we used ICD-9 codes because that was current when the database started. We're able to crosswalk those to ICD-10.  A larger, more representative samples needed to ensure that this model is not overfitted. But we think this is a version 1.0 that can be implemented electronically now while data is  collected for an improved version. And perhaps to go from logistic regression to other ways of evaluating the data. We still don't capture treatment limitations or adjust  for differences in discharge practices. So if I'm in a hospital where many of my ventilated patients are going out to an LTAC while still on a ventilator,  I'm going to look a little different than a hospital that waits until the patient is treated and liberated from ventilation. So future directions, additional testing validation,  both in the U.S. and worldwide. We're working on adjustment for treatment limitations including comfort measures only in DNR orders. We do have good data on DNR orders,  and it does affect the outcome. But there are other treatment limitations which frankly are just not well captured in the EMR. And hospital discharge, particularly if you're using  LTACs and SNFs, is a very arbitrary endpoint. And we would really like to have the data to assess 30-day and one-year outcomes. And we're expanding the predictive analytics  to look at the likelihood of ICU transfer and admission, risk for complications, and expected discharge destinations. So I want to acknowledge Maureen Stark, one of my co-authors  who's here in the audience as well. And also people who've contributed to this in a big way over the years, including Corey Nelson  who did the initial statistics, Andrew Kramer, Jack Zimmerman who was one of the original Apache developers for his contribution for this as well.  And other people who've contributed to the statistical methods and the diagnostic groupings. The disclosures are the same as in the previous talk.  And with that, I thank you for your attention.  Thank you.  So we actually are finishing a little bit early. So we have a few minutes. Questions for Dr. Higgins?  So maybe while people are thinking about questions, I might ask, you mentioned, I think, this. But do you feel that implementing other models  apart from logistic regression, specifically machine learning tools, might enhance the performance of your model? So there are pluses and minuses of that.  Certainly, machine learning offers opportunity to look at larger databases and to do things not only automatically, but to continuously update  the models over time, which is difficult with logistic regression because you basically have to go back and do the data again. I would say you have to use some caution, though,  because you start looking at data with machine learning. And unless it's been clinically validated and makes some sense clinically, you'll start to see all sorts of correlations  that may not make any sense. So you have to be very careful with that. Also, another problem that I've personally encountered  is that when we switched from mortality probability model to Apache, our SMR went from 0.83 to 1.01. And there were no changes during that period of time other than that.  And it took us a year to figure this out. But Apache was collecting the data from the Philips monitors. And most of the patients in the ICU then had ART lines.  But the nurses could label the ART line ART or ABP for arterial blood pressure. And only the ABP data was crossing to the system. And we had no idea that this was not happening.  And it actually wasn't me. It wasn't our engineers. It wasn't our IT people that figured it out. It was the data collector who used to do the data collection  manually who finally went and did the data collection. And we found out that this was why. So you have to be very cautious of mistakes like that.  And you could not have predicted that this was why the SMR went up. And frankly, had I just been implementing the system for the first time, I would have been satisfied  with a 1.01 SMR. I would have said, OK, I'm average. But because we had historic data looking at that, we knew that there was a problem. So I have one more question.  I think your principal outcome is mortality, is that correct? Yes. Yes. For this study, the principal outcome is mortality. And you may have touched upon this,  but I know that you express interest in longer term outcomes. Of course, that's important. But what about other outcomes apart from mortality? Do you have this in your database,  like functional status? And could these modeling exercises could also be applied to predicting functional? So I say this because I guess mortality  is, we know, declining in most critical illnesses. And I think maybe a more meaningful outcome would be the functional status of our patients.  So again, it's very difficult to get functional status data because typically that's not collected in the hospital. But as we move into the ACO environment  and this data becomes part of the enterprise data warehouse, that data does begin to get out there and see. And also, as you're partnering with LTACs and SNFs,  you can look at the transfer rates and the retransfer rates. You can look at ICU readmissions. That data we have looked at. And we do have predictive models for that.  You can look at prediction of renal failure as an outcome if it develops in the hospital. But one of the talks today was looking at, not at this session but earlier, was  looking at the outcome at three years in sepsis. And it was surprisingly high, the number of patients. And we don't typically have that kind of data looking out there.  But you would like to have that one year and three year data to be able to look at that. And I think functional status is obviously very important  because there probably are things that we can do, like early liberation from the ventilator or minimizing sedation that are going to make substantial differences in the outcomes.  I mean, the modeling part of this is easy. It's just getting good data that's the difficult part. Any other questions? OK, thank you very much.  So this brings to the close this session on predictive analytics. I want to thank all four speakers and also all the audience participants. Thank you very much.

Video Summary

The session on predictive analytics featured speakers discussing the use of machine learning algorithms to predict various outcomes in the healthcare setting. Dr. Rohit Gupta presented on the use of machine learning models to predict clinical deterioration in admitted patients. He discussed traditional prediction models such as the modified early warning score and highlighted their limitations. Dr. Gupta then introduced the Muse++ model, which uses a random forest classifier to predict clinical deterioration by defining escalation of care or unexpected cardiac arrest. The model incorporates 36 different features and provides a probability score for the likelihood of clinical deterioration within a six-hour time period. Dr. Thomas Higgins presented on inpatient deterioration and the use of artificial intelligence to predict which patients will be transferred to the ICU. Using data from the Philips EICU database, he developed models that could predict neurological outcomes and mortality within 24 hours of ICU admission. Key features in the models included vital signs, laboratory values, and diagnoses. Hanbin Kim discussed the prediction of post-cardiac arrest outcomes using machine learning algorithms. Using data from the EICU database, his team developed models that could predict neurological outcome and mortality. He found that the inclusion of physiologic time series data improved model performance. Dr. Laura Freseman-Freeman presented on the validation of an automated electronic inpatient mortality prediction tool. Using the Cerner Health Facts database, her team developed a model that could predict mortality in all hospitalized patients. The model performed well with an area under the receiver operating curve of 0.82. The session highlighted the potential of machine learning models to improve predictions in healthcare and emphasized the importance of incorporating relevant features into the models.

Asset Subtitle

Quality and Patient Safety, Administration, 2020

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2020 Critical Care Congress held February 16-19, 2020, in Orlando, Florida, USA.

17-MEWS++: Predicting Clinical Deterioration in Admitted Patients Using a Novel Machine Learning Model

Rohit Raghav Gupta

18-Inpatient Deterioration: Can Artificial Intelligence Predict Who Will Be Transferred to the ICU?

Thomas Higgins

19-A Machine Learning-Based Prediction of Cardiac Arrest Outcome Using a Large Multicenter Database

Hanbiehn Kim

20-Prospective Validation of an Automated Electronic Inpatient Mortality Prediction Tool

Thomas Higgins

Meta Tag

Content Type Presentation

Knowledge Area Quality and Patient Safety

Knowledge Area Administration

Knowledge Level Advanced

Membership Level Select

Membership Level Professional

Membership Level Associate

Tag Scoring Systems

Tag Informatics

Year 2020

Keywords

predictive analytics

machine learning algorithms

clinical deterioration

Muse++ model

ICU transfer

neurological outcomes

mortality

vital signs

model performance

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.