SCCM Resource Library-10-Early Identification of COVID-19 Patients Susceptible to Rapid Deterioration Using the Rothman Index

Video Transcription

Hello, my name is Joseph Beals and I'm the Chief Analytics Officer at ParaHealth. I'll be talking to you today about our work on early identification of COVID-19 patients  susceptible to rapid deterioration using the Rothman Index. This work was very much a team effort, and I want to acknowledge my co-authors from ParaHealth,  Sinai LifeBridge, the Yale New Haven Health System, and UCI Medical Center. In terms of disclosures, both myself and two co-authors are employees of ParaHealth, and  two of us are also shareholders. ParaHealth is the commercial entity behind the Rothman Index Acuity Score, which I'll be discussing throughout this talk.  Since submitting our abstract, we were able to extend this research, and I'll report the latest updated numbers in today's presentation, which is based on the data reflected in the  table here, covering analysis of over 3,000 adult COVID-19 patients and over 14,000 non-COVID adult medical patients.  Our data comes from four quite different hospitals, and as you'll see, the findings are remarkably consistent across each of them.  Our motivation for this work was succinctly captured in an article by Johnson & Gottlieb earlier last spring, who said about COVID patients, quote, the clinical course trends  toward two different stages. For some, the second stage starts between days five and seven, when sudden rapid clinical deterioration may occur.  And they went on to discuss the difficulty in identifying which patients were likely to deteriorate.  From the outset, it was clear that age and comorbidities are major risk factors for COVID-19 patients, but our thesis was that in many cases, risk and comorbidity were essentially  serving as proxies for patient acuity. And that makes sense. The older you are, the more likely you are to have comorbidities, and the more comorbidities  you have, or the greater the impairment from them, the higher your acuity. Now, obviously, not all comorbidities are equal in terms of their implications for COVID,  but broadly speaking, it's clear from the literature that there is a general correlation between age and comorbidity and risk of serious deterioration for COVID patients.  So we wanted to see if we could use the Rothman Index acuity score, which is a direct measure of acuity, to predict risk of deterioration in the COVID population.  For those of you who may be unfamiliar with the Rothman Index, it's a machine learning-based  acuity score that uses vital signs, common lab measures, and body system nursing assessment data.  It's been extensively validated in the peer-reviewed literature, it's FDA cleared, and it's widely deployed in hospitals around the US.  As we started to characterize patients in our data set, we found a number of interesting things that bear in how clinicians think about COVID patient deterioration risk.  For one, we saw that the average Charleston comorbidity index for COVID patients who expired  in the hospital was significantly lower than for non-COVID patients who expired, underscoring  the fact that comorbidity does have an outsized impact on risk for COVID patients, and implying  that normal clinical experience in thinking about the risks associated with a particular comorbidity burden don't translate to understanding the risks implied for COVID.  Even more telling is the difference in risk we found when comparing patients on the basis of physiological acuity as measured by the Rothman Index.  What you can see from this graph is that for any given acuity measured on the horizontal axis, COVID-19 patients have a much higher likelihood of dying.  For example, for patients with a Rothman Index value of 40, which is the level of acuity around which patients are often considered for IC-level care, we see that the mortality  risk for non-COVID patients is a little over 5%. That's the blue line. But for COVID patients, their mortality risk is around 25%, the red line.  And that's really a startling difference. And this has an important implication. It means that a clinician's experience with non-COVID patients and their sense of the  patient's risk based on how the patient's presented admission does not translate to an understanding of risk for COVID-19 patients on admission.  In other words, a clinician's experience and intuition could actually be misleading and needs to be recalibrated.  This has been anecdotally observed by numerous frontline clinicians. And what you see here is a quantitative validation of that idea.  So we then wanted to go from descriptive analysis to predictive analysis. To do that, we built single variable logistic regression models for the first Rothman Index  score at admission, as well as for age and for comorbidity. And we evaluated them as predictors of in-hospital mortality, looking at their performance using  the area under the rock curve. As you can see from the graph here, across all four hospitals in our study, the Rothman Index at admission is a substantially better predictor.  And this shouldn't be a surprise since the Rothman Index incorporates a wide range of clinical data that reflects acuity and by extension, patient risk.  Looking beyond mortality as an outcome, we also wanted to see how we could do using the Rothman Index at admission to predict the downstream risk of the patient ever spending  time in an ICU or being mechanically ventilated, as these are surrogates for severe deterioration.  Now, these are much harder things to predict than simply whether or not the patient will survive. But as you can see, the difference in predictive performance is even more stark.  The logical question is then, how do you use the Rothman Index acuity score at admission to categorize COVID patients as being either high or low risk?  To do that, we evaluated threshold values of the score and calculated the precision and recall for predicting in-hospital mortality.  And in this way, we identified cut points for risk stratification. We settled on two cutoff values for our COVID population.  Patients with a first Rothman Index score below 50 would be categorized as high risk.  This captures about 25 to 30% of COVID patients, and these are the ones who want closer attention or more intensive care.  And on the flip side, patients with a first Rothman Index above 70, we would categorize as low risk. That's about half the COVID population in our data set.  And these might be candidates for lower level of care or less intensive therapy. We can explicitly look at how these cut points perform in stratifying risk.  And so as you can see here, looking at mortality, our low risk population, the blue bars, have a mortality rate that is between 2% and 5%.  At most sites, this is not far off the typical overall mortality rate. But for our high risk COVID patients, the red bars, we have a mortality rate close to  10 times higher, well above 30% in all cases. So these patients really are a much, much higher risk.  We can also look at the level of ICU utilization or mechanical ventilation in our high and low risk groups.  And while the difference is not quite as big, we still see between a two and fivefold difference in risk between our high risk group and our low risk group.  So having identified high and low risk groups, we also wanted to determine the opportunity for using such risk stratification in future to potentially improve care.  We compared the high risk COVID patients with the high risk non-COVID patients, and we saw some interesting differences.  Among them was that the median time from admission to ICU transfer for those patients who are not directly admitted to the ICU was two to three times longer for COVID patients.  And the mortality rate for the transferred COVID patients was two to three times higher than for the transferred non-COVID patients.  This suggests that perhaps if risk associated with the COVID patients had been better appreciated  earlier on, they might've been transferred to the ICU sooner and possibly had better outcomes.  In wrapping up, I do want to note that among the large universe of predictive models that have been developed for COVID diagnosis and prognosis, we think that our approach has  certain benefits, namely that we're applying a complex but well-validated general model of patient acuity in a very simple way by using cut points that can be easily evaluated  and easily adjusted. Because we haven't created a model that is trained on a static dataset, we've avoided potential issues of overfitting or training to artifacts.  And this approach is robust to the changing dynamics of both care environments, as well as the changing characteristics of the COVID patient population itself.  And the generalizability of our approach is seen by the remarkably consistent results we get across different hospitals.  In conclusion, we've shown that for a given acuity, COVID patients have a higher non-obvious risk of subsequent deterioration, and that the Rothman Index acuity score at admission  can identify patients who may need closer monitoring or more intensive treatment, as well as those patients who may be appropriate for treatment in a lower level of care setting.  And finally, I would emphasize that the early recognition of patient risk is crucial to the effective delivery of care.  The need to determine appropriate level of care for patients holds true even as therapies change and outcomes improve. So thank you all very much indeed for your time and attention.  And I would invite anyone with questions to contact me by email. And I also look forward to speaking with you at the upcoming conference.

Video Summary

Joseph Beals, the Chief Analytics Officer at ParaHealth, discusses their work on early identification of COVID-19 patients susceptible to rapid deterioration using the Rothman Index acuity score. The study involved analyzing data from over 3,000 adult COVID-19 patients and over 14,000 non-COVID adult patients from four different hospitals. The findings indicate that age and comorbidities are major risk factors for COVID-19 patients, but the Rothman Index acuity score is a better predictor of risk and patient outcomes. The study also identifies threshold values of the score to categorize COVID-19 patients as high or low risk, providing an opportunity to improve care and treatment decisions.

Asset Subtitle

Quality and Patient Safety, Crisis Management, 2021

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2021 Critical Care Congress held virtually from January 31-February 12, 2021.

Joseph Beals, PhD

Introduction/Hypothesis: COVID-19 patients have a recognized susceptibility to rapid and severe deterioration. Early identification of patients at risk of deterioration is a clinical challenge. Characteristics associated with COVID-19 risk, including age and comorbidity, are crude predictors of ICU utilization and mortality risk. Real-time predictive models may aid in determining appropriate patient placement and care plans. We assess whether the Rothman Index (RI) acuity model can predict mortality risk and distinguish between high and low-risk COVID-19 patients at the time of admission.

Methods: We conducted an IRB approved study of 1,453 COVID-19 patients discharged from Yale New Haven Health System's New Haven, Bridgeport, and Greenwich hospitals from 4/1/2020 â€“ 4/28/2020. The initial RI score on admission was used to predict risk of ICU utilization, mechanical ventilation, and in-hospital mortality, and corresponding AUC values were computed. We also compared patient age, Charlson Comorbidity Index, and RI on admission as indicators of in-hospital mortality risk and calculated predictive performance for each. Precision and recall curves for mortality prediction using initial RI thresholds were determined and performance at selected operating points evaluated.

Results: As a direct measure of acuity the RI was a substantially better predictor of COVID-19 in-hospital mortality than age or comorbidity (AUC values of 0.87, 0.78, 0.70, respectively). The RI computed at the time of admission provided a high degree of discrimination to differentiate COVID-19 populations into high and low-risk groups. At selected RI cut-points, the high-risk segment constituted 18-27% of COVID-19 patients with a mortality rate of 39-46%. The low-risk segment constituted 48%-58% of patients with a mortality rate of 1%-3%.

Conclusions: COVID-19 patients exhibit elevated mortality compared to non-COVID-19 patients and may be subject to rapid deterioration following hospital admission. The RI is an excellent predictor to stratify COVID-19 patient risk at the time of admission. In particular we find that COVID-19 patients who present with raised but unremarkable acuity may be flagged as high risk for physiological deterioration. This can aid providers in aligning level of care decisions at admission with hospital and ICU capacity constraints.

Meta Tag

Content Type Presentation

Knowledge Area Quality and Patient Safety

Knowledge Area Crisis Management

Knowledge Level Advanced

Membership Level Select

Tag COVID-19

Tag Scoring Systems

Year 2021

Keywords

Rothman Index acuity score

COVID-19 patients

Risk factors

Patient outcomes

Threshold values

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.