Thank you, Ailey. So good afternoon, everybody. My name is Derek Wheeler. So I have no relevant financial relationships to disclose. I will disclose that if you attended the LEAD Conference yesterday, you will see some of these same slides. I apologize for that. I would also like to disclose that I didn't choose to be the con in this debate, so thank you very much whoever selected me for that, but hopefully we'll cover a good portion of that. I think Ailey and I agree on a lot more than we disagree. So the objectives are really to discuss the potential reasons why ICU metrics fail to measure quality of care and provide a list of what not to do when it comes to measuring quality in the ICU. So the first question that you have to ask yourself whenever you use a metric or when you evaluate whether a metric is good or not is what exactly are we measuring? And so as a really good example of that, I think everybody's heard the story that the U.S. pays more for health care than any other country in the world, but yet we do worse on every other possible health outcome compared to every other country in the world. And so this was a publication from just last year that showed that, and I'm going to show you a whole bunch of graphs here in a second that will support these two statements. Not meant to actually really dig into each one of these, but I just want to illustrate kind of the health-related outcomes that we're talking about. So here's the first, is that the U.S. spends more than any other country, and in fact spends three to four times more on health care than South Korea, New Zealand, and Japan, and this has been well known. It's continuing to grow. Health care costs are growing at an astronomical rate of inflation compared to the normal inflation rate, and it's really just not a sustainable picture for us long-term in the United States. And we're really doing worse on all of the so-called health-related outcome measures. So here is U.S. life expectancy at birth. This is avoidable deaths. This is showing that the U.S. has the highest rate of infant and maternal mortality compared to all of our peer countries. Rates of suicide are higher, maybe with the exception of Japan and Korea, but still pretty high. Deaths from assault are really significantly high compared to other countries. The obesity rate is nearly double the OECD average. Adults in the U.S. are more likely to have multiple chronic conditions. And so the question I would ask with all these, are all these health-related outcomes really under the control of our health care delivery system? And I think this was a great point made by Elizabeth Bradley. She published a paper. She's a health policy researcher out of Yale and wrote this book that is, I think it's been out for about five years now. It's an excellent book that I highly recommend. But it's really comparing what we spend in the United States on health care versus social services, and then comparing that with other countries. And so you can see here that countries like Sweden and Finland that really have these legendarily high-quality health care delivery systems, and they're frequently put forth as the examples of what we should all aspire to be in health care delivery systems. They actually don't spend very much on health care, but they spend a lot on health through their social services, and significantly more than even the U.S. when you consider our health care-related expenditures and our social service expenditures. So I think the take-home point of this is that perhaps some of these health outcomes that are frequently put forth are really not under the control of our hospitals and our health care organizations. And so I think people are beginning to appreciate that, and certainly this is a paper and a graph from Dr. Magnin, and really looking at the so-called social influencers of health. And so if you look at health-related outcomes, a lot of the outcomes that I just mentioned, about 80 percent of those outcomes are really not determined by what actually goes on in the clinic or the hospital setting. It's really all these other social influencers of health that may or may not be under the control of hospitals and physicians. So my point, first one, is really most commonly used measures of health care quality actually don't measure the quality of care. And so when you look at ICU-related metrics, you have to be really careful that what you're measuring is actually under the control of the ICU care that's happening in the hospital setting. I think, you know, organizations typically emphasize process measures, so, you know, if you look at process measures, outcome measures, and balancing measures, the best example of that is if you look at the outcome measure of central line infections, certainly Haley mentioned that, your bundle compliance, so your bundle of different initiatives or practices that can reduce central line infections, your compliance is the process measure. And then your balancing measure typically would be something like staff satisfaction or staff workflow. And so unfortunately, a lot of organizations emphasize process measures at the expense of outcome measures, and I would just say we need to be careful about how we measure those process measures. So this is, I used to be at Cincinnati Children's for 20 years before being at Lurie Children's Hospital in Chicago, and so this was the work that we did in ventilator-associated pneumonia. I won't comment on ventilator-associated pneumonia as a quality metric because that's fairly controversial and nobody is really following that as much anymore. But back in the day when we were really focused on reducing ventilator-associated pneumonia in the intensive care unit, we looked at our bundle compliance in the PICU in a stretch where we went 540 days without a ventilator-associated pneumonia. And our compliance with our ventilator-associated pneumonia bundle during that time was only 65%, so not very good. So when we compared that to the NICU and the cardiac ICU, you can see here during that same period of time, the bundle compliance in the NICU was 95%, and you can see on this graph they had significantly more ventilator-associated pneumonias than we had in the PICU. And then comparing to the cardiac intensive care unit, the same thing. The bundle compliance was upwards of 95%, and again, significantly higher rates of ventilator-associated pneumonia. So this would suggest to you that, well, maybe the bundle doesn't work, but what I can tell you is it all really is how we measured compliance in those three intensive care units. So in the cardiac ICU and the NICU, it was all staff self-report. So at the end of the shift, the nurse, the respiratory therapist, and the physician all said, yeah, I did this bundle, or no, I did not. And so they did a pretty good job, right? They didn't rate themselves 100%. So they were 95%, so maybe they were a little bit honest. But in the PICU, it was all direct observation. So a peer would observe the respiratory therapist and the nurse going through the elements of the bundle. When they failed on a particular element, they would actually correct that behavior in real time. Still rate them as being noncompliant, but they would actually correct it in real time. And so that, I think, is why we were able to have a 65% compliance, but yet have a really impressive rate of ventilator-associated pneumonias. So process measures don't always accurately reflect outcomes. And so I think, again, how you measure process really needs to be looked at and considered very carefully. So number three point. So I've been to hospitals where I walk into these large rooms, maybe not quite the size of this room, but they're pretty large, and every inch of the wall is covered with metrics, right? And they're really proud of it. They're saying, look at all of our metrics. And they're all printouts. They're all collected by hand. They're all posted up there with pens on a bulletin board or whatever. And they're really, really proud of that. And I always ask, it's like, OK, well, how much are you paying the person to collect all that, and how long is it taking them, right? And so I think our tendency is to get lost in metrics sometimes. And so this is actually from one of our inpatient units at Lurie Children's Hospital. We do know that being transparent about your data is helpful. Showing your data and displaying it prominently is also very helpful. And so this was an outcomes board that was in the break room, very prominent area. But I'd challenge you to look at this and try to figure out really what's going on. There's a lot there. It's pretty confusing. It's pretty crowded and really hard to discern what's going on there. And so I think our tendency is just to continue to pile on metric after metric after metric. And I think Goldilocks was right, OK? What is the right amount of metrics to have in your ICU? Well, you don't want too few, and you don't want too many. We just want to shoot for just right. And I think that's going to be pretty important. Another point I'd like to make. So looking at ED overcrowding, big problem. The British National Health Service looked at this. They set a goal in 2010 to evaluate, treat, and discharge 95% of all patients presenting to the ED within four hours. And so they incentivized hospitals, they penalized hospitals. And I think we all agree that ED overcrowding, you don't want a patient who's critically ill sitting down in the ED waiting on an ICU bed. And so when they looked at that, I'd call your attention to this graph right here. It's the time that they spent in the emergency department. And you can see here the patients who were admitted, there was a significant increase right up to four hours, right? So what was happening here is rather than kind of sitting on the patient and allowing and making a decision whether they really needed to be admitted to the hospital or not, they were just admitting them because they were being incentivized and penalized for that four hour threshold. So I think you just have to be really careful and be aware of something that's called Good Heart's Law. So raise your hand if you've heard of Good Heart's Law. So a few of you. So it basically states, and it's from an economist from I think the 1950s, that said when a measure becomes a target, it ceases to be a good measure, okay? And so there's some examples up here. I think one of the classic examples, if you look up Good Heart's Law on Wikipedia, it'll talk about this particular story. So way back when, when cobras, very deadly snake, lots of cobra bites, people dying of cobra bites. So the British incentivized the Indians to bring them cobras that were dead, right? And they would give them money for dead cobras, okay? So they did that, they were collecting all kinds of dead cobras, you know, they're paying out money, they're thinking, this is great, we're doing awesome. And the number of deaths from cobra bites didn't actually decrease, it actually increased. So what was going on is that people figured this out. Well, if I bring dead cobras, I get more money, I'm just going to start raising cobras. Because that means, you know, I can raise the cobras and then I kill them, and then I bring them and I get paid for it. So really good example of Good Heart's Law right there. The other example, and we were talking about this at the LEAD Conference yesterday, who here flew to Phoenix for Congress? Okay, did everybody get here on time? Did the plane leave on time? No, there's some shaking heads. So one of the things that we talked about is one of my colleagues actually left the gate and they sat in the plane for an hour and a half on the tarmac. And why do you think that is? So airlines are incentivized for on-time departures. How do they measure on-time departures? As soon as they pull away from the gate, they've departed on time, right? So it doesn't matter whether they sit on the tarmac, but it's really kind of a faulty incentive there. So be aware of Good Heart's Law. And then, have you seen this graph before? It's about the growth in physicians versus the growth of administrators. You can see here over time since 1970, administrators, I mean, I'm an administrator, so I look at this with a little bit of pause. But the number of administrators increased by 2,500% since 1970. And so, you know, there's a real cool thing called Google Ngram, and you can actually input on Google any word in the English dictionary, and it will tell you how many times that word has been used in all of the available published materials, both medical and non-medical. And so looking at the word metrics and benchmarks and performance indicators, I'll draw your attention to metrics. If you look at that graph versus the graph I showed you before, that increase is fairly similar, right? And I think what's happened is that we're throwing so many metrics on hospitals and healthcare organizations that somebody actually has to do that work to measure those and report them. And so that has led to the growth in healthcare administrators today. So point number six, and I think this is my last point, measure what matters. And this is one of the gurus, one of the founding fathers of the quality improvement, Edwards Deming. He basically cautioned us, and he had two points here, about cautioning about management by numbers and using numerical goals. So here's a guy who thoroughly believed in the utility of measurement to drive improvement, but he added the caution that we should be really careful about excessive measurement, and that's something that we should be wary of as well. And then I guess I think this is my last point, apologies, but accountability and authority must align. And this gets to that original point with the quality of the healthcare delivery system. Make sure that what you're measuring, the individuals that you're evaluating in those metrics actually have the authority to act on that data. Because if you're holding them responsible and accountable for data that they have no control over, that really is a wrong thing to do. And lastly, I'll close with a lot of this, and a lot of these stories came from these two books. They're really great. One is very pro-measurement, one is very anti-measurement, as you can see, but both very readable texts and both relatively cheap on Amazon. Thank you very much. Thank you.

ICU metrics

healthcare quality

Goodhart’s Law

social factors

outcome measures