SCCM Resource Library-Critical Care Research: Old Trials Through New Lenses

Video Transcription

My name is Alex Flannery from the University of Kentucky. Welcome to Critical Care Research, Old Trials Through New Lenses.  Today we'll talk about critical care trials through the framework of the traditional frequentist  approach and then our second speaker, Fernanda Zimperi, will talk about critical care trials through the Bayesian approaches.  And I'll kick us off here with common challenges of interpretation of critical care literature  as we take a dive into different ways to look at the literature that's been published within critical care here, especially in recent years.  These are my disclosures that we won't be discussing anything that would serve as a conflict of interest per se. These are the learning objectives for the session.  So I'll be taking the first one here, looking at really how the nuances in the way that  we analyze and present critical care literature can influence our interpretation as clinicians  of how to put this into practice, and then really introduce some examples of trials from the frequentist standpoint, at least, and then the second part of the talk today will  really get into how those interpretations may differ from a Bayesian framework. So just a roadmap, at least, for the session that I'll talk about.  I'll stick to interpretation of these trials in the traditional frequentist framework, which is probably what the vast majority of us are brought up on in academic medicine,  where if you run a statistical test and a p-value is less than 0.05, you have a statistically significant result.  But we're really going to be looking at this from the eyes of the clinician. No trial is perfect, but it's our job to basically take the data that we're given and be able  to apply it to our patients to make the most informed decisions possible, and in doing so, really try and be good stewards of evidence-based medicine.  We'll stick with the discussion of clinical trials just because clinical trials are the building blocks of the decisions that we make in clinical practice, but at times, it feels  like attempting to apply evidence-based medicine into practice can feel, at least, like putting a square peg in a round hole. No trial is perfect, obviously.  It might not be a perfect match between the patient population, but it seems like in recent  years, even at times, it's increasingly difficult to even make sense of some of the trial results based on some of the nuances that we'll talk about here in this presentation.  So to start, why is critical care unique? Why can't we be like cardiology? And I think one of the first things that comes to mind, obviously, is sample sizes.  We can have cool trial names just the same as cardiology groups can, but I think one of the other challenges, though, is that really in critical care, a lot of the times, we're  treating syndromes. We're not necessarily treating well-described pathophysiological processes at the macro level.  So if you think about syndromes like sepsis, acute respiratory distress syndrome, even acute kidney injury, and compare those to something that at the macro level is as definable  as a myocardial infarction or a stroke, they're just very different. So we're operating really in a syndromic world as opposed to something that is definite,  again, like with something like a myocardial infarction. Critical care trials are challenging as well because we know increasingly that a patient's  preexisting frailty may influence the outcome. We'll talk about mortality, but we know that there are other things important to patients  and families aside from mortality, like how well they do after the ICU stay. But critical care is also challenging, though, because especially compared to other specialty  groups and other specialty areas of medicine, critical care has higher mortality rates, and mortality can get challenging in several different ways, which we'll go into as we  go on later in the presentation. Also just to say, we'll talk about very specific examples of other trials to highlight this,  but we want to do so in a transparent and respectful way. It seems like discussing clinical trials and what to do in practice sometimes becomes just  as contentious as politics these days, but my goal today is really to show you how, given  the same trial, two people can make very reasonable, different conclusions depending on how you looked at it.  So just as the picture on the left side, half of you may see a vase, while half of you may see two faces looking at each other, and the picture on the left, half of you may see an  elderly woman facing leftward on the screen, while half of you may see a younger woman facing away.  Depending on how data are presented, you may come away with two different conclusions about  what you should do, but all in all, I think we all respect that clinical trials are extremely  difficult to do, and that patients and families volunteered at their sickest time to contribute this information to society.  We should also recognize that the ice we skate is really very thin. The fragility index is a concept that's been around for several years now, but newer in  critical care, and it's basically for the traditional trial that has a p-value less than 0.05, how many non-events in a trial treatment group with the lowest event rate  have to become events for the p-value to become above 0.05? And so, an easy way to think about it is how many non-events have to become events?  How many patient outcomes have to be different for a trial to become no longer significant when it was statistically significant?  Now obviously critics will say this approach overly relies on the p-value and so forth, but it is interesting to look at, and this was recently done a few years ago, looking  at 56 multi-center trials in critical care with a significant effect on mortality, you can see the median fragility index is 2, 20% of trials had a fragility index of 0, 40%  of trials had a fragility index of less than 1. If you think about all the things that could influence a patient in these trials, we talked  previously about the syndromic conditions, you know for a sepsis trial, sepsis may be  on the differential, but there could be other things as well, even how an outcome is adjudicated,  the median loss to follow-up, so in one of eight of these trials actually, the loss to follow-up in number of patients was actually greater than the fragility index.  And so, I think it's important to recognize that, especially for mortality, our results in critical care are alarmingly brittle, and so it's a good exercise, there's fragility  index calculators available, and if you, the next time you're reading your favorite landmark  or the next time you're reading a trial hot off the press, to really calculate a fragility  index for that, just to see how fragile those results are, at least again from this frequentist P equals, you know, less than 0.05 kind of approach.  But I think critical care also struggles with the question, is mortality still the best endpoint?  This is just an example of sepsis data, but you can see mortality rates are dropping over the years, to the point that, you know, is there a point where it becomes too difficult  for us to prove a mortality benefit in critical care research? Again, it's good in the way that it's dichotomous, it's easy to adjudicate, again, it amends  all those syndromes, sepsis, ARDS, it is something that's sort of constant, and it means the same thing across all syndromes, which is still the argument for using mortality, but  particularly as advances in palliative care, again, increasing recognition of post-intensive  care syndrome, and that what happens to patients after the ICU, mortality is being questioned as to whether or not it is the best outcome anymore for critical care research.  It's also difficult in many ways, which we're going to talk about now, because it's low  enough that it may be difficult to adequately power trials for mortality to detect a difference,  but mortality is also high enough or frequent enough that it really influences the outcomes  in significant ways, would really influence your interpretation, even if you're not looking  at mortality, and this is really the concept of death being a competing event to mortality. So one of the simplest ways that we've gotten around this has really been using the free  days concept. So there's several iterations of this, whether it's renal replacement therapy-free, vasopressor-free, probably the most common is ventilator-free days, though.  That started in the early 90s in ARDS research, and I think one of the challenges of using a ventilator-free day, which is really from the time you basically randomize a patient  all the way to 28 days, how many days free of mechanical ventilation do they have, is that it is a composite outcome, and one unit of a ventilator-free day is very difficult  to really determine what it means. So if you look at the top graph here, for example, all of these patients have zero ventilator-free days.  A patient even that was extubated on day 10 but passed away on day 20, those are often adjudicated as having zero ventilator-free days, which is obviously a very different  outcome than a patient, for example, that was intubated and had received mechanical ventilation for 28 days, was extubated and survived on day 30. Those are very different outcomes.  They'd obviously be very different for patients.  The same thing on the bottom. These patients all have 14 ventilator-free days. Whether or not a patient was extubated on day 14 and survived, or even whether a patient  was extubated on day 14 but then went on to experience death at day 30, again, they're categorized as the same, but they mean very, very different things depending on how you  look at what it means in the overall picture, and one of the nuances to using the ventilator-free days as an example is that mortality and the ventilator durations should really be  in the same direction. So if you have an intervention, for example, that you think might reduce mortality but  that as a result of that intervention, a patient may actually spend longer on the ventilator, it really would not be a good idea to use the ventilator-free days.  It would make it very difficult to interpret that, just kind of the same way that using a composite outcome of good and bad things mixed together would be very difficult to  interpret as well. There's other nuances, though, of using the free days. Again, the graph on the right shows kind of what we just talked about, that you can really  mix and match a lot of different mortality rates and a lot of different durations of mechanical ventilation to really come up with the same value of a ventilator-free day.  So it's very difficult to separate when you're talking about a ventilator-free day, when you're talking about mortality, or when you're talking about a reduction in actual duration  of mechanical ventilation. But the other thing that makes it challenging, though, is that their distributions are often highly skewed.  So on the left, this is just some example data from a cohort that had a very high mortality rate, so the data is obviously very skewed at zero, but you can also have less acutely  ill populations, as an example, where you may see the data sort of highly skewed on the other end, so the curves almost end up looking U-shaped.  And there are ways to do regression with these that are able to sort of account for this type of distribution, but even when you're talking about medians, it can be difficult  to really talk about what a difference between one ventilator-free day is and how this is interpreted, because there may be only, because you're talking about medians, there could  be, you took out one single patient in the middle part of the distribution, for example, and that could substantially really change your median value.  So interpreting all of these becomes very difficult for that reason. So one of the alternative ways to actually deal with death as a competing event is to  do competing risk regression, which basically, when you use this approach and you treat death  as a competing event, basically, if a patient dies before they experience the event, they can never really experience the good event in terms of how you process this.  So this is an example of the graphical approach that comes with it using a cumulative incidence  function, where you'd have the probability of a good event, like probability of extubation, probability of coming off of vasopressors, and you can present these sort of in the same  way, very analogous way that you would a Cox proportional hazards model, but this time you're showing sub-distribution hazard ratios with confidence intervals, you can adjust  for relevant covariance, and so forth. So when you have to deal with death as a competing event this way, you can use this approach.  You can continue, certainly, to use the free outcomes, which I think we will see people continue to do, but just how they define them should be very explicitly stated.  So even when people have gone and looked at how ventilator-free or vasopressor-free is  defined in the literature, there can be substantial differences in definitions, which obviously makes this much more difficult to interpret as well.  Obviously reporting them out separately helps to a great extent as well. So mortality should be reported out separately, as well as ventilator duration in survivors,  just to see what is really driving the outcome of this composite outcome. So death as a competing event is challenging, but how you handle mortality can also come up in other ways.  And to do this, we'll illustrate with the citrus ALI example. So I don't really necessarily want to start a vitamin C debate, but I think this is an  interesting trial of how you deal with death and what that does to interpretation of your results.  So as a brief recap, citrus ALI was a multicenter RCT with patients with sepsis that developed ARDS within 24 hours and were randomized to ascorbic acid versus placebo.  And their co-primary outcomes were a modified SOPA score at 96 hours, so this was a SOPA score without a bilirubin, basically scored 0 to 20, as well as plasma biomarkers of CRP  from a modulant. And they were powered for 170 patients, and over the course of three years, enrolled 167 patients. And these are really the top line results for the citrus ALI.  You can see this modified SOPA went down in both groups. The biomarker at one week were no different between groups, and so the top line is that  citrus ALI was non-statistically significantly different in its primary outcome of the modified SOPA and of these biomarkers.  We should also say that 43 of 46 other pre-specified secondary outcomes were also not different. Clinical and biomarker values at multiple time points.  But one of the ones that was statistically significant, and I think made this trial challenging to interpret, was that 28-day mortality was reduced in the ascorbic acid arm compared  to placebo. You can see 29.8% versus 46.3%, which in turn really drove differences in ICU and hospital-free measurements as well.  You can see the Kaplan-Meier curves there, they separated early, almost questionably too early maybe, but they were statistically significantly different over time.  And so when I think this first came out and people were talking about how to handle this,  I think traditional hesitations are it's underpowered for mortality, this has to be changed, it's  a secondary outcome, so it's hypothesis-generating, but I think there's still that sort of concern  that hey, this is mortality and it's statistically significant, why are we not treating it with sort of more weight basically because it's such an important outcome?  But one of the important things to notice though is that complete data for this modified SOFA component was only available for 62% of patients.  So if patients were discharged from the unit or patients died, they weren't included in the way that this was analyzed.  And so concerns existed that there was a survivorship bias, meaning that for two given patients  for example, one receives vitamin C and one receives placebo, let's say they're very severely ill, the placebo patient may die because they didn't get vitamin C, the patient that  survived because they got vitamin C may survive the 96 hours, but because they were so sick they may still have significant organ dysfunction.  And so I think the concern with excluding patients that died is that you're still sort of not using information necessarily that may influence the way that you think about this.  One approach to do is you can do a rank-based approach, which they did based upon a letter to the editor, where they assigned survivors a SOFA score of zero and where they assigned  patients that died a SOFA score of 20. And you can see there's no difference in SOFA scores there at 0 and 48 hours, but there  was at 96 hours, again a p-value of 0.03, it's a secondary exploratory post-hoc outcome so take it for what it's worth, but it is interesting to think about how the interpretation  for this trial might have been different if this was sort of the top-line analysis, if you will, as compared to the other way that it had been presented.  So this is one of the ways in which, again, death can really be an issue in some of these critical care trials.  Probably one of the ways, too, that trials are really frustrating to clinicians attempting to use information in practice is powering bias, and so we know there's really two things  that drive the sample size down for critical care, for any trials, really, and that's a higher baseline control rate and also a larger effect size.  So this was a recent study from December of 2020 in critical care medicine that was a secondary analysis of published multi-center studies that really looked at mortality as  a primary outcome, and you can see that the observed control mortality was much lower than was predicted based on sample size calculations.  Again, you can see the attributable or the absolute risk difference that most trials were powered for was 9%, which really brings to discussion what is the minimal clinically  important difference for mortality in critical care trials, and this is really difficult to tease out.  You know, patients would maybe tell us, well, even 1% difference in mortality is important. Only 19% of trials were powered to detect less than 5% difference, and two-thirds of  trials actually really included this 5% difference in there, so this is really that classic absence  of evidence is not necessarily evidence of absence conundrum that we sometimes get into with the frequentist approach, at least, to clinical trials, and so to set the stage for  the next part of the talk, we'll go through a couple examples that I think really highlight this issue quite well. The first example is the Andromeda SHOP trial.  This is a multi-center, randomized clinical trial looking at patients with septic shock  receiving vasopressors, and they were basically comparing randomization to peripheral perfusion pressure versus lactate.  You can see here, these are the details of their sample size calculation. The primary outcome was all-cause mortality at 28 days, and anticipating a reduction in  mortality from 45% to 30%, they had 90% power with 420 patients, and what's remarkable about  this trial was that with no funding that they did the trial this fast, which is really commendable, so 424 patients enrolled, and these are really kind of the top-line results.  28-day mortality was almost 10% lower in the peripheral perfusion group, but when you looked  at the hazard ratio, the hazard ratio barely crossed one, the risk difference barely crossed  zero, and so I imagine these are kind of the faces of the investigators when the statistician comes back to them and say, hey, here's your results.  Even more, what's interesting about these is even though the secondary outcomes of things  like death within 90 days or mechanical ventilation or vasopressor-free days, there was no difference,  there was a difference in SOFA score at 72 hours, there was fewer resuscitation fluids received in the peripheral perfusion group, and so the way that the frequentist approach  for this trial really reads is that among patient-specific shock, the normalization of capillary refill time compared to lactate did not reduce all-cause 28-day mortality,  so these findings do not support peripheral perfusion, et cetera, but I think when you see the totality of these results, even though we may have all been trained to read a trial  this way, I think there's still something that just makes us say, but it just seems like there's something here, right?  We're not supposed to use the word signal of benefit, we're not supposed to use trending toward significance, we're not supposed to use those terms, but as a clinician, it just  seems like there's something there. Very similar scenario to EOLIA trial, which was International Randomized Clinical Trial  of ECMO, basically, in severe ERDS patients that required mechanical ventilation for less  than seven days, again, early BB ECMO versus conventional treatment, details of their primary outcome, the primary outcome was mortality at 60 days, and anticipating their reduction  from 60% in the conventional arm to 40% with the ECMO arm, they had 80% power, planned with 331 patients, obviously, a trial like this takes several years to enroll, this is  what had sort of planned to happen, EOLIA was actually stopped early for futility based on interim analysis rules, and again, I imagine sort of the same faces from folks when they  got these results back, at least, you look at 60 day mortality, 35% in the ECMO group versus 46 in conventional, the relative risk, and in hazard ratio actually both crossed  one with P values that are just slightly above, again, our threshold of 0.05, and again, very  kind of similar results that there does seem like there's a definite signal here, at least,  your treatment failure, there was less treatment failure with ECMO, mortality at 90 days was  numerically lower, but obviously, the hospital length of stay was longer, so when we make conclusions in this framework, again, we say, among patients with very severe ARDS, that  60 day mortality was not significantly lower with ECMO than with the strategy of conventional ventilation that included ECMO as a rescue therapy, and there's certainly other issues  with this trial, too, with crossover and things that made interpretation a little bit difficult  as well, but I just wanted to set the stage, at least, of how these trials are interpreted  in the frequentist approach so that in the next talk, we can look at how they're interpreted  initially from a Bayesian approach and how that may change the way that you look at them. So I think, as clinicians, a lot of the times, we're really left with, the decision should  we change practice based on negative RCTs? I don't really like the word negative, we should maybe say neutral or non-statistically  significant randomized clinical trials, and there's a great editorial in JAMA this past summer, there was a great line that really talked about it, that the burden of proof  to do more should probably be more than the burden of proof to do less, particularly if doing more costs more, and even for newer interventions, too, the authors argue that  the burden of proof should be higher because things that are newer are often more expensive, whether it's a drug price, equipment cost, whether something is more invasive and costs  more staff, but that those should maybe be considerations to think about based on these negative randomized controlled trials, and there's certainly different ways to conduct  clinical trials that, in the critical care research community, are evolving, things like adaptive trial design, platform clinical trials, and those things, but at the same  time, again, as clinicians, at the end of the day, it really comes back to how we view these trials and how we interpret this data to really make decisions clinically, and so,  again, we talked about primarily randomized clinical trials in this particular session, but there are other nuances very similar to multiple different types of critical care  research, whether it's observational, whether it's meta-analysis, big data, translational science, these types of things. I'll just put a plug in here for the SSCM Discovery  and the Pharmacy Sections Research Webinar series that have been going on for the last few years that have experts really come in  and talk about the different nuances of conducting but also interpreting these various types of  designs in critical care research. And so I'll close there with going through some of these  common challenges with interpretation and then we'll sort of switch over here into a different  framework for looking at some of these clinical trials. Hello, my name is Fernando Zampieri. I'm  an intensivist from Sao Paulo, Brazil and it's a pleasure to present here at the Critical Care  Congress. This session is entitled Viewing Through a Different Lens, a Bayesian Approach  for Clinical Trials. These are my contacts. I have the following conflicts of interest to declare.  So I have received investigator-initiated grants from IONIS, from Research Council in the United  States, and Bec-de-Garde Switzerland and the Brazilian Ministry of Health but I have not received direct funding or compensations of any kind in the past. So we have some initial  considerations before we go. So it's clear now and COVID made it even more clear that clinical  trials are the backbone of modern medical practice and it's unconceivable that we have an answer to  a clinical question without a clinical trial. So clinical trials are absolutely necessary.  However, in the past 15 years and more frequently in the past five years, we have seen some issues  with the design and interpretation of clinical trials which are related mostly to the bad  comprehension of the methods involved, sometimes poor reporting, and sometimes favoring simplicity  over complexity and uncertainty. And in this session, my main point here will be, which will  be to present how a different approach, a Bayesian approach, may aid interpretation of clinical  trials and how to make that easier for clinicians to directly translate the information the trial  gives to the bad side. So first of all, we have to start by comparing both approaches. So from  a frequentist endpoint, what we have is that the data is random and the effect is fixed. So  when you do a frequentist trial, what we're trying to do is to estimate a single true effect side  that can only be estimated through sampling. So you're going to sample a population of patients  and you're going to estimate a true unique effect side through sampling. And your conclusions are  going to be mostly based on null hypothesis testing. So null hypothesis testing provides you  the compatibility of the data you observed with the verity of the null hypothesis. So the p-values, which are, let's say, the more important information on null hypothesis testing,  they just provide a measurement of how compatible is your data with the null hypothesis that's  usually set at absence of effect. So the p-values are not the probability that your findings are  true, but the p-values are the probability that given that the null hypothesis of absence of  effect is true, that you find data as extreme as those you found. So the interpretation is a little  bit backwards here and that gives you sometimes that can give some insecurity to clinicians and  that can lead to troubles interpreting clinical trial data. Of course, as I said, the p-values are  tightly defined in your null hypothesis. So sometimes we are really bad at defining null  hypothesis. So we usually do that for commodity. We just say that, well, the null hypothesis is that  there is no effect for a given intervention, for example. But you can set null hypothesis for other,  in other scenarios, for example, in difference approaches such as non-inferiority, etc. And we are  actually pretty bad at defining null hypothesis. The p-values also have several caveats that are  mostly related to their bad interpretation. For example, they are erratic, they are sensitive to  sample size, etc. In that scenario, sometimes if you just make a quick transformation on the p-values  by just taking the negative base 2 log of the p-values, you have what is called a Shannon's  reprise values, which is the number of bits of information against the null hypothesis. So  simple tweaks, and we are not going to go through this, but this is just to say that some simple  tweaks can help in an aid interpretation of clinical trial data. As I said, clinical trials  and p-values are erratic. So in this plot, I have simulated 1,000 trials, including 2,500 patients,  and this trial was designed to check whether there was a reduction in an endpoint, let's say mortality,  from 30 to 25 percent. And all these simulations are based on a scenario where that difference is  true. And as you can see for all the trials, you have a wide range of observed values for the odds range and the confidence interval. Sometimes results will cross the magic boundary  of p05, and sometimes not. But you see there's a very big fluctuation here, and sometimes you  are going to refute the new hypothesis, sometimes you are not going to refute the new hypothesis  based on dichotomical p-values. But as I said, there's a lot of noise. And this is part of the problem, and when we say that there is a reproducibility crisis in clinical trials,  one of the reasons is that we expect some noise, and we are really bad at detecting and discussing  those noises. So two investigators may lead, may run the same trial, and the same trial may result, two trials may result in very different results. For example, one trial may say,  well, the intervention looks like harmful. The other may say, well, it's very beneficial.  And both results are right. So both results are compatible with sampling error that comes from  this type of reasoning. Of course, the p-values will go, will mirror the facts you have on this.  Well, I'm sorry. So this goes for, this type of interpretation has become even more, let's say, it came to our attention after several trials. And as discussed before,  two trials are the YOGA trial, which assessed the use of estrocorporal oxygenation in ARDS, and the other trial was the DROMDA trial, which checked for a different strategy,  aimed at clearing lactate versus peripheral perfusion in septic patients. So you can see in the right, in the probability of survival curves and the cumulative incidence curves,  that that seems to be a difference, right? So the ECMO group seems to go better, and the peripheral perfusion group seems to go better. But when we look at the conclusions,  state that this is a direct copy and paste from the trial, from the paper, you see that the conclusion here was that ECMO was not significantly lower associated with mortality,  or among patients with septic shock, resuscitation targeting capillary refill time did not reduce. And the p-value here is 0.06, and here is something like p07. So this results in,  when clinicians look at that and say, well, it looks different, right? So what's actually the chance that this is due to chance, and what is the probability that intervention may be  beneficial for the patient? So this indirect inference, for example, how likely the intervention is to be of benefit for the patients, is something that's not easily acquired based  on frequentist methods. And perhaps Bayesian methods here can be helpful.  So Ewan Gonger did perhaps the most remarkable Bayesian reanalysis of these trials. And what  he did was consider several priors, and we're going to go through what a prior is in the next  minutes. He considered several priors, which are the prior belief of effect size compatible with  the intervention, and he provided some probabilistic inference on the EOLGA data based on  prior information. So this is a new piece of information you have in the trial. So differently  from frequentist trials, you are not considering that all effect size are possible, but you are  just telling the model, well, I suspect that the effect size may be something that favors benefit,  or it's probably new, or it's probably harmful. And you can compute the posterior probability  of benefit. So for example here, they have shown that using several priors, you have a high  probability that the relative risk of mortality is below one for ECMO use. And this is easier  for the clinicians, right? So you can say, well, there is a 99, a 96% chance that ECMO is associated  with improved mortality in an RDS patient. So this is easier to understand than something like,  given that the null hypothesis is true, the chance that you observe data as extreme as that is 0.07, or something like that. So it's easier for the clinician.  We have also did something similar in the Blue Journal using the Andromeda trial. We also use  the family of priors. We're going to go through this in a couple minutes. And using these priors,  we concluded that there was a very high probability that peripherally perfusion target resuscitation would be associated with improved outcomes. So even when I told the model to be  pessimistic, which is here, so even when I said to the model, well, my prior belief is that  intervention is harmful, the data is so overwhelming that the posterior probability of benefit is still  above 90%, or around 93%. So this may not be conclusive, but this may, the clinician may look forward for confirming this in further trials. And this allows you to not make very  harsh conclusions and say that there is no benefit at all. So in Bayesian reasoning, we have a different approach. So here, data is fixed. So the data you collected is fixed.  But you assume that the effect size for an intervention is random. So there is no unique true effect size, but a family of possible effect size associated with the intervention.  And your model is going to provide all effect size that are compatible with your prior beliefs  and the data you collected from the trial. So for a binary outcome model, Y, for example, you may have an intercept and you may have an effect size beta 1 for an intervention here.  And you can set any priors you want for anything in the model. Usually for a binary outcome, such as mortality or something like that, you set the priors for the expected odds ratio  in the log scale. So this is a logistic regression. So most things work in the log scale. So you can set a prior for that, for the odds ratio. And then the output of that model will  be a distribution of all possible effect size in odds ratio scale compatible with your data and your prior. So mostly you are incorporating prior information on your results. So the  posterior, so the results of the study will no longer be only a result of the data you collected,  but they also incorporate some prior information. This prior information, you don't have to know  exactly what the prior is. So of course, this prior should be developed and considered in the  design phase of a Bayesian trial. But we are talking here about reinterpreting a trial. So you don't actually have to know the precise effect size. You may only have a glimpse,  or you can only consider the most possible range of effect sizes. For example, odds ratios below  0.5 or higher than two are very uncommon in critical care medicine. So you may set your prior  as shown here, so that the biggest probability mass will be around one, and you'll have a very  low probability that intervention is associated with an odds ratio below 0.5 or above two. So  you're just telling the model, well, I don't know what effect size is, but when you are going to run  your computation, please be skeptical of very large effect size for good and bad. And of course,  this prior is centered on absence of effect. So it's centered at one here, which in log scale is  zero. So this is an example of what we could call a neutral or a skeptical prior. How can this help  define the posterior probability? So I'm just going to pick up the same simulations I did before,  and I'm just going to tell the model, well, I believe the intervention is neutral. So my prior  was intervention was neutral, and there was a very low chance, so let's say a 5% probability  only that the odds ratio was associated with a very high or a very low effect size. And then I  can rerun all the iterations here, and you can say that you no longer observe those very high  swings in the output. This is because the priors normalized or regularized your results, so that your effect size are more stable over time and are more stable over iterations.  This is something that's very interesting, because since you're not going to estimate one point,  you're going to estimate a distribution of possible effect size. You can query that as you  want. So for example, after we run a trial, we may end up with this distribution here for the odds  ratio. So in the left here is the log odds, here is the odds ratio. And then the clinician may  wonder, well, what's the probability that the odds ratio is between 0.8 and 0.9? And I can calculate  that, so it's 48%. What's the probability that the odds ratio is above 1, for example? And I can say,  well, it's a 1.2%. So this type of reasoning, once you have the posterior, you can summarize and you  can query the posterior any way you want. And you can provide any measurement of effect size.  And you can even define, for example, regions of practical equivalence. For example, you can say,  well, I believe that odds ratios close to, let's say, 0.95 to 1.05 are pretty similar in terms of  effect size. So you can say, well, what's the probability that the odds ratio is within that  range? And you can get a result for that. You can also define a minimum clinical informative  difference, for example, or anything you want. And you can query that posterior by doing so. So you're  just free to do that. When we talk about trials and we want to reanalyze a trial, so the priors  were not defined a priori, so we have to run the analysis post hoc. So we believe that the most  correct way is to select a family of priors. So you can use a prior that is skeptical, as I  mentioned before, one prior that is pessimistic for intervention, one prior that is optimistic  for intervention. And you can vary the degree of belief of the prior according to what is known.  This degree of belief is just how wide the distribution of this prior is for the intervention. So sometimes you may be very certain that intervention effect is probably around this.  Sometimes you may say, well, I really don't know. And you may choose to use a very wide distribution  for effect size. And then you can perform the analysis and report the results for each prior  you have used before. So for example, these are neutral priors. So they are all centered at  absence of effect, so for the odds ratio. So the log is zero here for an odds ratio of one.  So it's centered. So half the probability mass is for benefit, half is for harm. And you can  see that you can vary the width or the standard deviation for these priors that are normally  distributed so that you can say, well, I believe that effect size distribution is wider or no,  I'm more skeptical. And the same, you can use optimistic priors that are centered on the  benefits. Here, for example, if the outcome is mortality, an optimistic prior will be centered in the odds ratio below one. So there's an optimistic prior and a pessimistic prior.  And actually, you can rerun the analysis of any priors you want. But these priors should be  based on something, right? They should be based on what you believe. But when you reanalyze a  trial, we believe that it's important that you provide a family of possible priors because the  trial was not Bayesian from the start. So these priors, they can be used to check how much your  data changes when you change your prior belief of the effect size of the intervention. For example,  for the overall recruitment trial, which was published in JAMA in 2017, for example, this trial was on the contrary of all the previous literature. So it suggested harm from overall  recruitment maneuvers. We can reinterpret the trial under a Bayesian framework. In this scenario,  I used an absolutely flat prior. So this flat prior is equivalent to a frequentist reasoning.  But either way, what I can provide is what's the probability of severe harm, the probability of  harm, the probability of benefit, and eventually what's a region of the area of practical equivalence  for the result. So it's easier to translate that for the condition. For example, it's easier to  report that, well, we still have, let's say, 54% probability that overall recruitment is associated  with a harm as severe as an odds ratio above 1.25, for example. This is just an example.  Of course, when we can change the priors, we are going to obtain different results. So what I did  here was in this first scenario, we use a neutral prior, here a pessimistic prior, and here a  pessimistic prior. You may know that this optimistic prior is a little bit more tightened than this  pessimistic prior. So this optimistic prior was designed so that there was still a 15% probability of harm, including the prior. The pessimistic prior here was designed so we had  still a 30% probability of benefit in here. And you can see that you have the resulting odds ratios  and the credible intervals for each prior. So for a neutral prior, the odds will be 1.25, and this is the credible interval. For an optimistic prior, we are going to have this  odds ratio, 1.19. And for the pessimistic prior, we're going to have 1.28. So you see that the  numbers are different, but overall, the results are very similar. So this suggests that the data  was so strong here, we had so much information here, that regardless of the prior you used, you reached very similar conclusions for the association between over-equipment maneuvers  and mortality, and all these results point towards harm of the intervention. Even when I use, I consider an optimistic prior, the results, they still point towards a signal of harm.  So this is just another way to reinterpret the trials and to discuss these trials. And of course, can report that on a table, the same results here, we have the three priors we  have used. You can report a probability of harm, the probability of important benefit, the probability  of important harm, the probability that the result is within a range that we consider of being of practical equivalence. This was defined arbitrarily as between 1 over 1.1 to 1.1.  And all you can see is that for most, for all the priors, we still have a very high probability  of harm. The chance that interventions were equivalent was around 20% or something like  that to change a little bit more with the priors. And you can have direct interpretations of these  results. For example, when we use an optimistic prior, you may just say that even when you assume  a moderate strength of mystic prior, the probability of harm of the intervention given the data is above 0.9. So there's a very high probability of harm. And the probability of  severe harm defined as a non-analyzable 1.25 is still very, it's still important. So it's still around 34% or 35%. So this type of interpretation is perhaps easier for the  clinician than just reporting the null results and the compatibility of the data with a null hypothesis.  Of course, there are several ways to do Bayesian reanalysis. So some authors advocate for using  several priors and these priors are locked in the, for example, the distribution of defect size.  But of course, Bayesian analysis is very flexible. So there is no obligation that you have normal  priors. There is no obligation that you have a binary outcome. So everything I have mentioned  can be transposed for time to events, survival, counting, or any type of distribution.  There's another paper by Chris Yarnel and Inigo Gollinger published recently also in last research on medicine. And they checked whether there was a concordance between  Bayesian analysis and Frequentist analysis. And most of the time, both analysis, there is a,  let's say, a reasonable degree of concordance on this analysis. They defined the priors in a  slightly different way than we did. But again, this is all flexible. And if it's properly justified,  it's all good. And they reported that the concordance was not very bad. It was actually  pretty good. So this, once again, points to my final slide. That is, Bayesian methods can indeed  aid trial interpretation. But there is not right and wrong here, right? So it doesn't really matter,  actually, if you're going to do a Frequentist or Bayesian trial if you use proper methods and if  you interpret your methods correctly. So regardless of your approach, if you use it properly, and  there are several other ways to analyze the data and to enrich Frequentist methods, for example,  mostly you're going to reach similar conclusions. So again, the problem is in the way we misinterpret  data and we use the wrong methods. And there is no clash between Bayesian and Frequentist. What  we have are different approaches to reach and interpret data. Sometimes Frequentist reasoning  may be preferable. Sometimes Bayesian reasoning may be easier to translate for the clinicians.  But neither of these approaches are wrong. And there is no clash and they can all live well together. So this was my last slide. Thank you very much for the opportunity to present.

Video Summary

The video consists of two presentations on the topic of interpreting clinical trials in critical care through both frequentist and Bayesian approaches. The first speaker, Alex Flannery, discusses the challenges of interpreting critical care literature using traditional frequentist methods. He emphasizes the importance of nuances in analyzing and presenting data that can influence clinicians' interpretation and decision-making. He highlights examples of trials from the frequentist standpoint, focusing on issues such as sample sizes, treating syndromes in critical care, high mortality rates, and the use of composite outcomes like ventilator-free days. Flannery also mentions the concept of fragility index, which measures how many non-events in a trial group would need to become events for a previously statistically significant result to become insignificant. He concludes by questioning the continued use of mortality as the primary endpoint in critical care trials.  The second speaker, Fernando Zimperi, introduces the Bayesian approach as an alternative way to interpret clinical trials. He compares frequentist and Bayesian methods, highlighting the limitations of frequentist approaches such as p-values and the reproducibility crisis. Zimperi explains that in Bayesian reasoning, the data is fixed while the effect size is assumed to be random, and prior beliefs are incorporated into the analysis. He demonstrates how Bayesian analysis can be used to estimate the probability of various effect sizes and interpret trial results. Zimperi presents reanalysis examples of trials like ECMO in ARDS and perfusion targets in septic shock, showing how Bayesian analysis can provide more meaningful and interpretable results for clinicians. He concludes that Bayesian methods can aid trial interpretation and suggests that different approaches can coexist as long as they are used properly.

Asset Subtitle

Research, Quality and Patient Safety, 2021

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2021 Critical Care Congress held virtually from January 31-February 12, 2021.

Learning Objectives

Identify best practices for analysis in critical care literature and downstream effects of nonadherence to these analyses in particular situations
Recognize instances in critical care research in which frequentist and Bayesian interpretations of the same clinical trial may be at odds with each other
Interpret the results of a Bayesian analysis given the prior probability reference frame

Common Challenges With Interpretation of Critical Care Literature
Alexander H. Flannery

Viewing Through a Different Lens: A Bayesian Approach
Fernando Zampieri

Meta Tag

Content Type Presentation

Knowledge Area Research

Knowledge Area Quality and Patient Safety

Knowledge Level Intermediate

Knowledge Level Advanced

Membership Level Select

Tag Outcomes Research

Tag Mortality

Tag Scoring Systems

Year 2021

Keywords

interpreting clinical trials

frequentist approach

Bayesian approach

critical care literature

sample sizes

composite outcomes

fragility index

reproducibility crisis

Bayesian analysis

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.