SCCM Resource Library-Translating Preclinical Data Into Practice: Why Therapies for Neurologic Disease Don't Work

Video Transcription

Good afternoon. I'm Tom Bleck. My co-chair, Huda Bair, is down here. This is the session on Translating Preclinical Data into Practice, Why Therapies for Neurologic  Disease Don't Work. We have three speakers over the next hour, so we should have a little time for questions either after each speaker or at the end.  Our first speaker will be Dr. Fred Rincone from Thomas Jefferson. Fred. Fred.  So thank you very much for the invitation and welcome to this phenomenal session at four o'clock before dinner in this phenomenal city to talk a little bit about bench research.  And I had the talk on models and why I think models are the problem that we have currently  in science and explain the lack of translation in terms of therapies for acute brain injury. These are my disclosures. I don't have any conflicts of interest related to this talk.  I recently learned that the National Board of Medical Examiners is going to have the approach for the USMLE Step 1 of pass and fail.  That means that knowledge in terms of an exam is changing. So we will see in the future more and more of these changes in terms of, for example,  admissions to medical school and how medical students are selected on the basis of different approaches. So that means that knowledge is an ever-changing field.  Is it possible to find the truth? The answer is maybe. And we have employed different philosophical approaches to answer our questions, test our  hypotheses, finding solutions for our problems. It doesn't matter which philosophical approach you want to use, but over the last 2,000 years  we have used inductive and deductive reasoning from large observations, large problems down  to specific conclusions, and then from specific problems, specific issues down to generalizable conclusions.  So the scientific method has a very rigorous systematic approach that is inherent to threats, and those threats impact and impinge on the robustness and the sound of the data.  So when you start with a problem, you generate a question, you have your background, you formulate a hypothesis, a null hypothesis, and you go about how to test it.  It doesn't matter how you do it, observational, experimental, you end up with a hypothesis  and an experiment, and then you go about testing it, and finally you get your results and then your analysis of those results, and then you end up with a conclusion.  Can you refute or accept your null hypothesis? That's what we do in research in general. So when it comes to animal models, preclinical models in human disease states, we have to  understand that a model is defined as a non-human biological system that is used to obtain a better understanding, more knowledge of a human disorder, a disease state.  And by this definition, we have to understand that it should display exactly the same phenotype as you see in the condition that you are trying to study or test.  So for this particular talk, I'm going to concentrate on the disease state that has had the most exposure to research at the preclinical and clinical phases.  I'm talking about focal ischemia reperfusion, I'm talking about stroke. If you review the literature, you will see the number of compounds and therapies that  have been tested for neuroprotection in ischemic stroke. And if you believe me, right now we only have two strategies that confer a degree of improvement  or benefit after ischemic stroke. I'm talking about intravenous TPA, intra-arterial endo-vascular thrombectomy, and perhaps I  could mention some of the anti-platelets, but that's it. And then beyond that, all of the strategies and things that we have done over the last  30 years have not been able to be translated on human face. So the internal validity of your research process has a lot of threats, and I'm going  to concentrate on some of them that have been studied in or by empiric research. So your animal model has to have internal validity in order to be generalizable.  Internal validity and external validity are both necessary for the translation of the experimental therapy that you're trying to use.  So human and also animal models are at risk of being threatened by problems or issues with the scientific method.  So if you think about animal models that have looked at, for example, the exposure to hypothermia as a neuroprotectant in focal models of ischemia and reperfusion, you will  see that a meta-analysis can tell you the robust effect that hypothermia has on those models of focal ischemia and reperfusion.  So when we talk about an outcome here, we're talking about infarct size, which is probably  a reasonable benchmark for both clinical and preclinical research in terms of neuroprotection. There is no question about it. We have used infarct size as a benchmark for research.  So when you look at studies like this, this meta-analysis published in 2007, looking at all the data for the last 30 years, this is from the UK group in Glasgow, has shown that  there is a robust effect of hypothermia in the focal ischemia reperfusion model. At the preclinical phase.  However, when you look at the clinical experience, you don't see that translation. As a matter of fact, the Europeans had to shut down the EuroHype study on the basis  of lack of efficacy, perhaps related to the robust effect of thrombectomy that they didn't anticipate was going to affect their study.  So when we look at design, for example, empiric research applied to this meta-analysis of preclinical data have shown that these preclinical models have been prone to these threats.  So when you look at randomization, for example, which is necessary for clinical trials, you look that there is really no effect in the animal world, right?  When you look at analysis of this meta-analysis and you look at the effect of randomizing versus not randomizing, there is no effect. Well, there is an explanation for that.  Most of these animals, right, are symmetric in the sense that they are from the same strain, same sex or gender, age, weight, et cetera.  So randomization may not play a role if everyone looks the same, and it will be probably similar in human trials as well.  However, randomization may be important at the time or after you induce your experimental therapy.  If I clip both arteries and I'm trying to test the hypothesis that therapy X is associated  with better outcomes, randomization perhaps may have a role after I induce the model because I don't have any control of what could happen to the animal.  The animal could have hypotension, the animal could have hypoxia, and I don't have any control of that. So perhaps randomization in that setting may be important.  But empiric research has shown us that because most of these studies have been done in animals that look the same, randomization may not play that important role.  The conduct in your clinical, I'm sorry, in your scientific methodology is also important. For example, when you blind your experiment, right, empiric research in models of stroke  at the preclinical phase have shown that if you are not blinded, you're more likely to  produce bias away from the null hypothesis. So this is a problem and has been observed in these animal models, and I'm talking about specifically of stroke.  So 13% higher risk of being away from the null if you don't consider blinding of the experiment or the experimental strategy that you're using. So this is crucial.  It's crucial also for clinical trials in humans.  About a third of animal models that have been analyzed through these meta-analyses have actually provided some information about blinding.  So two-thirds of the animal models that we have actually relied on for translation in ischemic stroke have not reported on blinding schemes or strategies, for example.  So when you have a significant deviation from the null, from just a blinding strategy, then you can consider this as a significant threat for your animal models.  Blinding and randomization, all of them are tried to or designed to limit bias at many stages of the scientific method.  So it's very important to consider these things when we are interpreting data from animal models. What about the effect of comorbidities?  If you think about an animal model resembling a human disease state, it's a big leap of faith.  Thinking that a rat probably has been exposed to tobacco, has diabetes, hypertension, and by the way, has a sleep apnea syndrome in a model of ischemic stroke is, you know, sort  of like crazy. But that doesn't happen in the animal model. So comorbidities, when you look at the empirical research, if you do not adjust for these  co-founders that can deviate your effect away from the null, then you enter significant  bias. So empirical research have shown that perhaps, you know, comorbidities should be addressed in these animal models.  And about, as I said before, two-thirds of these animal models have not shown to have reported adjustment to this.  So perhaps in the future, you may need to have a model where the rat will have also exposure to diabetes, hypertension, and all these risk factors for a stroke for a strategy  that you're using to produce neuroprotection.  The analysis, it doesn't seem to be important based on empirical research, but sometimes you look at animal models and reporting, and then you see how investigators don't have  a clue of how to, for example, handle linear variables or categorical variables. And you see reports using t-tests, for example, for nonparametric data that is not normally  distributed, that sort of thing. It gives you an idea that perhaps there is really not a lot of understanding in some  of these experiments, but at least from empirical research, it does not provide a sense of bias away from the null.  Blinding of outcomes, extremely important for the analysis of human data, clinical data. Outcomes in animal models or preclinical models are usually automated.  So computerized ways of determining infarct size, for example, are set in stone. So there's really not a lot of bias from measuring outcomes in this way.  But recent approaches to the problem of interpreting animal model data have suggested that perhaps  having a repository where investigators in a multi-center, for example, way of conducting animal studies are submitting information and these outcomes are measured by a totally  blinded assessor.  Reporting, it doesn't seem to affect them much based on empirical research. I'm talking about does the research meet criteria based on consensus, meaning is the  study a crappy study or is the study a very good study? It doesn't seem that that affects that much, at least from empirical research, of course.  You have to look into the details of the study and as I said, based on, for example, statistical  analysis, you can actually figure it out if the investigators have or have been using the appropriate analysis. Now, why is this important?  Because as I mentioned at the beginning, there is really not a lot of interventions out there  that are available for this particular example for ischemic stroke in terms of neuroprotection besides revascularization strategies, as I mentioned before.  This is an example of it. So this was a very promising medication that was sort of like designed to be a neuroprotectant after ischemic stroke.  It had robust data from animal models. It translated into a phase one study that was positive and then the phase two study was totally negative, catastrophic results.  Post hoc analysis of the animal data that led to the approval of the medication for the clinical phase showed that most of these clinical studies were confounded or affected  by those issues that I mentioned before, primarily lack of reporting of blinding strategies and the use of healthy animals that didn't have comorbidities.  So when the post hoc analysis eliminated that, then the results became actually negative. So it was a blunder.  Now, those are the things that empiric research have informed us in terms of the threats to the scientific method specific to animal models.  What about things that are out there but you don't know? So if you are a bench researcher and you are doing studies on the effect of a therapy A  for tumor growth and you are not considering the effect of inflammation, then when you  see reports like these, for example, the temperature of the housing at which the animal lives can  have an impact on that inflammatory profile of that animal and by decreasing the temperature  of that animal affect the profile of inflammation and therefore the growth of the tumor, then  you start thinking that perhaps you need to be more knowledgeable about all these co-founders, all these variables that may be associated with your exposure and your outcome and you  may need to account for it. So these things are not reported. These are just observations that are out there that makes you think, gosh, you know,  how different would it be to perform an experiment in the summer or during the winter, perhaps  an experiment in Brazil close to the equator than in Iceland or in Denmark, you know, far  away from the equator in terms of the environmental confounding that could occur in animal models.  So it seems that there is a problem. Fortunately, we have recognized this, at least we start sort of like becoming more aware  of this around 2007, 2008 when people start thinking about are these animal models really representing what we do in terms of understanding disease states in humans.  And fortunately, there's been a movement in the scientific community to move towards a better way of developing a testing strategy for questions that we have using models.  So the lack of internal validity is, I think, the primary reason why animal models don't represent that well our understanding of human diseases.  And that may be the reason why we haven't been able to translate a lot of therapies that are promising in terms of neuroprotection.  And I'm just talking about one, any ischemic stroke. I'm not talking about TBI. I'm not talking about seizures. Cardiac arrest, perhaps a lot of translation, right.  But even though we have a signal right now in the clinical phase, we still don't understand  things about dosing, duration, degree of hypothermia, and the impact of several other variables that we are starting to understand more and more right now.  So this is what I'm talking about. Fortunately, there is a movement right now to look into more rigorous reporting  and try to limit the threats that we have with animal models, particularly looking at  randomization strategies, blinding and allocation and concealment of allocation groups for animal models as well.  Blinding of outcome measures with central repositories, perhaps doing multi-center animal status is something that is occurring and happening. And I think it makes a lot of sense.  And then finally, sample and power calculations. Extremely important. When I publish my cohort studies looking at databases, the first question that comes is,  what was your power for your end? 3% of the studies perform in stroke at the preclinical level had reporting on sample and power.  So even if you have a P value, but you don't have the power, right, to reject or accept, you know, that no hypothesis that it becomes into question, right?  If the study has really internal validity. So can we arrive to the truth? The answer is yes.  And this is the truth index that I sort of like thought about yesterday is directly proportional to the robustness of the science and the robustness of the science by limiting those  threats can be accomplished. Right. But there are several things that also need to be taken into consideration, right?  What are the financial pressure to run a therapy through a preclinical model and then through different models and then finally into the clinical phase?  So that's a lot of pressure for the investigator. And then finally, the pressure that we sometimes have in the academic setting of publication.  You know, should we also report negative studies and so forth? And the answer is probably yes. Having said this, I'm done.  I will probably do questions at the answers at the end. Right. Thank you.  Hello, everyone.  My name is Julia Bayer. I'm from University of Pittsburgh. And our next speaker is Dr. Joshua Levine.  He is from Permanent School of Medicine. And he's the Chief of Division of Neurocritical Care at University of Pennsylvania.  His research is on goal-directed cerebral resuscitation and multimodal neurological monitoring.  Thank you.  Maybe we can take questions if you have any questions  Here, I'll ask a question. OK, there we go. Sorry about that. So I'm going to talk to you about failure to launch, why randomized clinical trials fail in clinical neuroscience.  And really, the remarks I'm going to make don't just apply to clinical neuroscience, but they apply to critical care and many other fields of medicine.  How do I advance the slide?  There we go. OK, so I do have an unrestricted educational grant on a topic that has nothing to do with this talk. But I do have skin in the game for this talk.  While I've participated in many clinical trials, both as an investigator and as a subject, I don't consider myself a clinical trialist. I have, however, as all of you have,  funded numerous clinical trials, almost all negative through my federal taxes. So the objectives of this talk are going to be to describe the failure of randomized  clinical trial in neuroscience, to give you an understanding of the flaws in the way preclinical data is used to inform human trials, and then  to recognize key failure points in trial design, execution, analysis that all contribute to negative results. So we're going to start out with rocket science.  And the reason we're starting out with rocket science is because I think there are a lot of analogies between the way you launch a rocket and the way you launch a clinical trial.  Rockets are very expensive to launch, whether they're funded by the government or now by private industry. They require a lot of foundational science  and engineering before you put a rocket on a launch pad. And importantly, a flaw at any stage can lead to catastrophic failure. And everyone here is aware of that.  This was the Challenger disaster. And this was a headline from The Washington Post. And it says that a major reason for the space agency's failure  to heed the warnings that the O rings were faulty was pressure to meet an overambitious schedule. And I want you to remember that, an overambitious schedule. It was rushed.  Here's another one a little bit more recently where a Mars probe crash landed onto the surface of Mars instead of making a gentle touchdown. Why?  Because the Americans were collaborating with the British. And one team was using imperial units and the other metric. And they failed to convert one to the other.  And it ended in disaster. And so RCTs have an abysmal track record in critical care. And this is not true in every field. There are certain fields, like infectious disease,  where a single organism causes a single disease. Or even some fields of oncology, where a single gene mutation or a single cell type causes a very well-defined disease.  But in critical care, in neuroscience, in neurocritical care, where you're dealing with multiple pathophysiologies and poorly defined diseases, RCTs have an abysmal track record.  The paper that I cite down below estimates that 95% of randomized clinical trials that attempted to show a reduction in mortality in critical care have been negative.  And there's close to a 100% failure rate in neurocritical care to find a neuroprotective strategy,  the glaring exception being targeted temperature management. So this is not a secret. And it also applies to many fields of medicine. If you open up the newspaper on any given week,  you're likely to find a story about this. This was from three weeks ago in The New York Times. Childhood obesity is a major problem. Research isn't helping.  Something is missing with many study methods. This is from last week. An Alzheimer drug trial fails. We don't have anything now.  So this is now a problem that the public is aware of. And the problem is that there's been an abject failure to translate therapies with promising preclinical data  into safe and effective human treatments. And there's been a stepwise decline in efficiency from preclinical work to human studies. This is a graph from a paper entitled  Why Most Acute Stroke Studies Are Positive in Animals but Not in Patients. And what it shows, the red bar, is how many positive studies there are in animal studies.  And it's close to 70%. But then you look at the orange bar, early clinical trials, it drops down to about 30%. And then phase three trials are almost never positive.  And so a stepwise decline in efficiency. So why are randomized clinical trials negative? There are multiple levels of failure. So as Dr. Rincon talked about, there  are problems with preclinical studies. This is undoubtedly true. But we're also testing therapies that don't actually work, probably because of false data from animal studies.  Maybe we don't understand the pathophysiology. And we have to admit that there's a possibility  that some diseases may just be too inherently unmodifiable, as unpalatable as that seems. We're also testing and abandoning therapies that actually do work because of the way  we conduct research. And I'd like to focus on these last two points. So what I'd like to posit to you is that the reason we're testing therapies that don't actually  work and the reason we're testing and abandoning therapies that do work are, number one, problems with the way we use preclinical data to inform or justify human trials.  And then number two, problems with randomized control trial design, conduct, and analysis. And so there are multiple stages of failure. We'll start out with how preclinical data  informs human trials. So we tend to overestimate treatment effects of animal studies, leading to underpowered human trials. Why? How does this work?  Well, number one, we use animal studies that lack external validity. And this was touched upon in the last talk. We study young, healthy rats or mice. We find something that works.  And then we design a trial with old, morbid humans. Or we take an animal study that uses a middle cerebral artery occlusion model of stroke and uses infarct volume  as their final endpoint. And we design a human trial that has multiple types of strokes and uses functional outcome as the endpoint. We overvalue false positives.  So how does this work? This is based on a very often cited paper by Ionitis in PLOS Medicine that was entitled, Why Most Published Research Findings Are False.  And basically, the argument goes like this. We do animal trials, lots of them. They're mostly negative, except for the one that's positive. Then because of a publication bias,  that's the one that's published. We end up with an excess of published false positive studies in the literature. That leads to overenthusiasm for therapies and overestimation  of effect sizes, which ultimately leads to underpowered human trials. And this argument was put forth in an editorial in the New England Journal of Medicine as a major reason  that two very sort of storied trials, Synapse and Protect 3, failed to show a benefit for progesterone in traumatic brain injury.  Now, the fact that there are numerous false positive studies in the literature has also been demonstrated. In this paper from this year, the authors  use funnel plots to give you a sense of whether or not there was bias in the literature from unpublished studies. And what they found is that there's  a significant publication bias in experimental or animal studies, a moderate publication bias in early clinical trials, and a much smaller one in phase 3 trials.  And this was in stroke. And then the bottom row is just a trim and fill analysis where they sort of give you a sense of the magnitude of how large that bias is.  And this paper suggested that roughly 50% of the animal studies in stroke are probably false positive results. So then we get to randomized controlled trial design  and patient selection. So how many patients should you include in a trial? Well, human trials are frequently severely underpowered.  In part, it's because we overvalue false positives, as I just described. But it's also partly due to a lack of appreciation for target pathway-specific attributable risk.  So what do I mean by this? Well, let's take a hypothetical disease. Let's imagine this. You really don't have to imagine it  because you could say this is sepsis or ARDS or maybe even TBI. So we wish to study an intervention for a disease that's associated with a 35% mortality rate.  We might power a randomized controlled trial to detect a 20% relative risk reduction in poor outcome. So you therefore do a calculation  and find that it would take 1,378 patients for 80% power to demonstrate this effect. And this is what we do day in, day out in clinical trials.  But this trial design neglects a fundamental question, which is which fraction of the total poor outcome is attributable to the pathway being studied. So what do I mean by this?  So let's take severe TBI as an example. No patient has just severe TBI. They have comorbidities. Those might include things like alcoholism or substance abuse,  prior head injury, HIV. How about poverty, lack of education? All of these things might affect outcome. They also develop complications, sepsis, ARDS, renal failure,  PE, et cetera, et cetera. And the top arrows in blue are all those pathways that have nothing to do with TBI that actually impact our outcome, in this case, death  or poor functional outcome. And on the bottom in red are the pathways where TBI leads to poor outcome. And there are probably many of them.  There are many pathophysiologic pathways in TBI that lead to poor outcome. And if the one on the very bottom is the one that your drug is targeting,  you're not really taking into account all of the other factors that impact outcome. And to give you a sense of what this means, if the pathway attributable fraction  for risk of poor outcome is 0.5, then you actually need 5,700 patients. If it's 0.25, you need 23,000 patients. And so what if two different patients  have two different pathways towards poor outcome? It makes it even more complicated. And so the typical trial that has 500 to 2,000 patients is grossly underpowered because we  don't think about the fraction of the risk of poor outcome that's attributable to the pathway that we're trying to study. So we talked about how many patients.  How about which patients should we include? So patient selection for studies in TBI or stroke are plagued by poor definition of disease.  So one author looked at how many definitions of concussion there were in the literature. And at his count, there were something like 40 to 45.  So if we can't define the disease we're trying to study, we're in big trouble. All of these diseases are heterogeneous. And Dr. Rincone touched upon that.  Is someone with diffuse axonal injury the same as someone with a unilateral subdural? Does the location matter, whether it's bihemispheric? Is there edema? Is there bleeding?  These are things we all lump together, even though they're completely different diseases, and study is one. We misclassify injury as well.  And confounding comorbidities are not excluded. Trials do their best. They might exclude a few. But I've personally enrolled people in clinical trials  where a catastrophe happens in the hospital that you know is going to impact outcome, but they're still included. And when you look at some of these fragility scores  for studies, all it takes is one or two patients miscategorized for the entire trial to be undone. And we don't account for confounding therapies.  There was a study published in 2012 in the New England Journal looking at amantadine for outcome after severe TBI. And a third of the patients in that study  received confounding medications, antipsychotic medications. And so if we're not accounting for other medications that act on the same pathways, how can we possibly  study a specific target pathway? So how about therapy? In the neurosciences in particular, we suffer from too little, too late. We don't incorporate preclinical data  to optimize treatment dose, duration, frequency, route of administration, or therapeutic window. We don't incorporate data about the effects of age and sex on response.  And there are animal studies that show that it might work in males, but not in females. But yet we design a human trial that includes both. And we don't individualize therapy  based on individual patient pathophysiology. So this is a comparison of time to treatment between animal studies, early human clinical trials, and phase 3 trials.  And what you can see is in red that the majority of the animals get treated in the 0 to 3 hour window. In the early clinical trials, the majority of human subjects  get treated very late. And in the phase 3 clinical trials, it's all over the place. And this is for stroke. And we know, or at least we think  we know for neurologic conditions, time is brain. And the earlier you treat, the better. And so how can we possibly take animal studies where there's early treatment and then  design a human trial with late treatment and expect it to have the same effect?  So how about standardization? We try our best to standardize therapies, but we don't do a good enough job. And this study, I think, was very eye-opening.  And it looked at between-center differences in outcome in major TBI trials. And what they found is that the difference in outcome between one center and another  was massive, especially in Europe. And if this difference in outcome between centers dwarfs any effect of the thing you're studying,  you're never going to have a positive clinical trial.  So another example of this is we don't account for differences in access to or quality of rehab. And I think this is a huge oversight. Rehab makes a big difference.  There's large disparities in the way patients are treated in rehab within this country and then both between this country and other countries.  Having had a patient that I discharged from my ICU to Scandinavia, where they underwent a year of government-funded rehab, there's clearly a difference there.  And if those patients are all combined into one trial, there's no accounting for the effect that rehab has. How about our outcome measures? And this is sort of a face plant.  The choice of outcome measure can determine the success of a clinical trial more than the actual intervention being studied.  And so there's an utter lack of consensus or standardization across trials in terms of which outcome measure to choose. This is especially problematic if you  want to do a meta-analysis. And commonly used metrics, like the GOS or the GOS extended, are very crude, blunt instruments that are insensitive to clinically important findings.  Furthermore, they're subjective. And a patient might answer a questionnaire one way, based on circumstances that day, and a completely different way  if circumstances change, like family support or family issues. And this can lead to arbitrary placement in a given category. Very problematic, and not like the animal literature,  where infarct volume, something objective and measurable, is being studied. And then we take these ordinal scales and we dichotomize them into good and bad outcome.  And this leads to further loss of valuable information that's contained in the ordinal scale. So this was an interesting paper that basically  showed that the choice of an outcome measure can determine how you perceive the patient did. So what percentage of your patients in your trial recovered?  Well, it depends on what outcome measure you used. And they took commonly used outcome measures. And they showed that your perception  of how many patients did well ranged from something like 30% to 70%. These are the same patients, just changing the good outcome definition. And so, of course, that's also highly  problematic for randomized clinical trials. And then how about follow-up duration? And this is a quote from the paper that I mentioned below by Donald Stein, which  I thought was spot on. There's no law of neurology or neuroscience, but there may be such a principle for health insurance companies declaring that if recovery does not  occur within three to six months, it will never occur. We know, and we know increasingly, that severe brain injury can recover over a very protracted time course.  The longest stroke in TBI trials typically measure outcome at six to 12 months. And I would guess that this probably isn't long enough.  So in terms of executing clinical trials, this was mentioned also in Dr. Rincone's talk. There's pressure to do it quickly. And remember the Challenger disaster.  That was also fed by pressured timetables. Clinical trials are rushed. They might not feel like it, but there's tremendous financial pressure. If you have a government grant, you  have three to five years of funding before you have to apply for continuation. And every investigator knows that funding budgets change, and often not for the better.  And it's no better if you're industry funded, because industry is under pressure to maximize the amount of time left on patents. And by the time phase three trials are conducted,  there's often very few years left on the patent. So what does this lead to, this rush through the clinical trials? It leads to shortening of the follow up interval,  adding centers on an ad hoc basis, and that worsens quality control. So data handling or quality control is a problem we don't talk about a lot.  We all know about the impact of medical errors on patient outcomes after this publication to air as human. But so too are research errors in data entry and data handling.  This was a study in Boston. I think it was at the Dana-Farber, where the investigators had a trial where they entered data from the same subject  into two different duplicate databases. And these investigators went back and they said, let's compare the databases and see how many errors were made.  And what they found is that the errors were as high as 27%, or a 13.5 error rate in each database. And what makes this worse is you could say, well,  if the errors are randomly distributed, maybe it doesn't matter. But they weren't. They found things like, if you make a data error entry on one patient, you're more  likely to make more errors for that patient. And so these errors are not always randomly distributed, and they absolutely have the impact to completely alter a clinical trial.  Finally, how do we analyze randomized clinical trial? We could have an entire symposium on p-hacking and the manipulation of statistics, or more commonly, the misuse of statistics.  And we're not going to go through all of that. But what I think everybody knows is that you can take the same data set, and depending on which test you do,  come up with completely different conclusions. Oh, the data is tapering off. Or, oh, the data is exploding uncontrollably.  And this is obviously a problem for randomized clinical trials. And so what we're left with is that when we launch a clinical trial and we conduct a clinical trial,  there's a whole checklist of all of these things I've mentioned that have to go right. And if any single one of them goes wrong, the rocket blows up on the launch pad,  and the clinical trial just fails. So a single point of failure is sufficient to blow up a randomized controlled trial. So I'm going to leave you with some final thoughts.  I think we set ourselves up to fail often because we ask the wrong questions. We ask questions like, what's the optimal blood pressure  after fill-in-the-blank, intracerebral hemorrhage, acute ischemic stroke? What's the optimal ICP in fill-in-the-blank, patients with TBI? What's the optimal hemoglobin for?  You can fill-in-the-blank there. And I think these are the wrong questions. It's sort of like asking, if you drive a sedan, what's the optimal speed limit?  That's a nonsensical question. It depends greatly on the local conditions. Is this a superhighway? Is this a New York City street? Is this a country road?  And then if you tell me, well, but we narrow our definition and we say we're just talking about patients with ARDS, even that's not good enough.  What if you just limit yourselves to country roads? Well, is it nighttime? Is it raining? Is it snowing? And it's almost impossible to get down to an answer  unless you're examining an individual driver on an individual road with individual conditions or an individual patient with his or her own physiology and pathophysiology.  And so what we do is we neglect the immense heterogeneity there is in individual patient genetics, environmental influences, physiology, and pathophysiology.  And even worse, we take these populations like TBI patients, we lump them all together. We may find that on average, people do well with an ICP of less than 20,  but then we misapply this by saying every patient we manage must have an ICP of less than 20.  It's sort of like my surveying American men and asking what's the average waist size, figuring out it's 38. And then my response to that is issuing size 38 pants  to every American man. Obviously, there would be a lot of unhappy men. And so this quote, which is misattributed to Einstein and I think probably comes from some academic at Yale  in the 60s says, if I had an hour to solve a problem and my life depended on it, I'd use the first 55 minutes determining the proper questions to ask.  And I think we could stand to do a little bit more of that. So to conclude in clinical neuroscience as in other fields of medicine, there's been a failure to translate  promising preclinical research into a safe and effective clinical therapies. And the causes include problems with clinical trials,  how they're justified based on the animal literature, how they're designed, executed, analyzed, and we're not always asking the right questions.  And the result of that is that we've invested a lot of time and resources in testing therapies that should never have been tested in the first place.  And we've abandoned therapies that actually work. And this middle diagram strikes me because it seems like in the aerospace industry,  they try, they fail, they try again and they succeed because they're very good at analyzing what went wrong and learning from it.  But in medicine, sometimes it feels like we're stuck in this loop of try, fail, try, fail, try, fail. And we're not quite as good at learning from our mistakes.  And so with that, I'll stop.  All right, well, we've heard from Fred that the models are the problem and from Josh that the trials are the problem. Now, Pat Kahanek, the head of the Safra Center  at the University of Pittsburgh and editor of Pediatric Critical Care Medicine is gonna tell us we don't have a problem at all.  Well, I don't know about that, but I think I'll try to give you a few ideas of how we and others are thinking about solving some of these problems.  Am I all set to go here?  Sorry about that, he didn't think he had the presentation ready yet.  Hopefully I'll actually address some of the points that were made, actually.  Here we go.  I have a number of conflicts, but none of them have really anything to do with this presentation.  I also thought that what I would do, since all of neurocritical care is such a broad problem,  is that I'll use traumatic brain injury, one of the areas that we focus on to try to define a path to a therapeutic breakthrough.  I'll start by saying I really believe we need more support for translational research,  but I think the current situation would probably dictate to say something like that.  First, you don't need preclinical research on traumatic brain injury because it's just a headache.  And I think that I might follow it up by saying that, in fact, you don't need any research at all. Just use the treatment and I'll take care of the regulations.  But I really think that if we go to Pittsburgh, maybe we can get some answers.  And in fact, if you think of the most important event in the history of Pittsburgh, other than the Immaculate Reception,  it was the development of the polio vaccine.  And if you look at the development of the polio vaccine, it obviously was translational research in the green monkey kidney virose cell line.  It took 50 years from the discovery of the polio virus to the use of that cell line to develop the polio inoculum with the formalin-inactivated virus.  And so for a very simple translational model, it took 50 years, in essence, to develop what was probably the greatest thing that Pittsburgh has contributed.  But what about TBI or what about other acute neurocritical care disorders?  And we heard the blame game, the clinical trials, a litany of lists, and also the preclinical data.  But rather than blame, I'm going to try to talk about some solutions.  And to me, the first thing that we have thought about, and this is to, well, let's go to other acute neurocritical care diseases,  not to point out what has failed, but to look carefully at the winners.  And so if you look at the winners, obviously, it was just briefly mentioned that the big winner, of course, was mentioned targeted temperature management.  But the real big winner is actually the use of mild hypothermia in perinatal asphyxia for neuroprotection.  And you can look at the meta-analyses. They're just all positive, positive, positive.  And it doesn't take a rocket scientist, actually, to recognize that.  But what's really interesting, you see basically about a 20 percent effect size for this.  And remarkably, the exact protocol that was used to treat and is currently employed in perinatal hypoxia ischemia was developed in an animal model in perinatal sheep.  And with a 90 minute potential delay and three days of therapy, the protocol is nearly identical to what was used clinically.  And remarkably, this also included something that we talk about a lot.  But these guys were pretty prescient in terms of patient selection for this.  It's only been shown in the term newborn.  And you can look through the list of other potential selection criteria.  And so they were really ahead of their time in terms of, if you want to call it, phenotyping and careful patient selection and following the animal model to a T.  And this really led to a breakthrough.  It is standard of care in term newborns.  And when I was a resident in the 1980s, it was malpracticed to prevent, to fail to prevent hypothermia in a term newborn.  If you allowed the baby to get cold, you'd be thrown out of the residency.  And it's why incubators were developed.  And what's really interesting is, ironically, it's probably some unique molecular neuroprotective facets of the cold stress response in the newborn that really gives it powerful beneficial effects of hypothermia and maybe the importance of apoptosis in newborn brain development.  But recognize the clinical trials would have never happened without the preclinical research.  So it's a perfect example of translation.  And another one, we talked a little bit about stroke, but I think people in this room have missed what I think is the important point about stroke.  And that, look at clot retrieval.  Clot retrieval has been successful. It has, what, about 20 to 30 percent effect size.  And so you say, wow, you need a pretty big effect.  And you'd say, well, what about the animal studies?  All the drug studies have failed. But look back at Beau Sigeaud's 1992 paper.  And he showed his statement that the time window for salvage of penumbral tissue and focal ischemia is thus about two hours in rats, two to three hours in cats, and three to six hours in monkeys.  And that's really kind of what was found in the initial studies with clot retrieval.  Obviously, people have been able to push the window even further now.  But the way I look at it is the therapeutic potential of clot retrieval was staring us in the face for decades in stroke.  But the powerful temptations of drugs derailed this somewhat.  And so I think the translational potential was there.  And we didn't recognize it as well.  And obviously, you could say, well, we didn't have the devices either.  But I think that's another really important example. Peter Safra used to always say, in stroke, you need to fix the plumbing.  And amazingly, he was always surprised that everyone was trying drugs rather than trying reperfusion strategies.  He was ahead of his time in that also.  What kind of effect sizes in TBI?  Between negative eight and plus eight, if you look through all the studies.  And that probably won't work.  Well, how do we develop a preclinical strategy? If we have a preclinical strategy, then that might help us identify breakthrough therapies.  And this was one of the things that I've been involved with for the last 10 years.  And that's Operation Brain Trauma Therapy, a preclinical multicenter drug and biomarker screening consortium for severe TBI.  Are we treating TBI as if we were treating cancer as a single disease concept?  We put together multicenter, multi-model, highly rigorous approach to this with people with no axe to grind on any given therapy.  We had two hypotheses.  One, if we had a robust therapy that crossed all the animal models, that would be the one to take to a regular trial.  And even recognizing all the risk of other comorbidities and all the weaknesses we've heard of,  it would still be a big step forward that, hey, this didn't just work in one lab or two labs.  This worked in a rigorous multicenter consortium.  The other is that if a therapy worked in one model, pick the patients that were like that model.  Classic example, one of the models is controlled cortical impact. Something that showed only benefit in that, doing contusions in a clinical trial.  The Operation Brain Trauma Therapy was also very highly rigorous.  I'll give you an example.  There is a different technician that makes the drug and administers the drug from the one doing the injury,  from the one doing the behavioral outcome assessment, and the one doing the histology.  And you need very experienced labs to be able to do that.  We have done in Operation Brain Trauma Therapy 1,500 rats and about 5,000 biomarker samples.  We picked 12 therapies, trying to cover the gamut of drugs that had a lot of preclinical support  or drugs that targeted a mechanism that we wanted to kind of cover the spectrum of mechanisms.  And we used the literature to base our dosing.  Have we found a magic bullet? Here is what our Operation Brain Trauma Therapy scoring matrix showed.  It was really eye-opening.  Some drugs, such as EPO, with 24 positive literature reports in traumatic brain injury models,  actually had negative points.  Some total across Operation Brain Trauma Therapy.  Two therapies showed the most promise.  One, levatoracetam, actually given as a single bolus. Quite remarkable that it was really the only truly neuro-targeting drug that we tested.  And the other, gliburide, which Marc Simard's group has suggested is a drug that's targeting  the SIR1-TRIP-M4 channel and is a drug targeting edema, contusion expansion.  It generated its greatest number of points in contusion. And I know a Phase III trial is being launched for it.  We're really pleased because we were hoping to see that happen.  In fact, you could think of, this has been a big gap in preclinical research.  Where is the multicenter rigor to match the multicenter rigor of clinical trials?  And in fact, you could think of this needing to be done in many ways.  This was, OBTT is just this one little hourglass here.  But you could have delayed and chronic therapy.  You could have only delayed therapy.  Test rehab agents.  Look across age, sex, TBI, phenotype, etc.  And so we really think this strategy is important.  You might say, well, why not just do this in one lab?  Do it in one lab.  But this is the way it's been described in Operation Brain Trauma Therapy.  It's like you have a cop car driving behind you.  There's this hypervigilance.  And we really think it adds something to the story.  Also, how we select therapies.  We use the literature review, but maybe some new high-throughput screening strategies should be employed.  And this is a strategy that we're employing right now.  People have also suggested do immediate rapid screening in lower vertebrate models. As I mentioned, we're using a stretch model now to screen therapies with RNA-seq  to try to get the Computational Drug Discovery Institute to come up with some totally novel therapies that might be tested.  Another problem we heard a little bit about, but we thought after Operation Brain Trauma Therapy is particularly important,  is what we call target engagement. Have you really affected the mechanism that you're suggesting that you're affected with your therapy,  whether it's in an animal or in a human?  And in Operation Brain Trauma Therapy, we measured UCHL1 and GFAP,  the neuronal and glial biomarker in all the animals.  And our data was actually used by the FDA in the ultimate approval of these two biomarkers, more as a diagnostic.  But what about a pharmacodynamic response to see, or some people call a theranostic response of target engagement?  We have a repository with all these samples, and obviously biomarker people have been testing it. We have samples at 4 hours and 24 hours, and we've been looking at,  can you predict the effect of a drug out to 21 days in behavior and histology?  And you can see here, this is a paper this year published by Kevin Wang, a leading authority in biomarker development.  And this is with phosphoneurofilament H, showing when we broke the code of the Keppra animals,  showing that benefit that we saw in two models with Keppra. And look, in both of those models, Keppra at both of the doses tested reduced phosphoneurofilament H,  and you start to ask yourself a question, is this just a general biomarker that might be able to be used?  Boy, it would be really nice in animal studies to be able to say at one day,  this is a promising therapy and not have to go out to 21 days, screen 20 drugs,  and then take the winners and really develop those. Or even more exciting, might this be a specific target engagement biomarker for axonal injury?  And that would obviously be even neater, something like neurofilament light, which is also very promising.  What about target engagement in humans?  And Julia, one of our moderators, published a paper now a decade ago,  showing depletion of antioxidants in brain.  This is in CSF, ventricular CSF, of kids with severe head injury, progressive and enduring.  And the loss of glutathione shown here.  Our group just completed and published two papers on, if you want to call it, target engagement study.  You may or may not be aware that N-acetylcysteine has showed some promise, particularly in blast injury in a paper that was published in TBI in humans.  But the problem with N-acetylcysteine as an antioxidant is that it is transported out of the brain  by the organic acid transporters 1 and 3.  But the drug probenicid can inhibit those transporters.  Both are FDA approved. So we carried out, Bob Clark and Phil Empey, in a collaboration between critical care and pharmacy,  carried out a really nice feasibility trial in kids with severe TBI.  And this was first shown in rats, that if you give rats this combination,  there was a 1.6-fold increase in blood levels of N-acetylcysteine, but an even more dramatic 2.4-fold increase in brain levels, consistent with our overall hypothesis.  And you can see the first paper on this showed that N-acetylcysteine was readily detectable in CSF.  So it wasn't being transported out, or at least not as readily.  The probenicid was also detected and we saw no safety concerns, although it's a small study.  But more exciting was a pharmacometabolomic analysis.  Here's the principal component analysis of control patients, TBI vehicle, and TBI treated.  And you can see that, just for the sake of time, what was really exciting was that  there was over 450 metabolites different from the placebo and the treated groups. Sorry about that.  And a further analysis of the pathways showed that the glutathione pathway was enhanced by this treatment.  And so, again, showing maybe the level of sophistication we need to really try to determine,  are our drugs affecting the targets in brain?  That has been a really big gap. And you might say, people talk about, oh, who can do combination therapy?  You know, you have to have each drug approved.  Here's an example of a preliminary study of combination therapy in critically ill kids.  So if we can do it in kids, it should be able to be done certainly in adults.  And what about genetics?  We heard a little bit about genetic differences.  This is another area we're interested in with cerebral edema, based in part on operation brain trauma therapy. Might we get something new for edema?  And there are a lot of potential exciting new candidates.  As I mentioned, we've been interested in SIR1-TRIP-M4 now after operation brain trauma therapy.  What about the genetics of this?  Why does one patient with a contusion explode and need a craniectomy  and the other with a contusion just kind of sit there and not explode?  There could be a lot of reasons, but one may be genetics. And might we, with the genetics of a pathway like this, be able to predict who's destined to swell?  And so this is the work of Ruchira Jha, one of our adult neurocritical care faculty,  who's done just beautiful work on ABCC8, is the SIR1 gene.  And this is in our single center DNA bank.  And she identified several SNPs that were strongly associated with CT edema, peak ICP, or the need for decompression.  And even more exciting is in a subsequent follow-up study,  she showed that the polymorphisms are all occurring at the SIR1-TRIP-M4 interface.  If you remember, glyburide has an effect on SIR1 in diabetes,  but that's in the opposite end of the chromosome. And all of the diabetes polymorphisms are in the KIR channel,  which is the potassium channel related to glucose.  So this seems very unlikely to just be by chance.  And we are really hopeful to be able to confirm this in the center TBI database  with the 5,000 patient genetic database that they are accumulating.  And we have a proposal. And so maybe TBI phenotypes, TBI genotypes.  And I just mentioned drug therapy.  I'll just close with a couple of comments.  One is that, are there diseases in the neuro realm where drugs have been successful?  Obviously, the one that comes to mind is multiple sclerosis,  because you have the EAE model.  And 20 years ago, people were saying exactly the same thing about the EAE model. Several trials have failed. Nothing's translating.  But now look at the list of therapies that are FDA approved for multiple sclerosis.  And in fact, here's just one example of one of the drugs.  This BG12 dimethyl fumarate that was effective in EAE,  that was effective clinically,  and it had a 51% reduction in the treatment effects.  So again, big effects.  But what may be more important is, once again, like with the neonatal brain injury,  the MS investigators have done careful phenotyping.  And in these clinical trials, they have primary, inactive, primary progressive,  relapsing remitting, secondary progressive phenotypes  that the different drugs have been tested in.  And so again, maybe we need to just do a better job,  and that there really is hope.  And finally, why hasn't preclinical research in TBI done as well?  And getting back to Peter Saffer,  when Peter was first testing in brain resuscitation clinical trials, several therapies,  and in fact, when I first was working with him,  he said he wasn't sure that we ever can get a therapy  in acute neurocritical care or in acute resuscitation.  He, as we heard in our second lecture today,  he felt there were just too many potential confounders,  successful translation might be impossible.  And so I do think, though, that if I were to summarize,  here's what I think that might work.  And then on the preclinical front, really improve the rigor,  the reproducibility, but do it with multicenter preclinical testing  and ensure target engagement in the animal studies.  And in the clinical arena, we heard define the phenotypes.  And I didn't even mention physiologic phenotypes in TBI.  That's a whole additional area. I just didn't have time.  Incorporate genetics if we can, not just predicting who will swell,  but who may respond to a therapy.  Optimize outcome assessment, as we heard. Ensure target engagement in brain PK and PD.  And use advanced trial design.  And if I had to have an overall vision for the failure of translation,  I guess I'd end with this slide.  They're coming to test their new traumatic brain injury device on us.  They think it'll translate.  Thank you.  I apologize, because of our technical glitches, we're going to have to end the session here. I'd like to thank our speakers for three really insightful presentations.  In about 15 minutes, actually, the ACCM convocation will be held over in the Regency Ballroom in the Hyatt, so please join us for that. Thank you all very much.

Video Summary

The session on Translating Preclinical Data into Practice discussed the challenges of translating therapies for neurologic diseases from preclinical studies to clinical practice. Dr. Fred Rincone from Thomas Jefferson University emphasized the importance of models in preclinical research and explained how models that do not accurately mimic the human disease state can hinder translation. He also discussed threats to the internal validity of preclinical models, such as lack of randomization and blinding, and the impact this can have on the robustness of the data. Dr. Joshua Levine from Penn Medicine discussed the failures of randomized clinical trials in neuroscience and highlighted problems with trial design, conduct, and analysis. He proposed solutions such as improved patient selection, preclinical testing of therapies that have a robust effect across multiple animal models, and the use of biomarkers to assess target engagement. Dr. Patrick Concannonek from the University of Pittsburgh shared examples of successful translation in neurocritical care, including the use of mild hypothermia in perinatal asphyxia and clot retrieval in stroke. He also discussed the importance of conducting preclinical research with rigorous methods and ensuring target engagement in both animal models and clinical trials. Overall, the session highlighted the need for improved preclinical models, better trial design and conduct, and personalized medicine approaches to increase the translation of therapies for neurologic diseases.

Asset Subtitle

Neuroscience, Quality and Patient Safety, 2020

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2020 Critical Care Congress held February 16-19, 2020, in Orlando, Florida, USA.

Learning Objectives

Recognize pitfalls in translating data acquired in preclinical models to the clinical setting
Explain ways in which trial design may be responsible for failure to reproduce findings in preclinical models
Describe both preclinical models and the results of clinical trials to advance the field

The Models Are the Problem

Fred Rincon

The Trials Are the Problem

Joshua M. Levine

There Is No Problem; You Just Have to Understand the Challenges and How to Interpret the Data

Patrick M. Kochanek

Meta Tag

Content Type Presentation

Knowledge Area Neuroscience

Knowledge Area Quality and Patient Safety

Knowledge Level Advanced

Membership Level Select

Membership Level Professional

Membership Level Associate

Tag Stroke

Tag Evidence Based Medicine

Year 2020

Keywords

Translating Preclinical Data into Practice

neurologic diseases

preclinical studies

clinical practice

models in preclinical research

internal validity of preclinical models

randomization and blinding

trial design, conduct, and analysis

preclinical testing of therapies

target engagement

personalized medicine approaches

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.