So good morning, everyone, and welcome to this session on real-world evidence. So just as a bit of disclosure, I do have funding from NHLBI, CDC, HRQ, and I do serve on DSMB for pragmatic trials, and I'll talk a little bit about them, and non-pragmatic trials, as well as actually serve on a scientific advisory panel for Regeneron and Philips Healthcare. So over the course of the next 15, 16 minutes or so, we aim to understand what real-world data and real-world evidence is, and to be able to describe the different types of real-world evidence, and to understand the strengths and limitations of this evidence. So first let's start by understanding what are we talking about when we talk about real-world evidence. FDA defines real-world evidence as clinical evidence about the usage and potential risk or benefits of a medical product derived from the analysis of real-world data. So what's real-world data? It's basically data that is generated in the course of day-to-day life. For most of us who are doing medicine, we already know that a lot of clinical data in our EHR system, in our insurance claims, Medicare, Medicaid, are all part of this real-world data that can be examined. But this data now is kind of going through transitions in the sense that more and more of this data can now be linked. An example of that is this merger of CVS Health and Aetna means that data are not just your prescription medicine, but also actually your over-the-counter use medications can now be part of this real-world data for analysis. And we used to think omics and detailed family history is something that you give your doctor and it's only in research. Well, that's not true, because almost all of you guys know somebody who has participated in 23andMe as well as actually a family tree genealogy. All of that is real-world data and they're actively participating in research now. Many of you are like me and has either a Fitbit or an Apple Watch, these mobile devices, as well as actually all of the data that is coming through the ICU in terms of your monitors, your vents, all of those are real-world data now that can be used and analyzed. And it's not just healthcare data. So this is an example of using particulate measurements of the atmosphere and pollution and linking the Canadian wildfire smoke to increase asthma symptoms presentations to ED in New York City. So environmental data. Most of the data that is out there is actually data that we have generated when we interact with the internet, the social media, what we look up in terms of reading and PubMed or all of the other things that we do in the internet. And this is an example of the use of real-world data of looking at search terms and seeing how frequently in an area people will search out symptoms that are related to an upper respiratory infection and how that might predict for surges in influenza in that area and to the emergency department. So use of real-world data. So how is this real-world data used most commonly in medicine and critical care? So one example of that is pragmatic clinical trials. So pragmatic clinical trial aims to slide in the interim between traditional efficacy trials that Jonathan had just mentioned about how long it takes to put together and how long it takes for the results to translate to practice and actually what happens in the real world. These trials aim to inform policy and decisions to test new intervention in the real-world setting. So as opposed to traditional efficacy trials, they tend to have very broad inclusion criteria and very few exclusion criteria so that you have large heterogeneous populations similar to what we see in clinical practice. There is no randomization to a placebo and there's no blinding because that's not happened in real world. So usually your control arm is standard of care. And interventions tend to be simple and can be implemented in clinical setting and we oftentimes use the EHR data to collect the outcomes that also is clinically relevant and simple. In the last five to ten years or so, there's been basically an explosion of pragmatic clinical trials that has dealed with critical care both in the adult and pediatric setting and this is just a few of those that you have seen. All of these have used EHR data as part of their ability to collect the data and outcomes on this. But this is not just EHR data. So this is the hospital airway resuscitation trial or the HART trial that's led by Dr. Ari Muscovitz at Montefiore Medical Center. This is a trial looking at in-hospital cardiac arrest and whether the airway strategy of endotracheal intubation versus laryngeal mask airway can improve the outcome of return of circulation as well as mortality. So yes we're using EHR data on this but in addition to the EHR data, we're able to actually pull data from the zone monitors which is connected to every single patient who has a cardiac arrest and that gives us real-time high granularity data in terms of actually chest compressions, how deep, how frequent. Connecting it to the end tidal CO2 monitor allows us to also understand how well we're doing in terms of perfusion and when we get return of circulation. This gives us much better data than actually an observer can even collect or a research order can collect that will help us understand the mechanisms by which airway strategies can help with resuscitation. But the larger use of real-world evidence is in post-marketing surveillance studies, a form of phase four studies. So whenever a drug or a device is approved and put on the market, the FDA and many other regulatory agencies around the world require that the company then collects data on its use in clinical setting to determine safety, tolerability, and effectiveness, meaning how well does it work in the real life of this newly approved intervention. An example of that is Zygris that was actually originally put on the market after the publication of the POWERS trial. And then the company then collected data both in terms of observational cohorts as well as subsequent randomized control trials. So there's a couple things to note there. So one is when they looked at real-world data, okay, and observational studies, they find that the odds ratio for mortality from Zygris is not as striking as it was in the POWERS study, so a little bit not quite as effective. But more interestingly is that the risk of severe bleeding outside those tight confinements of the first randomized control trial was nearly double. And subsequent to this, the vendor themselves, not the FDA, did pull it from the market. But what you're going to hear about today and what you're going to hear more about as the next hot, sexy topic is target trial emulation. And what target trial emulation is is the use of real-world data and applying trial design and analysis principle that you would normally apply to a randomized control trial to the same way that you would analyze observational real-world data. That means, for example, just like in a randomized control trial, I have strict inclusion in exclusion criteria that must be met within a particular time frame in the course of the presentation of the patient. Similarly, you look at your observational data and you select those patients to match the same kind of an environment, inclusion and exclusion criteria, as well as the timing of when they come into the trial, when to study, when to get the intervention, and what outcomes you're looking at. Now there are different applications to these target trial emulations. One is to inform the design of randomized control trials. This is an example. This is from the STOP COVID investigators looking at tocilizumab and mortality in patients hospitalized with COVID-19. Subsequent to this observational study, RemapCAP published on their randomized control trial on IL-6 receptor antagonists in critically ill patients with COVID-19. Both found a benefit to TOSI, but if you look at actually the odds ratio for mortality, which is a secondary outcome in the Remap trial, you'll see actually it is very similar to what was seen in the STOP COVID investigators using real-world data. This kind of similarity and being able to anticipate but also help plan and also mimic what you might see in a clinical trial has become more important because real-world data is now used to inform regulatory decisions around new drug approvals, label revisions, and supplemental approvals. And this is an example of all the drugs that has been given this approval just on real-world observational data. Now this is usually on drugs that has been on the market and is being used off-label in a different patient population or for a different indication, so that you have real-world data to see patients that some who got it and some who didn't. But this is of interest to the FDA, enough so that the FDA actually funded a project known as the RCT Duplicate to see actually how well we can use real-world data to basically replicate well-known published randomized control trials. And we'll talk a little bit more about that in a second. Now the other way you'll see real-world evidence is in rare conditions where you're just not going to have enough patients to be able to do a randomization for a new drug or a new intervention, in which case you might have a single-arm trial but they'll use historical or synthetic controls to be able to replicate a randomized control trial. But what you'll see more often in critical care is comparative effectiveness studies as well as heterogeneity or treatment effects, where you look at drugs that are being used to compare them but also actually look to see if there are subgroups of patients who may have more or less benefit or harm. An example of that is a study looking at actually warfarin versus actually DOACs in preventing strokes and in bleeding, causing bleeding, in patients with atrial fibrillation. One that was comparative effectiveness of warfarin versus these DOACs and the different DOACs, but they were also able to look at it within a subgroup of patients, the chronic liver disease, that were often excluded from these trials because concerns about bleeding. And they found that it was still beneficial even in this population. Now I know what you guys are going to say. You're going to say, this is opposite of what we've always been taught, that in this hierarchy of pyramid of evidence, these observational studies is lower. They're not as good as randomized control trial, right? Because there's all these biases that are there. And that is why we're always supposed to, and I know when I've published, the editor keeps coming back and say, don't say cause. Association, not causation. That is true. However, there has been advances on several fronts that now make us much better able to account for these biases and therefore strengthen the level of evidence that we're finding. Some of this is from the analytical tools, propensity score analysis, inverse probability weighing, all that has been there before. But there's also been an exponential increase in the use of machine learning and artificial intelligence analysis. Part because it's actually really good at handling a large amount of data. But also actually because certain analysis such as convoluted neural networks or large language models allows us to look at unstructured data that we couldn't measure easily before or capture images that we see on the monitors and on the ventilators, waveforms and sound. But the biggest reason, okay, it's not necessary to tools because, you know, you can use the right tool, but if you use it on the wrong material or use it wrongly, you're still going to end up with garbage. I think actually the biggest advance in this area is really from the exponential growth of data and the quality of the data that we're getting. This allows us not just to have a large number of patients and higher quality data to have the power to do some of the studies, but it also allows us to have a larger pool of patients that we can be more selective about selecting them to better match or emulate like what we would do in a randomized controlled trial, okay, and to be able to have the enough other data that we can adjust for for confounders. In critical care especially, the higher granularity data, especially if you have device integration, really helps with regards to finding a causation because you know exactly when the intervention is and you can see exactly what happens afterwards. So giving you that pre to predict the outcome. But that's not to say that real-world evidence is going to replace randomized controlled trials because not everybody is going to predict the outcome. Sometimes the drugs that are not currently in use cannot be studied this way. And interventions that requires expertise, those are not things that you can capture sometimes in a real-world data so well. And just like it has always been before, garbage in, garbage out, however much we have more data and the quality has improved, missing patients, missing data, unreliable data are still going to affect your outcomes. And as we'll hear more about, there are still challenges in terms of linking data from different sources, avoiding duplication, making sure that you're linking them not only on the right patient, but in the right time frame of that health of that patient as well as privacy. And I think you'll hear more about that and what we have been able to do later on. But just like we have done with everything else, poor study design is not going to be But just like we have done with everything else, poor study design is going to limit the usefulness of real-world evidence. So that means inappropriate selection of patients, inadequate adjustment of confounders. And we know that this is true. When we looked at what happened during the pandemic, you can see actually the methodological quality of COVID trials, observational studies, was actually poorer than it was in the period before. And this is important because remember what I said about the RCT duplicate? When they had looked at how well they can use real-world data to emulate randomized control trial, they found that there were some studies that they could emulate well. The data is there for them to match on the patients and match the study design and match the outcome. And then there are trials that they couldn't match so well. It turns out, actually, the quality and the correlation falls rapidly between those two. Trials that can match really well, 88% of those trials that they emulate in the real-world data would agree. The real-world data agree with the trial. But in those that they couldn't, only 50% of them ended up agreeing. So in conclusion, we will have more data and real-world data will grow and there will be more tools for us to analyze them. And it's going to be very effective because it provides effectiveness evidence in actual practice versus efficacy. But like all clinical research, real-world evidence will be limited by incomplete or poor quality data or poor design. With that, I want to thank you for your attention. So I think we have time for maybe one or two questions. If anybody has some from the audience, please step up to the microphone, introduce yourself, and go ahead. and more heterogeneous inclusion and are challenged with heterogeneity of treatment effects and missing potentially signals for subgroups of patients in a field that's seen a lot of quote unquote negative trials but may be missing important therapeutic benefit for subgroups. I wonder if you can talk about that potential conflict there. Yeah, so that is another talk all in of itself, I have to say. So another thing that you can hear a lot about is this heterogeneity of treatment effects and that there are subgroups of subphenotypes in ARDS or in sepsis or in others that may respond to one treatment versus another. So this is where there are many different ways to skin a cat, if you will, okay. And in my opinion, we haven't yet settled on one way only to answer that question. So there's different ways. One way is let's personalize this. Before we go and do the whole group of real-world data, let's actually phenotype these patients and then randomize these patients based upon their phenotype to the intervention to see if it helps. So you can see if this works in one patient population that theoretically should work, right? So that's the personalized medicine. Another way to deal with this is do it in a larger group but have a priority definitions of like subgroups that you think may have better effect or more harm or whatever and then analyze that. Which way is best? I think it depends on the intervention. It depends on your goal. It depends on your power, okay? My big concern about both of these is disparity because one of my concerns of personalized medicine is that I'm not sure I want to know about a drug that only works on 5% of my patients. That costs a lot of money. You know, I work in the Bronx. I pretty much can guess that none of my patients are going to end up getting that treatment. So either way that you do it, we also need to be able to design it with the goal of making sure that we have an idea about what that intervention will be in terms of applicability for my total population and to be able to avoid disparity. One more quick question. That was a, thank you, that was a fantastic overview. Chris Horvat from University of Pittsburgh. You know, the point that you made about the quality deteriorating during the pandemic compared to non-pandemic times related to studies leveraging real-world data is obviously very important and some of that's probably because things are rushed but at least part of it seems like there's not a great North Star set of guidelines about how to actually be careful about the pitfalls of real-world data in some of these study designs. And I'm wondering if there are any efforts that you know of or could speak to to actually provide some formal guidance on leveraging real-world data for the types of study designs that you mentioned. So that's a great question and maybe I should clarify my statements a little bit. So the, during COVID, the methodological quality issues is not necessarily because of the data quality, okay? Because actually what we also saw in COVID and stop COVID was one of those example is that a greater impetus on pulling EHR data, okay? But what there was a lot of in terms of methodological quality was design and I'll give you an example of that, was lead time bias. That was in a lot of the COVID observational studies where they kind of take patients and compare them based upon whether they got a drug and versus not, you know, avoiding the issue actually that, you know, they had to survive to get that drug to begin with, okay? And that doesn't happen in a randomized controlled trial. So if you were to do a real targeted trial emulation, okay, you wouldn't be designing your target trial emulation using the observational data that way. Instead what you would be designing is, okay, I'm going to look at all patients on ICU admission, let's just say, okay, whether they have the drug or not have drug, let's just take a look at those who didn't have the drug and then that's how they get into the study. You don't select them on the basis that they got the drug or not. You select them on the basis that they have filled these inclusion and exclusion criteria of a randomized controlled trial and then see if they got the drug afterwards and then have, okay? So the data itself, okay, I'm not sure that it was any different, you know, what was pull and stop COVID versus actually what was collected in ReMAP, but the design is very different by some of these other COVID trials, studies, observational studies. Thanks, Michelle, for a wonderful talk and that last question was actually a great segue to our next speaker, Danielle Boyce, who's a data scientist, instructor, patient advocate and researcher from Tufts University, is going to talk about the EDGE tool, a practical approach to data harmonization. Thank you. Can I just click here? Am I doing it wrong? There we go. I have a couple of disclosures. I had a grant from Pfizer. I'm a consultant for the Critical Path Institute and I hear an echo. Can you hear that? And Emory University Morningside Center for Drug Repurposing. So I should give you a little bit of background about myself. I have been a data analyst, well, you're saying data scientist. I've been a data scientist since before they called us data scientists and they just called us nerds about 30 years and I made my living helping researchers, clinicians like yourselves, put together your databases, analyze your data, do biostatistics and so on. And then in my 40s, I had a child with rare disease and decided I wanted to do something with real-world data. And so I went back to school and that's why I'm a very old assistant professor. But I still feel very passionately about supporting these projects where these brilliant clinicians want to realize their vision but they're using new tools. And so I play a very small and humble role in these projects where I help to develop and teach the tools that we use. And one thing I wanted to mention is this edge tool that was mentioned in the title of my talk. That is really kind of a way that we described a bundle of mostly ODYSSEY and OMOP tools. You may have heard of ODYSSEY and OMOP. ODYSSEY is the Observational Health Data Science and Informatics Initiative and OMOP is the name of the original project, Observational Medical Outcomes Partnership. But now it's come to be known as the Common Data Model that is used by the ODYSSEY community. And so, sorry. So the pathway to real-world evidence is a squiggly one as you saw. So I've been a knitter for 45 years. And I want to give you this analogy to help you get your mind around what I'm asking you to do with your real-world data. So in knitting, you knit with two sticks. And I can make anything with knitting. I can make a sweater. I can make a hat. And in my community, in my family, if somebody wants something made, they contact me and say, Danielle, I want to make this sweater. Can you do it for me? So I decided I was going to take up crochet last winter. You make sweaters and hats, but you use a hook. You still use yarn, make the same thing. But it was very different and very difficult. Different stitches, different language, different communities that I had to join in order to understand more about crochet. And it was very difficult and humbling. And this is a lot how I think about the work that we do now with real-world data and real-world evidence with the software that I'm asking you to use. So I come in with all of this jargon and these new tools, and you've been doing your research with SPSS and Stata and flat files. And now I'm asking you to use relational databases and GitHub and our shiny apps, right? And it doesn't take away from the fact that you're a knitter, right? You know how to knit. You know how to make sweaters. And it's very, very frustrating sometimes. And so my goal is to teach you crochet painlessly and then get you back to knitting. So this is a graphic that I borrowed from Patrick Ryan from the Odyssey community. This describes what we're doing here with our Odyssey tools and real-world evidence. So think of these sources at the top as either databases from three different hospital systems. So it could be like Epic, Cerner, and then a different Epic implementation. Or it could be within your hospital system. It could be your Epic data or Cerner or VA data, and then your claims data, and then a REDCap database. But as you notice, these little boxes, these tables and how they relate to one another are all different. And in source one, you might have mail as M. And in source two, you might have mail in a completely differently named table, and it might be O1. And in source three, it might be the word mail written out. And so I made my living for many years going in and understanding these different sources and helping them to talk to one another, writing proprietary code and software to pull them all together just so that you could analyze them together. So what we're doing here in this blue section, we're transforming the data into a common data model. That's the crochet part, and that's where my work lives. So we use tools and code, and sometimes a combination of things, different software, different code, whatever works for your institution and for your dataset, to transform. So it's called ETL, extract, transform, and load your data from those sources into this blue area. And you can see now they're all the same, which allows us to analyze all of them with the same code and the same software. And that's the brilliant thing about OMOP. So now you don't have to hire an analyst to decode all of these three different databases, figure out how they go together, make staging tables, combine them all. You don't have to do that. All you need is somebody who knows how to use Odyssey tools. And within moments, I can pull all of your data together and analyze it, and then share the results with others. And you know you're comparing apples to apples, because everything has been standardized. So this is the OMOP common data model. This is the scariest thing I'm going to show you today, I promise. But it's not nearly as scary as your source data schema, I guarantee you. So the blue side is your clinical data tables. And that's probably what you're, if you've worked in the OMOP common data model at all, you've probably focused on that. So OMOP is a person-centric model. And everything else that's collected about that person falls in one of these tables. Every single site that has an OMOP common data model instance is labeled exactly like this. So I can go into your site and figure out exactly what's going on and analyze your data instantly. And we can all share code. And we know that I'm analyzing the data on, let's say, diabetes, the same as you are. And we're using the same codes and references. And then we also collect data and transform data on economics. So the payer plan period collects insurance information, and there's drug cost, and so on. Not everybody ETLs all their data at once. Sometimes people, institutions might ETL their data incrementally based on certain use cases. Maybe they have funding. So what we found in our COVID work was that some people who already had OMOP instances had never ETLed their ventilator data before. It just never came up. And it's a lift. I'm not going to lie. It's a lot of work to do that. And so sometimes they do their devices later. That's okay. But what I want you to understand here is that every single site that has an OMOP instance looks exactly like this. So I can come in and analyze their data and provide them with code that I wrote that works on my OMOP CDM, and it'll work on yours as well. And I think this is really brilliant for lower resource settings. Because like I said, I'm almost always in short supply. Everybody wants Danielle to analyze their data. There's always this long list, and this solves that problem. So as I mentioned, here's a tool that we use for the standardized concept vocabularies. So anyone can go to the Athena website and look up their favorite codes or favorite diseases. So if you have an ICD code that you use all the time, you can go in there and see how it's represented in the OMOP CDM. This is also where you download the vocabularies that you use in your own ETL process. But it's free. You just go to this website and download it. This is the Perseus ETL mapping support tool that bundles together a number of different Odyssey tools, and they're trying to just make everything simpler and easier to use. But basically, on the left, you have your source data. That's your source data. When we say source, we mean your REDCap database, your EPIC data or clarity data, CERN or so on. And then on the right is the target, which is your OMOP CDM. And so these tools help to support you to kind of guess and work out which of your source data should match to your target data. So the point is, there's a lot of really easy no-code ways of supporting sites who are doing this work. This is the classic USAGI manual mapping tool. I use this a lot with, let's say, like a REDCap database that doesn't use ICD codes or, you know, LOINC or HL7. And I use this. It helps me to work out what might this question actually represent in the OMOP CDM. And again, these are our shiny apps. You can download from GitHub. I know that sounds like I'm making it sound really easy, but it's free, and your technical group should know how to do it is the point, but these are all free tools. This is my favorite Odyssey tool, the Atlas tool. When we were talking about developing cohorts and inclusion and exclusion criteria, this is the tool that allows you to do that. So Atlas, again, is a free tool. Once your data are transformed, you point this at your own Odyssey instance, and you can develop very complex inclusion and exclusion criteria. And at a glance, you can see, for example, if an industry partner might come to you and say, I work a lot in rare epilepsy. I want to know how many people you have with Dravet syndrome who also take this drug. I can do that within seconds. And in the olden days, that would be something I would have in a long waiting list with a lot of people who wanted me to look that up, and I wouldn't always be certain I was actually capturing everything accurately because it hadn't been vetted through the ETL process like it is here for OMOP. So this is just an example of a really complex cohort definition. This is something that I developed just to show you how we develop these concept sets. So as I mentioned, I'm interested in rare epilepsy. So I might develop a concept set called infantile spasms diagnosis, and there might be a few different codes for that. So I'll make a concept set for that, and then a concept set for the different drugs that I want to capture or exclude. So then in my cohort definition, I say, just like you would if you were writing your protocol, I want to include everyone with infantile spasms diagnosis. And it bundles up all of those concepts that I had previously defined and puts them into my cohort definition. Then I can click on that export tab and get a JSON file that I can send to a colleague to get a different institution that has ATLAS and an OMOP CDM incident. And in seconds, they can tell me how many patients they have that match this criteria. And that is the basis of a network study. And no patient data needs to leave the building, which makes IRB really fast and efficient. So I was 16 years at Johns Hopkins, so I borrowed this slide from Paul Nagy, who you may know. This is the data quality dashboard, again, free, downloadable from GitHub. Don't be scared. And you point this at your electronic medical records, and it runs the con at all data quality checks. So there's a variety of checks on plausibility and completeness and conformance. And it will automatically scan your whole CDM and tell you where some of the issues might be. This comes up a lot when we have issues with discordant units that, you know, the plausibility of the units of measurement aren't making sense. And you can see here the kind of report that you get. And then you could send that to me, no patient data involved. You send this to me, and then I can look at it, you know, upload it into my app, and I can give you advice on what you need to do to go back and improve your ETL. Because believe me, there can be a lot of mistakes. You know, it's a human multidisciplinary process, but things slip through the cracks. So you want to do this as just one of your many checks of your data quality. And finally, I just wanted to provide you with some resources. You're always free to contact me. I love talking about them, and I love helping people to learn how to crochet. So here's the Book of Odyssey. The Odyssey website is very helpful. Eden Academy is out of Europe, and they have a number of great classes that you can take and learn really in-depth how to do at your own ETL or use Atlas. And this is the page specifically for the Odyssey software tools. Thank you. So thanks for giving us some practical tools to think about this. We have time for one question, if there are any from the audience. Is there, so I'm hopelessly naive at this stuff, is there like a collection of institutions that have taken their data and transformed it in this way such that you kind of know who to work with and in order to get like big data sets? Yeah that's a wonderful question and actually Smitty is going to be talking about this a little bit too. So yes, so the Odyssey community is the place to start and every year at the annual symposium they publish a list of all of the partners and they have this ever-growing map. Smitty and I have helped maybe 25 sites maybe more just in the US alone transform their data into the OMOP CDM and many many many are working on it and looking for that project that you know would justify the engineering time, the you know the support that they need in order to ETL their data. But the place I would start is the Odyssey community website and the Odyssey community forums and and there's they're really working to try to bring everyone together and find ways to connect each other because that's what the real vision is the network study. Especially in some of these more rare outcomes you really want to be able to use these really sophisticated tools. I don't even fully understand where they use epidemiological methods almost like a meta-analysis to take the output of the analysis throughout all of these sites and look at them as one piece and understand the evidence better. Thank you so much. Thanks so much. So our next speaker is Laura Evans who's medical director of critical care at the University of Washington Medical Center and she's going to talk about leveraging real-world data in adaptive and embedded trials. Thank you John and before I start I just want to give a huge shout out to Danielle because I'm not somebody who identifies as a data scientist and it's kind of magic to me how you manage to sort of demystify some of these processes so thank you for making me seem feel less dumb. I really appreciate that. It's a real pleasure to be here for this session and with all these great partners. So in terms of disclosures in 2022 I did serve on the scientific advisory panel for Endpoint Health which is a commercial health entity looking at precision diagnostics and therapeutics for sepsis and just as sort of a system-wide disclosure I happen to be the incoming vice chair for the SCCM Discovery Network so I'll talk a little bit about that. So Dr. Gong already kind of covered some of this territory but I just want to sort of bring this back up and again as somebody who doesn't sort of primarily identify as a data scientist I think it's always helpful for me as I think through this to kind of review what are the concepts that we're talking about here and so there's gonna sort of this idea of real-world data, real-world evidence and just to reemphasize what Michelle was saying earlier real-world data is just routinely collected data often from the health care delivery process and it can have myriad different sources right electronic health record data, registries, the apps on our wrists, claims data etc and real-world data has the potential to generate real-world evidence and in that setting can help hopefully bridge this gap between clinical research in a sort of a very traditional RCT controlled environment model into clinical practice. It has obviously potential for designing and conducting confirmatory trials and Dr. Gong already talked about some of this target trial emulation work that I think is really fascinating and I think in a really interesting way and this kind of also tags on to what Danielle was just saying allows us to answer questions that would be really difficult to address with sort of traditional trial design methodology. So I think the appeal of real-world data and real-world evidence is really very appealing and very evident to us and we'll talk I think in some of the subsequent sessions about some of the potential limitations of this that were alluded to earlier. So shifting just a minute from real-world data and real-world evidence to this idea of an embedded trial and these are intrinsically pragmatic trials right so when you look at that sort of precise two diagram of a trial and it's pragmatic things a better trial is intrinsically pragmatic. It should be set in routine health care delivery. Most of them will have very broad eligibility criteria and use routinely collected data most oftenly from the electronic health record in order to minimize or the additional burden and infrastructure requirements of conducting the trial and fascinatingly they can be you can do different trial designs with embedded trials right you can randomize by patients in this you can do observational trials clearly with this you can do cluster randomized trials where you randomize and Michelle showed some examples of some of the fluid trials right a cluster randomization of to balance crystalloids versus normal saline for resuscitation so about cluster randomization by unit or hospital or whatever unit you choose. There are lots of advantages some probably that have panned out some that are largely theoretical to doing embedded trials and there are some disadvantages and I know one of the subsequent talks in this session will sort of delve more deeply into some of the cautions around the idea of doing embedded trials and unused and the use of real-world data. So I think one of the big advantages to the idea of doing an embedded trial is access and this is something that probably I don't personally think is talked about enough in terms of the access to research and the access to participate in research and it is you know if you attended the morning session this morning there are the keynote speaker or the thought leader speaker focused a lot on health equity and I would put broader access to participation in research as a another endpoint that we should be seeking health equity for. So you know there is equity in institutions that may be able to participate ideally right an embedded trial with support from tools like the edge tool and others may require less infrastructure and may require fewer resources to be locally available. So for those who practice in community hospital settings where you don't necessarily have a big research machine at your institution the idea of an embedded trial where you don't necessarily need as much local resources on the ground to be able to participate and offer research opportunities to your patients is very appealing. Sort of echoing some of the earlier comments as well embedded trials are hopefully more generalizable right they really do reflect more real-world circumstances rather than the idealized world of a highly controlled traditional RCT model. And I think John mentioned this in his his early comments both the reduced cost as well as hopefully a sort of tighter turnaround time to trial results when you an embedded trial into clinical practice. And something near and dear to my heart is in coming out of COVID and the sort of flood of research that came out of COVID is also this idea that an embedded trials and particularly embedded trial networks are more readily scalable to do high-quality research in an emergency. So that building the infrastructure in this sort of inter-emergency time period that we're now hopefully back in I think is a great idea in order to better prepare us for the next emergency. There's also tons of disadvantages to the embedded model and I think we'll touch on this later is that you know back to this garbage in garbage out concept is that anything that we do that is relies upon the embedded biases that we already have in clinical care delivery when we design a trial around that risks both reinforcing and perpetuating those existing biases in clinical care. There are methodologic concerns that can be sort of addressed with some statistical analyses but it's not the sort of pure traditional RCT model. There's lots of issues around data collection and privacy concerns that tools like edge are trying to help us overcome but I think institutions still have a great deal of caution about their data leaving the proverbial firewalls of their institutions and I think there will be a learning curve around that in terms of getting folks to a place where we can feel confident that we have the best data protection and privacy protections in place but also getting institutions familiar enough with these types of tools that I think are relatively new and although growing in prominence and getting people comfortable enough to be able to be willing to potentially you put their data into these systems and then another comment that John I think mentioned in his introductory remarks is that the funding model for this type of work is not well defined and I think that's particularly here in the United States from that so how do these potential trials get funded I think remains a challenge here in the US particularly. There are lots of examples of recent embedded trials because this is a session sort of about moving beyond COVID I just want to talk about these two kind of platform adaptive embedded trials and so remap cap is one that I think within the critical care community we are very familiar with from it and you know not the embedded elements in this trial we're really kind of just you can distill them down into embedding the concepts of patient identification and eligibility so using embedded elements to identify enroll and follow patients all from the EHR the remap cap platform does embed the data acquisition piece so the research outcomes and endpoints are all aligned with clinically available outcomes and evidence integration so results come out really quickly and so you saw that I think relatively nicely in the pandemic that remap cap was publishing as clinical care was evolving and that this timely publication of high quality research enabled us to change clinical practice on the ground if you look at I spy to which is a different platform adaptive randomized trial it's open label it's a really about breast cancer looking at investigational repurposed drugs for breast cancers and they're looking at does adding repurposed drugs to sort of standard of care new adjuvant therapy it does that increase the likelihood of a pathologic response for breast cancer patients and if you look at the embedded elements in that trial platform it's really again this idea of data acquisition and interventions and then evidence integration as well so there's really nice you can think about what aspects of a trial may be embedded and there probably is some trials out there that sort of do a hybrid approach where some elements are embedded and others are not they're all fall under this umbrella of pragmatic trials those of you who know me outside of this setting will know that one thing that I'm particularly interested in obviously is sepsis care for patients that and I wouldn't take a moment to kind of suggest to you the use case that sepsis is a syndrome that is particularly attractive for us in critical care as the basis for an embedded platform trial from that and I'll tell you my rationale for that I'm not the only one who thinks this it's not I'm a sort of proselytizing as a sole voice for this but as we all know sepsis is common it's very deadly it's extremely expensive and for those of us who have been practicing critical care for the last 20 some years we also know that advances in treatment have been exceptionally difficult to come by particularly using our traditional clinical trials model of conduct a freestanding RCT of a single agent or intervention for patients with sepsis and there's a lot of reasons for that and a lot of theories about why that might be and secondly I would advocate to you that many of the research questions that we think of in terms of how we take care of patients with sepsis are difficult to fund as standalone trials and those to me are questions around systems of care, right? So designing, we've put a lot of work into designing fluid trials for sepsis, right? And then they don't show any difference in outcome and we critique the design of like, oh well that's not how care is practiced, that's not what we actually do from that. So I think the strategies of care and supportive care delivery are something that blend themselves very nicely to embedded trials. And obviously this idea of using repurposed drugs as well, not just sort of in the latest greatest R&D agent. One particular aspect I think that also, and I believe the question from the audience to Dr. Gong got at this as well, is this idea of heterogeneity has limited our traditional trials in sepsis. And so I deliberately chose the framing that sepsis is a syndrome, not a diagnosis, right? It's a constellation of clinical findings. And there's been beautiful work that's coming out obviously about identifying phenotypes, endotypes, sub-phenotypes of patients with sepsis. And there's many lovely position papers and panel discussions that have been held that talk about the failure of RCTs in sepsis to show a difference may be in part attributable to this heterogeneity of treatment effect, right? That we're treating people with a heterogeneous syndrome all the same with a trial drug and not showing a difference. And that if we were better at predictive enrichment or better at prognostic enrichment, that we would be better able to design these trials from that. And I think embedded trials have a real role in helping us define that as sort of an exploratory piece and looking at the heterogeneity of treatment effect amongst different phenotypes or endotypes of sepsis patients. And lastly, I think going back to the advantages of embedded trials as a tool for advancing health equity, using an embedded trial in sepsis will hopefully allow us to enroll populations of patients who have been traditionally under-enrolled in clinical trials and who experience disproportionately bad outcomes from sepsis compared to those who have been better represented in clinical trials from that. So, I'm going to move on from my pick up the sepsis torch in the clinical trial and go back to some of the challenges with embedded trials in critical care. And I'm specifically doing this within the U.S. context because I think others around the world have done this better than we have. And one is, as Danielle was alluding to, there are a multitude of electronic health records out there and everybody has their data definitions separately. We do not yet have a clearly standardized data structure. We don't collect the same variables at different clinical sites. And so, we are left to do sort of post hoc trying to integrate this, which is an amazing effort, but it's not necessarily the same as if we define these things ahead of time a priori. And that's a challenge, I think, in a country where we have numerous, numerous, numerous health systems and every installation of an electronic health record, even though they do talk to each other, are fundamentally different at a local level. We also are in a country where many institutions and sites don't have an established, robust infrastructure to conduct research. This makes meeting regulatory standards a challenge for folks. It does heighten some of the data privacy concerns as well. And when you're looking at, for example, at something like the REMAP-CAP trial, right, they had extensive training, right? They have the—Australia, in particular, has a research coordinator at each facility that has an intensive care unit. And so, they have the opportunity to really standardize some of the training around enrollment of patients. And so, even the thought of being able to do that in the United States and saying that, you know, we can train staff on site at each of these institutions to be able to do a proper consent process, that is a heavy lift in terms of what—I think there's real space there for us to make a lot of progress. And lastly, of course, the idea of funding. Again, that this is not—we are looking for our traditional investigator-initiated research funding model does not lend itself well to what is fundamentally a research infrastructure project that then can help us answer questions that are hypothesis—generated by hypotheses. Wearing Discovery hat, I want to just take a moment and kind of talk about what SCCM's Discovery is working on in this space, and it's largely around collaboration, which I think is a really great role for SCCM and Discovery here. So, there's ongoing work that's going to help advance the real-world data, real-world evidence generation, and embedded clinical care trials. We're going to hear later about the collaboration with the CURE Drug Repurposing Collaboratory. We just heard about the collaboration with the EDGE tool to help sort of democratize access to data. And then there are other efforts around standardizing data definitions, specifically the C2D2 effort to kind of look at data elements and try to get us to talk about the same data in the same way. And going back to, again, some of Dr. Gong's comments from her initial talk, this matters when we're talking about timing of therapeutics, right? Dosing—what does pre-hospital corticosteroids mean? Does it mean the same thing to me as it means to Dr. Gong, or does it mean something different? So, we have to kind of decide on the meaning of different data variables and come to a consensus around that. And you can see a lot more about what Discovery is doing through the SCCM website. Hopefully, now it's up working again. I also do want to call your attention, as sort of a proof of concept of some of this work, there's a really cool abstract being presented tomorrow—or, sorry, on Tuesday, the 23rd—about basically doing a clinical trial emulation or simulation of the recovery dexamethasone trial. So, a fairly—you know, recovery was a fairly simple embedded trial, not a million data elements, but a really cool abstract basically saying that using the EHR data and using the EDGE tool, they were able to replicate these trial results from over 10,000 patients who are hospitalized with COVID, mostly here in the United States. There's a propensity matching analysis of this, and the results align really nicely with the recovery trial. Again, recovery is a fairly simple trial structure, but it is a nice proof of concept that using these tools, you can emulate the results. And again, the point estimates of benefit with reduced risk of death for those patients who were on oxygen or mechanical ventilation receiving corticosteroids compared to not, there was a very similar point estimate to the recovery trial. So, I'm going to just wrap up by saying that I think we have a ways to go to really leverage real-world data and embedded trials to generate real-world evidence that impacts our critically ill patients at the bedside. I do think we're on the journey, that it's a long journey, though. And there's a lot of advantages to this approach. Again, I would say sepsis is a great case example of this. It's a syndrome that we all struggle with, where the winds have been sort of hard to come by. This approach really does address some of the limitations of traditional trials, particularly, hopefully, around heterogeneity of treatment effect and access to research. The reduced cost is obviously a huge appeal, as we think about this, and the timeliness of this. There are also, I think, barriers to this approach for research in the absence of a single health care system works against us. The lack of common data structure and elements, we don't have an incentivization structure for institutions to participate in research. We are really relying on people's intellectual curiosity and altruism to do this, for the most part. And research infrastructure varies very substantially across hospitals and regions within the United States. I think there are unanswered concerns about data protection and privacy that still need additional work around that. And I think there are very legitimate concerns about perpetuation of existing bias in clinical care through the application of these. I think that the EDGE tool, the CDRC, SCCM, the C2D2 effort may help overcome some of these barriers. And I hope to see many of you on this journey together as we hopefully improve care for our patients. So I'll stop and say thanks. So we've got time for one question, and it looks like we have Andre Holder to ask the question. Hi, Laura. So that was an excellent talk, and definitely an area of interest for me. I wonder if you could speak to some of the challenges of conducting these kinds of trials, particularly places that may not be well-resourced, either financially or otherwise, to conduct them. As you mentioned, these kinds of trials, some of the advantages are the fact that it reduces the cost of doing clinical trials with high-quality, high-quality data. Doing clinical trials with high-quality research. But there is still a cost associated with that. And because the NIH is more reluctant to fund these kind of, well, clinical trials in general, what are some of the resources that folks that may be at institutions that may not have that kind of support, that financial support, what are some of the resources that they can take advantage of? You know, foundational grants. I know there's some like Kaiser Permanente and the AIM-HI initiative that are funding this kind of work. But are there any other resources that you can mention that allow people to do these kinds of trials at their institutions? Thank you. Yeah, thanks so much for the question, Andre. And I think it's really a work in progress, from my perspective, at least. And I'd be really open if others are aware of other existing resources to do this. I will say doing this type of work even in high-resource university settings is not easy. And in fact, Karen and I were kind of having a whisper in between sessions here that our own institutions are, you know, there's barriers to doing this in a major research university as well. So I think in terms of, I think we really do need to continue to develop these tools, to decrease the sort of barrier to entry, right? Because if you say, if I'm at a community hospital where I'm super interested in this, right, but I don't have a research coordinator, I don't have access to this, where it's going to be sort of dependent on my initiative and I'm going to be doing this in addition to my full clinical workload, how do I make that barrier the lowest that it can be, right? So how do I facilitate that we're really using data that can all be automated, the extraction can all be automated so that I don't have to go back and enter, hand abstract data into a separate study CRF, right? So I think really sort of supporting these efforts is a big part of that. I think there are other efforts around, you know, ensuring that, you know, the NIH is doing a lot of this, right? Ensuring that you, you know, central IRBs, right? And to sort of decrease the barrier to entry from that. One of the, I think, challenging parts about this is from a funder perspective, the idea of lower cost research is really lovely, right? Because you're like, I get, I can fund this and give you either less money or, you know, get a lot more for the same amount of money than a traditional model. It also means there's less money to go to sites then, right? So if when you think about our sort of traditional model of, you know, if you enroll a patient in a trial, you get a per patient fee for that. Those amounts may be very small in these types of trials, right? Because the cost has diminished from that. But that means the cost, the funds that go to a participating institution may also be very low from it. So I think it's really about trying to diminish that barrier from that. In terms of specific resources, I don't have a lot to offer from that. But if others do, I'd be very happy for them to speak up. Thanks, John. Thanks so much, Laura. So our fourth speaker is Karen Lutrich, who is Associate Professor and Vice Chair of Research in the Department of Family and Community Medicine, the College of Medicine, Tucson at the University of Arizona. And she's going to talk about bias and stigma in clinical research, how real world evidence helps us care for real patients. Thank you very much. And like everyone else, I'm going to figure out how to Hi, everyone. Thanks for coming. Is there a keyboard? There's no keyboard. Oh, here it is. OK, great. It just wasn't loading. OK, perfect. So thanks for coming. I am in family and community medicine. So I have the joy of getting to do ICU trials as well as community trials. So I'm going to kind of talk a little bit about bias and stigma and be maybe a slight wet blanket, but with the caveat that I think this is really important work. It's really essential work. We just have to be kind of careful and thoughtful about it. The mouse may or may not work. So with some disclosures, I have research support with NIH and CDC. I also am a consultant for the Eureka Institute of Translational Medicine. Oh, gonna be a lot of, okay, there'll be a lot of clicking. So, some objectives. I'm gonna try to be a little bit practical. I want, the goal is to sort of think about what bias might be in these real-world data because there's bias in clinical research and research in the world. And, I'm not gonna buy a lottery ticket today. I feel like maybe today is not my day. Um, and then also really just try to think through how to understand and maximize the potential of real-world data in diverse communities. I'm in Tucson, Arizona. I'm also an equity researcher, so these are things I think about a lot. Next slide. I guess I'll just have to tell, can I just go like this, maybe you can click? Okay, sorry. So, bias and stigma in clinical research. Can you go ahead and click? Clinical care and clinical research exist in the real world. There's bias and stigma, oppression in the real world. We don't get to ignore that. It doesn't stop at the doors of our institutions, right? It's sort of pervasive throughout, and I think we just have to realize that, acknowledge that, and just sort of accept that as the world that we're living in when we think about research. Next. So, big picture bias. I'm not really talking about statistical bias here. I'm talking about sort of like the larger bias, right? This idea of lack of objectivity. I have bias. I'm a social scientist at heart that lives in a health world, so it's hard for me, I see things as a social scientist. I'm an equity researcher. It's really hard for me to not see things with an equity lens. That's my bias. It's not bad, it just is what it is, and it's useful to acknowledge that it exists. Next, stigma. Stigma, people are devalued because of an attribute, right? And so, when we think about stigma, we often think about people feeling shame or discredited. This is something that we see in clinical world. In the clinical world, we see it in clinical care. We also see it in clinical research. Next slide. So, what it can look like in clinical research. Lack of diversity in screening. A lot of times we think about lack of diversity in our actual enrolled participants, but then if we go back and we go upstream and we look at our screening pool, that pool often looks very similar, right? It can be more diverse in the people that we enroll, which tells us that our inclusion criteria are consent processes, right? We're not developing trust, we're leaving people out, but sometimes we're also just not going to where people are and where we want them. Next. There's biased inclusion criteria. Sometimes we have this inclusion criteria that sort of like an administrative or the PI says they may not be a good fit, and that can be a place where we see a lot of bias and stigma, and we're losing people kind of along the way there. We also make some very practical and appropriate and financially supported language and legal authorized representative decisions that exclude people from our trials. Next. And then sometimes we have selective retention. Time and capacity are finite. We can't follow up on everybody. Sometimes we unintentionally follow up on or try to retain people that look like us, that look like our patients, and maybe we're just kind of creating a more standardized and less diverse sort of patient population which increases our bias. Next. And then I see this a lot. We have some maybe like scientifically unnecessary follow up visits which overly burden participants. We don't need them to come in. We could do a phone call, but we want them to come in, and people with childcare and transportation issues aren't gonna be in that study anymore because they can't feasibly see themselves doing that. Next. And then just in general, right, a lack of diversity in our enrolled participants. We all sort of say and we're expected to say that we're going to have an ethnically and gender and age distribution that matches our population. We so rarely don't because that's really complicated. It's very expensive. Next slide. And it limits generalizability. Go ahead, click. So we have biased research, and we've talked about biased healthcare data. Go ahead. Or biased healthcare. Those lead to biased data. Next. Which leads to biased real world data. So we can't really like stats our way out of this bias. Right, if we're just kind of living in this bias world, we have to just acknowledge that that's gonna continue to exist, and it doesn't mean that it's not the right thing to do. It just means we have to understand the limitations and be cautious about what we're doing. Next. So I wanna give an example. This is from our SARI Prep, which is a discovery project funded by the CDC Foundation. So it's a multi-site observational trial that utilizes EHR collected demographic data. EHR data is great. The demographic data is suspect, right? So we have age, sex at birth, race, ethnicity, insurance status, and zip code. So can we answer this question? Go ahead. Are there racial ethnic differences in COVID-19 ICU length of stay? Okay. Next. What about what's driving racial ethnic differences in COVID-19 ICU length of stay? I work with a lot of residents, and this is their question. And they wanna use EHR data to answer this question. Can we? Go ahead. Next. Hopefully you know, but sometimes we don't think through the details here. So, go ahead. So what's missing when we think about this EHR data that we're collecting? We're missing folks that have a non-traditional or non-standardized by the EMR sex at birth gender. We're missing gender. We're missing transgender. We're missing individuals on home run therapy and if our research question is related to hormones, we're not gonna be able to answer that question. We're missing things like intersectionality. I'm a social scientist. If Danielle can use GitHub, I can use intersectionality. So we're missing the complexity of like multiple social structures and influences that like compound, right? So it's not just race, ethnicity, and or gender, it's intersectionality, right? It's this multiplicity and sort of complexity of these things working together. We're missing social determinants of health. We're missing structural, financial, and access to preventative primary care, continuity care, right? We're missing all of those kinds of things. We're missing social vulnerability. There's these really brilliant and great and so useful social vulnerability indexes that we cannot use with zip code. And we're also missing all these environmental impacts, which we know impacts health. We don't have that if we're just looking at this. Next slide. So we really have to think about who's also missing from the data. So we understand maybe the difficulties with using the data, the biases within the data, but we need to know who's missing. Go ahead. So we know that there are structural, political, and financial barriers to accessing care. People can't come to see us. And they can't come to see us when they really, really want to, or sometimes until they really need to. And sometimes even when they do really, really need to, right? Maybe there isn't an ICU in their area, or they don't have access. They're undocumented, right? All of these kinds of barriers. We also know that people that are not represented by standard demographic questionnaires are missing from our data. And sometimes our questionnaires are so overly specific because they're trying to capture everybody that we have to remove groups because they're too small and we can't power on them. So we have kind of this, like we're not asking or we're asking too many, and there really isn't a perfect answer, but we kind of are missing them on both sides. Go ahead. So I wanna just share some preliminary findings from Sorry Prep. There's also a talk on Tuesday afternoon that Dr. Savransky is gonna do a little bit about this, but go ahead. So using this data, are there racial ethnic differences in COVID-19 ICU length of stay? And can we answer that question? Go ahead. So yes, go ahead. We can. We know that from our data, we identified that our Hispanic participants, and we're really lucky. We have a fairly demographically representative cohort that our Hispanic patients had a longer length of stay than non-Hispanic white, non-Hispanic black, non-Hispanic Asian. And then unfortunately, other, which is just a bunch of other people, a large amount of people in the cohort, and also very different from a clinical characteristics perspective, but that's sort of where we had to end up, right? Next slide. Now, if we look at the second question, go ahead. What's driving racial ethnic differences? Can we answer that question? Go ahead. It depends. My favorite answer. I'm a social scientist. It's my absolute favorite answer because it really does depend. Go ahead. If we're sort of thinking, again, social scientists, what's our conceptual framework? What's our causal pathway? If it's driven by treatment, like, go ahead, probably. We can probably answer that question. Maybe not necessarily, but we can probably answer it as long as we were consistent and thoughtful in how we were collecting that treatment data and we were keeping up with the COVID treatments. Go ahead. What if it's driven by clinical risk factors? Go ahead. Maybe. And I'll show you an example of this. Depends on what we think is driving this, but we might be able to do this. Go ahead. My favorite question. Is it driven by social determinants of health? Go ahead. Unlikely. Can't really say much about this. I know my hospital. I know who we recruited. We had conversations with them. I know some of those patients, but I still can't maybe use the data. Very, very likely, and I'd be very skeptical of somebody using exclusively EHR data unless they're really sophisticated. My healthcare system is not. Go ahead. So this is preliminary findings. So this is kind of our looking at are there risk factors that might be driving? Go ahead. You'll see some orange boxes here. We can see some differences here in comorbid conditions in this cohort that could potentially be driving those racial ethnic differences, right? So I might be able to make a strong case here if I look really closely, but again, maybe. Go ahead. So what can we do? This is where I'm gonna get really practical, and this may be very obvious, but I work with a lot of clinicians and a lot of community-based trialists where we have this conversation repeatedly. So I'm just gonna use this opportunity to share this. Go ahead. You can just kind of click through because these are meant for me to click, and it'll be really annoying. So we need to reduce bias and stigma in healthcare and clinical research. Done. We can oversample and target groups traditionally underrepresented in research. This is hard. This is where the funder and the time and the contract delays and the IRB delays, and we're like, we just gotta get 200 people in the door, and we're just gonna take the first 200 people, and those first 200 people usually look like us. They're usually, you know, I'm gonna be real honest. My dad, he loves to sign up for clinical trials. He's retired. He's got free time. He's got friends. He does it, right? Do we need more white men in their 70s? Probably not, but he's gonna be there right there, right? But if I wanna get somebody that looks like my patient population, looks like my community, that takes time. It takes energy. We can't rush that, right? And so there's this kind of constant tension. Go ahead. We have to create inclusive recruitment materials. There's really good guidance on this. It's not that hard, but we do a really bad job of it, right? So we want somebody in their, very classic example, we want somebody in their 70s. So we put a photo of somebody, somebody in their 70s on our flyer. That's gonna recruit somebody in their 90s. We want somebody in their 70s. We put somebody in their 50s playing tennis, right? It's how do we think about ourselves versus how other people see us. There's really good evidence on this, but this is a really, really important piece. Go ahead. We engage communities to increase trust in research and healthcare. The first time we go into a community isn't because we have a trial and we have one month to get people in, right? We really need to be a part of that. The easiest way to do this is to collaborate with people in public health, social work, family and community sciences, all of these other places, they're there already. Partner with them, have them invite you in as a trusted speaker and build that, work on that, expanding that relationship instead of going in for this one trial and then bailing because you're then making it harder for the next person that goes in, right? Next piece. And then taking a really thoughtful and critical approach to the data that we're using, making sure that we're not doing the work to go into the community, doing the work to pull these folks in so that they show up in our science and then when we send them the newsletter with the results, we've edited them out of the analysis because there weren't enough of them, right? So they don't exist anymore. We've edited them out. Next. So this is, just click all the way through this really quick. These are just some questions to think through. These are very common questions. I think they're common sense, but I like to make sure that we're thinking about this. What does the data include? What's missing? Who's missing? What was its original purpose? Am I trying to use some data that was collected for something completely different? Who owns it? Can it be repurposed and can it really answer the question that I'm asking, right? So again, just trying to think about how can we reduce that bias and stigma by taking a little bit more of a holistic kind of approach. And that's it. Thank you. Thank you, Karen. Do we have time for one question? So maybe I can ask you a question. Knowing nothing about social science. We're human. We live in society. So the first couple of speakers spent a lot of time telling us how more efficiently and cheaper and perhaps more representatively we might be able to enroll people into clinical trials. It sounds as though if we just do that, but don't think about some of the issues that you did, that we won't be able to have real data representing all of the patients we see and that just cheaper and more efficient isn't going to do it. If we want to do a well thought out clinical trial. Yeah, I think it depends on your healthcare system. My healthcare system really discourages uninsured patients. And we're separate from them. I'm an academic. They're separate. We sort of have a lot of agreements that bridge us. But they don't ask us, they don't ask me who I think they should treat and who I think they should sort of transfer out. So if we were to do this in my health care system, I kind of know what those results would be. And I know who is missing because we send them to other very under-resourced hospitals that don't have the infrastructure to do this research. So because there is that divide, that divide just continues. Again, we can't like stats our way out of basically systemic racism, which is like in my community in Arizona, we love segregation. The whole cities were built around segregation. We all just kind of grew together. We just have to deal with that. We can't reroute the buses. It's impossible to get to our academic health care center by bus from some parts of town. And that's intentional. And I can't build my research to undo that. So it's about finding partners. It's about lifting up groups. It's about sort of just knowing that and making sure that that's up front and center and the limitations. And then really pushing the people that are building those backbones to change the way that they're thinking because of these unintended consequences of not understanding how these treatments impact everybody in our community, not just the people we want in our health care system because they have really good insurance. It's kind of like a downer. I'm sorry. I'm really an optimist at my heart. I just have to live in this world. Sorry. Thanks, Karen. And I should say it's a marker of a good speaker who can handle a technological challenge when their slides are being presented. So our final speaker today is Smitty Hefner, who is the senior scientific director of the Critical Path Institute Cure Drug Repurposing Collaboratory. And he is, surprisingly, going to talk about the Cure Drug Repurposing Collaboratory. Thank you, John. We were placing bets on how far I get into my speech before the whole computer just catches fire at this point. So let's see if it'll work for me. And it's working for now. So thank you, John. Thank you to my colleagues. As John mentioned, I'm going to talk a little bit about what we're doing and where we've been collaborating and where we'll go in the future. I want to note that I am also on the social science side of things. I'm actually an implementation scientist by training. My background is in ER and critical care nursing. But I left the bedside before COVID. And then I like to say, mediocre white manned my way into my current position because I joined the SCCM virus registry, opened my mouth in a meeting, and ended up leading a team. And it snowballed from there. And through that work, I've learned enough data science to embarrass myself at cocktail parties. Thankfully, I have colleagues like Danielle to teach me and the rest of the wonderful colleagues in this panel to teach me more about clinical research. By way of my first disclosure, I do want to mention that I do co-chair one of the sub panels out of the data science campaign and serve on another working group. I also want to briefly mention what the Critical Path Institute is. We are a public-private partnership with the US Food and Drug Administration. CDRC is a subsidiary of that, a sub-portion of that. Our funding does come from the FDA and the Department of Health and Human Services. That does not, of course, in any way mean that anything in this speech should be seen as an endorsement by any part of the government. So why work for the FDA? Short answer is no. But some of my colleagues have already mentioned drug repurposing. As the name suggests, CDRC exists to look at how drugs are used off-label, how they can be used in new ways, and how we can advance the evidence around how those treatments need to be delivered to patients. As previously mentioned, repurposing often happens in diseases that are rapidly emerging, such as COVID-19. Those that are extremely rare, those that impact vulnerable populations, or those that are already treated by standardized guidelines. Something has already emerged. So for example, a rare subtype of angiosarcoma, which is another piece that we work on in our team, or something affecting pregnant women where you have a hard time developing and actually standing up a trial, or on neonates, or something like congenital cytomegalovirus, where there is a standard of care that exists that is not on any label, and so you can't actually withhold that treatment to conduct an RCT. All of those can be addressed through a few different strategies. If you go to some of the conferences that my boss and I, Mark Oskito, go to, we have a lot of really wonderful debates about what do we mean by repurposing. And it's delightful and horrifically boring for everyone who isn't like four of us sitting around that table. But broadly, we can think of conceptually a drug that has already been approved and labeled for something else, and we find a new use case for it. The most recent, widely publicized, would be like Wigovi, goes from diabetes to weight loss. Now we're looking at potentially using it to treat addiction. We might reformulate. We add a new component to the molecule itself that might extend the availability, might improve the availability, or make it last longer in the body. We can combine drugs. We see this a lot in psychiatric treatments and in cancers to either reduce side effects or to just make it more palatable. We don't have as many pills to make it easier for the patient to comply. There are also shelved assets. We might rescue some of those because it was developed. We decided it wasn't a good use, or it failed trials in one place. Then we found a new use for that molecule later on and resumed study of that drug. Through all of that, there are a number of issues because the system that we have in the United States is designed to incentivize the development of novel molecules. And so what we end up with is a field where there are thousands of medications that have hundreds of potential off-label uses. And we see them in the clinical space, especially in critical care. But because the system is not set up to incentivize the pharmaceutical company to pay to submit that to the regulators, it becomes a lot harder to actually get that data collected and the evidence submitted to the FDA for review. So we were formed in 2020 with the FDA to become a group that can look at how we can leverage real-world data to advance drug repurposing research in alignment with the guidelines and evidence that have been previously mentioned by my colleagues. And we focused on a pilot study in COVID-19 looking at, can we, through the systematic collection of real-world data from the electronic health record, begin to formulate hypotheses that can be later addressed in clinical trials? And can it be a resource for physicians where no approved products exist for the new use? Sounds great, or it sounded great in 2020. We know COVID is no longer quite as clinically exciting. We know a lot more about the disease than we did now. But there's a lot we can learn methodologically. And so CDRC, across its many functions, many groups that we're looking at, many different conditions, we have an area of focus. And I think I know if I don't have a circle on it. Well, I'm just going way too far. But you can see here in emerging and re-emerging infectious diseases, we have COVID-19, and especially building infrastructure to prepare as we enter this inter-pandemic period to be ready for disease X when it emerges, major concern for regulators and other government organizations, for example, the CDC. All of this is intended to support a platform that the FDA has hosted for about 10 years now called CureID, which is a web-based and mobile application where physicians can submit, and now patients and other individuals can submit cases where they've used a drug off-label. We all know how much everyone loves manually entering cases. And so the idea was to have a project that would start to build pipelines where data could flow in automatically, make larger pools of data available for exploration and for research, again, in line with the FDA's real-world evidence guidelines. There's a great, for those of you that haven't taken the time to read the riveting 75-page real-world evidence program report, keep a copy on my nightstand, just in case I need a little bit of help. But there's also a really wonderful primer. John Kakedo and Jacqueline Corrigan-Curry published in New England Journal, citation down here at the bottom, where they briefly, in three pages, describe the use cases that my colleagues mentioned earlier to explore clinical practice, develop and refine hypotheses, and provide external controls. And that observational research then becomes part of the infrastructure and the environment of developing and understanding how drugs can be used off-purpose. So now that I've got the elevator pitch out of the way, what have we been doing with SCCM? Why have we brought this group together? It's been my absolute pleasure to work with this group of individuals over the last couple of years and learn from them, as I've mentioned. And out of these ideas, we partnered and applied for and received an $8.3 million grant from the Department of Health and Human Services, from the Office of the Assistant Secretary of Planning and Evaluation. The funding came directly from the Patient-Centered Outcomes Research Trust Fund. And collaborators included, FDA is the primary applicant, subaward to CDRC. We included collaborators from Emory University, from Johns Hopkins, the Mayo Clinic, Oxford University, and of course, SCCM Discovery. This project was intended to build that pipeline to help build the infrastructure to get data out of the EHR. And we've made some pretty good progress. Together with our team, we leveraged the connections and the network through the virus registry, recruited institutions to implement the tools that Danielle described and pieces of those tools and helped them get to OMOP. The thing that we've found and the vision that we have is once we can implement the CDM, it's much easier to utilize it going forward. And the wonderful thing about being in critical care research is the amazing amount of overlap between the data sets we need to study different conditions, so the idea is we can expand from there. But you see here our structure with our receiving guidance from the FDA and NCATS at the NIH. We've been able to collaborate with these institutions to implement at more than 15 sites, have successfully implemented significant portions of the tool. We have 25 total sites that have engaged with us. More than 100,000 patient records have been submitted to SCCM's Data Coordinating Center. And just to clarify, that's not 100,000 cross-sectional, 40 variable data sets. That's the ultimate deliverable we get out for the FDA. On these 100,000 patients, we have every blood pressure that is captured. We have every medication that is administered. We have every FIO2, every PAO2, the devices that they are interacting with to receive oxygen support. Danielle mentioned the oxygen devices are frequently not mapped. Well, the data standards didn't exist prior to this project. So that's one of the things that we've contributed to the field. And so it gives us the ability to move forward and study things like ARDS, because we now have a pipeline that includes all of those metrics around the ventilator, all of the settings, all of the various pressures. And we get a much more granular view than any traditional registry would let us have, because we essentially have the raw EHR data on 100,000 patients. We have one publication out, but there are five coming up, some of them. Dr. Evans mentioned, and our colleagues here sitting over here on the right side of the room, just behind the projector, will be presenting two abstracts later this week on some of our ongoing studies, which include trial emulation, a major area of consideration, as well as multiple conference presentations. I think I managed to get the logo of every conference we've presented at on this slide, but I'm not sure. And that begs the question of future directions. As we collaborate with SCCM, as we deepen our partnership, there's always the question of funding. And so with a pipeline like we have built, the goal now is to enrich and maintain. This is my implementation. Scientists starts to come back to the surface. And the idea is looking at all the different ways we can leverage that. So there's obviously drug development, guideline development. There's sepsis research. Not only is sepsis a major public health concern, it's also a major concern to health care systems, because sepsis has so many quality metrics around it that it significantly impacts their reimbursement. So a health care system has a significant incentive to invest in infrastructure to let them understand their sepsis patients better. That same pipeline, because of all of the information on viral diseases, can be leveraged for syndromic monitoring, could be of interest to the CDC and other groups. And the pipeline can also let us get a really deep, granular look at various laboratory measurements and their interactions with medications, potentially giving us avenues to leverage that as hypothesis generation for industry stakeholders. And so pulling together a very broad collection of use cases, we're able to start building a strategy for how to keep funding available for these sites, not only to maintain with the sites that have already implemented these tools and are submitting data, but to expand to additional sites. And then downstream, we're able to leverage that to address the equity issues by having a way to intentionally oversample on some of those under-resourced or underserved populations. And there was a question earlier about where do I get started or something like that. Dr. Boyce mentioned the Odyssey community. The other place that people really should get involved is the data science campaign. In our work together, as was mentioned earlier, building data standards for SCCM, the Society for Critical Care Medicine is leading the way for how do we actually build a larger ecosystem, a larger pipeline for data across different institutions to study a wide range of critical care illnesses and to look beyond the spectrum of care. There's lots of interesting opportunities with ER data to look at those outcomes beyond the post-ICU syndrome, the recovery from various complications. And as we build and enrich the data hub at SCCM, there'll be more and more opportunities to leverage that, not only for grant applications, but also just to advance the evidence in general. And with that, I think I'll wrap up and kick it back over to questions.

real-world evidence

clinical research

real-world data

electronic health records

pragmatic trials

data privacy

OMOP common data model

adaptive trials

COVID-19 response

CURE Drug Repurposing Collaboratory

healthcare biases

data standardization

equitable access