Thanks for having me. And I'm also going to start with a concession. I will concede to Piyush that AI is going to win in the end. And why is that? Because hundreds of billions of dollars are being poured into AI to infiltrate every sector of society. So why are we here talking about AI? AI is not the first technological innovation that we have encountered. But AI is very different from all the other innovations even in health care. And why is that? It has this capability to really upend every human task that involves thinking. It promises to augment clinical decision-making. And this means that it will find its way in different corners of the clinical workflow. So what that's what does that mean? It means that this will be very hard to oversee. It will be very hard to regulate. As clinicians we don't understand how MRIs work unless you're a radiologist. But we order it almost every day in our practice. We don't understand how GPS works. And yet it's now part of our lives. But these are very narrow focused when it comes to applications. And for that reason you could have the FDA oversee MRIs. So that's the biggest difference between previous technological innovations and artificial intelligence. But that is not the biggest difference. What is most important to understand about AI is how it is developed. Underneath the facade of AI is a reflection of all the systems that we have in place. So AI is nothing but a product of the systems that deliver care. Systems that educate. Including both the good and the bad parts. And unfortunately what we are discovering is that the bad parts of the different systems are actually being augmented as you build artificial intelligence. There is a fundamental problem with creating a product that we evaluate looking at its accuracy based on real-world data. Because that means that the system that we are producing that incorporates AI would be just a mirror of all the system structural inequities that we see. And to highlight the bias that Piyush already alluded to earlier, our group decided to perform a project where we were using GPT-4 to help clinicians come up with differential diagnosis and work up suggestions and treatment recommendations. And what we did was we use exactly the same text prompts. We got case vignettes from the New England Journal of Medicine. We would ask GPT what are the differential diagnosis? What do you recommend for treatment? But each time we were changing the sex and the race ethnicity of the patient. This is a busy slide. We're not going to go through all this. I'm just going to give you the most salient and most almost frustrating findings that we have. When we gave GPT-4 a case of someone with pulmonary embolus, again we use exactly the same symptoms, duration, lab tests. When the patient is a woman, AI would say panic attack or anxiety disorder as one of the top differential diagnosis. You don't see that when we say that the patient is a man. When we gave GPT-4 a case of a college student presenting with sore throat, if the student happens to be a black male, it's sexually transmitted infection would bubble up in the differential diagnosis. More alarmingly, when we were asking about workup in the emergency department, given a certain description of symptoms and lab tests across the board, if the patient were black, GPT-4 was less likely to recommend a CT scan or consultation with a specialist. So as you could see, AI quickly learned some of the implicit biases that we have as reflected by electronic health records. This is another study that came from a group at Stanford and what they did was, does AI retain all the race-based medicine that we have been trying to remove from medical education? And the answer is yes. They use eight case scenarios and they were asking them about skin thickness. As you know, in the past we were giving more radiation when we are performing x-rays for black people, thinking that blacks have thicker skin. We were asked, they were asking about how do you calculate the lung volume of a black man? So again, these are debunked understanding of differences between black and white patients that we have tried to remove from our books. And it's all there. AI GPT-4 thinks that indeed black people have thicker skin, that their lung volumes are smaller. As you remember, this was used back in the past to justify slavery, so that blacks should be working in the farm so that they increase their lung volume. So all of that were encrypted, incorporated into large language models. And we fear that the decisions that are going to be augmented will reflect this race-based or sex-based medicine. Another group from the Brigham and Women's Hospital looked at whether AI could answer patient questions. And what they found is that one-third of the responses of GPT-4 to patients asking about cancer were not supported by the NCCN guidelines. And about one-third of that involved hallucinations. So AI just making up stuff, just like we do as doctors. And then when they asked GPT-4 to come up with responses to questions from patients to in a form of a letter, what they found is that in 58% of the time, AI generated a text that did not need any editing. Like, amazing. But there were 7.7% of the answers were potentially harmful or could even kill the patient. So the question is, are we going to expect the busy clinicians to look for those 10% of AI responses that could harm the patient? And mind you, these are the same clinicians who have implicit biases. So will they be able to pinpoint that, hey, this response is not good? There's such a thing as automation bias. Once AI becomes part of our lives, it's going to be hard for us to not trust it. It's going to be hard for us to scrutinize every answer that is formulated by an AI. And this is what we are concerned about. We've only talked about large language models. What's even more concerning would be AI that is being built for numeric data. So the data that we are more familiar with from the vital signs of the lab test. And here is a list that is not comprehensive. It's just a sample of some of the data issues that we have explored leading to bias in algorithms for prediction, for classification, for optimization. I highlighted bias from medical design because that's the one that I'm going to talk a little bit. There are others in the list that I wouldn't have time to discuss. But everyone is familiar with the pulse oximeter not working well for people of color. In this review article that we published last summer, we counted about 10 devices that are used in the intensive care unit that we know perform differently depending on who you are. That means that the same number would mean different depending on your body mass index, depending on whether you are black or white, depending on the type of your hair. So of course I'm alluding to EEG where you get more artifacts if you have a certain kind of hair. So the strategy where we just dump gobs and gobs of data, even if they represent blacks or Native Americans, and then build a model, see what sticks in terms of accuracy, it's perfect recipe for all the structural inequities to be permanently cemented in the way we deliver care. Here's another problem. We are discovering more problems faster than we are able to address them. And this is from two years ago now where we discovered that AI could easily infer the race ethnicity of a patient based on a medical image alone without even having any clinical data. And the most perplexing thing is we couldn't figure out how the AI is learning whether this chest x-ray or MRI is from a black patient or a white patient. It wasn't the disease distribution, it wasn't the mass the body mass index. At one point we were inputting random crops of the image and it still could tell whether the image belongs to a black person or a white person. We think it's something about the technology and phantom images of how we calibrate the quality of the x-rays of the MRIs and that most of the phantom images that are being used are coming from white individuals. And of course it's not limited to radiology. There are studies that have shown that AI could also infer the sex based on the fundus photo. So it could tell if the fundus photo is coming from a man or a woman. And of course ophthalmologists were shocked because they themselves couldn't tell if a fundus photo belongs to a man or a woman. And most alarmingly, and we replicated this study, I thought this study was a little, it was cheating. But what they did was can you can AI tell if you're rich or poor? So they train an algorithm to go through x-rays but I thought that it was kind of cheating because more likely you're gonna find abnormalities in minority patients. So what our group did was can we train an AI looking at normal chest x-rays without any findings to tell you if that chest x-ray belongs to a patient who is on Medicare Medicaid versus private insurance? And not surprisingly it could. So AI can tell if you're black or white. It could tell whether you're a man or a woman. It could tell whether you're rich or poor. Why do we worry about that? Well we worry about that because chances are it's also going to learn to make decisions based on those features even though we should not be looking at them when we make a decision of who gets chemotherapy, who gets to be admitted to the ICU. Especially in this age of multimodal modeling we don't know anymore what the machines are learning and what features are being used to make a prediction, a classification, or optimization. So consider this. Some of the phones that you have now would have an app that can take a picture of a suspicious mole and tell you the likelihood whether it's malignant or not. But consider this. During development of the app what the AI learned is that the images taken by an iPhone is less likely to be cancerous than an image taken by a cheap phone. And it starts using that feature because it could tell right away if it came from an expensive phone and would use that as a variable in determining the likelihood of a melanoma. You could see how this could be disastrous. It would scale medical errors and mistreatments. So how do we move forward? Two things need to happen. The first is we need to educate society at large. And why is that? AI is moving so fast it's very complicated that it will be wrong to depend on the FDA. It will be wrong to depend on a special group of people to oversee its development, deployment, and monitoring. Which means that everyone here in this room, everyone here in this world, would need to step up and learn a little bit about AI. How is it developed? What are the risks associated with it? But how do you educate society or intensivists or high school students in a field that is rapidly moving? Well, there is a key component of that system that will need to be in place. And that is bringing together people with different perspectives and different expertise. So this is the reason why we organize Datathons. It's transforming the way we learn from each other, we teach each other. We need classrooms that would bring together people with different backgrounds. We need conferences like this, but not just bringing in clinicians but also bringing in computer scientists and engineers, as well as social scientists. This is how we're going to advance. This is how we're going to scale the way we teach artificial intelligence. It's through Datathons or some version of it. We should have more of that in conferences. We should have more of this for continuing medical education. We are now involved in teaching high school students in inner-city Providence in getting them to understand data sciences applied to health care. We've heard this saying before, every system is perfectly designed to achieve exactly the results that it gets. The way we are designing AI now, where it is more profit-centered rather than patient-centered, it's very well poised to preserve the inequities that we're seeing now. But what does system mean? System is not just the organizational chart or a compendium of policies. System is the unwritten set of rules of how we communicate, how we work together, how we learn from each other, how we teach each other, how we solve problems. So those are at the heart of what we need to change so that we can have an artificial intelligence that truly delivers equitable care. The biggest impedance to AI delivering its promise is what I call AI, arrogance and ignorance. As long as the power dynamics exist in the legacy systems that are really controlling health care delivery education, then there is no hope for AI to truly provide care that will produce the best outcomes for our patients. And with that, I am happy to fight with Piyush. Thanks.

AI biases

healthcare

systemic inequities

Datathons

patient-centered AI