SCCM Resource Library-Pro/Con Debate: Can Simulation Be Used for Assessment of Procedural Competence for Hospital Privileges?

Video Transcription

Well, it's 4.30. I think we'll get started. Good afternoon, everybody. My name is Greg Boats.  I'm from MD Anderson Cancer Center in Houston, and I'm moderating this session along with my friend and colleague Heather Meissen from Emory University.  Welcome to a pro-con debate on, can simulation be used for assessment of procedural competency for hospital privileges?  Our speakers today are Dr. Jeffrey Lighthall from Stanford University and Michael Pisa from University of Pennsylvania.  We thought that this would be a fair battle on a pro-con debate, so we're going to start off with a coin toss to see who gets to go first.  We know that the pro sometimes has an advantage if it's just pro-con, so we're going to do a fair thing here. So Heather, go ahead. You want to choose, Jeff?  What kind of coin you got there? It's a dime. It's a rigged dime. Yeah, it's got tails on both sides. You'll take a what? Dead present.  Tails, would you like to go first? Here you go. Please welcome Mr. Michael Pisa. So I'm Mike Pisa. I'm from the University of Pennsylvania.  My clinical practice is in surgical and trauma intensive care. I am also the program director for our advanced practice fellowship there. I am a course director for FCCS there.  And I use simulation a lot. So I include those points off my CV just to let you know that I have no financial disclosures, but I am a fan of simulation.  I use it extensively in teaching both physician learners, advanced practice learners, and I'm a proponent.  However, for the reasons listed here, you cannot use simulation in the establishment or ongoing monitoring of procedural competency.  First off, competency itself is ill-defined.  Even if we could define competency and measure it, the technology associated with simulation just isn't to the level at which we can use it to establish competency with hands-on procedures.  We try to get around that by creating high-fidelity simulation and situation. That's a myth.  Further, there's little evidence to support the use of simulation as a teaching tool in general. And then surrounding procedural competency, the body of literature is even smaller.  And then in light of all of this, you've got to rely on your gut. My gut says no.  And I think by the end of this talk, your guts, while probably won't be as big as mine, will agree with mine. So the dictionary definition of competence is quite easy to understand.  But when you apply that definition or try to apply that definition in critical care and specifically around the procedural privileging, it becomes quite difficult.  One proposed method of defining competence is to simply put a threshold number of procedures out there that need to be completed before competency is declared.  Or a threshold number of procedures that need to be performed over a set period of time that allow for competency to be demonstrated on an ongoing basis.  When you look at the very little data that is out there surrounding this, you see that there's discrepancy between what the experts think in the field and what actually gets set as those numbers for threshold performance.  Despite that, our societies recommend specific numbers. But the data behind that is not great.  Further, there's wide variability across training programs and how they individually define competency.  This is a study looking at training programs for pediatric intensivists and their competency surrounding central line insertion.  Quite a surprisingly large number of those programs use no formalized assessment method or written record of assessment to establish competency for this pretty dangerous procedure.  And very few, only 9% of the studies or programs queried employ a method for ongoing competency assurance.  Even if we could define competency, even if we could figure out a way to measure it, the technology surrounding simulation in that regard is lacking. It's just not up to the task.  This is so even in the setting of having a wide variety of simulation modalities to choose from. There's commercial task trainers. There's high fidelity training mannequins.  And when you get into the world of homemade simulation modalities, the options are infinite.  I myself have spent hours as a vegetarian sticking mayonnaise-filled balloons into chicken breasts to decompress abscesses. Here is a setup from our institution.  It's a task trainer setup for the instruction of paracentesis, thoracentesis, radial artery line insertion, lumbar drain, and lumbar puncture simulation.  This is not without utility. These trainers are very good at giving participants exposure to some crude representation of the anatomy associated with doing these procedures.  They're very good at giving participants a chance to interact with institutional-specific kits and pieces of equipment.  They're very good at allowing participants to practice the steps and sequence of events associated with these bedside procedures. However, they in no way mimic reality.  They come nowhere close to a real patient. Just the tactile feel of the skin on these things is nowhere near real.  There's no soft tissue overlying the anatomic structures to move around and slosh you off target as you're sticking a needle in. It's just not there.  The fidelity isn't there to support using this as a means to demonstrate competency. Furthermore, these things are real expensive, staggeringly expensive.  It really limits our ability to create any kind of generalizable method and standardized practice across institutions and environments in the testing and establishment of competency.  Here's an example of some homemade trainers. Homemade trainers are created because these other things are so darn expensive. The example on your left is from the military.  It's using a SAM splint and a piece of red rubber catheter to simulate arterial line insertion. It's because these trainers are really expensive. And this is great.  It's really portable. You can stick it in a backpack. You can teach a medic out in the field how to put in an arterial line. But it in no way mimics reality.  The second example there is a neonatal model for the performance of thoracentesis. Again, very good at giving the learner the option to use the equipment, identify anatomy.  But doing it on a piece of plastic like this just doesn't allow the operator to interact in a way with the mannequin that mimics reality. There's no emotional involvement.  The stakes are real low. You know you're not going to hurt this SAM splint with a red rubber catheter in it.  So you may say, hey, Mike, there's these high-fidelity trainers that are super fancy and do a really good job at doing a lot of the things that our patients do.  This is another representation from our institution. We have an incredible resource.  We have a very well-equipped simulation center with several really high-tech, high-fidelity mannequins.  We're staffed with primarily clinicians in that center who understand patient physiology, who understand the clinical environment,  and put a great deal of effort into creating simulations that mimic the clinical environment.  As participants, you can really interact physically, cognitively, and emotionally with the mannequin and the scenario that you're immersed in.  A vic-vasculopath here, our outside hospital transfer comes complete with personal effects and an outside hospital chart that's largely full of useless information with a few embedded nuggets that give a clue to diagnosis.  But still, the stakes here are nonexistent, really. Sure, learners might get low marks on their performance, or they might get criticized and debriefed for missteps.  But you can't hurt these patients.  It's not the same stress that you're under in the clinical environment when you're exposing patients to potential harm with your bedside procedures.  As such, you can't use this to establish or monitor for ongoing competence. You have to stress learners or demonstrators in a way that is real in order to establish competence.  So high fidelity in this regard is a myth. Even in the most elaborate simulation setups, like this one pictured here from the military using the surgical cut suit,  where they fly these patients in, drag them through a dusty landing strip, and land them in a military operating room,  do all kinds of crazy stuff to this piece of plastic that's laying on top of a real person, the stakes are just not high enough. You're not going to hurt this person.  Everyone knows that. So you can't establish competency. This is excellent practice.  This is excellent practice in team leadership, team management, excellent practice in logistics management,  excellent practice in equipment organization, but it can't establish competency. Is there evidence out there?  Is there evidence that demonstrates using simulation can, in fact, demonstrate competence? Is there evidence out there that says simulation is an effective teaching tool even?  Well, I think the strongest evidence, and I apologize for those of you who are sitting here expecting a data-rich presentation.  The strongest evidence really shows that the evidence is poor surrounding this topic. So we don't have the support from the evidence to use simulation in the establishment  or ongoing monitoring of procedural competency. So when there's no evidence, you have to trust your gut.  And in this circumstance, I think your gut has to say no, or else you end up with an ulcer, right?  So in conclusion, and this is a relatively short talk because the points are easy to make. We can't measure that which we can't define.  The tech isn't there to support it, even if we could measure it. There is no evidence to support its use, and our gut has to say no.  So we cannot use simulation for the ongoing assessment of procedural competence for hospital privileges. That's it. Good luck, sir. Thank you, Mike, for that great presentation.  I'm beginning to regret the year I spent as a medical simulation fellow.  Our next speaker is Dr. Jeff Lighthall, and he's going to present the pro position for the use of simulation for procedural competence. Please welcome Dr. Lighthall.  Yes, I was given the affirmative to this question. And just to, you know, think about a little bit why we're here.  I went to a seminar a couple of days ago where they ask you to consider your audience.  So I assume that you're like me, have an interest in simulation capabilities, but maybe a little bit of, maybe some knowledge,  or you've been tasked to participate in the credentialing process, but maybe a little bit of ambivalence about how to connect those two. So if that's you, you're in the right spot.  Like Michael, I've developed a lot of different curriculum around simulation, and I'm a huge fan of it as an education modality.  And I actually thought maybe I got held, you know, handed the wrong side of this argument because I certainly was offended initially to try to defend this for credentialing.  But I will try to do that. I do not have any conflict of interest.  And again, a goal is just to try to get some ideas out there that we can all discuss or argue about a little bit later.  So kind of as a group exercise, I want to first look at, you know, one is, is this technology that's looking, you know, for a problem or are we trying to solve a problem?  So I'm going to give you a problem just for a few moments. Picture the worst code that you've ever gone to. Just a complete cluster. Someone got Hep C because needles were flying.  Patient did poorly, etc. I bet we could get about, you know, 50 different stories. And I could tell you there's one thing they all had in common.  They're run by somebody with an ACLS card. They were credentialed to practice. Somebody decided at some point that someone was good enough to go ahead and run cardiac arrest.  And we gave them a card. And yet, as we know, these were terrible, terrible events run by somebody that was credentialed to do that.  So how we define competence, I think, is a little bit missing. This is a simulation study I did a long time ago where we just looked at people measuring,  where it measured people's ability to manage septic shock. A huge gradient, statistically distinguishable levels of performance within a group of cohorts,  whether you're looking at interns, residents, or even, you know, how they look globally or how technical aspects as well as non-technical aspects, such as leadership.  So there's a lot of different gradations of expertise out there. And we're saying everybody's okay. More of a problem. Mainstream medical press.  Four articles about a month ago or earlier this month talking about what do we do about this decay curve of skills that people have.  We all know about the learning curve, but they're worried about the decay curve. And this was in the context of a group, New Haven Hospital, I believe,  that ended up doing cognitive behavioral testing on some of the physicians that were approaching either 70 or 75 years. A big controversy now.  It was a lawsuit that started about three days ago because of that. And so it's out there. People are wondering.  People are seeing that we're questioning how we, you know, develop competence. If you're like my daughters who, you know, share podcasts, this is a great one to listen to.  So the person that promulgated, you know, acquired neurologic, iatrogenic neurologic injury throughout Texas was credentialed in three different hospitals and got referrals.  And somehow he graduated from residency and fellowship as well. So, again, this is out there.  People are worried about what are you doing in your hospital to make sure people are doing the right thing. What else do we have to my fellows in the back there on the screen here?  When procedures happen, these guys swoop in and get the procedures. I don't get to do anymore. So what do I get to do to do central lines and pacing and all this other stuff?  This doesn't go back to California, by the way. Get rid of it for calling my fellows vultures. Anyway, you know the story.  So we've been tasked administratively and otherwise for initial privileges and for ongoing privileges to show that people have some kind of competence and or some, you know, minimal level of skill and different procedures that we do.  This is a big problem.  I've been filling out forms for ongoing professional credentialing for years, and they're just like the most over general, not specialty specific forms that that you can possibly imagine.  I didn't even realize that this FFP and OPP were like these really hard and fast criteria. I was just answering these, you know, idiotic questions that the administrators give us.  You know, and asking the core questions like, you know, if your daughter was getting this procedure, would you want this person to be put in the line?  And yes, no, but instead it's, you know, do they have psychomotor repair, you know, impairments?  And so, you know, basically it's good for catching catching people with new strokes and who have disabilities that are floridly unable to perform procedures, but not really the kind of skills that we're after.  So I agree with you, you know, develop defining competence is a huge issue here. What else do we got here?  So when you have somebody that definitely that is dangerous, like Al Capone, rarely do you get to get rid of these people for what they've done wrong.  And instead, we put people through these remediation programs. And eventually what gets them are these technicalities. For Al, it was tax evasion or something like that.  And so for a couple of cases I'm aware of, it was things like poor credentialing and ministry of noncompliance.  And this is after people just got dragged through hours and hours of meetings and paperwork and legal defense work, trying to find out how to cover their butts to get rid of somebody that they would not want to have taking care of their loved ones.  So, you know, the current state of the art works great until there's a bad outcome. We're all going along with it. I'm getting, you know, credential this year.  Another hospital, same thing.  Don't have a paper trail of death following me, so I'm sure they're going to let me do whatever I want to do. So is there a better way to go about this?  Well, we could do psychomotor testing at a given age. They do that kind of at the DMV. I'm not sure in your state.  We can steal procedures back from our trainees and have fellow colleagues observe those.  And that's a little bit, sounds really easy in academics because you can, you know, there are like eight people I can call around.  But I bet in private practice that's really hard because, you know, your clinic, your colleagues are in clinics and so forth.  Or can we leverage the capability of simulation systems to do some of the work for us?  So my proposition is that simulation activities can be used to, you know, to acceptably determine some skills that we have to maintain and for some privileges, not everything, not all.  Nice thing about simulation, there are, as opposed to other cases we take care of, we know what's wrong with them.  These are, you know, when we're talking about different kinds of scenarios, these are known disease processes that we've designed into our scenario.  We know that there are standards of care for different kinds of scenarios.  We know there's certain hazardous actions that if you do certain things for certain diseases, they're, you know, the kiss of death.  So, you know, and again, back to, you know, what you're saying is that kind of autonomic response of how you feel when you do these simulations.  Really, the fidelity is the quality of the experience and does it get you sweating like I'm sweating now and your heart rate up and so forth.  And I think most people find, at least for full mission simulations, that they are engaged at that level. And when they can't think straight, it's because they're stressed out.  So I think it's not as bad as it's being depicted. And you can explore people's thoughts through debriefing and you have it on film.  So the actions that were performed are not based on subjective recall and hearsay. They're on film if you have to look at them.  And as I showed before, with carefully designed rating systems, you can actually differentiate levels of performance. And again, that gets back to how we want to define competence.  Is there some low bar that we all have to cross or is there some kind of gray areas that we really want to differentiate as far as performance goes?  So things that you're probably used to simulations, number of different activities.  This could be me just taking get out a pen and writing, you know, kind of a sample case on the board here.  Or you can hijack some software and kind of dynamically change vital signs in front of a group of trainees discussing a case.  There can be clinical reasoning done through kind of computer based STEM scenarios. And you're familiar with standard patients probably to hybrid simulations.  Use combination of standard patients along with different sim modalities.  This is me with a blown pupil, little phenylephrine eye drops in one eye and a sim man software giving me hypertension and bradycardia.  And, you know, see how people manage it. So that's that's a hybrid model. This is a line sepsis here, as well as some hep C.  This person is going to get because they're doing this bare handed.  But this is central line trainer probably familiar with that person is going to get line sepsis as well as pacing.  And then you can have full mission simulations where this is a GI bleeder where we basically activated the house, our T system and we've got blood everywhere.  And we just have to see how, you know, looking at how the environment interacts with the blood bank and transfusion principles.  This is kind of down into the DIY level, but there's some people can build sensors and haptic feedback into a lot of some of the hands on procedures.  And it's kind of gets a little bit closer to the idea of just like basic procedure competence.  People build sensors into hands so they could track, you know, how fidgety someone is with a needle, how deep, how many misses they have.  And then discrete motions of the needle hand, as well as the ultrasound hand.  And this is basically showing that you can differentiate experts from inexperienced people in a lot of these different kind of aspects of procedures if you want.  Now, is that someone that's competent or not? That's kind of up to you decide. This is some of the stuff that comes out of the robotic simulators.  These are computer based interfaces that record movies. They record every last hand movement that surgeons have used doing operations.  And they can definitely give you a sense of, you know, what kind of motions are associated with tissue damage, bowel injuries, longer length of stay, etc.  So in terms of like the interface between recording motor capabilities and clinical differences, those capabilities are emerging.  And this is definitely a platform not only for training, but actually for kind of looking at different gradations of performance and getting feedback on that.  So in general, you know, standard patients are great for communication and people skills. Mannequins and sim labs are great for crisis management, leadership, behavioral skills.  Task trainers are good for psychomotor skills. Computers are pretty good.  You can interrogate judgment and compare people's actions to how they respond and plan cues within your stem.  Great for cohort ranking. And finally, hybrids kind of overcome some limitations of simulators themselves by trying to combine task trainers with humans.  So in the ICU, in the critical care environment, you can envision that, you know, you might be able to differentiate some levels of performance,  at least, you know, assume that people are reasonably skilled in the hands-on aspects of central lines, pacing, various kind of drainage procedures in the chest and the gut, bronchoscopy, and maybe even tracheostomy.  But these skills are multidimensional. Again, you've got to figure out if you're a hospital administrator what you want to do.  You know, is it getting, you know, green gas down the right hole if you're looking at intubation? Or is it trying to get the wire and the catheter into the right vessel?  Or is it something a little bit greater? And I would, you know, again, submit to you that some aspects you can address with simulation,  but maybe not how somebody manages crises, the environment, whether the lines even indicated,  whether the information is even used properly once someone's been subjected to all the risks of the procedure.  So these are things that are out there that people have to use to try to define what competence is, and you can build those into simulation models and have it be part of it. But again, you can't just, I agree, you just can't plunk a mannequin down here and say,  you know, get the wire in and I'll let you do it to humans. Definitely the skill requires a lot of different dimensions. So, a few comments about assessment here.  There's, you know, formative is what we're probably used to in the educational realm, that's where we're kind of coaching people along using our activities  to try to get people to be better. And then summative is past fail, and those are kind of the higher stakes exams. And definitely the characteristics of higher,  of kind of summative rating scales  brings in the whole field of psychometrics where people actually look at what is being scored. Does that discriminate between experts, non-experts? Does it discriminate on the basis  of race and class and gender? So, you know, again, when you go ahead and decide that you're gonna do an evaluation scheme, including simulation or not,  it's really incumbent upon those that design the test to make sure that they achieve all these performance characteristics that have been pretty well described.  And Jack Boulay's kind of sort of our national guru here in the US that's been behind a lot of the different exams that have been offered by the USMLE.  So, I mentioned there's checklist rating systems. Again, those can, those are not, those are okay, but again, they're fraught with problems.  Checklist just tells you whether you did something or not. It doesn't say whether you followed the right order. There's some people that do really well on checklists.  There's been some studies in simulation of, I think it was maybe line placement where people scored incredibly well on the checklist, but expert rater said,  I don't want this guy touching me. It's terrible. So again, you have to kind of work out the inconsistencies and the problems of evaluation.  And, you know, there's obviously auto-fail and desired actions that you can build into those.  The experts in the field basically say that global ratings by experts are not bad. And so again, this might open up the capability of having direct observation  or in a simulation kind of environment, having films reviewed by quote experts that are out there. It also turns out to be that surgical skill  when looked at by experts in the field, which are pretty time consuming when you have to look at somebody's skill, those ratings are about equal to those  when those films have actually been crowdsourced. Just people like you, people in the Orlando airport, like does this guy look like he has high, low, or medium levels of skill?  It turns out to be the crowdsource global ratings are about the same as experts, amazingly enough.  So there's different ways of rating things. And again, the idea is that you use these flexibly so you don't get caught in some trap where people know that there's a checklist  and just hit the items, even though they might not necessarily be the most competent souls on earth. So it's already been used. In ACLS, BLS, we're using mannequin feedback  to determine whether you continue on. This is the RQI platform of ACLS certification that some hospitals use.  It uses both computer and demonstration-based. If you're gonna get through general surgery residency, you gotta do FLS on a simulator.  In the VA, there's various levels of credentialing of respiratory therapists and others for out-of-OR airway management. And the lower levels of permission to manage airways  are based on simulation experience.  And in board certification, high-stakes environment,  the Israeli Board of Medical Examiners, the British Anesthesia Society, the Family Practice, I don't know, board, for years has been using STATR in patients.  And recently, the American Board of Anesthesiology has been using OSCEs. And the whole idea there is trying to identify those levels of performance where you can show how  and where you can actually do it rather than just say that you know how to do it. So the world is actually turning this way. And some of these are very highly valid,  very combed-over tests that have great content validity,  great ability to discriminate between people that really should be there or not. I don't know if anybody's watched this show. It's kind of like my late-night junk.  Remediation, simulation remediation, that's the other part about this. Many have been proposed.  This is a lot of hard work. It's gotta be really customized. There are two books on it. That's all I really wanna say. I gave you a couple references there if you wanna look at it.  But taking people that you're trying to bring back into practice really is based on how committed the institution is to those individuals. It is tons of work and tons of money.  But some people have used simulation to help out.  So what is this whole thing gonna look like? I think it's gotta be proactive. I think the whole idea of you showing up to do five lines on a mannequin  is not gonna be very acceptable.  I think what you have to do is make this part of training and part of ongoing practice where you, to maintain it, you have to show up to the simulator. You can get some coaching.  And you can actually see how your performance compares with peers. You can get that feedback. Like I have no idea how my skills compare to my peers for the most part and vice versa.  And yet if I am working in kind of a controlled environment and controlled cases, that capability might actually be able to provide me a reality check.  It might actually let me know that I need more practice. It might actually tell me that it's time to retire. One can only hope.  So we can talk a little bit about implementation. If you want, I don't think there's any roadmap for how people have done it. But I think it really depends.  You need to kind of, what we have to do is if we're gonna have this kind of system, we need to be involved and we need to be there  from square one and talking about what competency is and making sure that those are built into whatever programs that we want. As Michael said, competency is very hard to evaluate.  And yet we're granting people, we're saying that people are competent by even vaguer standards. So here's our opportunity to actually design into training activities what we want.  So my proposition is that we get involved in some level with this and go ahead and accept it to some degree where it can solve our problems.  I think we can look for ways of bringing it in rather than resisting it. Someone else is gonna do it for us and the product is not gonna be as acceptable.  And I think with that being the case, we can incorporate simulating training part of ongoing practice. So things are gonna change. I mean, there's people who did critical care  before ultrasound was an everyday practice. So how do you work in some of these new modalities? Do you throw up your hands and say, you guys do it  or do we actually take part in that?  New protocols will come out, new standards for treatment. And if you're locked into some time warp where you can't work on those with some of your colleagues  and bring the whole level of playing up, I think we're gonna be lost. And so there's some pretty good examples of where it happens right. And that's some of the ongoing training  that airline pilots have with simulators. And when people have cut corners and said, let's not do simulation training, I think we've all seen how that's turned out too.  So anyway, I'll turn this over back to my partner here  for further discourse. Thank you very much. Thank you.  Anything more? We could take questions. Well, I'd like to thank both of our speakers for very enlightened discussions about the use of simulation.  I'd like to open the forum to our audience who might want to ask questions of our speakers. Sir. Hi. Sam Simash from Omaha, Nebraska. I'm pretty involved with simulation.  And I think there are plenty of articles in the literature that show that in various areas I work in the surgical simulation, you can learn the skill by practicing on the  simulation. However, there is very literature that shows that once you learn the skill on the simulator, you can translate it to the clinical arena.  So I want to know your thoughts about that, because that's an area where we still need to work. Well, I'll make one comment about that.  There's tons of variability in anatomy, even skin trigger, and all that. So let's just take a central line, for example.  So maybe I want to get my repetitions. So let's just say I'm a new learner, right? So there's a lot of stuff I need to learn. The kit, how to set it up, the guide wire, how to be  efficient, how not to stab myself, et cetera. And so why do I have to have a human being with a tent over their face, head down, listening to a bunch of naive  conversation, when it's clearly that I have not put this damn kit together or anything like that before. So I think some parts of that can be done on a simulator  where you can just get things second nature. And then you know you've learned all that stuff. There's nothing to learn there. You've learned it.  You've learned it in a controlled setting. And then all of your focus is on how this guy's anatomy is different, why this guy doesn't have a right IGA,  where else you got to go, et cetera. And I think the learning that you finally do on patients, one is you're prepared to do it. You're going to be a lot more efficient.  Patient has to stand on their head a lot for fewer minutes. And I think your ability to really take advantage of the interaction with a real human is going to be a lot better.  So that's how, you know, if I have a family member, do that, okay?  I don't think you could have said that any more eloquently. I think the use of simulation is very well tailored to  getting that experiential learning down with the kit and the mechanics of the procedure. And I think you agreed with me in the end that, you know,  competency can't be established through that. Sir, please.  Hi, Matthew Mormon from Cleveland, Ohio. So I appreciate your one slide with the aviation example, because I think comparing ourselves to other professions is the key.  If you shun simulation for high stakes environments, you will never get on another commercial airliner in your life because that's the sum total of their training.  Sully Sullenberger landed a jet in the river, not because he did it in a jet, but because he did a dual engine failure hundreds of times in a simulator.  Compare his talk with any of our hospitals codes that we do all the time. And you'll see the value of simulation yet proving it objectively in a new field is a challenge.  So the question for you guys is when you look at other fields that use it in an evidence-based fashion,  where do we break down in healthcare? Why, why can't we make a one for one match with a central line or anything else? And how did, how do you then create simulation in healthcare?  Because clearly simulation works, it's evidence-based in high stakes environments, but we haven't figured out how to model our world because we're doing it in our own little silo.  So where does it break down from say Sully Sullenberger's experience? Why can't we do it? There are no randomized controlled trials on the use of parachutes.  So would you like to address that issue?  The question about learning from other. Yeah. You know, I think that's all, you have just as much expertise as I do on that. It just comes down to the will of the institution and  therefore thought you, you can get away with not doing it right now and save a whole bunch of money. And if that's what's kind of dictating the industry then,  or your hospital, then that's the way it's going to be. You know, does the court of public opinion change things? It, it could, in some cases,  I haven't seen it really turn that around much, but I mean, those are the forces is if we don't, you know, internally change our behavior,  someone else is going to tell us what to do. The joint commission might come down and say, you got to do so many simulations.  You know, you got to do five take back hearts in an ICU. If you're a cardiac center, you name it.  You know, if we, if we don't see the light, someone else is going to shine in our face. I'm afraid.  I think the tech associated with, you know, flying a plane is easier to simulate than the tech associated with taking care of a, an organism, a patient.  It's just very difficult to create that technology. I can see it coming down the pike. I mean, there's a very impressive virtual reality trainers  that allow you to feel things that you're looking at in the virtual world.  And I can see those kind of lending some aid in the, in this effort, but it's just not there yet. I mean, there's a fundamental difference  between us and aviation in that, if that plane crashes, so does the pilots, so do the pilots. There's tons of motivation to be as good as you possibly can.  You have tons of skin in the game.  I think we're still accustomed to blaming bad disease outcomes on the patients and the diseases that they show up with.  Certainly humans don't come with an owner's manual that defines things the way that an aircraft was designed by engineers. Your question, please. Yeah, hi, Brian Kelton from NYU.  I run a simulation center there that's geared for critical care. I agree 100% with our first speaker that in terms of credentialing for central lines  and other procedures, it's not there yet. I agree with all the, you can get three quarters of it down on how to do the procedure in terms of a checklist,  but I think it's really good for identifying the person who shouldn't be credentialed, right? And then I can call up the program director and say,  you know, so-and-so needs to work on central line skills with more direct supervision and then come back. So I think that may be a good use for it. Yeah. Great.  Thank you, sir. Hi, I'm Neil Chasen from Cleveland Clinic. So I'm the critical care program director, and I think that it's very easy for me to mandate  for our fellows to use simulation. We do it all the time.  We, at the Cleveland Clinic, we offer the exact same courses for our staff, and they're what I would say sparsely populated,  and that's probably being generous. The reasons I think for that is that physicians aren't really comfortable teaching their peers in many cases,  especially when the people that need that are older than us and probably more senior on that ladder. And so the question for you is, how do we actually change the paradigm  of physician self-improvement procedurally? We talk, I mean, this whole conference is full  of improvement, but it's mostly lecture-based. How do we do that with ICU procedures? So the topic of psychological safety in the use of simulation is a big topic.  Do you want to address how we do that for our peers?  Yeah, now I don't know if this is gonna solve your problem, but I think the best way is to do it in centers where you're not doing it in front of all the other ancillary staff,  your trainees and all that, and create kind of rules of engagement, which basically, in our shop a lot of times, involves signing confidentiality agreements  where, you know, worst scenario, you're gonna say this person's, you know, a total klutz and stabs everything.  But the other thing, too, is just trying to have leaders and facilitators of these sessions that just talk about, you know, maybe personalize the whole struggles  of keeping up with skills and some of the aspects of just, you know, aging and decay and hand-eye coordination and just, you know, the fact that there's this muscle memory  that we all had at one point, and sometimes you don't have it anymore because our procedures are going elsewhere. So I think it's a lot of that just kind of preparation  that really makes it possible. And then there's gonna be some people that are refractory to it all, and I don't know what you do there.  Yeah, I think, you know, you touch on something that relates to creating a feedback-rich environment in your clinical setting, in your health system,  and it's a difficult thing to do.  And I think it's a really,  it's really a cultural thing that needs to be built, and it takes, you know, a very complex paradigm of leadership and buy-in from everyone else in the system.  I think there's another motivation there as well, in that, you know, the urban myth of the physicians involved in a legal action where the, under discovery,  the attorneys asked for the films for their ACLS training to see if, in fact, they were competent at the time there, which is why the American Heart Association  does not provide you with a certification from their courses. You have a course completion card. You're a provider, not a certified ACLS provider, if you will. Your question.  Yeah, Kyle Briggs from Emory.  One of my thoughts is similar to what we're talking about here, and that is, which vulture is training me for that day? Because when I was learning my procedures,  I had a variety of different experiences. Some of them were very good at teaching. Some of them were not so good at teaching. I'm sure they were all clinically competent  and fine in their own procedures, but as far as how they could direct me and teach me, I got what I got, and it was kind of luck of the draw that particular day.  And so, recognizing that we seem to agree that defining competency is hard, we can't do it, what about those who are teaching us? And would having something like simulation,  where I could have more control over who's teaching and how they're teaching me, would that sway any of your thoughts on whether we use simulation or not?  I think simulation could be a fantastic tool to use to teach the teacher,  to give people practice in teaching procedural things.  I know in my institution, I've helped people along in their teaching techniques for invasive procedures at the bedside.  Again, I think it takes creating an environment where we're comfortable taking feedback from each other.  Yeah, I mean, that's a whole other dimension of skill I didn't really consider, is do you have the ability to teach others now that you are good enough to do it yourself?  Again, how many eggs do we put in that basket? Meaning, like we're trying to just make sure people are competent when they're doing it. But I think one thing you could do is,  if that's a concern, is you could say, hey, listen, just pretend I'm naive here, how would you teach this to me?  I'm an anesthesiologist, and sometimes what I task my residents with when I'm in the OR is say, hey, listen,  can you pretend I don't know too much about hemodynamics, teach me how to go ahead and troubleshoot somebody with a low blood pressure. And I kind of want to hear how they do it.  And then I kind of coach them on how they teach and how to structure knowledge and all that. I don't know if that gets at what you're getting at,  but I think if the skill is teaching, then you have to do something that's very teaching oriented in how you develop that person.  Any other questions?  I'm gonna ask a question. The current standard for credentialing or perhaps privileging for procedural competence, perhaps in institutions is based on my chairman  initialing next to my request for credentials or privileges, the ability to do such. And my chairman may have never seen me do one of those things before.  So is using simulation any worse than that? Are we actually raising the bar at all with simulation to say at least we're looking at process issues  and psychomotor issues and perhaps non-technical skills that might be valuable in evaluating someone  and or as you say, is someone else going to design the test that was used to evaluate whether we're competent to provide these procedures?  Tag, you're it.  Well, I'm an advocate of having this be kind of continuous and proactive where it's not a question of, does Dr. Boats have it or not?  It's just like, you are gonna be part of this system where you're gonna hit this every couple times a year. And perhaps that can be built with some rules if there's deficiencies,  people identify areas of additional practice. And certainly that's been, there are some precedents for doing that in military medicine where people come out of say,  ENT residencies and all of a sudden, they're put on an Air Force base where they have to deliver babies. Well, guess what? They are drilled till they know how to do that correctly.  We hope.  They say.  Yeah, I mean, of course, simulation can be used in that regard. I think it's a great tool to both practice  and have someone kind of take a look at how you're practicing and coach you on that practice. I think the ability to have ongoing assessment  of performance is a great plus for the use of simulation. Now I feel better about doing that year of simulation training. Your question. Hi, Eduardo Mireles, also from Cleveland.  I think that the challenge that we have is that we try to use one tool, and you poised it very well of saying, is simulation what will tell you that you're competent or not?  And I don't think that that's where simulation lives for all of us in the postgraduate world. Where it lives is to do one specific task training for new procedures, that's fair.  But the other one is there's items that we need to do and observe that may give you ideas or key features that tell you, ah, he may have an issue,  which assesses the skills with the hand, the way that you move across the room. How do you do this? Not necessarily all the knowledge base.  Those are assessed by your peers or by your outcomes or by the board exams, if you may. I mean, it's a multi-modal assessment. I think that what we are,  and it depends what you want to assess. If I want to assess central line skills, just put him how to grab the ultrasound in a patient.  Gives you totally much more on a standardized patient, much more information about how that human being will behave or how much he has retained.  So it's not an all or none in my mind. It's just a tool that the system, we have to embrace to help us recognize those things. Because me, when I'm signing those things  for my fellows as they're going out, have I seen them place pacers? Yeah, maybe one in a crisis. Not really. And we send them out saying,  yeah, he is ready for put a pacer on patients or manage ECMO or manage mechanical ventilations and general assessment. That's just a comment.  Sorry. Thank you. Any other questions or comments?  I think it's important to know that one of our colleagues at Stanford has said for 25 years that simulation is a technique, not a technology.  And it has its place in medical education and in assessment. Grace? I appreciate that. Yes, very true. Grace Arteaga, Mayo Clinic, Pediatric Critical Care.  Having been involved in education for many, many years, one of the things that I have noticed is that along the years, I had to learn a lot about adult education,  which is something that we were never taught and that we're not taught at all. So how do adult people learn? And recently, I became very aware  that we are not considering generations because the way I learn is very different from the way my new fellows are learning. And we are not considering that in simulation.  So when you assess competency, the expectations for one group is those are very different from the other group. So how can we merge all those generational differences  in one educational competency assessment?  You have thought about this tons more than I have. I don't know. I mean, so I think it would be interesting to see different generations of checklists for procedure  and then also global ratings and see what's going on. Because I think global ratings are really,  do you inspire confidence in me as I watch you doing a procedure and was the procedure completed in some semblance of normalcy in my mind? But maybe how you got there is different.  I don't know. So I think that's a fascinating area of investigation, but I don't have any great insights.  I've got a couple of kids, so help me, you know? I have no great insight. I have a millennial and a centennial at home and they learn very differently  and the approach that we use with them is completely different. They're 11 years apart and the things that worked with my son have no effect on my daughter  to try to guide her learning and her behavior. But it's an interesting thought to make it contextually sensitive to the learning style of our trainees as they progress.  Any other questions or comments?  Well, I certainly wanna thank all of you for joining us on the late afternoon. I'd like to thank our speakers, Dr. Lighthouse and Prasad. Thank you for your welcome. Thank you.

Video Summary

This is a summary of a pro-con debate on the use of simulation for assessing procedural competency in hospitals. The debate was moderated by Greg Boats from MD Anderson Cancer Center and Heather Meissen from Emory University. The speakers were Dr. Jeffrey Lighthall from Stanford University and Michael Pisa from the University of Pennsylvania. Dr. Pisa argued against the use of simulation, stating that competency itself is ill-defined and the technology associated with simulation is not advanced enough to establish competency with hands-on procedures. He also highlighted the lack of evidence to support the use of simulation as a teaching tool and the high cost of simulation devices. Dr. Lighthall, on the other hand, argued that simulation can be used to assess procedural competency and suggested using simulation for ongoing training and practice. He mentioned the benefits of simulation in creating a safe environment to practice procedures and the ability to provide feedback and assess performance. Overall, the debate emphasized the need to define and assess procedural competency and the potential benefits of incorporating simulation into the credentialing process.

Asset Subtitle

Procedures, Professional Development and Education, 2020

Asset Caption

The Society of Critical Care Medicine's Critical Care Congress features internationally renowned faculty and content sessions highlighting the most up-to-date, evidence-based developments in critical care medicine. This is a presentation from the 2020 Critical Care Congress held February 16-19, 2020, in Orlando, Florida, USA.

Learning Objectives

Explore the challenges in assessing procedural competencies in our current healthcare delivery model
Discuss the fidelity of procedural simulation models, such as partial task trainers
Debate the pros and cons of the use of simulation for the assessment of procedural competencies for the purpose of maintaining hospital privileges

Simulation Cannot Be Used to Assess Procedural Competence for Hospital Privileges

Michael A. Pisa

Simulation Can Be Used to Assess Procedural Competence for Hospital Privileges

Geoffrey Lighthall

Meta Tag

Content Type Presentation

Knowledge Area Procedures

Knowledge Area Professional Development and Education

Knowledge Level Intermediate

Knowledge Level Advanced

Membership Level Select

Membership Level Professional

Membership Level Associate

Tag Simulation

Tag Catheters

Year 2020

Keywords

simulation

procedural competency

debate

technology limitations

evidence

teaching tool

cost

training

credentialing

	Society of Critical Care Medicine 500 Midway Drive Mount Prospect, IL 60056 USA Phone: +1 847 827-6888 Fax: +1 847 439-7226 Email: support@sccm.org	Contact Us About SCCM Newsroom Advertising & Sponsorship DONATE	MySCCM LearnICU Patients & Families Surviving Sepsis Campaign Critical Care Societies Collaborative	GET OUR NEWSLETTER
© Society of Critical Care Medicine. All rights reserved. \| Privacy Statement \| Terms & Conditions The Society of Critical Care Medicine, SCCM, and Critical Care Congress are registered trademarks of the Society of Critical Care Medicine.