ACR 2019 Informatics and DSI Report



dr. Bibb Allen one of our new gold medalists and a CRS data Science Institute chief medical officer and dr. Keith dryer a CRS DSi chief science officer will present an informatics and DSi report doctors Allen and dryer well good morning and thank you for being here thank you Tim enrich I am Deb Allen the the DSi is chief medical officer with no commercial disclosures to disclose I'm a member of the Birmingham radiological group community practice in Birmingham and all of my all of my stuff is on file so as we started the data Science Institute and Geraldine said in her remarks on Sunday that educate define and lead in the education arena I want to point just one thing out importantly and that is the multi the multi Society ethics paper that has been in the comment period is going to be published in a shorter version in radiology and in JCR that gives the data Science Institute now the ability to have a code of ethics we wanted to find the beneficial uses of data science which of course led to our initial work in creating a I use cases and then finally to help radiologists become a leader in data science and how we did that where we looked at three important questions what are the most important clinical tasks for AI how do we make sure that AI is safe and validated for clinical use and then how do we make sure it's working in the real world and we leverage three historical attributes of the college really our resources around the appropriateness criteria is how we saw the ability to it was the model for our use case panels to develop structured use cases for for AI we looked at our clinical trials experience and how do we validate this and how can we leverage the multicenter trial atmosphere into how we validate AI algorithm so we can make sure that they're safe for criminal practice and then of course our registry program is how we're thinking we can can monitor and make sure that AI is working in the real world and if you look at our timeline of how we've done all of this we've had numerous meetings with the FDA we've set up numerous demonstration projects with both researchers and academic institutions and developers and the industry sphere to look at and validate these these tools that we're making for clinical use but now we've turned our attention to what do radiologists need for the future to adapt for AI and this is going to be the the the the the purpose of what Keith is going to talk about in a minute the a AI lab but basically we believe this pyramid sort of describes learning the basics of a I evaluate AI for use in your clinical practice these are things and I think all radiologists whether you're community practice or academic whether you're data scientists are just like me a regular clinical guide to get the work done but I think in the middle they're gonna be those of us who want to contribute to use case development to be able to create annotated data sets that will help us participate in the validation of AI for preclinical use or what to do when a developer comes calling it says I want to use your data I want you to mark up some cases for us to use but at the tip of this pyramid it's going to be those who are creating AI and our practices and even some that want to take the step into the venture of going through the whole process to get it approved by the FDA so with that I'm going to turn it over to Keith to talk to you a little bit about the AI lab Thank You Bev and thank you all there's a demonstration of this that you can see up in the mezzanine by the coffee break area so I please invite you to go up and take a look over the time that you have available it take a few minutes but also the goal of this is to make this available to you and your practices at home or on the plane home so there's a mobile that you can look at you can just go to the website a iLab ACR org but let me just step you through this here like BIM bib I have minimal conflicts here but they're all on file and exposed so the goal here is to what we're calling democratize AI the challenge that I've seen so far in this industry is that there's a lot of data that we all have behind firewalls that is difficult to move for obvious patient safety reasons data scientists are at various universities and companies and then companies that want to try and put these solutions together don't have a lot of that data and then there we are wanting to receive the solutions that they make and we have the data and we have the ability to request what it is we want but all these actors are in different places and so the goal is to actually create a framework for all of those groups to communicate similar to and some of you may not know but ACR was instrumental in the 80s for creating DICOM because we ran into the same problem back then we're trying to make PAC systems and reporting systems and EHRs or reese's at the time I'll communicate was very difficult without standards and so those standards got created this is the same thing we're doing for AI but we're also putting a layer on top an application layer so you all can become familiar with what this technology is and actually start to use it so that that is the introduction of AI lab as not only the framework but an application on top of that to empower radiologists to easily implement evaluate and create AI we want to have a component where you can actually learn pieces of this and we'll talk about that to define which is as Ben mentioned these use cases now we want to crowdsource use cases so everyone can contribute and interact and edit evaluate so you can run models that may be out that are coming from vendors or that can be created by colleagues you can create this yourself without having to code up like you do today code up using Python or R or different code sources now you can just do this by selection and you can see that up here or I can show you here and then finally to be able to collaborate so it's been a challenge for people to be able to share their data as I mentioned for moving it through the firewalls but now you can create models at a institution a model at institution B and then just share the models and not have to share the data and you can get stronger more robust algorithms so we want to make that available for people to do as well and again this is not that we're entering into the software business this is to create the platform so that industry can actually understand just like in daikon I make a printer I make a monitor I make a workstation and make a storage system the same kind of thing here so we had a meeting last night with industry about 50 members were there folks were there and we talked to them about this framework and got a lot of good feedback and are continuously interacting with them so let me just step you through this application real quick it gives you these kind of functionalities and components as I say are exposed through this simple straightforward application so here's first is the home page it guides you through some of the things that you can do all the data right now is for mammography for breast density coming from the dmoz trial about 62,000 cases available for training so it steps you into the different just basically how to learn this so it's just a couple minute vignettes of exactly what it is that we mean by democratization or what the modules are inside of here or some of the details of you know how you build a model through this process here's an example of all the use cases that we've created and you can go deeper into it and take a look in this case is classifying suspicious micro calcifications and this is the use case that's been constructed or could be constructed going forward for the other areas as well this is annotate so it shows you how to create a data set for training or for testing or for evaluating in this particular case we let you go for breast density as I say and you can on the right side here select the grade of density and just keep continuing to annotate cases this is kind of the rocket fuel that's necessary that you need to build artificial intelligence once you've done that you could say look at other people's models that they have and compare those so you could say I've created a dataset to analyze two different companies models and I want to be able to test those out here's performance metrics and now for the first time you can actually see how these algorithms compare back and forth to each other because we're all going to be in the situation where we're trying to decide which vendor solution is best or most appropriate for our our facilities for our patients data if you want to go as example of going up that pyramid and you want to actually start to create some AI models with your data you'll be able to actually define the problem prepare the data and configure the model no programming it's all done kind of underneath and when you say okay I'm ready to test and train that you can do that and it comes back and says here's your accuracy here's the model here's what's called a confusion matrix and again push a button and it gives you a video explanation of what these things are so we can all become better versed I think of this like a new modality so for those of you that train before Mr came out then Mr came out and we all had to get up to speed on that it's the same kind of thing here we want to let everybody be familiar with what this technology is now in this case I can just select run in that bottom right corner and you'd actually see the results of that AI running on data that I could have on premises finally you can just run these algorithms on data as I say right now all this theta is from the cloud it's from the dmoz trial the plan over the summer and into the fall is to allow you to do this on premises with your with your own data we're also running challenges and there's a challenge going on upstairs right now you can do it so I again suggest you go up there you can win an iPad but basically it's to go through and annotate these cases just to get you familiar with the annotation process and then finally as I mentioned if there's datasets being created at various places you want the ability to share those with your colleagues and create an even more robust data say I model the plan is to allow you to do that through this process through the foundation through either institutions industry or through these simple solutions that we have to let you create and combine those models to make them even stronger as I mentioned there is an iPad version so you can go I phone version you can go to the store at the App Store and download that and the goal here is as I say to take this through a development pathway so if you wanted to actually create something and put it to commercialization you can go through just like vendors do or partner with vendor to be able to put these things out into a commercial use through the FDA process that we have in place through a process called certify AI and then assess AI afterward and the goal as I say is to allow this to be another one of the instruments that we all have to be able to use ai as a appropriate either developed by others mouse to verify or possibly even developed by herself for in use inside of our own institutions and just to mention for those that are interested in more of this we have a symposium that's running on October 5th and 6th of this year here in DC where we'll go deeper into the use of all of this technology and it'll I'm sure it will have advanced quite a bit by then six months in this field is like a hundred years so thank you very much [Applause]

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *