AI Monitoring AI with LLMs: The American College of Radiology's Imaging AI Registry.
Authors
Affiliations (12)
Affiliations (12)
- Senior Data Scientist, American College of Radiology, Reston, VA, USA.
- Chief Radiology Resident Department of Radiology, Brigham & Women's Hospital, Harvard Medical School, Boston, Massachusetts, USA, Data Science Institute Scientist, American College of Radiology, Reston, VA, USA.
- Radiology Resident, Department of Medical Imaging, McMaster University, Hamilton, Ontario, CA, Data Science Institute Scientist, American College of Radiology, Reston, VA, USA.
- Radiology Resident, Department of Radiology, University of South Florida, Morsani College of Medicine, Tampa, Florida, USA, Data Science Institute Scientist, American College of Radiology, Reston, VA, USA.
- Radiologist, Department of Radiology, Mayo Clinic, Rochester, Minnesota, USA, Data Science Institute Scientist, American College of Radiology, Reston, VA, USA.
- Associate Director of Artificial Intelligence, Mosaic Clinical Technologies, Nashville, TN, USA, Data Science Institute Scientist, American College of Radiology, Reston, VA, USA.
- Data Science and Informatics, American College of Radiology, Reston, VA.
- Chief Medical Officer, Data Science Institute, Reston, VA, USA, Chief Strategy officer & Chief Medical Information Officer, HOPPR, Chicago, IL, USA, MSK Radiologist, VA Medical Center, Palo Alto, CA, USA.
- Vice Chair ACR Commission on Informatics, Reston, VA, USA, Associate Professor and Vice Chair of Practice Transformation, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA.
- Chief Science Officer, Data Science Institute, Reston, VA, USA, Chief Data Science Officer, Vice Chairman of Radiology, Mass General Brigham, Boston, MA, USA.
- Vice Chair, ACR Board of Chancellors; Chair ACR Commission on Informatics, Reston, VA, USA, Professor of Radiology & SA Consultant Radiologist, Department of Radiology, Mayo Clinic, Rochester, MN, USA.
- Associate Chief Science Officer, Data Science Institute, Reston, VA, USA, Senior Director, Mass General Brigham, Boston, MA, USA.
Abstract
To describe the technical workflow enabling scalable automated artificial intelligence (AI) monitoring in the first national imaging AI registry, Assess-AI. Large language model (LLM) prompts are developed to extract clinically relevant findings from radiology reports through collaboration between data scientists and subspecialty radiologists. Prompts are optimized using tuning cohorts of use case-specific radiology reports and LLMs available through AWS Bedrock. Such cohorts are used to evaluate prompt accuracy and consistency across 10 repeated runs. Report-AI result pairs submitted to the Assess-AI registry for actively monitored use cases are additionally used to further optimize corresponding prompts. Prompts were developed for nine use cases. In Stage 2 development cohorts, final-prompt agreement with hybrid report-derived reference standard labels was 0.985 for ICH and 0.997 for PE. Because these cohorts informed prompt refinement and label construction, they were not independent validation sets. The workflow demonstrates feasible report-finding extraction at scale; independent accuracy and clinical utility remain unestablished. LLM-based extraction within Assess-AI enables scalable, report-anchored AI performance monitoring in radiology.