Prospective Shadow-Mode Evaluation of an Artificial Intelligence Tool for Intracranial Aneurysm Detection on CT Angiography: Incremental Yield and Operational Impact.
Authors
Affiliations (12)
Affiliations (12)
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Director of Artificial Intelligence, Department of Radiology, Northwell Health, New Hyde Park, New York; Chair, Quality and Safety Committee, New York State Radiological Society; Co-Chair, ACR Recognized Center for Healthcare-AI (ARCH-AI) Learning Community Committee. Electronic address: [email protected].
- Northwell Health, New Hyde Park, New York; Institute of Health System Science at The Feinstein Institutes for Medical Research, Manhasset, New York.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York.
- Chief of Radiology, South Shore University Hospital.
- Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Frank G. Zarb School of Business, Hofstra University, Hempstead, New York.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Interim Chairman, Department of Radiology, NuVance West; Associate Chairman, Department of Radiology, Staten Island University Hospital/Northwell Health.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Associate Chief of Informatics, Northwell Health, Department of Radiology, New Hyde Park, New York; Vice Chair, Quality and Safety Committee, New York State Radiological Society.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Chief of Informatics, Northwell Health, Department of Radiology, New Hyde Park, New York.
- Executive Director, Harvey L. Neiman Health Policy Institute, Reston, Virginia.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Chair, Department of Radiology, Zucker School of Medicine at Hofstra University; Executive Vice President and Chief Learning + Innovation Officer, Northwell Health, New Hyde Park, New York.
- Northwell Health, New Hyde Park, New York; Department of Radiology, Zucker School of Medicine at Hofstra/Northwell, Hempstead, New York; Vice President & Chief Medical Information Officer (CMIO), Northwell Enterprise Clinical Shared Services; Vice Chair, Radiology Informatics, Department of Radiology, Northwell Health, New York, New York.
- Northwell Health, New Hyde Park, New York; Institute of Health System Science at The Feinstein Institutes for Medical Research, Manhasset, New York; Vice Chair of Research, Department of Radiology, Northwell Health; Associate Dean for Student Affairs, Zucker School of Medicine at Hofstra University.
Abstract
To evaluate the operational performance of an FDA-cleared artificial intelligence (AI) algorithm for brain CT angiography (CTA) aneurysm detection and the incremental value of combining AI with radiologists. Prospective shadow-mode study of consecutive brain CTAs (November 7, 2023, to December 19, 2023) was performed with radiologists blinded to AI. Aidoc's (Tel Aviv, Israel) AI algorithm processed CTAs, and natural language processing extracted radiology report results. Discordances underwent neuroradiologist adjudication with AI unblinding. Performance metrics included AI:radiologist incremental detection ratio, relative enhanced detection rate (rEDR), gain-to-pain ratio (GPR), and number-needed-to-examine (NNE). Performance metrics are estimated under a hybrid reference standard wherein only discordances are adjudicated. Among 3,856 CTAs, examination positive rate was 5.1% (195 of 3,856). Radiologist-AI concordance was 96.3% (3,714 of 3,856). Sensitivity of AI alone (0.846, 0.787-0.894) exceeded radiologist alone (0.718, 0.649-0.779), with similar specificity (radiologist: 0.985, 0.981-0.989; AI: 0.987, 0.983-0.991). AI surfaced additional aneurysms not identified by radiologists, yielding an rEDR of 39% (55 AI-only true-positives per 140 radiologist true-positives) and projecting a higher combined sensitivity for radiologists with AI. AI:radiologist incremental detection ratio was 1.83 (55 of 30), favorable for AI. Operationally, GPR was favorable at 1.20 (55 of 46), and NNE was 70.1 (3,856 of 55). Operational metrics were most favorable in inpatients (rEDR 78.3%, GPR 2.57, NNE 28.9) and emergency (rEDR 37.1%, GPR 1.0, NNE 85.3) and were unfavorable in outpatients (rEDR 14.1%, GPR 0.67, NNE 130.3). Most AI-only aneurysms were <3 mm (31 of 55, 56.4%); radiologist-only were mostly 3 to 5 mm (18 of 33, 54.5%). The AI tool demonstrated favorable operational performance, supporting clinical deployment in real-world practice, although setting-specific metrics were variable.