Demonstrating the False-Positive Paradox in Radiology with Artificial Intelligence: Estimating Clinically Anticipated PPV, FDR, NPV, and FOR of FDA-Cleared Devices Using Performance Metrics from 510(k) Summaries and Prevalences from Local and Published Data.
Authors
Affiliations (12)
Affiliations (12)
- SEI (Systems Evolution, Inc.), Philadelphia, USA.
- Faculty of Medicine, University of Freiburg, Freiburg im Breisgau, Germany.
- The Warren Alpert Medical School, Brown University, Providence, USA.
- vRad® (Virtual Radiologic), Edina, USA.
- Department of Radiology, Penn State Milton S. Hershey Medical Center, Hershey, USA.
- American College of Radiology Data Science Institute, Reston, VA, USA.
- Palo Alto VA Medical Center, Palo Alto, CA, USA.
- The American College of Radiology, Reston, USA.
- Department of Radiology, Mayo Clinic/Mayo College of Medicine and Science, Rochester, USA.
- The Warren Alpert Medical School, Brown University, Providence, USA. [email protected].
- Department of Radiology, Brown University Health, Brown University, Providence, USA. [email protected].
- Brown University Health AI Center of Excellence, Brown University Health, Providence, RI, USA. [email protected].
Abstract
The false-positive paradox (FPP) describes the counterintuitive phenomenon whereby a test can achieve high sensitivity and specificity yet still produce mostly false-positive results when applied to low-prevalence populations. This phenomenon is particularly relevant to radiology artificial intelligence (AI), where devices are typically marketed and cleared using diagnostic accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (ROC-AUC). However, these metrics do not indicate the proportion of false-positive and false-negative outputs clinicians should expect in local practice. This study estimated the positive predictive value (PPV), false discovery rate (FDR), negative predictive value (NPV), and false omission rate (FOR) for recent Food and Drug Administration (FDA)-cleared radiology AI devices by combining publicly reported 510(k) sensitivity and specificity with local or literature-derived prevalence estimates. Public FDA summaries for 38 radiology AI devices authorized in 2024 or 2025 were reviewed, yielding 57 target pathology entries. Median reported sensitivity, specificity, and ROC-AUC exceeded 90%; nevertheless, clinically anticipated FDRs were often high. For example, a large vessel occlusion detector with 90.6% sensitivity and 88.8% specificity was estimated to have a PPV of 29.3% and an FDR of 70.7% at a prevalence of 4.87%, meaning most positive AI alerts would be false positives. These findings do not evaluate device performance but instead illustrate the FPP in radiology AI. Public 510(k) summaries should consistently report sensitivity, specificity, and test set prevalence so that radiologists and other stakeholders can estimate PPV, FDR, NPV, and FOR for their own patient populations before implementation.