Evaluation of artificial intelligence algorithms from Lunit for detection of breast cancer with mammogram in a South-East Asian population.
Authors
Affiliations (2)
Affiliations (2)
- Department of Diagnostic and Interventional Imaging, KK Women's and Children's Hospital, Singapore.
- Division of Clinical Support Services, KK Women's and Children's Hospital, Singapore.
Abstract
PurposeTo conduct a pre-clinical evaluation of a commercially available artificial intelligence system, Lunit AI, as an independent mammography reader and compare its performance with human readers.Materials and MethodsThis retrospective study analyzed a double-reader population-based mammographic screening cohort of women at an academic women's hospital in Singapore from 2010 to 2017. Lunit AI performance was compared with single-reader (SRR) and double-reader radiologist (DRR) assessment using AUC-ROC, specificity, sensitivity and accuracy.Results1,982 women 35-87 years without breast implants or prior breast cancer were included. The cohort comprised 193 histopathologically confirmed breast cancers, 1,299 healthy controls and 490 histopathologically proven benign recalls. Mean age was 55.7 ± 9.7 years for with cancer and 52.9 ± 8.5 years without cancer.Lunit AI achieved an AUC-ROC of 0.874 (95% CI: 0.843-0.905) compared with 0.811 (95% CI: 0.783-0.840) for SRR assessment and 0.863 (95% CI: 0.846-0.880) for DRR assessment. Compared with SRR, Lunit AI demonstrated higher specificity (81.0% vs 72.6%, p < 0.001), lower sensitivity (80.3% vs 89.6%, p = 0.013) and higher accuracy (80.9% vs 74.3%, p < 0.001). Compared with DRR, Lunit AI demonstrated higher specificity (81.0% vs 72.6%, p < 0.001), lower sensitivity (80.3% vs 100%, p < 0.001) and higher accuracy (80.9% vs 75.3%, p < 0.001). The apparent 100% sensitivity of DRR reflects the retrospective study design and should not be interpreted as true screening sensitivity. Lunit AI detected 156 of 193 cancers and missed 34 cancers of which the majority of the missed cases were 22 cases (64.7%) of microcalcifications.ConclusionLunit AI model shows promising performance in a Southeast Asian population with high AUC-ROC, specificity and accuracy. However it shows lower sensitivity for subtle microcalcifications. These results suggest that Lunit AI may not be suitable as a fully independent reader but as an adjunct to enhance existing clinical workflows in mammographic screening.