External validation of a top-ranked model from the RSNA pulmonary embolism detection challenge: assessment of generalizability.
Authors
Affiliations (9)
Affiliations (9)
- Institute and Polyclinic for Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany. [email protected].
- Helmholtz AI / Helmholtz-Zentrum Dresden-Rossendorf, Department of Information Services and Computing, Dresden, Germany.
- Institute and Polyclinic for Diagnostic and Interventional Radiology, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TUD Dresden University of Technology, Fetscherstraße 74, 01307, Dresden, Germany.
- Else Kroener Fresenius Center for Digital Health, TU Dresden, Dresden, Germany.
- OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TU Dresden and Helmholtz-Zentrum Dresden-Rossendorf, Dresden, Germany.
- National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TU Dresden and Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany.
- Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus Dresden, TU Dresden, Dresden, Germany.
- German Cancer Consortium (DKTK), Partner Site Dresden, and German Cancer Research Center (DKFZ), Heidelberg, Germany.
- Helmholtz-Zentrum Dresden-Rossendorf, Institute of Radiooncology - OncoRay, Dresden, Germany.
Abstract
To externally validate the RSNA 2020 challenge 2<sup>nd</sup>-place deep-learning (DL) algorithm for detecting pulmonary embolism (PE) on computed tomography pulmonary angiography (CTPA) scans with additional focus on subsegmental-only PEs (SSPE). Between 2015 and 2022 1,038 CTPAs were retrospectively enrolled. An experienced radiologist (> five years) labelled CTPAs for the presence, location (central, lobar and segmental, subsegmental-only) and side (right-sided, left-sided, bilateral) of PE. The DL model was tested for its ability to predict these labels by analyzing accuracy, sensitivity, specificity and area under the receiver operating characteristic curve (AUROC) using different model outcome probabilities for each analysis. Overall, 136 central, 375 peripheral PEs (303 lobar and segmental PEs, 72 SSPEs), and 527 patients without any PE were analyzed. The model correctly predicted the presence of any PE in 921/1,038 patients (88.7%), yielding a sensitivity of 0.80, specificity of 0.97 and AUROC of 0.94. No central PE was missed, whereas 100/375 (26.7%) peripheral PEs remained undetected. Using the corresponding model output probability, the central PE status was correctly identified in 999/1,038 (96.2%) patients, while peripheral PE was correctly identified in 785/1,038 (75.6%) patients. This corresponded to sensitivities of 0.94 and 0.77, specificities of 0.97 and 0.75, and AUROC values of 0.99 and 0.74, respectively. The model performed better in detecting right-sided compared to left-sided PEs (AUROC: right- vs. left-sided: 0.95 and 0.92, p < 0.05). SSPE status was correctly predicted in 796/1,038 (76.7%) patients, yielding a sensitivity of 0.94, a specificity of 0.22, and an AUROC of 0.48, respectively. The model demonstrated high performance in detecting any PE, performing best on central and slightly lower on lobar and segmental PE. Predicting the presence of SSPEs was difficult, likely due to the model not being explicitly trained for this subtype.