Ensemble-based uncertainty enables large-scale 3D segmentation quality control in population imaging.
Authors
Affiliations (22)
Affiliations (22)
- Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany; Pattern Analysis and Learning Group, Department of Radiation Oncology, University Hospital Heidelberg, Germany. Electronic address: [email protected].
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany.
- Institute for Diabetes Research and Metabolic Diseases (IDM) of Helmholtz Munich at the University of Tübingen, Tübingen, Germany; German Center for Diabetes Research (DZD), Neuherberg, Germany.
- HIP Applied Computer Vision Lab, German Cancer Research Center (DKFZ), Heidelberg, Germany.
- Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany; Medical Faculty Heidelberg, Heidelberg University, Heidelberg, Germany.
- Department of Diagnostic and Interventional Radiology, Medical Center - University of Freiburg, Faculty of Medicine - University of Freiburg, Freiburg, Germany.
- Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany.
- Berlin Ultrahigh Field Facility (B.U.F.F.), Max Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Berlin, Germany.
- Team Clinical Epidemiology, Department of Epidemiology (EPID), Helmholtz Centre for Infection Research, Braunschweig, Germany.
- Institute for Medical Epidemiology, Biometrics, and Informatics, Interdisciplinary Center for Health Sciences, Medical Faculty of the Martin Luther University Halle-Wittenberg, Halle (Saale), Germany.
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Augsburg, Augsburg, Germany; Centre for Advanced Analytics and Predictive Sciences, Augsburg University, Augsburg, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Augsburg, Augsburg, Germany.
- Research Group Metabolic Imaging, German Diabetes Center, Leibniz Center for Diabetes Research at Heinrich Heine University Düsseldorf, Germany.
- Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Essen, Germany; Department of Computer Science, University of Applied Sciences and Arts Dortmund, Dortmund, Germany.
- Department of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
- Max-Delbrueck-Center for Molecular Medicine in the Helmholtz Association (MDC), Molecular Epidemiology Research Group, Berlin, Germany; Max-Delbrueck-Center for Molecular Medicine in the Helmholtz Association (MDC), Biobank Technology Platform, Berlin, Germany; Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
- NAKO Study Center Augsburg, University Hospital Augsburg, Augsburg, Germany.
- Institute of Social Medicine, Epidemiology and Health Economics, Charité - Universitätsmedizin Berlin, Berlin, Germany.
- Institute for Community Medicine, SHIP-KEF - Division Quality in the Health Sciences, University Medicine Greifswald, Germany.
- Institute for Diabetes Research and Metabolic Diseases (IDM) of Helmholtz Munich at the University of Tübingen, Tübingen, Germany; German Center for Diabetes Research (DZD), Neuherberg, Germany; Section of Experimental Radiology, Department of Radiology, University Hospital Tübingen, Tübingen, Germany.
- Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany; Pattern Analysis and Learning Group, Department of Radiation Oncology, University Hospital Heidelberg, Germany.
Abstract
Modern Deep-Learning facilitates automatic segmentation in large imaging studies. To prevent segmentation errors from impacting data analysis, automated quality control (QC) for filtering segmentations and identifying errors (segmentation selection) is essential. QC must be fast and require minimal expert delineations for application to various ROIs in large datasets. Uncertainty quantification methods capable of estimating segmentation error without ground truth offer a promising foundation. However, their implicit error estimation may lack accuracy, making suitability for segmentation selection unclear. Unfortunately, commonly employed evaluation measures cannot capture their segmentation selection capability. Furthermore, selecting or rejecting segmentations based on uncertainty alone can be non-trivial. We propose a new method for efficient segmentation QC in large imaging cohorts. We comprehensively evaluate uncertainty-based error estimators with respect to key goals of large-scale QC, specifically assessing segmentation selection capability through the area under the risk-coverage-curve. To optimize selection and rejection, we employ a binary search for decision thresholds, balancing the risk of including errors and unnecessarily rejecting segmentations. In >11 000 MRI liver and lung segmentations in the German National Cohort, error estimation using the pairwise Dice of ensemble outputs achieved excellent QC, including segmentation selection. Numerous manually hard-to-detect errors and abnormalities (n>400) were identified during high-quality result selection, measurably restoring integrity of volume and fat measurements. QC was fast (0.29-4.48s/case) and required minimal annotated image data (n<125). Our method can help tackle the challenge of reliable automated QC in large-scale segmentation studies.