A multimodal cross-temporal fusion system for complementary evaluation of neoadjuvant chemotherapy in breast cancer.
Authors
Affiliations (11)
Affiliations (11)
- Breast Disease Diagnosis and Treatment Center, The Affiliated Hospital of Qingdao University, Qingdao 266000, Shandong Province, China.
- School of Computer Science and Technology, Tongji University, Shanghai 201840, China.
- Department of Interventional Radiology, The Affiliated Hospital of Qingdao University, Qingdao 266000, Shandong Province, China.
- Department of Radiology, Qingdao Women and Children's Hospital, Qingdao 266000, Shandong Province, China.
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan 430000, Hubei Province, China.
- Department of Radiology, The Affiliated Hospital of Qingdao University, Qingdao 266000, Shandong Province, China.
- Department of Breast Surgery, The Affiliated Hospital of Jining Medical University, Jining 272029, Shandong Province, China.
- Department of Medical Imaging, The Affiliated Hospital of Jining Medical University, Jining 272029, Shandong Province, China.
- First Department of General Surgery, Weihaiwei People's Hospital, Weihai 264200, Shandong Province, China.
- Breast Disease Diagnosis and Treatment Center, The Affiliated Hospital of Qingdao University, Qingdao 266000, Shandong Province, China. Electronic address: [email protected].
- Breast Disease Diagnosis and Treatment Center, The Affiliated Hospital of Qingdao University, Qingdao 266000, Shandong Province, China. Electronic address: [email protected].
Abstract
Neoadjuvant chemotherapy (NAC) has been established as a standard treatment for breast cancer. We aimed to develop a deep-learning multimodal system using longitudinal cross-temporal dynamic contrast-enhanced MRI (DCE-MRI) and clinical data to provide complementary evaluation of NAC. Here, we developed a multimodal automatic NAC assistance system (MANAS), which leverages a dual-pathway attention-based multi-scale feature fusion framework and a multicenter dataset. This dataset comprises pre- and post-NAC DCE-MRI and associated clinical information of 1,515 patients with locally advanced breast cancer. The performance of MANAS was evaluated using various metrics. Confidence-based decision criteria were employed to defer low-confidence cases to clinicians. MANAS consisted of three modules from breast cancer diagnosis to follow-up treatment, designed for predicting molecular subtype, NAC response, and residual tumor burden, respectively. MANAS achieved area under the curves (AUCs) of 0.8066 (95 % CI: 0.7552-0.8599), 0.7748 (95 % CI: 0.7210-0.8275) and 0.7661 (95 % CI: 0.6980-0.8252) on the internal, pooled external and I-SPY2 test sets for molecular subtype prediction; AUCs of 0.8400 (95 % CI: 0.7903-0.8896) and 0.8299 (95 % CI: 0.7886-0.8687) on the internal and pooled external test sets for NAC response prediction; and AUCs of 0.8885 (95 % CI: 0.7891-0.9657), 0.8628 (95 % CI: 0.8047-0.9132) and 0.8695 (95 % CI: 0.7856-0.9432) on the internal, pooled external and I-SPY2 test sets for residual tumor burden prediction. Confidence-based decision criteria could enhance their predictive performance. Our MANAS serves as a valuable complementary tool, providing longitudinal supportive evidence throughout the NAC process, from initial diagnosis to surgical intervention.