Deep learning multimodal fusion model based on ultrasound images and clinical-ultrasound texts for differentiating benign and malignant renal tumors.
Authors
Affiliations (7)
Affiliations (7)
- Shanghai Institute of Infectious Disease and Biosecurity, Fudan University, Shanghai, China.
- Department of Ultrasound, Zhongshan Hospital, Institute of Ultrasound in Medicine and Engineering, Fudan University, Shanghai, China.
- MedAI Technology (Wuxi) Co., Ltd. Wuxi, China.
- Department of Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, China.
- Shanghai Institute of Imaging Medicine, Shanghai, China.
- State Key Laboratory of Integrated Chips and System, Fudan University, Shanghai, China.
- Data Science in Earth Observation, Technical University of Munich, Munich, Germany.
Abstract
To develop a deep learning multimodal fusion model based on ultrasound (US) images and clinical-US texts to differentiate benign and malignant renal tumors. The retrospective dataset from January 2012 to February 2023 in a tertiary hospital was used for model development and randomly allocated to training and validation sets. An independent prospective test set was obtained from March 2023 to October 2024. We developed Model-G using grayscale US (GUS) images, Model-GC using GUS and color Doppler flow imaging images, Model-T using clinical-US texts, and Model-UT using US images and clinical-US texts. ResNet50 was used for image feature extraction, BERT for text encoding, and ViT for multimodal classification. In total, 1008 patients (740 in training set, 186 validation set, and 82 prospective test set) with a single renal tumor were included. In the validation set, Model-UT achieved the highest diagnostic performance in comparison to Model-G, Model-GC, and Model-T with an area under the receiver operating characteristic curve (AUC) of 0.89 (95% CI, 0.86-0.93) (all p-values < 0.05). In the prospective test set, Model-UT yielded an AUC of 0.92 (95% CI, 0.86-0.97), outperforming high-level, mid-level, and low-level radiologists (all p-values < 0.05). The multimodal fusion model presents promising potential in differentiating benign and malignant renal tumors. Deep learning model integrating US images with clinical-US texts improved benign-versus-malignant renal tumor differentiation. Decision-curve, controlled ablation, and SHAP analyses support its potential use as a clinical decision-support tool.