An Interpretable Multimodal Deep Learning Framework Integrating MRI and Clinical Data via Bidirectional Cross-Modal Attention for Improved Cardiac Disease Detection.
Authors
Affiliations (7)
Affiliations (7)
- Department of Radiology and Medical Imaging, College of Medicine, King Saud University, King Saud University Medical City, Riyadh, Saudi Arabia.
- Department of Diagnostic Radiology, University of Tabuk, Tabuk, Saudi Arabia.
- Department of Radiology, King Abdulaziz University Hospital, Faculty of Medicine, King Abdulaziz University, Jeddah, Saudi Arabia.
- Department of Diagnostic Radiography Technology, College of Nursing and Health Science, Jazan University, Jazan, Saudi Arabia.
- Department of Internal Medicine, Division of Radiology, College of Medicine, Jouf University, Sakaka, Saudi Arabia.
- Department of Otolaryngology, Division of Facial Plastic, King Saud University, P.O. Box 245, 11411, Riyadh, Saudi Arabia.
- Department of Clinical Laboratory Sciences, College of Applied Medical Sciences, Shaqra University, Shaqra, Saudi Arabia. [email protected].
Abstract
Cardiovascular diseases continue to be the primary cause of mortality worldwide; however, precise diagnosis is often limited by dependence on individual data modalities and the unclear nature of traditional deep learning models. In this study, we introduce an explainable multimodal deep learning framework that simultaneously analyzes cardiac magnetic resonance imaging (MRI) and structured clinical variables through a bidirectional cross-modal attention mechanism that operates on token sequences. The architecture utilizes modality-specific encoders ResNet-50 for imaging and a multilayer perceptron for clinical data interconnected through attention that dynamically aligns complementary information, reducing feature dilution and modal dominance. A multitask objective that includes classification, contrastive alignment (InfoNCE), and regularization further reinforces semantic consistency across modalities. Integrated post hoc explainability modules, such as Gradient-weighted Class Activation Mapping for spatial localization and SHapley Additive exPlanations for clinical feature attribution, offer transparent decision support. Tested on a cohort of 12,847 patients using fivefold cross-validation, the framework achieved an accuracy of 94.2% and an area under the receiver-operating-characteristic curve of 0.97, significantly surpassing unimodal and traditional multimodal baselines (p < 0.01). The explanations generated showed 91% concordance with independent cardiologist evaluations. The findings presented herein were derived through internal five-fold cross-validation conducted at a solitary tertiary care institution, thereby substantiating internal validity while limiting multicenter generalizability; consequently, external validation is requisite for broader applicability. These results demonstrate that attention-driven multimodal fusion, combined with post hoc explainability, provides both strong diagnostic performance and clinically useful transparency, positioning the framework as a promising internally validated approach and a foundation for future external validation.