CardioCLIP: sequential multimodal prediction of echocardiographic abnormalities and natriuretic peptide elevation from electrocardiogram and chest radiographs.
Authors
Affiliations (5)
Affiliations (5)
- Kobe University School of Medicine, Kobe, Japan.
- Division of Molecular Epidemiology, Kobe University Graduate School of Medicine, 7-5-2, Kusunoki-cho, Chuo-ku, Kobe 650-0017, Japan.
- Division of Cardiovascular Medicine, Department of Internal Medicine, Kobe University Hospital, Kobe, Japan.
- Division of Medical Informatics, Department of Internal Related, Kobe University Graduate School of Medicine, Kobe, Japan.
- Faculty of Data Science, Shiga University, Shiga, Japan.
Abstract
Diagnostic reasoning in clinical practice is inherently sequential and multimodal, with physicians integrating heterogeneous examinations under constraints such as missing data and variable test order. However, most multimodal machine learning models assume simultaneous availability of all modalities and fail to reflect this workflow. We propose CardioCLIP, a sequential multimodal framework that embeds electrocardiograms (ECGs) and chest X-rays (CXRs) into a shared pathological representation space. The model consists of (i) a self-supervised embedding stage aligning ECG-CXR pairs from the same patient and day, and (2) a gated recurrent unit-based prediction stage that estimates clinical indicators as data are sequentially observed. The model was trained on 9458 patients from a single tertiary centre and evaluated for B-type natriuretic peptide (BNP), left ventricular ejection fraction (LVEF), and <i>E</i>/<i>e</i>'. External validation was conducted using MIMIC-IV.CardioCLIP achieved strong cross-modal alignment and improved discrimination for selected echocardiographic abnormalities and natriuretic peptide elevation (AUC: 0.90 for BNP > 100 pg/mL, 0.87 for LVEF < 50%, and 0.80 for <i>E</i>/<i>e</i>' > 15). It also showed discrimination for subsequent severe BNP elevation (BNP > 400 pg/mL within 3 years) with an AUC of 0.79. External validation in MIMIC-IV showed an AUC of 0.89 for N-terminal pro-BNP elevation, supporting potential transportability for this endpoint. CardioCLIP demonstrated the feasibility of order-flexible sequential multimodal prediction. By leveraging routinely acquired ECG and CXR data, it may provide a non-invasive adjunct for identifying echocardiographic abnormalities and natriuretic peptide elevation in cardiovascular care, subject to prospective clinical validation.