Visual prompt engineering for multimodal and irregularly sampled medical data.
Authors
Affiliations (10)
Affiliations (10)
- Department of Cardiology, Angiology and Pneumology, Heidelberg University Hospital, Heidelberg, Germany. [email protected].
- Informatics for Life Institute, Heidelberg, Germany. [email protected].
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany. [email protected].
- Department of Cardiology, Angiology and Pneumology, Heidelberg University Hospital, Heidelberg, Germany.
- Informatics for Life Institute, Heidelberg, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany.
- Preventive Cardiology and Preventive Medicine, Department of Cardiology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany.
- Clinical Epidemiology and Systems Medicine, Center for Thrombosis and Hemostasis, University Medical Center Mainz, Johannes Gutenberg University Mainz, Mainz, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Rhine-Main, Mainz, Germany.
- Systems Medicine, Institute of Molecular Biology (IMB), Mainz, Germany.
Abstract
A patient undergoes multiple examinations in each hospital stay, where each provides different facets of the health status. These assessments include temporal data with varying sampling rates, discrete single-point measurements, therapeutic interventions such as medication administration, and images. While physicians are able to process and integrate diverse modalities intuitively, neural networks need specific modeling for each modality complicating the training procedure. We demonstrate that this complexity can be significantly reduced by visualizing all information as images along with unstructured text and subsequently training a conventional vision-text transformer. Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), simplifies data preprocessing and modeling by unifying clinical measurements, medications, X-ray images, and electrocardiography scans into a single visual representation. ViTiMM outperforms current state-of-the-art methods in predicting in-hospital mortality, phenotyping, and decompensation on two datasets, the MIMIC-IV and COVID Data for Shared Learning (CDSL) dataset. We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training. The source code is publicly available at https://github.com/Cardio-AI/ViTiMM .