Foundation Models and Multimodal Learning for Medical Data with Applications in Oncology.
Authors
Affiliations (15)
Affiliations (15)
- Department of Machine Learning, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA. [email protected].
- Department of Cancer Epidemiology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA. [email protected].
- Department of Machine Learning, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA.
- Department of Molecular and Medical Pharmacology, University of California, Los Angeles, Los Angeles, CA, USA.
- Department of Radiology, Mayo Clinic, Rochester, MN, 55905, USA.
- Departments of Radiology and Medical Physics, University of Wisconsin School of Medicine and Public Health, Madison, WI, 53705, USA.
- Department of Computer Science, Vanderbilt University, Nashville, TN, USA.
- Department of Information Engineering, University of Pisa, Via Girolamo Caruso 16, 56122, Pisa, Italy.
- Department of Pathology, University of Pittsburgh, Pittsburgh, PA, 15232, USA.
- MedAiConsult, Cleveland, OH, 44120, USA.
- Department of Diagnostic Radiology & Nuclear Medicine, University of Maryland Medical Center, Baltimore, MD, USA.
- Department of Pathology & Cell Biology, University of South Florida Health, Tampa, FL, USA.
- Bellini College of Artificial Intelligence, Cybersecurity and Computing, University of South Florida, Tampa, FL, USA.
- Morsani College of Medicine, College of Engineering - Department of Medical Engineering, University of South Florida, Tampa, FL, USA.
- Department of Cancer Epidemiology, H. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA.
Abstract
The rapid expansion of medical data, including radiological imaging, digital pathology, molecular profiling, electronic health records, and longitudinal clinical narratives, has created new opportunities for AI in healthcare, while also exposing the limitations of traditional unimodal modeling approaches. Multimodal learning addresses these challenges by integrating heterogeneous data sources, and recent advances in foundation models, large-scale pretrained models designed for broad task transfer, have further accelerated this shift. Together, these developments have given rise to multimodal foundation models that aim to learn generalizable, aligned representations across diverse clinical modalities. In this review, we provide an overview of multimodal foundation models in medical AI, with a particular focus on oncology. We first outline key medical data modalities and the machine learning objectives that motivate multimodal integration, including predictive, representational, and alignment-based goals. We then examine core architectural design patterns and training paradigms underlying multimodal foundation models, such as modality-specific encoders, fusion strategies, attention mechanisms, contrastive pretraining, and scalable self-supervised learning. Building on this foundation, we survey representative multimodal models across radiology, pathology, clinical text, and molecular domains. We further review applications of multimodal foundation models across the cancer care continuum, including early detection, diagnosis and subtyping, prognosis, therapy response prediction, and longitudinal survival modeling. Finally, we discuss critical challenges that limit clinical translation, including cross-institutional heterogeneity, temporal misalignment across modalities, weak supervision, interpretability, regulatory constraints, and workflow integration, and outline future directions for advancing safe and effective deployment in real-world oncology practice.