A clinical multimodal vision language foundation model with fine-grain explainability.
Authors
Affiliations (44)
Affiliations (44)
- Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore, Singapore.
- Singapore Eye Research Institute, Singapore National Eye Center, Singapore, Singapore.
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore.
- School of Computer Science, Wuhan University, Wuhan, China.
- Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford, UK.
- International Centre for Eye Health, London School of Hygiene & Tropical Medicine, London, UK.
- Department of Anesthesia and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore.
- Department of Ophthalmology, Faculty of Medicine and Graduate School of Medicine, Hokkaido University, Sapporo, Japan.
- Department of Inflammation and Ageing, College of Medicine and Health, University of Birmingham, Birmingham, UK.
- Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK.
- Byers Eye Institute, Stanford University School of Medicine, Palo Alto, CA, USA.
- Cumming School of Medicine, University of Calgary, Calgary, Alberta, Canada.
- Massachusetts Eye and Ear, Department of Ophthalmology, Harvard Medical School, Boston, MA, USA.
- SingHealth AI Office, Singapore, Singapore.
- Cleveland Clinic Cole Eye Institute, Cleveland, OH, USA.
- Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, OH, USA.
- Heart, Vascular and Thoracic Institute, Cleveland Clinic, Cleveland, OH, USA.
- Department of Ophthalmology, National University Health System, Singapore, Singapore.
- Centre for Innovation and Precision Eye Health, Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
- Department of Ophthalmology, National Healthcare Group, Singapore, Singapore.
- Lee Kong Chien School of Medicine, Nanyang Technology University, Singapore, Singapore.
- SERI-NTU Advanced Ocular Engineering (STANCE) Program, Singapore, Singapore.
- School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore, Singapore.
- Department of Clinical Pharmacology, Medical University of Vienna, Vienna, Austria.
- Aier Hospital Group, Changsha, China.
- Fondation Ophtalmologique Adolphe De Rothschild, Paris, France.
- Center for Medical Physics and Biomedical Engineering, Medical University Vienna, Vienna, Austria.
- Department of Ophthalmology and Visual Science, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Korea.
- Department of Cardiothoracic and Abdominal Radiology, Singapore General Hospital, Singapore, Singapore.
- SingHealth Community Hospitals, Singapore Health Services, Singapore, Singapore.
- Department of Anatomical Pathology, Singapore General Hospital, Singapore, Singapore.
- School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Tsinghua Medicine, Tsinghua University, Beijing, China.
- Beijing Visual Science and Translational Eye Research Institute, Beijing Tsinghua Changgung Hospital Eye Center, Tsinghua Medicine, Tsinghua University, Beijing, China.
- School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China.
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore.
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore.
- NUS AI Institute, National University of Singapore, Singapore, Singapore.
- Division of Medical Oncology, National Cancer Centre Singapore, Singapore, Singapore.
- Genome Institute of Singapore, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore.
- Generative AI Research Program, Icahn School of Medicine at Mount Sinai, New York, NY, USA.
- Singapore Eye Research Institute, Singapore National Eye Center, Singapore, Singapore. [email protected].
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore. [email protected].
- Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK. [email protected].
- Massachusetts Eye and Ear, Department of Ophthalmology, Harvard Medical School, Boston, MA, USA. [email protected].
Abstract
Despite their considerable promise, many current medical foundation models remain modality specific and lack transparent reasoning, limiting clinical adoption. To address this gap, we introduce Explainable Vision Language Foundation Model for Medicine (EVLF-FM), a multimodal vision language foundation model that combines broad diagnostic capabilities with fine-grained explainability. The development and testing of EVLF-FM encompassed more than 1.3 million total samples from 18 global datasets across 10 imaging modalities related to six clinical specialties: dermatology, hepatology, ophthalmology, pathology, pulmonology and radiology. External validation employed 13,443 independent test samples from 17 additional datasets across 10 imaging modalities. EVLF-FM supports disease diagnosis, visual question answering and pixel-level visual grounding with reasoning capabilities. In internal validation for disease diagnostics, EVLF-FM achieved the highest average accuracy, 0.895, outperforming leading generalist and specialist models. In medical visual grounding, EVLF-FM also achieved reasonable performance across nine modalities with average accuracy at 0.5 intersection over union ([email protected]) of 0.842. External testing further confirmed consistent zero-shot and few-shot performance, with competitive F1 scores despite a smaller model size. A hybrid training strategy combining supervised and visual reinforcement fine-tuning enables EVLF-FM to generate clinically interpretable explanations aligned with visual evidence. Human evaluation of these reasoning outputs using the S.C.O.R.E. framework yielded a high overall mean score of 4.25/5, supporting their clinical interpretability. Current limitations include the need for additional validation across diverse healthcare settings. Prospective clinical studies are required to evaluate the safety, effectiveness and clinical utility of EVLF-FM before routine clinical deployment.