MerMED-FM: Multimodal, Multi-Disease Medical Imaging Foundation Model.
Authors
Affiliations (24)
Affiliations (24)
- School of Computer Science, Wuhan University, Wuhan, China; Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A∗STAR), Singapore.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore.
- Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A∗STAR), Singapore.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore; Artificial Intelligence Office, Singapore Health Services, Singapore.
- Department of Anesthesiology, Singapore General Hospital, Singapore; Duke-NUS Medical School, Singapore.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore; Ophthalmology and Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore; Academic Unit of Ophthalmology, Institute of Inflammation and Ageing, College of Medical and Health, University of Birmingham, Birmingham, UK; Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK; Academic Ophthalmology, School of Medicine, University of Nottingham, Nottingham, UK.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore; Centre for Innovation and Precision Eye Health and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore.
- Byers Eye Institute, Stanford University School of Medicine, Palo Alto, CA, USA.
- Cumming School of Medicine, University of Calgary, Calgary, AL, Canada.
- Massachusetts Eye and Ear, Department of Ophthalmology, Harvard Medical School, Boston, MA, USA.
- Artificial Intelligence Office, Singapore Health Services, Singapore.
- Lee Kong Chien School of Medicine, Nanyang Technology University, Singapore.
- Lee Kong Chien School of Medicine, Nanyang Technology University, Singapore; Department of Ophthalmology, National Healthcare Group, Singapore.
- Department of Rheumatology, Allergy, and Immunology, Chang Gung Memorial Hospital, Taipei, Taiwan; Center for Artificial Intelligence in Medicine, Chang Gung Memorial Hospital, Taipei, Taiwan; School of Medicine, Chang Gung University, Taoyuan, Taipei, Taiwan.
- Department of Ophthalmology, Chang Gung Memorial Hospital, Linkou, Taiwan; College of Medicine, Chang Gung University, Taoyuan, Taiwan.
- Department of Ophthalmology, Jen-Ai Hospital, Taichung, Taiwan; Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei, Taiwan.
- Duke-NUS Medical School, Singapore; Department of Cardiothoracic and Abdominal Radiology, Singapore General Hospital, Singapore; SingHealth Community Hospitals, Singapore.
- Duke-NUS Medical School, Singapore; Department of Cardiothoracic and Abdominal Radiology, Singapore General Hospital, Singapore.
- Department of Anatomical Pathology, Singapore General Hospital, Singapore.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore; Beijing Visual Science and Translational Eye Research Institute (BERI), Beijing Tsinghua Changgung Hospital Eye Center, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University, Beijing, China; School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China.
- Centre for Biomedical Data Science, Duke-NUS Medical School, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore; NUS AI Institute, National University of Singapore, Singapore.
- Duke-NUS Medical School, Singapore; National Cancer Centre, Singapore; Genome Institute of Singapore, Agency for Science, Technology and Research (A∗STAR), Singapore.
- Byers Eye Institute, Horngren Family Vitreoretinal Center, Department of Ophthalmology, Stanford University School of Medicine, Palo Alto, CA, USA.
- Singapore National Eye Centre, Singapore Eye Research Institute, Singapore; Artificial Intelligence Office, Singapore Health Services, Singapore; Duke-NUS Medical School, Singapore; Ophthalmology and Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore. Electronic address: [email protected].
Abstract
Current artificial intelligence (AI) models for medical imaging predominantly focus on a single imaging modality and a single disease. Attempts to create multimodal and multi-disease models have resulted in inconsistent clinical accuracy. Furthermore, training these models typically requires large, well labelled datasets, which are costly and labour intensive to prepare. We aimed to train and evaluate an AI model that can interpret diverse imaging modalities across specialties while maintaining robust performance within each modality. We developed Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM), a multi-specialty model trained using self-supervised learning and a memory module. MerMED-FM was pretrained on publicly sourced, unlabelled medical images from 12 specialties and seven imaging modalities: chest x-rays, CT, ultrasound, histopathology, colour fundus photography (CFP), optical coherence tomography (OCT), and dermatoscopy. After pretraining, the model was fine-tuned, validated, and evaluated for the diagnosis of a range of diseases on 26 public datasets and five private datasets comprising radiology, histopathology, and ophthalmology images. MerMED-FM was compared against a general-domain vision foundation model, various specialist single-modality foundation models, and a multispecialty foundation model. Models were fine-tuned using 10%, 30%, 50%, and 100% of data, with primary comparative analyses conducted using a 10% label fraction. The primary outcome was the area under the receiver operating characteristic curve (AUROC), which was summarised by imaging modality. MerMED-FM was trained on around 3·3 million images from 53 publicly available, unlabelled datasets, comprising 713 931 chest x-rays, 292 353 CT slices, 389 885 ultrasound frames, 1 017 712 pathology patches, 333 099 CFP images, 176 719 OCT slices, and 401 059 dermatoscopy images. Strong performance was achieved across all modalities at a label fraction of only 10%, with mean AUROC values of 0·844 for chest x-rays, 0·906 for CT, 0·818 for ultrasound, 0·908 for histopathology, 0·810 for CFP, 0·962 for OCT, and 0·827 for dermatoscopy. MerMED-FM has the potential to be a highly adaptable, versatile, cross-specialty foundation model that enables robust interpretation of medical imaging across diverse medical disciplines. National Medical Research Council, Singapore and the Agency for Science, Technology and Research, Singapore.