Adapting Segment Anything Model 3 for concept-driven lesion segmentation in medical images: an experimental study.
Authors
Affiliations (4)
Affiliations (4)
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, 75390, USA.
- Medical Image Processing Group, Department of Radiology, University of Pennsylvania, Philadelphia, PA, 19104, USA.
- Division of Digestive and Liver Diseases, Department of Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX, 75390, USA.
- The Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, 75390, USA. [email protected].
Abstract
Accurate segmentation of lesions is fundamental for medical image analysis. However, most existing approaches are designed for specific anatomical sites and imaging modalities, limiting their applicability in heterogeneous clinical environments. Recent vision-language foundation models have enabled concept-driven segmentation of natural images, offering a promising path toward more flexible and generalizable medical image segmentation. However, to the best of the authors' knowledge, concept-prompt-based lesion segmentation in medical images, particularly using the latest Segment Anything Model 3 (SAM3), has not been thoroughly examined. In this paper, the authors present a systematic investigation of the SAM3 framework for lesion segmentation. An extensive evaluation of SAM3 was first conducted using geometric bounding boxes and concept-based text and image exemplars as prompts for lesion segmentation across multiple cancer types and imaging modalities, including magnetic resonance imaging, computed tomography, ultrasound, dermoscopy, and endoscopic imaging, with particular emphasis on volumetric medical imaging datasets. To improve robustness and segmentation accuracy, additional knowledge into the input representation was further integrated, including predictions from adjacent slices, multiparametric imaging information, and prior annotations. In addition, the study compared different SAM3 fine-tuning strategies on the BraTS2023-MET dataset, including partial module fine-tuning, adapter-based optimization, and full-model fine-tuning, under different training objectives. Extensive experiments conducted on 13 datasets covering 11 lesion types demonstrate that SAM3 achieves strong cross-modality generalization and accurate instance-specific lesion delineation. These results highlight the potential of concept-driven foundation models for scalable and practical medical-image segmentation in real-world clinical settings. The source code and dataset splits are publicly available at https://github.com/apple1986/lesion-sam3 .