Lesion Segmentation from Imperfect Medical Data via Weakly Supervised Learning with SAM and Explainable Vision Transformer.
Authors
Abstract
Accurate lesion segmentation remains a critical yet challenging task in medical imaging due to scarce pixel-level annotations, inter-observer variability and modality-specific noise. Conventional fully supervised models require exhaustive manual labeling, which limits scalability and clinical applicability. To address these challenges, this paper proposes a hybrid weakly supervised framework that integrates the Segment Anything Model (SAM) with an Explainable Vision Transformer (ViT) for robust and interpretable lesion segmentation. The SAM module converts weak annotations such as bounding boxes and sparse points into high-quality pseudo-masks refined through ensemble consistency, shape priors, and confidence masking, reducing dependency on dense labels. A ViT-based multiple-instance learning (MIL) module then performs weakly supervised classification and mask refinement using uncertainty-aware optimization and self-training, ensuring resilience to noisy supervision. A SAM-ViT alignment loss enforces consistency between pseudo-masks and transformer attention maps, improving anatomical plausibility. Additionally, an explainable AI (XAI) mechanism aligns attention rollout and gradient-based saliency with lesion masks, embedding interpretability directly into the training process. This unified pipeline delivers noise-robust segmentation, clinically transparent predictions, and efficient label utilization under imperfect supervision. Experimental results on CAMELYON16/17 and BraTS datasets demonstrate state-of-the-art performance, achieving AUCs of 0.972, 0.949 and 0.941, with a Dice coefficient of 0.883 and mIoU of 0.826, outperforming existing weakly supervised and SAM-only baselines. This framework establishes a scalable and interpretable foundation for clinical-grade lesion segmentation, reducing annotation cost while improving reliability in real-world medical imaging workflows.