Autoregressive Generative Medical Image Segmentation.
Authors
Abstract
Recent years have witnessed rapid progress in deep learning-based medical image segmentation. Existing approaches can be broadly categorized into discriminative models, which formulate segmentation as a pixel-wise classification problem, and generative models, which treat segmentation as a image conditioned mask generation task. While discriminative models often achieve strong performance on in-distribution data, they tend to generalize poorly under domain shifts. In contrast, generative models exhibit stronger generalization by modeling the joint distribution of images and labels, but often sacrifice fine-grained boundary accuracy due to limited sensitivity to local structural details. In this work, we propose AutoRegressive Generative Medical Image Segmentation (ARGMS), a unified framework that formulates segmentation as a coarse-to-fine, image-conditioned autoregressive mask generation process. ARGMS leverages a pretrained vision foundation model to project both images and labels into a shared latent space, and progressively generates multi-scale label representations via a causal Transformer. To bridge the semantic gap between image and label representations, we introduce a Spatial-Channel Feature Aggregation (SCFA) module that provides semantically informative conditioning signals throughout the generation process. Furthermore, we design a Conjunction Adversarial Network (CAN) to perform multi-scale latent alignment and rejection-based inference, effectively mitigating error accumulation during autoregressive decoding. By integrating multi-scale label generation, latent-space alignment, and adversarial consistency constraints, ARGMS achieves accurate and structurally coherent segmentation across diverse datasets. Extensive experiments on five public benchmarks demonstrate that ARGMS consistently outperforms both discriminative and generative baselines, while exhibiting strong generalization and scalability across different imaging modalities and dataset scales.