LLaMIA: Few-Shot Medical Image Lesion Localization Analysis with Large Language Models.
Authors
Affiliations (3)
Affiliations (3)
- School of Computer Science and Engineering, Guangdong University of Technology, Guangzhou, 510006, China.
- School of Computer Science and Engineering, Guangdong University of Technology, Guangzhou, 510006, China. [email protected].
- School of Computer Science, Guangdong Polytechnic Normal University, Guangzhou, 510006, China.
Abstract
Accurate lesion localization in medical imaging is essential for reliable diagnosis and treatment planning, yet existing methods often require large-scale annotations that are costly and time-consuming. Few-shot lesion localization alleviates this challenge by requiring only a small number of annotated samples; however, performance remains highly dependent on limited support information and annotation availability. To address this, we propose LLaMIA, the first large language model LLM-driven framework designed specifically for few-shot lesion localization. Unlike prior approaches that only treat LLMs as auxiliary tools, LLaMIA adopts an LLM-guided coarse-to-fine localization framework, where CLIP and Q-Former provide lesion-aware multimodal representations, the LLM predicts coarse lesion coordinates through structured prompts, and the Localization Refinement Network further improves spatial precision. We design three modules: a lesion-aware regional encoding network for fine-grained multimodal guidance, an instruction construction pipeline for disambiguating lesion attributes, and a localization prediction mechanism with bipartite matching, coordinate regression, and confidence supervision. Furthermore, a contrastive inference strategy is introduced to mitigate hallucination. Extensive experiments are conducted under multiple few-shot settings (1-shot, 3-shot, 5-shot, and 10-shot) on three public datasets to comprehensively evaluate localization capability under annotation scarcity. LLaMIA achieves state-of-the-art lesion localization performance, outperforming existing baselines in terms of accuracy, robustness, and generalization. The results demonstrate that LLaMIA effectively leverages lesion-aware multimodal reasoning to address annotation scarcity, while mitigating hallucination through contrastive inference. This highlights the advantages of integrating structured prompts with spatial reasoning in medical imaging tasks. LLaMIA introduces a novel paradigm for LLM-driven lesion localization, showing the potential of multimodal instruction-based reasoning to advance medical image analysis with minimal annotation cost.