Blank-Image Ablations Can Mislead: A Mismatched-Image Control for Visual-Requirement Audits in Medical-VQA-Evidence from ERA-TurkMed, A Turkish Generative System.

Authors

Abstract

Whether a Medical Visual Question Answering (Medical-VQA) system uses its image is usually tested by replacing it with a zero tensor: if performance does not drop, the image is deemed unused. We show that this test can mislead. We built ERA-TurkMed, to our knowledge the first generative Medical-VQA system for Turkish radiology (frozen BiomedCLIP, Q-Former, LoRA-adapted Llama-3.1-8B), trained by a three-stage curriculum on a single-center, ethics-approved dataset (~ 5000 chest radiographs; ~ 17,476 question-answer pairs). On the full test set (n = 3454, three seeds), BLEU-4 is higher without the image (Δ =  - 0.024 ± 0.007, seed-level p = 0.025). A mismatched-image control-an in-distribution radiograph of a different patient-reverses this conclusion: the real image is better on polarity accuracy (+ 0.035, p = 0.030), chrF (+ 0.018, p = 0.020), and exact match (+ 0.015, p = 0.028) in all three seeds. The blank-image advantage is confined to the BLEU family and the hallucination rate, and reverses on polarity accuracy, a measure validated against manual annotation (95.0% agreement) that exposes the negation blindness of clinical-term F1. The visual contribution is nonetheless small: 77.2% of test questions recur verbatim in training, and an image-free retrieval baseline matches the model on clinical-term F1 (0.496 vs. 0.510), a consequence of template-based, AI-assisted dataset construction rather than of Turkish. We therefore recommend verbatim-overlap auditing, an image-free baseline, a mismatched-image control, and negation-sensitive metrics as standard. Code is available; model weights follow upon publication. The clinical data cannot be shared under Turkish data-protection law (KVKK).

Tags

Topics