Multimodal visual question answering in neuro-oncology and gastroenterology: A domain-specific feasibility study.
Authors
Affiliations (4)
Affiliations (4)
- Institute for High-Performance Computing and Networking (ICAR), Research National Council of Italy (CNR), Naples, Italy; Department of Electrical Engineering and Information Technology (DIETI), University of Naples Federico II, Naples, Italy.
- Institute for High-Performance Computing and Networking (ICAR), Research National Council of Italy (CNR), Naples, Italy.
- Malaysia-Japan International Institute of Technology (MJIIT), Universiti Teknologi Malaysia, Kuala Lumpur, Malaysia; Faculty of Science, University of Hradec Kralove, Hradec Kralove, Czech Republic; Regional Research Center, Iwate Prefectural University, Takizawa, Japan.
- Institute for High-Performance Computing and Networking (ICAR), Research National Council of Italy (CNR), Naples, Italy. Electronic address: [email protected].
Abstract
Medical Visual Question Answering (Med-VQA) requires models to combine image content with the clinical intent expressed by a question, yet domain-specific resources and controlled analyses of modality use remain limited. This study introduces two expert-validated Med-VQA datasets for brain tumor MRI and gastrointestinal endoscopy and evaluates a dual-stream architecture that combines BERT question representations and ViT image features through Modular Co-Attention on Attention (MCAoA). The model formulates answer prediction as closed-vocabulary classification and is assessed through overall, question-type, disease-class, baseline, ablation, and usability analyses. The complete fine-tuned model with product fusion achieved accuracies of 80.86% (WUPS 81.32%) on the brain tumor dataset and 75.95% (WUPS 76.77%) on the gastrointestinal dataset. It exceeded the question-only and image-only baselines by 23.86 and 43.16 percentage points, respectively, on brain tumor MRI, and by 18.54 and 35.92 percentage points on gastrointestinal endoscopy. Product fusion provided a small but consistent advantage over concatenation, while end-to-end encoder fine-tuning improved accuracy over frozen encoders on both datasets. A complementary study with clinical experts indicated positive perceived usability in a controlled setting. These findings show that performance cannot be reproduced from either modality alone and support the complementary use of image and question information. However, the ablations establish modality-level dependence rather than spatially correct anatomical grounding, which requires targeted counterfactual and attribution-based evaluation.