Source-linked AI summary
MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA
Haowen Gu, Gensheng Pei, Zeren Sun, Mingwu Ren, Xiangbo Shu, Yazhou Yao, Fumin Shen
TL;DR
Medical VQA needs useful visual-language reasoning despite limited annotated data and the resource constraints of clinical deployment. MedFG-VQA combines synthetic supervision with frequency-memory fusion and graph-aware cross-attention, achieving competitive performance with substantially fewer parameters than mainstream large models. The results support the feasibility of efficient vision-language models for practical medical applications.
Problem
Medical VQA lacks sufficient high-quality annotated data and must satisfy strict model-size and computational constraints in clinical settings.
Method
MedFG-VQA combines a learnable frequency-domain memory bank for low-frequency structural priors with graph-aware cross-attention for visual-textual alignment and local context aggregation.
Results
Competitive performance is achieved across multiple medical VQA benchmarks with significantly fewer parameters than mainstream large models.
Takeaways & Limitations
The compact architecture demonstrates the feasibility of efficient vision-language models for practical medical applications.
Abstract
from arXiv · showhide
Medical Visual Question Answering (Med-VQA) holds significant promise for clinical decision support, yet faces challenges due to limited annotated data and the high computational demands of existing large vision-language models. We propose MedFG-VQA, a lightweight framework that leverages a memory bank to augment DCT-based low-frequency features and employs graph-enhanced cross-attention for effective visual-textual alignment. Specifically, our approach features two key components: Frequency-Memory Fusion (FMF), which enhances low-frequency features by retrieving from a learnable memory bank built on DCT decomposition, and Graph-Aware Cross-Attention (GACA), which aligns visual-textual features via cross-attention and refines them through graph-convolutional aggregation. To address data scarcity, we construct SynMed-VQA, a large-scale synthetic dataset comprising over 2 million question-answer pairs across 9 imaging modalities and 10 major organs, generated with GPT-4o. Extensive experiments on SynMed-VQA and three other standard biomedical VQA benchmarks demonstrate that MedFG-VQA achieves competitive or superior performance compared to much larger models while maintaining significantly lower computational costs, highlighting its efficiency and potential for clinical deployment.
1. Introduction
Medical VQA is motivated by clinical applications but constrained by limited annotated data and the computational demands of large models. MedFG-VQA addresses these challenges with synthetic supervision and lightweight frequency- and graph-based modules.
- Medical VQA links medical images with natural-language queries for diagnostic assistance, image retrieval, and clinical decision support.
- Limited high-quality annotated data and strict deployment resource constraints make it difficult for existing models to retain strong diagnostic capability in lightweight configurations.
- SynMedVQA contains 2.059 million GPT-4o-generated question-answer pairs spanning 9 imaging modalities and 10 major organs.
- FMF uses a learnable frequency-domain memory bank to retrieve low-frequency components and inject global structural priors through residual fusion.
- GACA combines cross-modal attention with a feature-adaptive KNN-based GCN, using gated fusion of global semantic and local topological information.
- Experiments and ablations across multiple medical VQA benchmarks report competitive performance with significantly fewer parameters than mainstream large models.
2. Related Work
Related work spans large vision-language models and emerging small vision-language models. Existing approaches align images with language but often incur substantial computational costs, while lightweight research has focused mainly on general-purpose benchmarks.
- Vision Language Model: Vision-language models align image features with language models for comprehensive visual understanding and commonly use autoregressive architectures for vision-language tasks.
- Vision Language Model: Many existing vision-language methods depend on large visual encoders or complex feature-alignment modules, resulting in high parameter counts.
- Small Vision Language Model: Small vision-language models reduce computational overhead through compact visual backbones or low-rank adaptation while preserving multimodal reasoning capabilities.
- Small Vision Language Model: Most small-model research has emphasized general-purpose models and broad-domain benchmarks, with comparatively limited attention to domain-specific medical applications.
3. Method
MedFG-VQA combines frequency-domain memory fusion with graph-aware cross-attention to enhance global structural representations and align visual-textual features. Its pipeline reconstructs refined visual features, aggregates local spatial context, and trains with text-generation and memory-diversity objectives.
- FreqMemoryFusion: FMF applies DCT decomposition, retrieves similar low-frequency entries from a learnable memory bank, and residually fuses them into enhanced representations.The retrieved entries are weighted by cosine similarity, while λ balances the original and memory-enhanced features; λ is fixed at 0.7 in practice.
- FreqMemoryFusion: The enhanced low-frequency features are concatenated with high-frequency components, transformed by IDCT, and gated with the original input for feature refinement.A learnable gate and linear projection adaptively combine reconstructed and original features.
- FreqMemoryFusion: FMF maintains a learnable global memory whose vectors are updated during optimization to capture representative low-frequency priors.A diversity loss is introduced because unconstrained memories may become redundant or overlapping.
- Graph-Aware Cross-Attention: GACA uses image features as queries and text features as keys and values in multi-head cross-attention to obtain semantically enriched visual representations.This provides global cross-modal alignment before local visual-structure modeling.
- Graph-Aware Cross-Attention: A dynamic KNN graph connects similar image patches, and graph convolution propagates neighboring information to enhance local contextual coherence.The adjacency matrix is symmetrized and normalized before graph convolution.
- Graph-Aware Cross-Attention: A learned gate fuses cross-modal semantic features with locally aggregated structural features, balancing global alignment and local context.The gate is generated from concatenated feature types through a learnable projection and sigmoid activation.
- Training Strategy & Loss Function: Training combines cross-entropy text-generation loss with a diversity loss that keeps FMF memory vectors distinct.The objective is a linear combination of Ltext and Ldiv.
4. Dataset
SynMedVQA is constructed from 11 public medical imaging datasets spanning nine modalities and ten anatomical regions, with GPT-4o generating clinically tailored Q&A pairs and a multi-stage quality-control process. The resulting dataset contains 2,059,020 clinically grounded questions covering diverse medical visual reasoning aspects.
- Data Source: 11 public medical imaging datasets span nine modalities and ten major anatomical regions for multimodal medical visual understanding.The modalities include MRI, CT, X-ray, ultrasound, dermoscopy, histopathology, and fundus photography, among others.
- Data Generation: GPT-4o generates open-ended and multiple-choice questions from cleaned image-label information using prompts adapted to dataset, modality, organ, and disease characteristics.The prompting framework incorporates original annotations and presents GPT-4o as a senior medical imaging expert.
- Data Generation: Multiple-choice questions use either one correct answer with three distractors or four distractors without a correct answer.The first format accounts for 75% of multiple-choice questions, while the second accounts for 25%.
- Quality Control: Generated Q&A pairs undergo rule-based filtering, Qwen2.5-VL and human review, and consistency checking against original image annotations.These stages target duplicate, ambiguous, incomplete, unclear, irrelevant, or annotation-inconsistent examples.
- Dataset Statistics: 2,059,020 Q&A pairs are equally split between open-ended and multiple-choice questions and cover anatomy, imaging characteristics, pathology, and clinical manifestations.Figure 4 visualizes category proportions with datasets in the inner circle and within-dataset category ratios in the outer circle.
5. Experiments
Experiments evaluate MedFG-VQA on SynMedVQA, three public medical VQA benchmarks, and targeted ablations. Results show strong accuracy with 795M parameters, while FMF, GACA, and frequency-domain design choices materially affect performance.
- Benchmark Results: 0.6441 average accuracy surpasses Qwen3-VL by approximately 9.5% on SynMedVQA with only 795M parameters.The comparison uses identical data splits and metrics, while baselines use publicly available pretrained weights without SynMedVQA fine-tuning.
- Benchmark Results: 795M-parameter MedFG-VQA achieves competitive performance across SLAKE, VQA-RAD, and PathVQA, with the highest accuracy on open-ended tasks.The evaluation covers both closed-ended and open-ended questions.
- Ablation Studies: 0.6441 accuracy with both FMF and GACA rises from 0.627 without both modules, supporting complementary global feature modeling and cross-modal alignment.The ablation attributes global structural perception to FMF and image-text semantic interaction to GACA.
- Ablation Studies: Memory size 64 is adopted because increasing it from 16 improves accuracy, whereas 128 causes a sharp drop associated with redundant or noisy retrieval information.The authors describe 64 as a balanced choice.
- Ablation Studies: FFT reaches 0.6441 accuracy, outperforming direct input and DCT-based settings by preserving magnitude and phase information.The reported interpretation links DCT’s lost phase cues to reduced structural consistency.
- Ablation Studies: GACA improves accuracy over standard cross-attention, with the reported comparison showing CA at 0.6407 and GACA at 0.6441.The improvement is attributed to modeling local geometric relationships.
- Qualitative Results: Qualitative comparisons report more accurate, clear, and clinically relevant responses than Gemma3 and LLaVA-Med in selected medical-image questions.Examples contrast Gemma3’s lengthy redundant descriptions with MedFG-VQA’s correct lesion identification where LLaVA-Med is broad and nonspecific.
6. Conclusion
MedFG-VQA addresses data scarcity and computational constraints through a compact architecture combining frequency-domain priors with graph-aware visual-text alignment, alongside the 2.059-million-sample SynMedVQA dataset. The conclusion reports strong performance and identifies synthetic-data quality and single-view construction as boundaries on adaptability to emerging patterns and practical multimodal clinical imaging.
- Conclusion: MedFG-VQA combines FMF for global frequency-domain priors with GACA for visual-textual alignment and local structural aggregation.The architecture is presented as a lightweight approach for medical VQA in clinical settings.
- Conclusion: SynMedVQA contains 2.059 million samples and is constructed to facilitate medical VQA research and development.The dataset is part of the paper’s response to limited annotated data.
- Conclusion: The conclusion states that synthetic Q&A quality and diversity depend on the underlying model’s capability boundaries, which may hinder adaptation to novel imaging patterns.This limitation concerns generated-data coverage beyond learned representations.
- Conclusion: Because SynMedVQA uses single-view images, practical clinical scenarios requiring joint reasoning over multi-view and multimodal imaging remain outside its construction scope.The passage contrasts the dataset with broader clinical imaging inputs.