PaperlayerView exact revision on arXiv

Source-linked AI summary

KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering

Xianyao Zheng, Hong Yu, Hui Cui, Changming Sun, Xiangyu Li, Ran Su, Leyi Wei, Jia Zhou, Junbo Wang, Qiangguo Jin

arXiv:2604.00601v1cs.CV

TL;DR

Med-VQA methods have difficulty integrating domain-specific medical knowledge and supporting diverse free-form answers. KG-CMI combines fine-grained alignment, knowledge-graph embedding, cross-Mamba interaction, and free-form answer-enhanced multi-task learning, outperforming existing state-of-the-art methods on VQA-RAD, SLAKE, and OVQA.

  • Problem

    Med-VQA models struggle to associate lesion features with diagnostic knowledge, while closed-set classification limits adaptation to diverse free-form answers.

  • Method

    KG-CMI integrates fine-grained visual-text alignment, knowledge-graph embedding, cross-modal interaction, and free-form answer-enhanced multi-task learning.

  • Results

    KG-CMI outperforms existing methods across VQA-RAD, SLAKE, and OVQA; its open-ended evaluation includes a CIDEr score of 1.42 on SLAKE, surpassing MUMC.

  • Takeaways & Limitations

    The framework supports both closed-ended and open-ended Med-VQA while integrating image, question, and medical-knowledge representations.

Abstract

from arXiv · show

Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine. Recent methods fail to fully leverage domain-specific medical knowledge, making it difficult to accurately associate lesion features in medical images with key diagnostic criteria. Additionally, classification-based approaches typically rely on predefined answer sets. Treating Med-VQA as a simple classification problem limits its ability to adapt to the diversity of free-form answers and may overlook detailed semantic information in those answers. To address these challenges, we propose a knowledge graph enhanced cross-Mamba interaction (KG-CMI) framework, which consists of a fine-grained cross-modal feature alignment (FCFA) module, a knowledge graph embedding (KGE) module, a cross-modal interaction representation (CMIR) module, and a free-form answer enhanced multi-task learning (FAMT) module. The KG-CMI learns cross-modal feature representations for images and texts by effectively integrating professional medical knowledge through a graph, establishing associations between lesion features and disease knowledge. Moreover, FAMT leverages auxiliary knowledge from open-ended questions, improving the model's capability for open-ended Med-VQA. Experimental results demonstrate that KG-CMI outperforms existing state-of-the-art methods on three Med-VQA datasets, i.e., VQA-RAD, SLAKE, and OVQA. Additionally, we conduct interpretability experiments to further validate the framework's effectiveness.

I. INTRODUCTION

Med-VQA methods struggle to connect subtle pathological features with medical knowledge and to handle nuanced free-form answers. KG-CMI addresses these gaps through fine-grained alignment, knowledge-graph reasoning, cross-Mamba interaction, and free-form answer enhancement.

  • Med-VQA supports clinical decision support and telemedicine by combining medical-image analysis with natural-language understanding.
  • Existing methods struggle with subtle image–language associations, specialized medical knowledge, and complex cases involving multiple organs or diseases.
  • Closed-set classification limits adaptation to diverse free-form answers and may omit detailed semantic information relevant to clinical decision-making.
  • KG-CMI combines FCFA, KGE, CMIR, and FAMT to align images and questions, integrate structured medical knowledge, model cross-modal interactions, and enhance open-ended answering.
  • The framework uses bidirectional contrastive learning to strengthen semantic consistency between image regions and textual tokens.
  • KG-CMI was evaluated on three Med-VQA benchmarks, with module-level experiments and Grad-CAM visualizations supporting its effectiveness and question-relevant attention.

B. Knowledge graph-based visual question answering

Medical knowledge graphs provide structured entities and relationships for Med-VQA, while recent systems increasingly retrieve and reason over knowledge dynamically. KG-CMI extends this direction by aggregating disease relationships before question-guided knowledge retrieval.

  • Medical knowledge graphs encode entities, relationships, and attributes that support medical research and clinical decision-making.
  • Knowledge-guided Med-VQA methods have progressed from grounding textual knowledge in anatomical regions toward dynamic knowledge retrieval and reasoning.
  • KG-RAG integrates knowledge-graph retrieval into large vision-language models, including LLaVA-based KG-LLaVA.
  • KG-CMI uses graph attention to encode global disease relationships, then applies question-guided cross-attention to retrieve relevant knowledge.

III. METHODS

The methods formulate Med-VQA over visual-text samples and use a question-aware Q-Former to align medical images with questions before contrastive learning reduces their semantic discrepancy.

  • III. METHODS: The KG-CMI architecture comprises FCFA, KGE, CMIR, and FAMT modules.
  • A. Problem formulation: Med-VQA is formulated using medical image-question-answer triplets, with the goal of predicting an answer from a candidate answer set.The dataset contains N training samples, and external knowledge can provide richer textual information.
  • B. Fine-grained cross-modal feature alignment module: FCFA uses a ViT image encoder, a RoBERTa text encoder, and a question-aware Q-Former containing self-attention, cross-attention, and a feed-forward network.
  • B. Fine-grained cross-modal feature alignment module: The Q-Former first applies self-attention to query embeddings and text features, then cross-attention between visual features and learnable query vectors.This produces query outputs that incorporate information from the visual-text interaction.
  • B. Fine-grained cross-modal feature alignment module: Cross-modal contrastive learning minimizes semantic discrepancy between aligned visual representations and text representations.The bidirectional loss averages visual-to-text and text-to-visual losses using cosine similarity and a temperature parameter.

C. Knowledge graph embedding module

The KGE module injects structured medical knowledge into Med-VQA through a graph-attention representation of organ–disease relationships, then fuses that knowledge with image and question features. This supplies prior knowledge for subsequent cross-modal interactions and medical reasoning.

  • Knowledge graph encoding: KGE uses graph attention to weight neighboring knowledge-graph nodes and embed their structural and semantic information into a low-dimensional feature space.The module aggregates features from important nodes rather than treating all neighbors equally.
  • Role in reasoning: By injecting knowledge-driven constraints and implicit organ–lesion dependencies, KGE complements surface-level visual-text alignment for clinical reasoning.The paper describes this integration as supporting reasoning beyond associations directly observed in image-text pairs.
  • Knowledge graph construction: The SLAKE knowledge graph represents disease keywords and organ relationships with adjacency connections among diseases associated with the same organ or tissue.Edges connect disease nodes to related nodes and a root node, with binary adjacency values indicating relationships.
  • Cross-modal fusion: KGE uses graph features as queries and question features as keys and values, producing a representation constrained to the knowledge-graph semantic space while highlighting question-relevant nodes.The resulting representation is fused with image and question information through cross-attention.
  • Role in reasoning: The KGE module provides rich prior medical knowledge for later cross-modal interactions, enhancing reasoning about medical images and questions.This is the module’s stated contribution within the KG-CMI pipeline.

D. Cross-modal interaction representation module

The CMIR module uses cross-modal Mamba blocks to interactively process visual, textual, and knowledge-graph features while retaining Mamba’s sequential computation. Its fused representations support downstream tasks with linear-complexity multimodal interaction.

  • CMM design: CMM blocks address Mamba’s limited cross-modal correspondence learning by enabling interactions among visual representations, text embeddings, and knowledge-graph features.They retain Mamba’s sequential structure while extending it to multimodal fusion.
  • Computational design: CMM reduces the interaction complexity from standard cross-attention’s O(L^2) to Mamba’s linear O(L) through selective scanning.This is advantageous when visual tokens, text embeddings, and knowledge-graph features form a long combined sequence.
  • Feature fusion: The module constructs interleaved multimodal feature sequences and combines them through Mamba and a fusion function using element-wise multiplication and summation.The fusion function is implemented as a linear transformation.
  • Feature fusion: Two sets of CMM blocks are used before concatenating the resulting features into the final fused representation, with knowledge-graph contributions controlled by η and θ.The number of blocks is empirically selected to balance reasoning capacity and over-fitting risk on small medical datasets.
  • CMM design: CMM uses selective scanning to capture both intra-modal and inter-modal dependencies in fused visual, textual, and knowledge-graph sequences.The fused textual feature incorporates visual and knowledge-graph information, while the visual sequence is enriched with text.

E. Free-form answer enhanced multi-task learning module

FAMT augments classification with a generative auxiliary task for open-ended medical questions. It trains a T5-based answer-generation head on open-ended samples while retaining classification-based inference.

  • FAMT motivation: FAMT combines a classification head with an auxiliary head to improve handling of free-form answers beyond closed-ended prediction.The auxiliary loss is applied only to samples annotated as open-ended.
  • Auxiliary head: The auxiliary objective filters closed-ended embeddings from each batch and applies mask-guided cross-entropy to the remaining open-ended samples.The open-ended sample count O is no greater than the batch size B.
  • Classification head: The classification head uses linear, normalization, and GELU layers to predict both closed-ended and open-ended question classes with binary cross-entropy loss.It operates on the fused feature vector.
  • Auxiliary head: The auxiliary head uses a pretrained T5 decoder to generate answers from open-ended multimodal features.A learnable attention mask selects tokens relevant to the question before generation.
  • Multi-task optimization: The total training loss combines classification, visual-text contrastive, and auxiliary-task losses, while the generative auxiliary head is removed during inference.At inference, answers are predicted through the classification head.

A. Datasets and setting

KG-CMI is evaluated against several method families on SLAKE, VQA-RAD, and OVQA using repeated runs, accuracy comparisons, and significance testing. It achieves the strongest overall results across all three datasets and leads open-ended performance across datasets.

  • Datasets and metrics: SLAKE uses English medical VQA data split by image into 450 training, 96 validation, and 96 testing images; VQA-RAD contains radiological images annotated by medically trained volunteers.The supplied dataset description identifies the English SLAKE subset and the VQA-RAD annotation source.
  • Comparison protocol: KG-CMI is compared with pretraining, knowledge-enhanced, and architecture-specialized Med-VQA methods using open-ended, closed-ended, and overall accuracy.The comparison covers SLAKE, VQA-RAD, and OVQA.
  • Evaluation reliability: Five independent runs report mean accuracy and standard deviation, while two-tailed t-tests against the second-best baseline yield p-values below 0.05 on every dataset.The reported tests are intended to assess whether improvements exceed random fluctuations.
  • Overall results: 84.26% overall accuracy on SLAKE surpasses CCIS-MVQA’s 84.08%, while 78.21% on VQA-RAD exceeds LaPA’s 78.15%.The corresponding improvements are 0.18% and 0.06%, respectively.
  • Open-ended results: KG-CMI leads open-ended accuracy on all datasets, scoring 82.78% on SLAKE, 68.18% on VQA-RAD, and 71.77% on OVQA.The OVQA score improves over M3AE by 10.12%, while the VQA-RAD score is 0.6% below CCIS-MVQA’s open-ended result.
  • Component analysis: The coordinated FCFA, KGE, CMIR, and FAMT components are reported to support performance in both closed-ended and open-ended settings.The ablation study also reports improvements when key modules are added individually across the benchmark datasets.

2) Ablation study:

The ablation results show that KG-CMI’s modules contribute complementary gains, particularly for open-ended answer quality, while auxiliary-loss weights require careful tuning.

  • Module ablation: The full combination of QQ-Former, CMCL, KGE, CMM, and AHead achieves the highest performance across SLAKE, VQA-RAD, and OVQA.The study reports improvements across open-ended, closed-ended, and overall performance when modules are added.
  • Module ablation: AHead increases open-ended scores by 1.03%, 5.95%, and 7.08% on SLAKE, VQA-RAD, and OVQA, respectively.Removing AHead notably reduces open-ended accuracy because its generative auxiliary task encourages more robust multimodal representations.
  • Module ablation: KGE benefits medical knowledge-intensive datasets, while CMM improves adaptation to complex scenarios and CMCL enhances closed-ended accuracy.The reported examples include 85.96% on SLAKE and 80.51% on VQA-RAD for CMCL, and 83.24% overall on SLAKE with CMM.
  • Open-ended answer generation: KG-CMI achieves a CIDEr score of 1.42 on SLAKE, outperforming MUMC in open-ended answer generation.BLEU, METEOR, and CIDEr evaluate fluency and semantic similarity rather than exact-match accuracy.
  • Loss-weight sensitivity: Performance declines when α exceeds 0.2 or β exceeds 0.3, so the experiments use α = 0.2 and β = 0.3.The analysis attributes the decline to auxiliary losses disrupting the stable representation learned by the main branch.

2) Sensitivity of QQ-Former layers:

Sensitivity analyses select shallow configurations for QQ-Former and graph aggregation, while two CMM blocks provide the best overall accuracy and lower-cost interaction than standard cross-attention.

  • 2) Sensitivity of QQ-Former layers:: A 1-layer QQ-Former is adopted because deeper configurations offer only marginal SLAKE gains and slightly reduce VQA-RAD performance.The reported 2-layer gain on SLAKE is +0.12%, with the decrease on VQA-RAD attributed to possible over-fitting on smaller datasets.
  • 2) Sensitivity of QQ-Former layers:: Two CMM blocks achieve the best overall accuracy: 84.26% on SLAKE, 78.21% on VQA-RAD, and 79.58% on OVQA.Performance improves from one to two blocks, while the passage identifies two blocks as optimal.
  • 4) Comparison between CMM and standard cross-attention:: CMM improves SLAKE overall ACC by 1.15% while reducing FLOPs by approximately 24% versus Transformer-based cross-attention.The comparison supports both higher efficiency and stronger modeling of tripartite Med-VQA interactions.
  • 5) Sensitivity analysis of GAT layers:: A single GAT layer provides the optimal balance for distilling knowledge from the three-level medical knowledge graph.The restricted graph hierarchy has a small diameter, making extensive message passing unnecessary.
  • 5) Sensitivity analysis of GAT layers:: Using two or three GAT layers can dilute disease-finding specificity by over-incorporating neighborhood information.The paper describes this effect as a representation bottleneck in which broad anatomical priors mask unique clinical features.
  • 5) Sensitivity analysis of GAT layers:: Grad-CAM is used to visualize attention correlations between questions and image regions across methods.The visualizations assess whether attended regions are relevant to the question.

6) Interpretability analysis:

Interpretability analyses show that KG-CMI generally localizes question-relevant lesions and anatomical regions, while hard cases expose errors in fine-grained location and existence judgments.

  • Visual localization: KG-CMI focuses on target regions related to closed-ended questions, including lesions, organs, and signs.The reported Grad-CAM comparisons describe stronger localization than M2I2, M3AE, and MUMC.
  • Visual localization: For open-ended questions, KG-CMI locates lesions, key anatomical regions, and fine lesions such as metacarpal fractures.This includes reasoning-type and long-sequence description-type questions.
  • Case analysis: In Case 1, KG-CMI assigns 79.7% to the lower right lung, whereas M3AE assigns 54.2% to that region and 22.2% to the heart.The example illustrates more concentrated localization for a lung-abnormality question.
  • Case analysis: In Case 2, KG-CMI assigns 65.1% to “Yes,” while M3AE favors “No” at 44.6% and produces irrelevant categories.The case concerns head-organ detection.
  • Failure cases: The reported failure modes are location confusion in fine anatomical structures and errors judging disease or organ existence.The paper notes that the erroneous case nevertheless exposes intermediate reasoning information.
  • Overall interpretation: Across three benchmarks, the authors report relevant-region attention, interpretability, and practical applicability alongside strong performance.The conclusion links attention analysis with reasoning about medical concepts and relationships.
Loading 2604.00601v1…