Source-linked AI summary
Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Hangrui Xu, Zhengxian Wu, Yunyao Yu, Zhuohong Chen, Rui Cong, Xiangwen Deng, Zhifang Liu, Peng Jiao, Haoqian Wang
TL;DR
KB-VQA retrieval must find long-tail entity evidence, but CLIP-style pipelines can confuse visually similar entities and miss matches across appearance changes. KBMR uses an MLLM-based semantic retriever with discriminator-derived soft supervision and distillation, achieving stronger retrieval and end-to-end VQA results across benchmarks.
Problem
CLIP-style KB-VQA retrieval prioritizes surface-level visual similarity, which does not reliably represent entity-level semantic relevance under large appearance variations and visual ambiguity.
Method
KBMR maps images into an MLLM semantic space and uses continuous entity-consistency weights for hard-negative sampling, soft supervision, and semantic distillation.
Results
KBMR achieves the best retrieval performance and strongest end-to-end KB-VQA accuracy across benchmarks, including up to +14.7 points at R@1 and VQA accuracies of 54.7 on E-VQA and 50.8 on InfoSeek.
Takeaways & Limitations
Entity-aligned retrieval improves candidate evidence quality and benefits diverse downstream KB-VQA architectures, including systems with reranking or stronger reasoning modules.
Abstract
from arXiv · showhide
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
1 Introduction
KB-VQA retrieval is constrained by CLIP-style visual similarity, which can miss entity-level relevance under appearance variation and visual ambiguity. KBMR addresses this with an MLLM retriever, semantic discriminator, and continuous semantic distillation.
- KB-VQA first-stage retrieval is a critical bottleneck because systems must identify knowledge evidence corresponding precisely to queried entities.
- Large viewpoint, temporal, and stylistic changes can make the same concept look different, while distinct entities may appear visually similar.
- MLLM embeddings offer expressive entity- and concept-level representations, but KB-VQA requires finer entity-specific relevance than general cross-modal retrieval.
- KBMR encodes images into a shared semantic space and uses an MLLM-based discriminator to assign continuous entity-consistency weights to query–candidate pairs.
- These weights support hard-negative sampling and soft supervision, while continuous semantic distillation aligns retriever similarities with the discriminator-induced weight distribution.
- KBMR is presented as the first MLLM-based retriever tailored for KB-VQA and achieves the strongest retrieval and end-to-end VQA performance across multiple benchmarks.
2 Related Work
Related work improves multimodal retrieval representations and adds reranking, filtering, or query refinement, but these approaches generally retain CLIP-style first-stage candidate generation. KBMR instead targets the retriever's representation space and training process.
- MLLM research has expanded from image–text alignment toward multimodal reasoning, generation, temporal understanding, and instruction following.
- KB-VQA commonly uses retrieval-augmented generation to retrieve evidence from large knowledge sources before conditioning a vision–language model.
- Existing KB-VQA systems typically encode query and candidate images separately with CLIP-style retrievers and construct candidate sets by vector similarity.
- Prior work strengthens multimodal representations through image–text alignment or joint embedding spaces, while another line adds reranking, filtering, or query refinement.
- These methods still depend on CLIP-generated candidate sets, leaving retrieval constrained by visual similarity rather than fine-grained entity-centric semantic relevance.
3 Methodology
KBMR replaces surface-oriented retrieval with an MLLM-based, entity-aligned embedding space for KB-VQA. Its semantic discriminator, hard-negative sampling, and continuous semantic distillation provide soft supervision over confusing candidate neighborhoods.
- 3.1 Overview: KBMR uses an MLLM to encode images into a shared semantic space, using final-token hidden states as retrieval embeddings.The design targets entity-level relevance rather than surface visual similarity.
- 3.3 Semantic Discriminator as Soft Supervision: The Semantic Discriminator determines whether each query-candidate pair refers to the same entity and converts Yes/No logits into continuous consistency weights.A sigmoid with calibrated logit scaling produces weights whose smoothness is controlled by γ.
- 3.4 Hard Negative Sampling: Potential hard negatives are first retrieved by EVA-CLIP, then filtered using entity consistency and sampled across four weight-based difficulty strata.Candidates with high embedding similarity but low entity consistency are retained as informative semantic mismatches; the final set contains at least eight samples when possible.
- 3.5 Continuous Semantic Distillation Training: Continuous Semantic Distillation replaces one-hot labels with a soft entity-aware target distribution over the hard-negative neighborhood.The retriever posterior is trained to respect relative semantic relationships among candidates rather than treating all negatives as equally irrelevant.
- Experiments: The method is evaluated on VQA accuracy for E-VQA and InfoSeek, with KBMR replacing only the retriever in an existing KB-VQA pipeline.This setup isolates the end-to-end contribution of retrieval improvements.
- 3.5 Continuous Semantic Distillation Training: KBMR minimizes symmetric KL divergence between retriever and semantic-prior distributions, encouraging semantic alignment while preventing an overly sharp or biased posterior.The two KL terms respectively move probability toward semantically correct candidates and improve optimization stability.
4 Experiments
Experiments show that KBMR improves retrieval and end-to-end KB-VQA accuracy across benchmarks, while ablations support its semantic discriminator and continuous supervision design.
- VQA Results: KBMR improves end-to-end accuracy by up to +9.2 on E-VQA and +9.3 on InfoSeek, reaching 54.7 and 50.8, respectively.These gains remain substantial with reranking and across downstream architectures.
- Retrieval Results: On E-VQA, KBMR exceeds the best MLLM retriever by +12.8 points and EVA-CLIP-8B by +11.4 at R@1.On InfoSeek, it surpasses the strongest CLIP retriever by +14.7 points and the best MLLM retriever by +19.0 at R@1.
- Results on more benchmarks: KBMR improves OK-VQA answer accuracy by +12.7 points over the original OMGM and establishes a SOTA score of 79.3.The evaluation replaces OMGM’s retriever while keeping other pipeline components unchanged.
- Ablation Studies: The full SD-based setting performs best, while hard one-hot supervision or corrupted SD weights consistently reduces performance.The comparisons indicate that both SD-guided hard-negative construction and soft supervision contribute to retrieval learning.
- Ablation Studies: SD weights distinguish same-entity from different-entity image pairs more effectively than CLIP similarity, achieving AUC 0.91 versus 0.79.Both scores are higher for positive pairs, but SD yields larger positive–negative separation.
- Ablation Studies: Performance improves as the semantic sharpness coefficient γ increases from 0.9 to 1.1, with the best results at γ=1.1.Smaller values make weights too sharp, while larger values oversmooth candidate contrasts.
- Ablation Studies: Weight analysis shows that many CLIP-retrieved top-50 candidates receive low semantic consistency weights from the MLLM evaluation.This indicates that visual competitiveness does not guarantee entity-level semantic relevance.
5 Conclusion
The paper concludes that KBMR replaces surface-level visual matching with entity-aware semantic retrieval for KB-VQA. Across E-VQA, InfoSeek, and OK-VQA, it improves retrieval recall and end-to-end performance through MLLM embeddings and continuous semantic supervision.
- 5 Conclusion: KBMR is presented as the first MLLM-based retriever for knowledge-based visual question answering.It targets the mismatch between visual similarity and entity-level semantic relevance.
- 5 Conclusion: KBMR combines autoregressive MLLM embeddings, an MLLM semantic discriminator, and continuous semantic distillation for entity-centric retrieval.The discriminator supplies continuous entity-consistency weights for hard-negative construction and soft supervision.
- 5 Conclusion: Experiments on E-VQA, InfoSeek, and OK-VQA show consistent retrieval-recall improvements and substantial end-to-end KB-VQA gains.The conclusion emphasizes explicit modeling of entity-aware semantic relations beyond surface-level visual matching.