Source-linked AI summary
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
ZhengXian Wu, Hangrui Xu, Kai Shi, Zhuohong Chen, Yunyao Yu, Chuanrui Zhang, Zirui Liao, Jun Yang, Zhenyu Yang, Haonan Lu, Haoqian Wang
TL;DR
KB-VQA models struggle to identify long-tail entities and retrieve trustworthy evidence adaptively. ProMSA progressively alternates image and text search under budgets, and experiments on E-VQA and InfoSeek show improvements over strong RAG and search-agent baselines.
Problem
KB-VQA remains challenging for long-tail entities because models may struggle to identify them and assess whether retrieved evidence is trustworthy.
Method
ProMSA is a progressive multimodal search agent that alternates image and text retrieval, stops when sufficient evidence is collected, and avoids repeated results under explicit budgets.
Results
Experiments on E-VQA and InfoSeek show clear improvements over strong RAG and search-agent baselines.
Takeaways & Limitations
Progressive retrieval and reasoning supports more effective KB-VQA search behavior across E-VQA and InfoSeek.
Takeaways & Limitations
The search process assumes explicit separate budgets for image-retrieval and text-retrieval tool calls.
Abstract
from arXiv · showhide
Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.
1 Introduction
KB-VQA remains challenging because long-tail entities and diverse evidence needs limit MLLMs and fixed retrieval pipelines. ProMSA addresses these issues with budgeted, multi-round multimodal search and tool-horizon normalized policy optimization.
- Motivation: MLLMs perform strongly on general visual question answering, but their KB-VQA capability remains limited under long-tail entity distributions.The supplied passage identifies long-tail entities as a central challenge for reliable KB-VQA.
- Limitations of Prior Work: Fixed retrieval pipelines are not adaptive to whether a question needs no retrieval, entity identification, or missing-attribute retrieval.Existing methods manually select retrievers or combine image and text retrieval, despite diverse KB-VQA problem states.
- Limitations of Prior Work: Static retrieval also struggles to recover from incorrect evidence and support multi-hop reasoning through progressively expanded evidence.The passages describe failures in query rewriting, additional retrieval, and static top-K evidence injection.
- ProMSA: ProMSA formulates KB-VQA as budgeted progressive search-and-reasoning, letting the agent choose image retrieval, text retrieval, or stopping across multiple rounds.The agent couples retrieval and reasoning while accounting for time and compute costs of tool calls.
- Training and Results: TN-GSPO is introduced as a tool-horizon normalized sequence-level policy optimization method for stable search-policy learning.The introduction also reports that experiments on multiple KB-VQA benchmarks show consistent gains over strong baselines and state-of-the-art performance.
2 Related Work
Related work improves KB-VQA through multimodal retrieval, evidence filtering, and conflict mitigation, while search-based agents enable adaptive tool use and sequential evidence acquisition. However, existing KB-VQA methods largely retain static retrieval pipelines, and general search agents do not directly address uncertain or long-tailed visual entities.
- KB-VQA Methods: KB-VQA research combines visual understanding with external knowledge, emphasizing multimodal retrieval and evidence construction.OMGM coordinates modalities and granularities through hierarchical reranking and section-level evidence selection.
- KB-VQA Methods: CC-VQA addresses conflicts between parametric knowledge and retrieved evidence through conflict-aware reasoning.
- KB-VQA Methods: Existing KB-VQA methods improve retrieval precision, evidence filtering, and conflict mitigation but mostly use static retrieval-then-generate pipelines.
- Search-Based Agents: Search-based agents support adaptive tool usage, sequential evidence acquisition, and trajectory-level optimization beyond one-shot RAG.Search-o1 refines retrieved documents during reasoning, while Search-R1 formulates search-augmented reasoning as reinforcement learning.
- Search-Based Agents: General search-agent systems do not directly solve KB-VQA’s uncertain or long-tailed visual-entity retrieval, which cannot reliably be handled by one retrieval step.
3 Method
ProMSA formulates KB-VQA as a budgeted, progressive retrieval–reasoning process in which one multimodal policy adaptively selects image search, text search, or stopping while accumulating summarized evidence. The method warm-starts valid tool use with rejection-sampling SFT and optimizes whole trajectories with a tool-normalized GSPO objective.
- Progressive retrieval–reasoning: The agent alternates retrieval and reasoning under separate image-search and text-search budgets, selecting img_search, text_search, or stop at each step.When it outputs stop, it enters the final answering stage.
- Progressive retrieval–reasoning: A unified multimodal policy generates reasoning, an action token, and action arguments from the current state, jointly learning retrieval decisions, reasoning, and stopping.Arguments can specify query rewrites, top-k, and a de-duplication list.
- Evidence processing: Retrieved pages and text chunks are compressed into question-conditioned snippets that preserve relevant entity descriptions, attributes, and relation cues before being appended to the next state.This prevents raw retrieval content from causing severe context growth.
- Training: Rejection-sampling SFT retains at most one trajectory per question only when tool calls are valid, execution succeeds, and the final answer is correct.The resulting cold-start dataset makes the policy executable before reinforcement learning.
- Training: Tool-Normalized GSPO uses sparse whole-trajectory returns, masks tool-returned tokens from gradients, and incorporates interaction depth into normalization to reduce length bias.The reward combines answer correctness, format compliance, and a tool-cost term under the search budget.
4 Experiments
Experiments on E-VQA, InfoSeek, and OK-VQA show that ProMSA improves knowledge-based VQA performance while generalizing beyond its training benchmarks. Ablations and training analyses attribute these gains to adaptive tool use, TN-GSPO optimization, and balancing retrieval depth against evidence quality and cost.
- Main results: ProMSA achieves the best performance across all evaluation metrics on E-VQA and InfoSeek, with consistent gains across model sizes.The comparison includes zero-shot MLLMs, search-agent methods, and fixed RAG-style retrieval pipelines.
- Main results: ProMSA also improves performance on OK-VQA, indicating that its learned tool-usage policy generalizes beyond the training benchmarks.
- RL optimization: TN-GSPO stabilizes sequence-level reinforcement learning by normalizing updates using both generation length and tool-interaction depth, with asymmetric clipping supporting effective exploration.
- Training dynamics and tool usage: After reinforcement learning, tool-use distributions shift from text-search reliance toward retrieval behaviors better matched to task needs.
- Training dynamics and tool usage: TN-GSPO maintains tool-call counts within a reasonable range, whereas GRPO reduces them rapidly and may collect insufficient evidence.
- Budget and retrieval ablations: Increasing tool-call budgets or retrieval Top-k improves correct-evidence recall, but larger settings eventually show diminishing returns and can slightly hurt performance through added noise.
5 Conclusion
ProMSA addresses long-tail KB-VQA by jointly deciding answers and adaptive search actions, alternating image and text retrieval until sufficient evidence is collected. It uses explicit retrieval budgets, deduplication, and TN-GSPO, which normalizes sequence-level RL updates by generation length and tool-interaction depth.
- ProMSA targets long-tail KB-VQA, requiring models to decide both what to answer and how to search for missing knowledge.
- The progressive multimodal search agent alternates between image and text retrieval and stops when enough evidence is collected.
- Retrieval operates under explicit budgets and uses deduplication to avoid repeated results.
- TN-GSPO is a sequence-level RL objective that normalizes updates by generation length and tool-interaction depth.