PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Multimodal Planning Agent for Visual Question-Answering

Zhuo Chen, Xinyu Geng, Xinyu Wang, Yong Jiang, Zhen Zhang, Pengjun Xie, Kewei Tu

arXiv:2601.20676v1cs.CL

TL;DR

Existing VQA mRAG systems often rely on rigid, dependent multi-stage pipelines that waste computation. This paper trains a multimodal planning agent to select only necessary retrieval steps, improving average performance across six datasets while cutting search time by over 60% versus OmniSearch.

  • Problem

    Existing mRAG systems use rigid multi-stage pipelines with dependencies that limit practical efficiency and scalability for VQA.

  • Method

    The paper trains a multimodal planning agent to dynamically select necessary mRAG components for each VQA query.

  • Results

    The method improves average VQA performance over six datasets, reduces search time by over 60% versus OmniSearch, and is 3–4.5× faster than Deep Research agents.

  • Takeaways & Limitations

    Dynamic mRAG path selection can improve VQA performance while substantially reducing retrieval latency and redundant computation.

  • Takeaways & Limitations

    The evaluation does not report overall end-to-end time because unstable GPT-4o latency would obscure the comparison, and the agent model is assumed to match the annotation model.

Abstract

from arXiv · show

Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60\% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code will be released.

1. Introduction

Existing mRAG systems use rigid, dependent multi-stage workflows that can waste computation on diverse VQA queries. The paper introduces a planning agent that dynamically selects necessary steps, improving efficiency while maintaining or improving performance across six datasets.

  • 1. Introduction: Rigid mRAG pipelines may chain image grounding, image retrieval, query rewriting, and text retrieval despite dependencies that make them inefficient and data-agnostic.Query rewriting may depend on image retrieval, while text retrieval depends on query rewriting.
  • 1. Introduction: The planning agent dynamically selects only the mRAG components needed for each VQA query, omitting redundant operations while preserving task performance.It can bypass retrieval for simpler queries and selectively execute image or text retrieval for queries requiring additional evidence.
  • 1. Introduction: Across six diverse VQA datasets, the method improves average performance while reducing search time by over 60% versus OmniSearch.The comparison also includes lower tool-call latency than Deep Research agents.
  • 1. Introduction: The agent is 3× faster than WebWatcher-7B and 4.5× faster than WebWatcher-32B while achieving better performance.These results indicate lower tool-call latency than both Deep Research reference settings.

2. Method

The method constructs decomposed VQA training examples, labels queries by the retrieval they require, and fine-tunes an MLLM to choose an efficient inference path. At inference, the agent selects among no retrieval, text retrieval, image retrieval, or both.

  • 2.2. Agent Training Data: Training data expands each question-answer pair into an image query q_i, image answer a_i, and gold query q_g that combines visual and textual information.The image query identifies image content, while the gold query more comprehensively describes the required information.
  • 2.2. Agent Training Data: The agent classifies each VQA query into four categories: no mRAG, additional text contexts, additional image contexts, or both.For the both-context category, image retrieval precedes gold-query rewriting and text retrieval.
  • 2.3. Agent Training and Inference: The method fine-tunes an MLLM with category labels so it selects an inference path according to the input query and prompt.The objective minimizes the negative log probability of the category label over the training set.
  • 2.3. Agent Training and Inference: The training setup assumes the agent model and annotation model are the same, enabling use of open-source models.The paper explicitly states this shared-model setting as an assumption.
  • 2.3. Agent Training and Inference: At inference, the agent avoids gold-query generation for no-mRAG or image-retrieval paths, while rewriting q_g when text contexts must be retrieved.The task model then generates answers using q alone, q with k_i, q with k_t, or q with both contexts.

3. Experiment

Across six isolated VQA test datasets, the multimodal planning agent dynamically selects retrieval types and outperforms baseline settings while reducing unnecessary mRAG operations. It achieves comparable or better performance than strong fixed mRAG configurations with substantially improved efficiency.

  • 3.2. Main Results: The planning agent outperforms all baseline settings on the mixed dataset and in unweighted average performance.The mixed dataset combines all tested VQA query types, while the +ki,t configuration is a strong but computationally costly baseline.
  • 3.2. Main Results: The agent dynamically predicts retrieval types across datasets, often omitting mRAG for queries solvable by the MLLM alone.It predicts no mRAG for approximately 60% of NoCaps and Visual7W examples and uses image and text retrieval more selectively than the prompt-based baseline.
  • 3.2. Main Results: The method achieves 56.48 on Dyn-VQA (en), exceeding the 56.34 obtained by universally applying the +ki,t configuration.This shows performance comparable to or better than the optimal fixed mRAG configuration on that dataset.
  • 3.2. Main Results: On Private VQA, the method maintains performance while using simultaneous image-and-text retrieval in only 36.4% of cases.The result illustrates more efficient deployment by avoiding unnecessary joint searches across visual and textual content.

4. Analysis

The planning agent improves VQA efficiency against WebWatcher and OmniSearch while preserving or improving performance, and transfers across MLLMs without additional fine-tuning.

  • 4.1. Compare with Deep Research Agent WebWatcher: 3× and 4.5× lower tool-call latency than WebWatcher-7B and WebWatcher-32B, respectively, while outperforming WebWatcher on the Mix dataset.Table 4 compares performance and tool-call latency on Mix; WebWatcher’s multi-round operation can invoke multiple calls per query.
  • 4.2. Compare with OmniSearch: 66.7% average search-time reduction versus OmniSearch and 52% lower time even after including agent inference.The comparison measures search duration separately because unstable GPT-4o API latency makes end-to-end timing unreliable.
  • 4.2. Compare with OmniSearch: On Dyn-VQA (en), image-to-image and text-to-text searches decrease by 87.4% and 69.8%, respectively, while overall task performance improves.Reduced retrieval frequency shortens input sequences and lowers inference computation; image-to-image search is identified as a major latency bottleneck.
  • 4.3. Agent Applying to More MLLMs: The agent improves Qwen-VL-Max-latest to 69.59, versus 66.61 without mRAG and 66.00 for the prompt-based baseline.The fine-tuned 7B-scale agent also outperforms settings using GPT-4o and Qwen-VL-Max as task models on average and on Mix.
  • 4.4. Training Dynamics: LoRA with rank 8 lacks capacity, whereas LoRA with rank 32 and full fine-tuning converge strongly; rank 32 performs on par with full fine-tuning.The rank-8 evaluation loss spikes after 2500 steps, while the stronger configurations reach near-1.0 training token accuracy.

5. Related Work

Related approaches use multi-stage or prompt-based retrieval pipelines, knowledge-boundary detection, or adaptive planning; this paper extends knowledge assessment into actionable workflow selection.

  • Agent-based multimodal retrieval: Multimodal agents commonly retrieve and integrate external visual and textual knowledge through sequential operations such as image retrieval and query refinement.These systems use RAG mechanisms to support reasoning in multimodal tasks.
  • Prompt-based mRAG: Prompt-based methods select retrieval actions through predefined action spaces and recurrent prompting, with OmniSearch decomposing complex questions into sub-question chains.Such methods depend on pretrained model capabilities and prompt engineering.
  • Knowledge-boundary methods: Knowledge-boundary detection identifies whether a model knows a query’s answer but does not determine how to solve the VQA query with external visual or textual information.The limitation is that classification alone does not select the needed retrieval components.
  • Position of this work: This paper extends knowledge-boundary assessment by dynamically selecting necessary components in a predefined workflow, enabling actionable and adaptive multimodal planning.The proposed framing moves from static knowledge assessment toward inference-time workflow decisions.

6. Conclusion

The paper presents a multimodal planning agent that dynamically selects mRAG steps, improving VQA efficiency and average performance across six datasets.

  • 6. Conclusion: The agent dynamically selects only necessary mRAG processing steps, mitigating inefficiency in static VQA pipeline architectures.The method is evaluated across six VQA datasets.
  • 6. Conclusion: Search time decreases by over 60% versus OmniSearch, while latency is reduced by 3–4.5× versus Deep Research agents.The reported efficiency gains accompany improved task performance on average.
  • 6. Conclusion: The results support adaptive multimodal agents that improve efficiency without sacrificing average VQA task performance.The conclusion frames adaptability and effectiveness as jointly achievable within the evaluated scope.

A.1. Prompts for Visual Query Decomposition and Correctness Checking

The paper uses distinct prompts for training-time and inference-time visual query decomposition because gold answers are available only during training-data construction.

  • A.1. Prompts for Visual Query Decomposition and Correctness Checking: Training data construction uses gold answers to annotate gold queries, whereas inference-time decomposition must operate without those answers.The prompt design therefore differs between the two stages.
  • A.1. Prompts for Visual Query Decomposition and Correctness Checking: The prompts support Visual Query Decomposition as part of constructing and applying the agent’s query-processing workflow.The passage specifically distinguishes annotation prompts used for gold query qg during training and inference.

Prompts for gold query annotation without image retrieval information (at inference stage)

The appendix details prompt and training configurations, evaluation checks, failure cases, annotation verification, and comparisons between LoRA and full fine-tuning.

  • Prompts for evaluating the correctness of model output: The appendix reports the evaluation prompt and scoring scale alongside implementation details, rather than introducing additional task-level findings.
  • A.2. Training Examples: The training examples use special <image> tokens whose format depends on the multimodal language model input format.The appendix introduces VQA queries with prompts before explaining the token convention.
  • A.3. Training Setting and Cost: LoRA training used 2 NVIDIA A100 SXM GPUs for 20 hours, whereas full fine-tuning used 4 GPUs for 25 hours.The detailed hyperparameter configuration is provided in Table 7.
  • Prompts for evaluating the correctness of model output: LLM evaluation scores are scaled to a 0–100 range, with GPT-4o and Qwen-Max differing mostly within 3 points.The consistency comparison addresses variation between scoring models.
  • A.5. Failure Case Study: The planning agent fails to choose a proper mRAG strategy in three representative cases.These cases are summarized in Figure 5 for the Qwen2.5-VL-7B-Inst task model.
  • A.6. Human Verification of Annotation Process: Human verification evaluates query-decomposition and correctness-checking accuracy on 100 random samples from each training set.Three Ph.D.-level NLP researchers performed the evaluation, with results reported in Table 9.
  • A.7. Full Fine-tuning Agent: LoRA produces comparable performance to full fine-tuning, while the fully fine-tuned agent fails on standard VQA queries.The LoRA-finetuned agent remains effective as the base model for VQA tasks; the comparison appears in Table 10.
  • A.8. Token Accuracy Metric: Four of five reported models outperform all baselines on average under token accuracy, while reducing retrieval calls in both training settings.Token accuracy is included as a static metric because LLM-based scoring may be unreliable and internally variable.
Loading 2601.20676v1…