Source-linked AI summary
Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning
Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray
TL;DR
Multimodal agents struggle to use tools reliably because they must integrate noisy retrieved evidence while learning intermediate decisions from sparse outcome-level supervision. SVRL is an RL-only framework that teaches agents to self-verify evidence and control search, yielding consistent multi-hop VQA and efficiency gains while reducing reliance on external inference-time verifiers.
Problem
Reliable multimodal tool use lacks explicit verification signals and is challenged by noisy evidence, poor search calibration, and sparse supervision for intermediate decisions.
Method
SVRL combines self-verification of retrieved evidence with search-aware penalties and query-diversity rewards in an RL finetuning objective.
Results
SVRL consistently improves accuracy and the accuracy–search tradeoff across FVQA-test and InfoSeek, narrowing the gap to substantially larger proprietary models.
Takeaways & Limitations
Self-verification, calibrated search decisions, and query diversity provide learnable trajectory signals for more efficient tool-augmented multimodal reasoning.
Takeaways & Limitations
LLM-judge rewards were vulnerable to reward hacking, producing longer and less decisive answers, so training uses exact-match correctness instead.
Abstract
from arXiv · showhide
Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.
1 Introduction
Multimodal tool-augmented reasoning is difficult because agents must decide when to search, filter noisy evidence, and learn intermediate tool-use decisions from sparse rewards. SVRL addresses these challenges with search-control rewards, explicit self-verification, and stabilized RL training.
- Problem: Multi-hop VQA requires recognizing visual entities and retrieving missing facts from external sources.The setting combines perception, tool calls, and noisy evidence integration.
- Problem: Over 10% of analyzed cases needed search but did not invoke it, while over 30% triggered unnecessary search.These errors expose poor calibration about when external evidence is needed.
- Problem: Nearly 45% of retrieved results were irrelevant, and answers were incorrect in over 28% of cases despite appropriate queries and relevant evidence.Retrieved pages may be irrelevant, outdated, or internally inconsistent, requiring explicit filtering.
- Problem: Sparse outcome-level rewards leave intermediate search and evidence decisions unlabeled, making credit assignment brittle and sometimes collapsing GRPO learning signals.19.5% of MMSearch-R1 text-search queries were irrelevant, reducing information in the final correctness reward.
- SVRL: SVRL augments GRPO with rewards that discourage unnecessary search and reward informative, diverse query proposals.These terms provide trajectory-level feedback about both when to search and what to search for.
- SVRL: SVRL makes evidence selection explicit and learnable through structured verification scores applied during inference without an external verifier.The framework also adopts Dr. GRPO and reports improved accuracy when increasing candidate queries at test time.
2 Related Work
Related work has advanced visual question answering, tool use, retrieval, verification, and test-time scaling, but SVRL targets their combination in knowledge-intensive multi-hop multimodal reasoning.
- Visual Question Answering: Knowledge-intensive VQA benchmarks stress compositional reasoning, text reading, and evidence acquisition beyond the image.Examples include OK-VQA, A-OKVQA, InfoSeek, WebQA, and LiveVQA.
- LLM finetuning for tool use: MLLMs acquire interactive tool behavior through instruction tuning, explicit reasoning, retrieval-augmented generation, and data-induced tool use.The paper reports a RAG workflow baseline to distinguish always-on retrieval from selective agentic search.
- Verification, reward design, and test-time scaling: Prior reliability methods use additional inference compute, external verification, process supervision, verifier training, or RL stabilization variants.These approaches provide context for SVRL’s focus on internal verification signals for multimodal tool use.
3 Self-Verification via Reinforcement Learning
SVRL augments GRPO with search-aware calibration, query-diversity rewards, and self-verification signals that train multimodal agents to select useful queries and evidence. Its multiplicative trajectory rewards improve tool calibration, evidence filtering, and training efficiency while mitigating reward-hacking risks.
- 3 Self-Verification via Reinforcement Learning: SVRL frames multi-hop VQA as multimodal policy optimization over trajectories containing reasoning steps, tool actions, and final answers.The agent is finetuned with KL-regularized GRPO using group-relative advantages and clipped PPO-style updates.
- 3.2 When and What to Search?: Search-aware calibration downweights unnecessary search on instances the pretrained model can answer without tools, preserving search incentives when external evidence is needed.The method labels search-free instances with a no-tool forward pass and sets λsa = 0.1; experiments report improved accuracy–search tradeoffs and tool calibration.
- 3.2 When and What to Search?: Query-diversity rewards differentiate rollouts by the number of valid, unique query proposals, encouraging multiple well-formed queries and more specific search intent.The query-count factor uses max(ϵ, k̂/k) when text search is used, with a small floor ϵ to avoid zero reward.
- 3.3 Verifying Search Results: Self-verification trains agents to propose queries and assign binary usefulness scores to retrieved items, aligning both with verifier labels during finetuning.The external verifier accesses the ground-truth answer during training, while the verifier is discarded at inference and the agent performs filtering itself.
- 3.3 Verifying Search Results: The multiplicative SVRL reward gates query and snippet alignment by exact-match correctness, producing fine-grained trajectory-level feedback while reducing additive reward-hacking opportunities.The resulting signal targets query quality and evidence filtering; verification supervision is applied only when the final answer is correct.
- 3.4 Practical Considerations: Reward-hacking experiments show that LLM-judge rewards can induce verbose or hedged answers, while removing search-aware calibration can drive search on nearly every example.Adding the search-aware factor reduced unnecessary search, and the full self-verification objective further improved accuracy and tool efficiency.
4 Experimental Setup
The experiments finetune Qwen-2.5-VL-7B-Instruct with SVRL and compare it with GRPO-based baselines and GPT-4o across visual question-answering benchmarks.
- Primary experiments finetune Qwen-2.5-VL-7B-Instruct with SVRL and evaluate it across all benchmarks.
- 5,000 FVQA examples are used for training, with 1,800 reserved for testing.FVQA includes both direct-answer and multi-hop questions requiring tools.
- The study evaluates FVQA and InfoSeek as in-distribution benchmarks and additional out-of-distribution benchmarks.
- GPT-5 determines final answer correctness for reported accuracy, while GPT-4o provides inference-time comparison results.The evaluation protocol accommodates aliases, physical units, and date-time format variations.
5 Results
SVRL improves multimodal VQA accuracy and the accuracy–search tradeoff while encouraging stronger evidence filtering and more effective test-time scaling. Its gains reflect better calibration, query behavior, and verification than prior adaptive-search baselines.
- VQA generalization: SVRL improves accuracy across benchmarks while achieving a better accuracy–search tradeoff than adaptive-search baselines.Table 2 reports accuracy and search ratio as complementary measures of tool use.
- Ablations: SVRL’s self-verification recovers most of external test-time verification’s denoising benefit without verifier dependence at inference.The ablation identifies evidence selection as a dominant failure mode and separates verification effects from other components.
- VQA generalization: SVRL-full-7B gains roughly 8 points on FVQA-test and 9 points on InfoSeek over MMSearch-R1++ while reducing search ratio.It approaches GPT-4o on FVQA-test, reaching 65.3 versus 66.0.
- Qualitative analysis: Qualitative cases show SVRL handling no-search, image-search, and multi-tool reasoning examples, including both correct and incorrect outcomes.The examples illustrate different tool requirements and multi-turn evidence integration.
- Tool-use analysis: SVRL reduces omitted searches when search is required and improves search precision and recall under search-free annotations.Figure 5 also measures whether relevant evidence appears among retrieved content for incorrect answers, isolating evidence use from retrieval quality.
- Test-time scaling: SVRL continues benefiting from additional parallel search budget, whereas MMSearch-R1++ saturates after roughly 5 searches.The scaling experiment evaluates up to 15 parallel text searches per question on 256 FVQA-test examples.
6 Conclusion
The paper presents SVRL as an RL-only framework for improving tool-augmented multi-hop VQA through self-verification in compact multimodal agents. Across FVQA-test and InfoSeek, it reports better accuracy–search tradeoffs and reduced reliance on external verification, while identifying practical limitations and future extensions.
- Conclusion: SVRL improves tool-augmented multi-hop VQA by eliciting self-verification within compact multimodal agents’ reasoning traces.The framework couples calibrated search decisions and query diversity with verifier-aligned supervision for filtering retrieved content.
- Conclusion: Across FVQA-test and InfoSeek, SVRL consistently improves accuracy and the accuracy–search tradeoff over prior GRPO-based baselines.The method also narrows the gap to substantially larger proprietary models.
- Conclusion: Ablations indicate that search calibration and self-verification drive most of the gains, while test-time verification measures the benefit of denoising noisy retrieval.The framework requires no external verifier at inference.
- Limitations and future work: The approach depends on web tools and their failure modes, training-time judge labels that may introduce bias, and sensitivity to prompt and tool formatting.Future work targets richer tool suites, stronger self-supervised verification, and robustness under retrieval distribution shift.
A Overview
The supplementary document adds methodological, analytical, and benchmark details beyond the main paper. It covers datasets, search tools, experimental setup, prompts, judge-based evaluation, evidence verification, and results for additional models.
- Supplementary overview: The supplementary document provides additional details on datasets, search tools, experimental setup, and prompts.It also includes further analyses of judge-based evaluation and evidence verification.
- Supplementary overview: Supplementary results include evaluations on Qwen3-VL-8B and GPT-5.
B Datasets
The experiments finetune Qwen2.5-VL-7B on FVQA and evaluate transfer across in-distribution and out-of-distribution multimodal question-answering benchmarks. The suite tests search calibration, evidence use, modality and language variation, timely retrieval, and broader tool-use boundaries.
- Dataset setup: Qwen2.5-VL-7B is finetuned on FVQA and evaluated on InfoSeek, MMSearch, SimpleVQA, and LiveVQA.The benchmarks test complementary aspects of search-augmented multimodal question answering.
- Training dataset: FVQA mixes search-free and search-required examples, supporting learning when search is needed rather than encouraging constant tool use.An initial no-tool forward pass provides a proxy label for shaping search decisions.
- In-distribution evaluation: InfoSeek provides an in-distribution test of transfer beyond the specific FVQA training subset.It is broader in scope while remaining closely tied to the FVQA benchmark family.
- Out-of-distribution evaluation: MMSearch tests out-of-distribution multimodal web search with text-only and multimodal questions, noisy evidence, and search-result integration.
- Out-of-distribution evaluation: SimpleVQA broadens transfer evaluation across image-grounded and text-centric questions, multiple-choice formats, and English and Chinese.
- Out-of-distribution evaluation: LiveVQA stress-tests search behavior when timely external knowledge matters more directly.
- Benchmark scope: The benchmark suite remains centered on factual question answering with search and does not fully cover longer-horizon interaction, richer tool use, or persistent memory.Extending evaluation in these directions is identified as future work.
C Multimodal Search Tools
The agent combines image and text search with caching and randomized evidence sampling to manage retrieval cost and noise.
- Text search uses a search-parse-summarize pipeline, while image search returns visually relevant web results for an input image.
- Caching and efficiency: Caching exact and approximately matching queries reduces text-search cost by up to 40%.Approximate reuse requires Jaccard similarity above 0.85.
- Returned evidence and noise: The system extracts up to six webpage summaries and randomly returns two to six to avoid overfitting to a fixed evidence pattern.Returned summaries may be inaccessible, weakly related, or otherwise noisy.
D Experimental Setup
Experiments finetune Qwen2.5-VL-7B with short ReAct-style trajectories, explicit verification masks, structured rewards, and judge-based evaluation, including search-calibration and scaling analyses.
- Base model and prompting: Qwen2.5-VL-7B is finetuned with SVRL over trajectories allowing direct answers, one image search, or up to two text searches.Episodes use at most three interaction steps and an 8124-token context budget.
- Base model and prompting: The ReAct-style format requires reasoning blocks before actions, which can be image search, text search, or final answering.
- Verification and query generation: After each tool call, the model emits a binary verification mask; training samples one of five candidate queries, while test-time scaling can execute all candidates.
- Reward and evaluation: The trajectory reward combines exact-match accuracy, format reward, and multiplicative SVRL shaping terms.The reported reward is rsvrl = raware · rcount · rqalign · rsalign.
- Query diversity and test-time scaling: SVRL generates an average of 4.86 unique queries per text-search step on FVQA, and increased inference-time search budgets are evaluated with consensus aggregation.The aggregation first checks exact-string majorities, then uses embedding-based consensus when needed.
- Search calibration: 61.4% of FVQA examples invoke search versus 80.3% for MMSearch-R1++, while SVRL improves search-subset accuracy from 51.0% to 56.4%.No-search accuracy also rises from 78.6% to 79.4%.
- Evidence verification: Verification-mask agreement with GPT-5.0 labels is 71.3%, but the model under-filters noise far more often than it over-filters useful evidence.There are 1495 under-filtering disagreements versus 82 over-filtering disagreements.
F Additional Results
Additional experiments show that SVRL transfers to another multimodal backbone, outperforms stronger proprietary references in the reported search setting, and depends on evaluation choice.
- SVRL on Qwen3-VL-8B: 64.9 FVQA and 64.0 InfoSeek accuracy are achieved by SVRL-8B, versus 56.4 and 55.1 for MMSearch-R1++.These results indicate transfer beyond the Qwen2.5-VL-7B backbone.
- Comparison to GPT-5.2: SVRL-7B and SVRL-8B achieve higher GPT-5.0 judge-based accuracy than GPT-5.2-thinking on FVQA and InfoSeek under search-based evaluation.The comparison concerns compact open-source models and a larger proprietary reference in the reported search regime.
- Evaluation metrics: Exact-match accuracy is substantially lower than GPT-5.0 judge-based accuracy across nearly all methods, especially proprietary models.The judge metric is more tolerant of paraphrases and semantically equivalent answers.
H Limitations
The method remains dependent on verifier-generated labels and judge-based evaluation, while its evidence base and experimental scope constrain broader claims about multimodal agency.
- Training alignment rewards rely on verifier-generated labels with ground-truth access, introducing dependence on teacher-model biases.
- GPT-5.0 judge evaluation improves tolerance for aliases and paraphrases but is not fully reproducible and may depend on prompt phrasing.
- Web retrieval remains noisy and non-stationary, and the learned filter retains noisy evidence more often than it discards useful evidence.
- Experiments cover short-horizon search-augmented VQA with a restricted action space, not general multimodal agency.Test-time scaling also uses a simple consensus-based answer aggregation rule.