PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents

Shijue Huang, Hangyu Guo, Guanting Dong, Chenxin Li, Junting Lu, Xinyu Geng, Zhaochen Su, Zhenyu Li, Shuang Chen, Hongru Wang, Yi R. Fung

arXiv:2605.10832v2cs.CL

TL;DR

Multimodal deep-search systems struggle to reuse tool-produced visual evidence and to adapt training data to the evolving policy. This paper combines an image-bank harness with On-policy Data Evolution, raising Qwen3-VL’s average accuracy from 24.9% to 39.0% at 8B and from 30.6% to 41.5% at 30B across eight benchmarks.

  • Problem

    Existing multimodal deep-search approaches often treat tool-produced visual outputs as non-reusable evidence, while data synthesis lacks closed-loop modeling of agent search behavior.

  • Method

    The paper combines an image-bank visual-native harness with On-policy Data Evolution, which refines synthesized tasks using policy rollouts and rubric-based trace analysis.

  • Results

    Across eight benchmarks, ODE improves Qwen3-VL average accuracy from 24.9% to 39.0% at 8B and from 30.6% to 41.5% at 30B.

  • Takeaways & Limitations

    Reusable tool-generated images aid multi-step visual evidence gathering, while evolved data yields higher-quality and more diverse teacher traces than static synthesis.

Abstract

from arXiv · show

Multimodal deep search requires an agent to solve open-world problems by chaining search, tool use, and visual reasoning over evolving textual and visual context. Two bottlenecks limit current systems. First, existing tool-use harnesses treat images returned by search, browsing, or transformation as transient outputs, so intermediate visual evidence cannot be re-consumed by later tools. Second, training data is usually built by fixed curation recipes that cannot track the target agent's evolving capability. To address these challenges, we first introduce a visual-native agent harness centered on an image bank reference protocol, which registers every tool-returned image as an addressable reference and makes intermediate visual evidence reusable by later tools. On top of this harness, On-policy Data Evolution (ODE) runs a closed-loop data generator that refines itself across rounds from rollouts of the policy being trained. This per-round refinement makes each round's data target what the current policy still needs to learn. The same framework supports both diverse supervised fine-tuning data and policy-aware reinforcement learning data curation, covering the full training lifecycle of the target agent. Across 8 multimodal deep search benchmarks, ODE improves the Qwen3-VL-8B agent from 24.9% to 39.0% on average, surpassing Gemini-2.5 Pro in standard agent-workflow setting (37.9%). At 30B, ODE raises the average score from 30.6% to 41.5%. Further analyses validate the effectiveness of image-bank reuse, especially on complex tasks requiring iterative visual refinement, while rollout-feedback evolution yields more grounded SFT traces and better policy-matched RL tasks than static synthesis.

1 Introduction

The paper addresses multimodal deep search bottlenecks by making intermediate visual evidence persistently reusable and adapting training data to the evolving policy. Its framework combines a Visual-Native Agent Harness with On-policy Data Evolution (ODE) across supervised and reinforcement-learning data construction.

  • Motivation: Multimodal deep search extends MLLMs from static image question answering to open-world interaction with search engines and external tools.Agents gather evidence in real time to produce grounded answers.
  • Challenges: Two bottlenecks are persistent visual-state underuse in tool-augmented search and fixed training-data recipes that cannot track changing agent capabilities.Existing approaches often center reasoning on the original task image instead of treating returned visual evidence as reusable state.
  • Visual-Native Agent Harness: The Visual-Native Agent Harness unifies 9 core tools around an image-bank reference protocol, making tool-produced visual evidence persistently reusable throughout trajectories.Search, browsing, and visual manipulation operate over shared addressable image references.
  • On-policy Data Evolution: ODE repeatedly synthesizes candidate tasks, rolls out the target policy, analyzes traces with rubrics, and revises subsequent data synthesis rather than using a fixed recipe.The framework supports both SFT-style teacher-trace curation and policy-aware RL data construction.

2 Method

Section 2 introduces a visual-native harness that makes tool-returned images reusable across multimodal search, then builds ODE as a rollout-feedback loop that evolves data-generation configurations for SFT and RL. The method combines nine-tool evidence gathering, multimodal task construction, rollout diagnosis, and rubric-guided configuration updates.

  • Visual-Native Agent Harness: The image bank reference protocol registers tool-returned images so visual evidence can propagate across later tool calls.This addresses the limitation that existing approaches tie visual operations to the original task image and cannot reuse returned images.
  • Visual-Native Agent Harness: The harness models tasks as T = (q, I, a) and supports nine tools spanning retrieval, browsing, visual search, image transformation, and computation.The initial visual context I is loaded into the image bank, and the policy gathers multimodal evidence before answering.
  • On-policy Data Evolution: ODE separates a fixed System Config from an evolvable Config Ct that controls seed proposal, web exploration, graph organization, and task curation.These four forward stages convert open-world evidence into verifiable multimodal deep search tasks.
  • On-policy Data Evolution: Each candidate task is rolled out in the harness, producing a trace with message history, Image Bank references, final answer, and an LLM-judged success or failure label.SFT uses a teacher model’s successful rollouts as demonstrations, whereas RL rolls out the current policy to assess task suitability.
  • On-policy Data Evolution: ODE aggregates trace diagnoses into a round-level signal Δt that edits Ct into Ct+1, revising the flagged generation stage rather than optimizing only current-batch rollout success.Rubric-guided optimization steers later generation toward stronger evidence, source support, and training utility.
  • On-policy Data Evolution: The SFT demonstration set covers all eleven topical domains, while the two RL sets cover the same domain range.Figure 2 reports topical-domain coverage and curator-annotated difficulty for SFT, ODE-8B, and ODE-30B-A3B data sets.

3 Experiments

Across eight multimodal deep-search and reasoning benchmarks, ODE substantially improves Qwen3-VL agents under a visual-native harness. Analyses attribute these gains to reusable visual evidence and rollout-feedback evolution that produces more effective SFT data and deeper-search RL tasks.

  • Datasets and baselines: The evaluation covers 8 multimodal deep-search and related multimodal-reasoning benchmarks, including MMBC, HLE-VL, BC-VL, VDR, MMSearch, MMSearch+, SVQA, and FVQA.The paper compares proprietary and open-source multimodal models and agents under three evaluation settings.
  • Main results: ODE improves Qwen3-VL-8B average accuracy from 24.9% to 39.0% and Qwen3-VL-30B from 30.6% to 41.5% under the same visual-native harness.Gains are largest on benchmarks requiring iterative multimodal search, inspection, and evidence integration.
  • Main results: Tool access alone does not yield tool competence: standard agent-workflow baselines remain far below agents trained on ODE-curated data.ODE-trained agents learn when to search, what to inspect, and how to integrate multimodal evidence across multiple steps.
  • Visual-native harness ablation: The full harness preserves every tool-returned image as an addressable <image:N> reference, enabling later tools to consume intermediate visual evidence.The ablated harness displays returned images but removes reusable references, preventing their passage into later image-consuming tools.
  • Static synthesis versus data evolution: Evolved SFT data outperforms static-recipe data on most benchmarks, with clear gains on visually grounded and search-oriented evaluations.This supports feedback-driven evolution as more effective imitation-data construction than forward synthesis alone at the same sample scale.
  • Mechanism analysis: ODE evolution induces RL rollouts with substantially more tool calls, dynamic images, and image-input calls, indicating deeper policy-facing search.For SFT, evolution selects trajectories teaching the model to inspect, reuse, and integrate visual evidence rather than merely execute many tools.

4 Related Work

Prior work has advanced multimodal agents for web search, browsing, visual grounding, and provenance verification, while synthetic-data methods increasingly support tool-using agent training. However, multimodal deep-search datasets and trajectories largely remain dependent on predefined synthesis recipes or close supervision.

  • Multimodal deep-search agents: Multimodal search agents extend MLLMs with web search and increasingly emphasize deeper reasoning, fine-grained visual grounding, and provenance verification.Early systems include MMSearch and Vision Search Assistant, followed by benchmarks that raise these requirements.
  • Synthetic data for agent training: Synthetic-data frameworks generate diverse post-training data, verified function-calling examples, and multi-turn interaction trajectories for LLM-based agents.These approaches use agentic flows over raw documents and code or focus specifically on tool use.
  • Synthetic data for multimodal deep search: Multimodal deep-search datasets and trajectories span semi-automated search VQA, automated real-web-search pipelines, cold-start training, and long-horizon multi-tool visual-textual search.Examples include MMSearch-R1, DeepMMSearch-R1, WebWatcher, and Vision-DeepResearch.
  • Limitations of prior synthesis: Most existing multimodal deep-search methods still rely on predefined synthesis recipes or close supervision.The cited works demonstrate the value of synthetic supervision but retain largely fixed data-construction procedures.

5 Conclusion … A.5 Stage 3: Graph Organization

The paper combines a visual-native harness that makes tool-generated images reusable with On-policy Data Evolution, which adapts multimodal training data to the current policy. Its appendices instantiate this loop through configurable stages for seed proposal, web exploration, graph organization, and rubric-based evaluation.

  • 5 Conclusion: ODE raises average accuracy from 24.9% to 39.0% at 8B and from 30.6% to 41.5% at 30B across eight benchmarks.The gains follow SFT and RL, while reusable tool-generated images support multi-step visual evidence use.
  • A.1 Round Configuration Ct: ODE freezes the System Config while optimizing the four-stage Evolvable Config Ct between rounds.The frozen components enable matched round comparisons, while editable parameters govern seed proposal, exploration, graph organization, and task curation.
  • A.2 Mode-Specific Trace Rubrics: The analyzer scores each rollout on seven dimensions from −5 to +5 and computes a weighted overall score with stage-level attribution.Shared dimensions include Information_Complexity, Visual_Dependency, Shortcut_Leakage, and Verifiability; the remaining dimensions are mode-specific.
  • A.2 Mode-Specific Trace Rubrics: The RL rubric selects tasks that are judgeable, evidence-supported, and useful for optimizing the current policy.It rewards multi-source evidence integration and visual dependence while rejecting memorization, common-knowledge, single-query, or text-only solutions.
  • A.3 Stage 1: Seed Proposal: Seed proposal pairs a real entity with an information-bearing image capable of supporting visual extraction, multi-hop verification, and computation.The prompt prioritizes maps, placards, diagrams, archival documents, posters, charts, and timelines containing labels, dates, quantities, coordinates, or legends.
  • A.4 Stage 2: Web Exploration: The explorer expands each seed into a six-node information network by visiting twelve URLs and recording textual facts, source URLs, and image identifiers.Image-bearing nodes specify what can be extracted visually and why the evidence is useful downstream, with requirements for multiple tool calls, numeric or date facts, and independent sources.
  • A.5 Stage 3: Graph Organization: The graph organizer converts collected nodes into a multimodal evidence graph G and adds reasoning and perception nodes for consistency checks and fine-grained visual grounding.In the illustrated run, zoom_in passes recover Trust Territories shading and a Western Sahara administering-power annotation under perception_ratio = 0.40.

A.6 Stage 4: Task Curation

Stage 4 curates hard-but-learnable multimodal tasks by combining image evidence, web-only facts, and synthesis or computation. A worked example illustrates this process through explicit visual grounding, web retrieval, calculation, and complexity enhancement.

  • Worked Example: The worked example asks agents to identify and count depicted UN-administered trust territories from a September 1948 map, excluding ordinary dependencies, colonies, and Somaliland.The question then compares the depicted count with the original UN trusteeship-system set.
  • Worked Example: The task workflow uses image labeling, web_search, and calculate to extract map facts, retrieve the original-set count and Somaliland exclusion, and compute a rounded percentage.The visual evidence is registered in the image bank as I0 before rollout.
  • Worked Example: Curator complexity enhancement rewrote an easier percentage question by adding an explicit Somaliland-exclusion check and removing the verbatim total.This changed the task from directly asking the percentage to requiring evidence retrieval and verification.
  • Task Curation Prompt Fields: The curator prefers hard-but-learnable tasks that combine a key image fact, a web-only fact, and synthesis, comparison, or computation.The prompt also excludes trivial, brittle, or unsolvable tasks and encourages expert difficulty when the evidence path remains clean and judgeable.

A.7 Stage 5: Rollout and Stage 6: Backward Analysis · A.8 Optimizer Update to Ct+1 · A.9 Round t+1 Forward Under Ct+1

The pipeline evaluates policy rollouts, traces failures back to forward stages, and uses aggregated diagnoses to update the next-round configuration. Under Ct+1, the loop produces denser visual evidence and more fine-grained task grounding for the next rollout.

  • A.7 Stage 5: Rollout and Stage 6: Backward Analysis: The verifier marks the candidate answer incorrect, after which the analyzer scores the rollout and attributes failures to specific forward stages.The analysis evaluates the complete rollout rather than only the final answer.
  • A.7 Stage 5: Rollout and Stage 6: Backward Analysis: The seed proposer receives a moderate-severity diagnosis because insufficient identity, provenance, and resolution constraints allowed map drift and later visual misclassifications.Affected dimensions include Difficulty_Match, Learning_Utility, Visual_Dependency, and Verifiability.
  • A.8 Optimizer Update to Ct+1: The optimizer aggregates batch diagnoses into ∆t, applies edits to Ct, writes Ct+1, and appends analyzer-suggested rejection rules to the strategy and requirement prompts.The next-round configuration differs from Ct in four numerical fields.
  • A.8 Optimizer Update to Ct+1: The four numerical updates are seed_proposer.max_steps: 8 → 10, explorer.params.image_ratio: 0.50 → 0.40, reasoning_max_steps: 5 → 6, and perception_max_steps: 4 → 5.The changes target seed-stage drift, visual-evidence quality, and graph-organizer reasoning and perception capacity.
  • A.8 Optimizer Update to Ct+1: The graph-organizer increases provide room for legend-category nodes and an extra cross-source consistency check, while curator fields remain unchanged because upstream fixes address their diagnosed causes.The batch pass-rate signal also leaves curator.few_shot_difficulty_weights unchanged because the [too_hard] share is below threshold.
  • A.9 Round t+1 Forward Under Ct+1: In round t+1, the updated Ct+1 drives a new seed through the same four stages, exposing a different failure mode for the next optimizer update.The updated explorer surfaces a zoomed NOAA-chart segment, while the curator grounds the task on a fine-grained channel reach and denser visual evidence.

A.10 Round t+1 Backward and Update to Ct+2

Round t+1 exposes an informative failure mode: a visually dependent, multi-step USACE verification task scores 2.60 overall. The optimizer responds by slowing exploration, increasing graph-organizer reasoning and perception budgets, and rolling back the image-ratio change.

  • Round t+1 is rolled out by the same policy and scored with the same rubric, with the new failure mode treated as informative.
  • 2.60 overall rubric score is reported for the round t+1 candidate.
  • The candidate requires multi-step disambiguation of similarly named USACE FY2025 dredging efforts, official-source verification, cost-per-yard arithmetic, and chart-based program identification.Information_Complexity, Visual_Dependency, Verifiability, and Capability_Requirement each receive 5.0.
  • The optimizer reduces explorer.params.max_nodes_per_phase from 2 to 1, raises reasoning_max_steps from 6 to 7 and perception_max_steps from 5 to 6, and rolls explorer.params.image_ratio back from 0.40 to 0.50.The [too_hard] signal motivates deeper traversal of fewer nodes and additional graph-organizer enrichment headroom.

A.11 Additional Statistics of ODE-Curated Data · B More on Experimental Setup

The appendix reports that ODE preserves broad topical coverage across SFT and policy-specific RL data while adapting planned reasoning depth to policy capacity. It compares ODE-8B, ODE-30B, and the SFT demonstration set using topical-domain and reasoning-step statistics.

  • A.11 Additional Statistics of ODE-Curated Data: ODE-8B and ODE-30B span the same eleven topical domains as the SFT demonstration set.The comparison appears in the topical-domain donuts and covers both RL task sets.
  • A.11 Additional Statistics of ODE-Curated Data: ODE-30B limits per-domain shares to 8.43%–10.03%, with a coefficient of variation of 0.05.ODE-8B falls within a comparable topical-share band.
  • A.11 Additional Statistics of ODE-Curated Data: The forward exploration stage preserves topical coverage across SFT and policy-specific RL data construction.Difficulty distribution can still shift between SFT and RL construction.
  • A.11 Additional Statistics of ODE-Curated Data: ODE-8B concentrates 70.58% of tasks in the 5–6-step reasoning bucket.The planned reasoning-step distribution is shown as a depth ladder.
  • A.11 Additional Statistics of ODE-Curated Data: ODE-30B places 81.22% of tasks in the ≥9-step reasoning bucket.Its planned reasoning depth extends farther than ODE-8B’s concentration at 5–6 steps.
  • A.11 Additional Statistics of ODE-Curated Data: The SFT demonstration set averages 8.47 reasoning steps, inherited from the teacher.It sits at the deep end of the planned reasoning-step distribution.
  • A.11 Additional Statistics of ODE-Curated Data: Figure 10 combines topical-domain donuts for two RL task sets with planned reasoning-step distributions for SFT and RL data.The figure complements the statistics discussed in Section 2.3 and Fig. 2.

B.1 Data Construction

ODE constructs both supervised fine-tuning and reinforcement-learning data through an evolvable configuration initialized with GPT-5.2. The configuration is refined for up to five steps using 32 curated tasks and verified traces per step.

  • Data Construction: ODE curates both SFT and RL training data, using GPT-5.2 for most generation, analysis, and optimization stages.GPT-5.2 also serves as the SFT rollout policy, while the trained policy performs task-verification rollouts in RL mode.
  • Data Construction: The evolvable configuration is initialized with GPT-5.2 and refined for a maximum of 5 evolution steps.Each step uses curated tasks and verified traces for rubric-guided configuration updates.
  • Data Construction: 32 curated tasks and verified traces are used at each evolution step for rubric-guided configuration updates.The passage describes these per-step inputs as part of the configuration-refinement process.

B.2 Training setup.

The training setup uses Qwen3-VL 8B and 30B backbones, followed by ODE-curated supervised fine-tuning and reinforcement-learning refinement. SFT uses the same 64k-token, batch-size-64, two-epoch configuration for both models.

  • Training setup: The agent is instantiated with Qwen3-VL-8B-Instruct and Qwen3-VL-30B-A3B-Instruct backbones.
  • Training setup: SFT uses ODE-curated data with a maximum sequence length of 64k tokens, global batch size 64, learning rate 2 × 10−5, and 2 training epochs.
  • Training setup: Reinforcement learning further refines both agents starting from their SFT checkpoints.

B.3 Evaluation Setup

The evaluation uses standardized decoding and interaction budgets across all models, with a shared LLM-as-judge verifier producing terminal answer-level rewards. The verifier extracts final answers and grades semantic equivalence conservatively rather than requiring exact wording.

  • All models use temperature 0.6, top-p 0.95, at most 50 LLM calls, 8,192 tokens per turn, and 16,000 total tokens.
  • A shared LLM-as-judge verifier extracts each candidate final answer and computes the trajectory’s terminal answer-level reward.
  • The judge accepts answers that clearly match the reference semantically, while distinguishing exact, format, semantic, wrong, missing, and ambiguous outcomes.

B.4 Benchmark Details

The evaluation spans eight multimodal benchmarks covering browsing, academic reasoning, visual question answering, and long-horizon deep research. Unless otherwise specified, experiments use released questions with a unified agent harness and an LLM-based answer judge.

  • Evaluations use released benchmark questions with a unified agent harness and an LLM-based answer judge unless otherwise specified.
  • MMBC: MMBC tests multi-hop browsing that retrieves and reasons over web evidence in images or videos, with fine-grained multimodal dependency requirements.The evaluation uses the released MMBC evaluation set.
  • HLE-VL and BC-VL: HLE-VL measures visual interpretation combined with specialized academic reasoning, while BC-VL requires cross-modal inference, web search, browsing, and planning.HLE-VL uses an expert-level, broad-subject benchmark; BC-VL uses the full split.
  • Deep-search benchmarks: VDR emphasizes long-horizon, multi-turn, multi-entity, multi-scale evidence gathering; MMSearch requires external evidence retrieval; MMSearch+ adds provenance-aware visual cue extraction and cross-validation.The evaluation uses the test-mini split of VDR, all VQA instances in MMSearch, and the single-image subset of MMSearch+.
  • Visual question answering: SimpleVQA and FVQA each contribute 300 randomly sampled instances, evaluating factual visual question answering with reliable grounding or external factual knowledge.SimpleVQA focuses on short factual visual questions; FVQA associates each question with supporting facts.
Loading 2605.10832v2…