PaperlayerView exact revision on arXiv

Source-linked AI summary

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang

arXiv:2610.22240v1cs.CV

TL;DR

Long-horizon multimodal search needs challenging training questions and process feedback beyond terminal rewards. MMSearch-OPD refines existing questions and combines RL with on-policy distillation. Its 9B agent improves average accuracy over its agentic-workflow baseline, with additional gains from joint training beyond SFT.

  • Problem

    Training long-horizon multimodal search is constrained by questions solvable through parametric knowledge or short retrieval and by limited process feedback from terminal rewards and shared trajectory-level advantages.

    Source: §1 · p. 1

  • Method

    MMSearch-OPD refines existing multimodal questions to control search depth and evidence requirements, then jointly trains with answer rewards, behavior penalties, and reward-gap-weighted token-level OPD on imperfect student trajectories.

    Source: §1 · pp. 1–2

  • Results

    The 9B agent improves average accuracy by 15.3 percentage points over its agentic-workflow baseline, with joint training adding 2.3 points beyond SFT.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results support the value of upgrading existing multimodal data with refined questions for long-horizon search.

    Source: §5 · p. 8

    Open-web search relies on changing content and tool availability, limiting reproducibility.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.

1 Introduction

MMSearch-OPD addresses limited challenging training questions and insufficient process feedback for long-horizon multimodal search. It refines existing questions and combines reinforcement learning with on-policy distillation, with evaluation organized across ten benchmarks.

1 Introduction

  • Existing multimodal datasets often contain questions solvable through parametric knowledge or short retrieval, while harder-question construction requires evidence collection, image curation, and validation.Long-horizon investigation can also involve redundant verification, premature stopping, or incorrect answers, while terminal rewards and trajectory-level advantages provide limited process feedback.Source: §1 · p. 1
  • Refine reuses simple multimodal questions and candidate reasoning chains, applying configurable operators to increase chain depth, entity disambiguation, and visual evidence requirements.Expand extends chain depth, Blur increases disambiguation requirements, and Visualize introduces retrievable visual evidence.Source: §1 · pp. 1–2
  • Joint RL and OPD combine answer rewards and behavior penalties with reward-gap-weighted token guidance on imperfect student trajectories, while exponentially reducing the OPD coefficient during training.The method applies this training after question validation and supervised initialization.Source: §1 · pp. 1–2
  • Evaluation is organized across ten multimodal search benchmarks to assess agent performance and the roles of data refinement and joint training.Source: §1 · p. 2

2 Related Work

Related work extends multimodal search toward longer investigations and develops harder training data and feedback methods. The paper situates its approach alongside these efforts and their remaining challenges.

2 Related Work

3 Method

MMSearch-OPD treats long-horizon multimodal search as sequential evidence acquisition, refines existing question chains to raise evidence demands, and combines RL with selective teacher guidance.

3 Method

  • Refine expands existing multimodal questions, while joint RL–OPD uses answer rewards, behavior penalties, and reward-gap-weighted teacher guidance on student trajectories.The distillation coefficient decays exponentially as training proceeds.Source: §3 · pp. 2–3, §3.4 · p. 5

3.1 Problem Formulation

  • The task is sequential: later search actions depend on accumulated textual and visual evidence, and each designated chain item must be necessary for a unique answer.The criterion applies to the intended chain; later filtering checks for shortcuts through parametric knowledge or alternative web evidence.Source: §3.1 · p. 3

3.2 Data Curation

  • Seeds come from LiveVQA and FVQA training questions that the data-generation model answers correctly without tools, with successful reasoning chains retained as candidates.Source: §3.2 · p. 3

3.2.1 Refine Pipeline

  • BuildLink extracts entity chains, then Expand verifies and extends them, Blur adds disambiguating facts, and Visualize adds retrieved visual evidence under separate parameters.Question synthesis replaces selected entity names with indirect clues while preserving the chain, answer, and images.Source: §3.2.1 · pp. 3–4

3.2.2 Quantifying Search Difficulty

  • The refined-chain burden counts added textual and visual evidence, assuming additions remain necessary and shared facts are not double-counted.It is not a global minimum search route, tool-turn count, or model success-rate estimate.Source: §3.2.2 · p. 4

3.3 Quality Control and Trajectory Generation

  • Validation checks solvability, answer uniqueness, chain consistency, evidence provenance, image–text alignment, and actual use of designated evidence.Source: §3.3 · p. 5
  • Questions solved by a no-tool probe are discarded; expert Pass@k rollouts assess learnability, and successful trajectories are sampled across difficulty levels for SFT.The expert’s empirical Pass@k complements the chain-based evidence burden and informs the SFT/RL split.Source: §3.3 · p. 5

3.4 Agentic Training

  • A larger model is fine-tuned on successful trajectories and frozen as teacher; the student is SFT-initialized before joint RL and OPD on harder questions.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate MMSearch-OPD across multimodal search benchmarks and test data refinement and joint training. Results report improved accuracy after refinement and further gains from joint RL–OPD training, alongside changes in search behavior.

4.1 Experimental Setup

  • Evaluation uses seven multimodal search benchmarks, averages three correctness judgments per prediction, and reports their unweighted mean.The compared baselines include tool-free Direct Answer, Agentic Workflow, and trained search agents grouped by model scale.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 60.2 average accuracy: MMSearch-OPD-9B improves 15.3 points over its agentic-workflow baseline and exceeds DeepVoyager-VL-8B by 9.6 points on average.It also achieves the highest reported accuracy among compared 7B–9B search agents on all seven benchmarks.Source: §4.2 · p. 6, Table 1 · p. 6
  • The 9B SFT checkpoint rises from 44.9 to 57.9 average accuracy, and joint RL–OPD training raises it further to 60.2, improving every benchmark.The 2.3-point average gain supports joint online optimization after SFT.Source: Table 1 · p. 6, §4.2 · p. 7

4.3.1 Data Refinement Ablation

  • With identical initialization, SFT configurations, and training budgets, Refine improves accuracy on all four benchmarks, with a +15.0-point gain on BrowseComp-V3.SimpleVQA, FVQA, and VDR improve by 2.0, 6.3, and 6.8 points, respectively.Source: §4.3.1 · p. 7, Table 2 · p. 7
  • After Refine, average tool calls per trajectory increase from 6.61 to 33.71, while ImageSearch and FetchImage invocations per question rise from 0.38 to 4.88 and 0.51 to 3.47.The search-query distribution shifts toward longer searches, alongside the SFT gains.Source: §4.3.1 · p. 7, Figure 3 · p. 7

4.3.2 Refine Pipeline Ablation

  • Across Expand, Blur, and Visualize, accuracy falls from 79% to 65%, 44%, and 33% as search requirements strengthen.Blur raises average turns from 10.2 to 15.1; Visualize raises them to 20.2 and tool calls to 34.7 per question, while ImageSearch and FetchImage grow from 6.3% to 47.5% of tool actions.Source: §4.3.2 · pp. 7–8, Figure 4 · p. 8

4.4 RL Training Analysis

  • During joint RL–OPD training, mean answer reward rises from 0.347 to 0.587 over 19 steps, while later search uses fewer tools and tokens with comparable reward.Comparing steps 6–13 with 14–19, tool calls fall from 26.0 to 21.3 and response length from 29.0k to 23.4k tokens; mean answer reward is 0.525 versus 0.544.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Conclusion

MMSearch-OPD combines controllable data refinement with joint RL–OPD training for long-horizon multimodal deep search. Its results support upgrading existing multimodal data and combining outcome optimization with selective teacher guidance.

Conclusion

  • Across seven primary benchmarks, the 9B agent improves average accuracy by 15.3 percentage points over its agentic-workflow baseline, with joint training adding 2.3 points beyond SFT.Source: §5 · p. 8
  • Refine reuses existing multimodal question–image pairs and applies targeted transformations to control search depth, entity disambiguation, and visual evidence requirements.Source: §5 · p. 8
  • Joint training combines answer rewards and behavior penalties with reward-conditioned token-level guidance on imperfect student trajectories, while progressively reducing teacher influence.Source: §5 · p. 8
  • Data ablations support the training value of refined questions, and the results support sustained evidence acquisition and joint outcome optimization with selective teacher guidance.Source: §5 · p. 8

Limitations

The limitations include residual shortcuts, dependence on teacher and answer-judge quality, and reduced reproducibility in open-web search.

Limitations

  • Filtering cannot rule out all shortcuts from parametric knowledge or alternative web evidence.Source: Limitations · p. 9
  • Training depends on teacher and answer-judge quality, which may introduce noisy guidance.Source: Limitations · p. 9
  • Changing open-web content and tool availability limit reproducibility.Source: Limitations · p. 9

Related papers

Loading 2610.22240v1…