PaperlayerView exact revision on arXiv

Source-linked AI summary

OneSearch-VL: Unified Multimodal Deep Research Agent for Image and Video

Hongyu Li, Manyuan Zhang, Kaituo Feng, Shu Chen, Dian Zheng, Hao Li, Hao Yu, Zhangquan Chen, Zoey Guo, Ray Zhang, Shaofei Huang, Tianrui Hui, Linjiang Huang, Si Liu

arXiv:2610.12419v1cs.CV

TL;DR

Image and video deep research share a workflow, but existing evidence structures do not preserve dependencies from visual anchors through sourced facts to answer composition. OneSearch-VL uses VGEG to unify training, evidence-aware trajectory rewards, and operation-oriented benchmarks; it outperforms Qwen3-VL-8B on the new multi-image and video benchmarks.

  • Problem

    Image and video research require different visual operations but share grounding, retrieval, and fact composition, while prior structures do not jointly preserve dependencies through answer composition.

    Source: §1 · pp. 1–2

  • Method

    OneSearch-VL uses VGEG to connect visual anchors, source-supported facts, and answer operations for unified agent training, EVGR supervision, and task construction.

    Source: §1 · p. 2

  • Results

    20.2 and 17.6 points: OneSearch-VL-8B exceeds Qwen3-VL-8B on OneSearch-MI-Bench and OneSearch-Video-Bench, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OneSearch-VL consistently improves over evaluated open-source baselines across seven established single-image benchmarks and two operation-oriented benchmarks.

    Source: §8 · p. 12

    Changing webpages and reliance on external search tools can affect evidence availability and exact reproducibility.

    Source: §9 · p. 12

Abstract

from arXiv · show

Single-image, multi-image, and video deep research require different visual operations but share a workflow of visual grounding, external retrieval, and fact composition. A key challenge is to preserve the dependencies linking localized visual anchors, entity relations, source-supported facts, and answer-producing operations. We introduce OneSearch-VL, a unified agent centered on the Visually Grounded Evidence Graph (VGEG), which encodes these dependencies as a shared task-level reference for data construction, process supervision, and operation-level evaluation. Our VGEG-based data engine constructs and verifies multi-image and video questions and filters expert trajectories. Using these data, we assemble OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K for SFT and RL, respectively. We further derive the Evidence-aware Visual-Grounded Rubric reward (EVGR) from VGEG annotations to supervise evidence traceability and visual grounding during RL. For fine-grained evaluation, we construct OneSearch-MI-Bench and OneSearch-Video-Bench, organizing questions by the research operations encoded in their VGEGs. Experiments show that OneSearch-VL-8B improves over Qwen3-VL-8B with tool access by 20.2 and 17.6 percentage points on the two new benchmarks, respectively, while also achieving substantial gains across 7 image benchmarks and VideoDR. Project repository: https://github.com/appletea233/OneSearch-VL

1 Introduction

Image and video deep research use different visual operations but share grounding, retrieval, and fact composition; prior structures did not preserve dependencies through answer composition. OneSearch-VL uses VGEG for unified training, evidence-aware rewards, and operation-oriented evaluation, with reported gains on new benchmarks.

1 Introduction

  • Image and video research agents can share a workflow of visual grounding, external search, and fact composition despite requiring different visual operations.This motivates the paper’s question of whether one policy can jointly learn deep research across both images and videos.Source: §1 · p. 1
  • Earlier evidence structures did not jointly preserve dependencies from localized visual anchors through entity relations and source-supported facts to answer composition.The paper also identifies limited coverage of multi-image research in prior work, which had primarily considered single-image or video inputs.Source: §1 · p. 2
  • VGEG links localized visual anchors, real-world entities, multi-hop relations, source-supported facts, and the operations that compose facts into answers.Its retained dependencies support task construction and verification, expert-trajectory filtering, RL rewards, and operation-level evaluation; the data engine produces OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K.Source: §1 · p. 2
  • OneSearch-VL trains a unified agent across single-image, multi-image, and video inputs, while EVGR checks evidence traceability and visual grounding in complete trajectories.Traceability checks whether required facts are supported by tool observations; visual grounding checks whether relevant objects, regions, or frames are identified and used.Source: §1 · pp. 2–3
  • OneSearch-MI-Bench and OneSearch-Video-Bench organize questions by VGEG research operations to evaluate evidence retrieval and composition.This organization emphasizes how an agent retrieves and composes evidence rather than what the visual input contains.Source: §1 · pp. 2–3
  • 20.2, 17.6, and 27.0 points: OneSearch-VL-8B exceeds Qwen3-VL-8B on OneSearch-MI-Bench, OneSearch-Video-Bench, and VideoDR, respectively.Joint SFT across three visual input types achieves a six-benchmark average of 55.8 versus 51.7–55.1 for single-type training; EVGR raises the average from 57.3 to 61.1 over answer and query rewards.Source: §1 · p. 2

2 Unified Multimodal Deep Research

The paper formulates multimodal deep research as a unified policy that inspects visual input, retrieves external information, and synthesizes a supported answer. It uses a shared action–observation protocol while retaining input-specific operations and tools.

2.1 Problem Formulation

  • Given visual input and a question, an agent actively inspects visual content, retrieves external information, and synthesizes a supported answer.The input may be a single image, an image collection, or a video.Source: §2.1 · p. 3
  • At each step, the policy conditions on interaction history and generates a reasoning trace plus either a tool call or a final response.Tool observations can include cropped regions, video frames, recognized text, or retrieved information; the complete trajectory ends with the final answer.Source: §2.1 · p. 3
  • All three input types share an action–observation protocol but retain distinct behaviors: local-region inspection, indexed-image aggregation, or temporal localization.Source: §2.1 · p. 3

2.2 Tool Environment

  • OneSearch-VL exposes tools according to visual input type, while a common dispatcher and observation format support a shared trajectory representation.That representation is used for expert synthesis, reinforcement learning, and inference.Source: §2.2 · p. 3

3 VGEG-Centered Multimodal Data Engine

The data engine builds task-level VGEGs from visual sources, linking localized anchors to source-supported web facts and answer-producing operations. It uses these references to construct tasks and retain verified expert trajectories for training.

3 VGEG-Centered Multimodal Data Engine

  • The data engine turns raw visual sources into localized anchors, source-supported web facts, task-level VGEGs, verified question–answer pairs, and expert trajectories.Source: §3 · p. 3

3.1 Visual Source Curation

  • Source curation screens and balances candidate videos for identifiable entities, concrete event cues, and potential for external-knowledge retrieval.The process reduces an initial pool of 2.5 million web videos to 70,000 candidates.Source: §3.1 · p. 4

3.2 Dense Visual Anchor Discovery

  • Dense anchor discovery organizes frames into clips and events, removes redundant key frames, and localizes searchable objects in representative frames.The resulting caption tree links events to temporal ranges, key frames, and localized object instances.Source: §3.2 · p. 4

3.3 Web Evidence Graph Construction

  • Image search, OCR, and text retrieval connect visual anchors to verified entities, source-supported relations, attributes, and webpages in an input-level evidence graph.Source: §3.3 · p. 4

3.4 VGEG-Based Task Construction

  • A task-conditioned projection selects required anchors and facts from the input-level graph, then adds answer-producing operations and their dependencies.The VGEG records localized anchors and entities, source-supported facts, answer operations, and grounding and support relations.Source: §3.4 · p. 4
  • Task generation verifies questions and answers against selected evidence, rewrites entity mentions as visually grounded references, and filters leakage, ambiguity, irrelevance, and visually unnecessary questions.Multi-image tasks use an unordered image collection with remapped image indices; video tasks retain event, timestamp, frame, and region bindings.Source: §3.4 · pp. 5–6, Figure 1 · p. 5

3.5 Expert Trajectory Synthesis

  • Expert trajectories are retained only after answer-correctness and process-quality evaluation, then combined with single-image trajectories for SFT; the RL pool covers all three input types.These datasets are OneSearch-VL-SFT-110K and OneSearch-VL-RL-10K.Source: §3.5 · p. 6

4 Onesearch-Bench

OneSearch-Bench organizes multi-image and video questions by the research operation needed to answer them. The benchmarks share an operation taxonomy and quality criteria while preserving input-specific visual structure.

4 Onesearch-Bench

  • Operation-based organization isolates entity lookup, relation tracing, conditional filtering, and fact-composition demands that aggregate accuracy does not distinguish.The shared taxonomy enables consistent analysis across multi-image and video inputs.Source: §4 · p. 6
  • Multi-image examples use unordered image sets with at least two images, retaining image indices for visual anchors while removing playback references and unnecessary temporal order.Questions remain self-contained over the image set while preserving relations needed for fact composition.Source: §4 · p. 7
  • Video questions preserve temporal structure and link anchors to events, key frames, and web facts; both benchmark branches share sources, VGEGs, taxonomy, and quality criteria.Candidates are checked for visual relevance and non-triviality, and a text-only probe removes questions answerable confidently without visual input.Source: §4 · p. 7

5 OneSearch-VL Training

OneSearch-VL is trained first on expert tool-use trajectories and then refined through online reinforcement learning. Its shared action–observation format supports all three visual input types, while EVGR rewards evidence use and visual grounding.

5 OneSearch-VL Training

  • OneSearch-VL is trained in two stages: supervised fine-tuning learns a unified multimodal tool-interaction policy from expert trajectories, while reinforcement learning further optimizes answer quality, search behavior, and evidence use through online interaction with the tool environment.Source: §5 · p. 7

5.1 Agentic Supervised Fine-Tuning

  • Expert trajectories interleave reasoning, tool calls, observations, and answers in one action–observation schema shared across all three visual input types.One policy selects input-appropriate tools within a unified action space.Source: §5.1 · p. 7
  • The SFT loss targets policy-generated reasoning, tool commands, and responses, while serialized tool observations remain conditioning context rather than prediction targets.The token mask excludes tool observations from the policy loss.Source: §5.1 · p. 7

5.2 Reinforcement Learning

  • EVGR aligns each trajectory with an item-specific rubric and separately scores evidence traceability and visual grounding.Traceability checks support from tool observations and required fact hops; grounding checks whether relevant entities, regions, or video frames guide retrieval.Source: §5.2 · p. 8
  • The RL reward combines EVGR with answer-correctness and query-quality rewards, while format validity gates the resulting reward.GRPO normalizes rewards across trajectories for the same question; tool observations are excluded from the policy loss.Source: §5.2 · p. 8

6 Experiments

Experiments evaluate OneSearch-VL across single-image, multi-image, and video research, then test training-data mixtures and RL reward components. Results show gains over baselines and improved averages from joint training and EVGR.

6.1 Experimental Setup

  • Evaluation covers seven knowledge-intensive single-image benchmarks, two new operation-oriented benchmarks, and VideoDR, using GPT-4o to judge response correctness against reference answers.Training variants share the same tool environment and evaluation protocol.Source: §6.1 · p. 9

6.2 Main Results

  • 58.3 average: OneSearch-VL-8B exceeds Qwen3-VL-8B Agent by 16.3 points and OpenSearch-VL-8B by 1.7 points across seven single-image benchmarks.It improves over OpenSearch-VL on all seven benchmarks and leads listed agentic methods on six.Source: §6.2 · p. 9, Table 2 · p. 10
  • 20.2, 17.6, and 27.0 points: OneSearch-VL-8B improves over Qwen3-VL-8B on OneSearch-MI-Bench, OneSearch-Video-Bench, and VideoDR, respectively.It improves all six research operations on both OneSearch benchmarks.Source: §6.2 · p. 9, Table 3 · p. 10
  • Operation-level gains include 29.8 points for multi-image knowledge-conditioned counting and 25.5 points for video multi-hop retrieval.Multi-anchor arithmetic and comparison improve by 26.3 and 22.5 points on multi-image tasks; arithmetic and joining improve by 20.0 and 19.2 points on video tasks.Source: §6.2 · p. 9

6.3 Ablation Studies

  • Training on any single trajectory type raises the six-benchmark average from 40.7 to 51.7–55.1, while joint training across all three reaches 55.8.The video-only model averages 55.1 and improves all three single-image benchmarks; the joint model performs best on SimpleVQA, OneSearch-MI-Bench, and OneSearch-Video-Bench.Source: §6.3 · pp. 9–10, Table 4 · p. 11
  • Using Trace and Ground together yields a 61.1 average, 3.8 points above answer-plus-query rewards and 2.1 and 1.8 above either single-dimension variant.The full reward performs best on five benchmarks and ties for best on the remaining one.Source: §6.3 · p. 11, Table 4 · p. 11

7 Related Work

Related work develops multimodal search, video research, and unified visual reasoning, while this paper emphasizes a shared structural reference connecting task construction and process supervision.

7.1 Multimodal Search and Deep Research

  • Multimodal search combines visual inspection, external retrieval, and tool-use trajectories, extending active search from language models to image research.Prior work also explores active perception and joint optimization of reasoning and search.Source: §7.1 · p. 11

7.2 Video Deep Research

  • Video research requires locating relevant moments, identifying objects across frames, and linking them to external knowledge.Prior approaches combine temporal localization, spatial inspection, and retrieval, or stage visual grounding before web exploration.Source: §7.2 · p. 11

7.3 Unified Visual Reasoning

  • Prior unified visual models span spatial-temporal understanding or multiple image-and-video tasks; OneSearch-VL unifies deep research across single images, image collections, and videos.Its single policy combines input-specific visual operations with external retrieval and fact composition.Source: §7.3 · p. 12

7.4 Structural References and Process Supervision

  • VGEG preserves visual provenance and answer-composition dependencies in a shared task-level reference, connecting task construction and verification with trajectory supervision.EVGR derives evidence-traceability and visual-grounding criteria from that reference to evaluate complete research trajectories.Source: §7.4 · p. 12

8 Conclusion

The conclusion presents OneSearch-VL as a unified deep-research agent whose VGEG structure supports task construction, trajectory filtering, and process-level reward design.

8 Conclusion

  • Across seven established single-image benchmarks and two operation-oriented benchmarks, OneSearch-VL improves over evaluated open-source baselines; ablations show complementary input-type supervision and benefits from EVGR.VGEG links visual anchors, source-supported facts, and answer-producing operations; EVGR evaluates evidence traceability and visual grounding.Source: §8 · p. 12

9 Limitations and Future Work

The appendix describes a VGEG-centered data engine that curates visual sources, discovers anchors, and links them to verified web evidence. The authors also identify reproducibility, annotation reliability, and inference cost as limitations and future-work concerns.

9 Limitations and Future Work

  • External search tools and changing webpages can affect evidence availability and exact reproducibility; the authors call for retrieval snapshots and robustness evaluation.They also flag possible annotation errors or judging biases, and the added inference and tool costs of multi-turn interaction.Source: §9 · p. 12

A Details of the VGEG-Centered Multimodal Data Engine

  • The input-level graph supplies a common evidence context, while task-conditioned VGEG projections select anchors and facts and preserve their grounding and dependency links.Each extracted fact retains its supporting URL and quotation; entity-valued facts may be expanded through additional retrieval.Source: A Details of the VGEG-Centered… · p. 17, §A.3 · p. 19

A.1 Stage 1: Visual Source Curation

  • Video curation reduces an initial pool of 2.5M records to 70k candidates through metadata, category, task-suitability, and duration filters.The snippet assessment uses textual metadata; visual inspection occurs during later anchor discovery.Source: §A.1 · pp. 17–18

A.2 Stage 2: Dense Visual Anchor Discovery

  • The anchor-discovery pipeline samples and captions clips and key frames, deduplicates frames, groups related clips into events, and localizes searchable objects.It retains frame indices, timestamps, event ranges, and object bounding boxes for later retrieval and question construction.Source: §A.2 · p. 18

A.3 Stage 3: Web Evidence Graph Construction

  • The evidence-graph pipeline routes localized objects to image search or OCR and text search, verifies candidate identities against visual and source evidence, and extracts source-supported relations.The input-level graph deterministically links events, frames, objects, entities, attributes, and source pages.Source: §A.3 · p. 19

A.4 Stage 4: VGEG-Based Task Construction

Stage 4 constructs and verifies questions with task-level VGEGs, then adapts them to multi-image or video inputs. Stage 5 synthesizes expert tool-use trajectories and filters them for answer correctness and process quality.

A.4 Stage 4: VGEG-Based Task Construction

  • Candidate questions are generated from visual descriptions, anchors, graph facts, and relation chains, with VGEGs selecting task-specific anchors, facts, operations, and dependencies.Deterministic checks test evidence paths and task structure; a further evidence-only re-answer check tests whether the evidence supports the proposed answer.Source: §A.4 · p. 20

A.5 Stage 5: Expert Trajectory Synthesis

  • Questions are rewritten to conceal explicit and intermediate entity names, then instantiated as unordered multi-image collections or temporally bound videos and filtered for visual relevance and answer leakage.Expert trajectories are retained only when their answers are correct and their tool use is effective, observation-consistent, and not repetitious.Source: §A.4 · pp. 20–21, §A.5 · p. 21

B Benchmark Construction and Evaluation

The appendices detail benchmark construction and evaluation, tool execution, and supervised trajectory learning. The benchmarks preserve input-specific visual references while organizing tasks around answer-producing research operations.

B.1 Visual Input Construction

  • The multi-image benchmark uses unordered collections of deduplicated frames and requires evidence from at least two images; the video benchmark preserves temporal structure and anchor bindings.Video questions require localization before external fact retrieval and composition.Source: §B.1 · p. 21

B.2 Research Operations and Reference Structures

  • Each item receives one principal operation label based on its final answer-producing step, while its VGEG preserves intermediate dependencies.The benchmarks contain 608 questions, of which 555 (91.3%) are compositional; multi-image questions use 2–8 images, averaging 3.05.Source: §B.2 · p. 22

B.3 Difficulty and Visual-Domain Distributions

  • Difficulty scores combine operation type, reasoning depth, and required-fact count; the easy, medium, and hard groups represent relative complexity within each input type.The groups use empirical boundaries of 14.4 and 22.2, with balanced group sizes rather than independently calibrated human difficulty.Source: §B.3 · p. 22, Figure 4 · p. 23
  • All six visual domains span the six principal research operations, while their operation proportions reflect the benchmarks’ emphasis on compositional questions.Figure 4(b) presents operation composition with bars normalized within each domain.Source: §B.3 · p. 22, Figure 4 · p. 23

B.4 Human Review and Annotation

  • Human review checks visual-reference identity, source support, answer derivation, and operation labels; a final review rejects ambiguous references, unsupported facts, indeterminate answers, or unclear labels.Source: §B.4 · p. 23

C.1 Resource Registry and Tool Availability

  • The tool interface registers images, frames, crops, and clips as reusable handles and uses one tool call per turn; image and text search return candidate identities and source-grounded summaries.Video inputs additionally expose temporal tools for selecting frames or clips.Source: §C.1 · p. 24, §C.2 · pp. 24–25

C.3 Observation and Trajectory Serialization

  • Each tool turn records the policy output, parsed call, textual observation, and any new visual resources; errors are preserved as observations for later correction and filtering.Source: §C.3 · p. 25

D.1 Supervised Trajectory Learning

  • OneSearch-VL-SFT-110K contains approximately 110k expert trajectories across single-image, multi-image, and video inputs, with training targets covering model reasoning, tool commands, and final responses.Environment observations remain conditioning context rather than supervised targets.Source: §D.1 · p. 25

D.2 Reinforcement Learning Data Construction

The appendix describes RL task selection, VGEG-derived process rewards, and policy optimization, then illustrates evidence composition in multi-image and cross-frame cases. The examples show both successful answer derivation and a case with inconsistent numerical evidence.

D.2 Reinforcement Learning Data Construction

  • The RL dataset contains approximately 10k deduplicated tasks across three visual input types, selected by retaining candidates with some but fewer than eight correct baseline rollouts.Source: §D.2 · pp. 25–26

D.3 From VGEG Annotations to EVGR Rubrics

  • EVGR rubrics retain required entities, fact hops, sources, snippets, and visual locations; separate judges score evidence traceability and visual grounding from executed trajectories.The rubric avoids rewarding verbosity, unsupported reasoning, ineffective tool calls, or answer matching alone.Source: §D.3 · pp. 26–27

D.4 Reward Composition and Policy Optimization

  • GRPO updates the policy using relative advantages within groups of trajectories, with weighted answer, query, and EVGR rewards and a format gate.Optimization applies to model-generated tokens; for trajectories ending in tool error, only the valid prefix is used.Source: §D.4 · p. 27

E.1 Multi-image Evidence Composition

  • In multi-image examples, two identified manufacturers’ founding years yield a 58-year difference, while ownership-conditioned counting links three pictured cars to Bentley’s Volkswagen ownership before returning 3.Source: Figure 5 · p. 29, §E.1 · pp. 29–30, Figure 6 · p. 30

E.2 Cross-frame Evidence Composition

  • Cross-frame cases ground objects in two or three selected frames before comparing retrieved attributes; the three-building case returns 1,128 feet despite inconsistent height evidence and an inaccurate antenna height.Source: §E.2 · pp. 31–32, Figure 7 · p. 31, Figure 8 · p. 32
Loading 2610.12419v1…