PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepEyesV2: Toward Agentic Multimodal Model

Jack Hong, Chenxiao Zhao, ChengLin Zhu, Weiheng Lu, Guohai Xu, Xing Yu

arXiv:2511.05271v4cs.CVcs.AI

TL;DR

Agentic multimodal models need to invoke and integrate external tools rather than only understand text and images. DeepEyesV2 addresses this gap with curated cold-start supervision followed by reinforcement learning, and demonstrates strong performance across perception, reasoning, and search tasks with context-adaptive tool use.

  • Problem

    Existing multimodal models are largely passive and lack reliable integration of operation tools, information retrieval, perception, search, and reasoning.

  • Method

    DeepEyesV2 uses curated cold-start supervised fine-tuning, reinforcement learning, and a dynamic loop interleaving code execution and web retrieval.

  • Results

    DeepEyesV2 performs strongly across real-world understanding, mathematical reasoning, and search-intensive benchmarks, including 52.7% accuracy on MathVerse and 63.7% on MMSearch.

  • Takeaways & Limitations

    DeepEyesV2 selectively invokes tools by task, while reinforcement learning enables more complex, context-aware tool combinations.

Abstract

from arXiv · show

Agentic multimodal models should not only comprehend text and images, but also actively invoke external tools, such as code execution environments and web search, and integrate these operations into reasoning. In this work, we introduce DeepEyesV2 and explore how to build an agentic multimodal model from the perspectives of data construction, training methods, and model evaluation. We observe that direct reinforcement learning alone fails to induce robust tool-use behavior. This phenomenon motivates a two-stage training pipeline: a cold-start stage to establish tool-use patterns, and reinforcement learning stage to further refine tool invocation. We curate a diverse, moderately challenging training dataset, specifically including examples where tool use is beneficial. We further introduce RealX-Bench, a comprehensive benchmark designed to evaluate real-world multimodal reasoning, which inherently requires the integration of multiple capabilities, including perception, search, and reasoning. We evaluate DeepEyesV2 on RealX-Bench and other representative benchmarks, demonstrating its effectiveness across real-world understanding, mathematical reasoning, and search-intensive tasks. Moreover, DeepEyesV2 exhibits task-adaptive tool invocation, tending to use image operations for perception tasks and numerical computations for reasoning tasks. Reinforcement learning further enables complex tool combinations and allows model to selectively invoke tools based on context. We hope our study can provide guidance for community in developing agentic multimodal models.

1 Introduction

DeepEyesV2 targets multimodal reasoning that combines perception, search, and reasoning through active tool use. It uses curated training, a two-stage pipeline, and RealX-Bench to develop and evaluate this capability.

  • Motivation: Existing multimodal models remain largely passive, lacking autonomous invocation of operation and information-retrieval tools needed for integrated multimodal reasoning.Operation tools support fine-grained visual manipulation and quantitative computation, while retrieval tools provide up-to-date, verifiable external knowledge.
  • Approach: DeepEyesV2 integrates code execution and web search within a dynamic reasoning loop, iteratively incorporating tool outputs to refine and verify conclusions.The model decides whether internal reasoning suffices and, when necessary, emits executable Python code or web queries.
  • Training: Direct reinforcement learning did not reliably induce tool use, motivating cold-start training before reinforcement learning refinement.The curated dataset spans perception, reasoning, and search tasks and retains difficult cases where tools improve accuracy.
  • Evaluation: RealX-Bench evaluates integrated perception, search, and reasoning, addressing the lack of cross-capability benchmarks and exposing a gap between current models and humans.DeepEyesV2 coordinates the three capabilities more effectively than open-source models and models restricted to a single tool.
  • Results: DeepEyesV2 outperforms general-purpose and specialized reasoning models across real-world understanding, mathematical reasoning, and search-intensive benchmarks.It reports 52.7% accuracy on MathVerse with a +7.1 gain and 63.7% on MMSearch versus 53.8% for MMSearch-R1.
  • Results: Tool invocation is task-adaptive: image operations support perception, numerical analysis supports reasoning, and reinforcement learning enables more complex combinations.The contribution analysis characterizes this as selective, context-aware tool invocation emerging through training.

2 Related Works

Related work develops multimodal understanding, iterative visual reasoning, and search augmentation, while DeepEyesV2 combines these capabilities through interleavable tools. Prior work commonly uses cold start followed by reinforcement learning, whereas DeepEyesV2 extends tool use beyond cropping.

  • Multimodal Large Language Models: MLLM research progressed from visual encoders with lightweight adapters toward stronger architectures for multimodal understanding.The passage names Qwen2.5-VL, LLaVA-OneVision, and InternVL3 as more powerful architectures.
  • Thinking with Images: “Think with Image” interleaves reasoning with iterative visual analysis, and many approaches train with cold start followed by reinforcement learning.DeepEyes is contrasted with these approaches because it adopts reinforcement learning without the same stated two-stage setup.
  • Search-oriented Reasoning: Search-oriented reasoning augments multimodal models with external knowledge, but retrieval-augmented generation remains limited by static, finite corpora.Search augmentation is positioned as complementary to perception and other tool-use capabilities.

3 DeepEyesV2

DeepEyesV2 integrates code execution and web retrieval into a dynamic reasoning loop, using cold-start training followed by reinforcement learning to develop and refine tool use. Its pipeline combines curated data, executable trajectories, and outcome-driven rewards to support adaptive, multi-tool reasoning.

  • 3.1 Overall Pipeline: DeepEyesV2 interleaves code execution and web retrieval within one reasoning trajectory, appending tool outputs as observations for further reasoning.Code can produce transformed images, measurements, arrays, plots, and logs, while search returns ranked webpages with thumbnails, titles, or snippets.
  • 3.5 Evaluation: RealX-Bench statistics organize questions across five domains and overlapping ability subsets, including perception, reasoning, search, and integration.Integration denotes questions difficult across all three abilities simultaneously.
  • 3.2 Pioneer Experiments: Direct reinforcement learning initially produced buggy code, then bypassed tools; adding a tool-use bonus led to placeholder code and reward hacking.The experiment motivates cold-start training before reinforcement learning.
  • 3.3 Training Data Curation: The training corpus spans perception, reasoning, and search, while filtering retains difficult cases that the base model rarely solves and where tool use improves success.Qwen2.5-VL-7B generates eight responses per question for difficulty and tool-benefit assessment.
  • 3.3 Training Data Curation: Cold-start trajectories are synthesized with explicit tool calls, executed interactively, and retained only when final answers are correct and code is error-free.The originating model receives tool outputs and may issue additional calls before producing its final answer.
  • 3.4 Agentic Reinforcement Learning: After cold-start training, agentic reinforcement learning uses interactive tool decisions with a sparse reward combining accuracy and format compliance.The total reward is defined as R = Racc + Rformat.

4 RealX-Bench

RealX-Bench evaluates coordinated perception, search, and reasoning in challenging real-world scenarios rather than isolated capabilities. It contains 300 question–answer pairs with structured annotations, standardized answers, and automatic verification.

  • 4 RealX-Bench: RealX-Bench addresses the limitation of benchmarks that evaluate perception, retrieval, or reasoning in isolation by testing their coordinated interplay.The benchmark targets complex real-world scenarios requiring multiple capabilities.
  • 4.1 Design Principles: Its questions challenge precise visual localization, multi-hop evidence gathering, or multi-step reasoning with intermediate consistency checks.Every question exhibits at least one of these difficulty dimensions.
  • 4.2 Data Construction: The benchmark is built from openly available real-world images and user questions, followed by expert refinement, labeling, and quality control.The construction workflow includes data collection, QA annotation, difficulty and category labeling, and quality control.
  • 4.2 Data Construction: 300 question–answer pairs span five real-world domains and receive overlapping perception, search, and reasoning labels.24% of questions are simultaneously challenging across all three abilities.

5 Experiments

Experiments evaluate DeepEyesV2 across integrated perception, reasoning, and search tasks, showing strong performance and adaptive tool use after two-stage training. Ablations indicate that diverse cold-start supervision and reinforcement learning improve complementary tool-use behaviors.

  • RealX-Bench: 46.0% accuracy is achieved by the best proprietary model on RealX-Bench, while performance remains far below human capability and coordination is weakest on three-skill subsets.Gemini reaches 27.8% on subsets requiring perception, search, and reasoning together, versus 46.0% average accuracy.
  • RealX-Bench: DeepEyesV2 outperforms open-source and zooming-tool models on tasks requiring coordinated perception, search, and reasoning.The strongest gains occur on tasks requiring all three capabilities simultaneously.
  • Other Benchmarks: DeepEyesV2 shows substantial gains over Qwen2.5-VL-7B across real-world, OCR, and chart benchmarks, surpassing Qwen2.5-VL-32B on some tasks.Dynamic tool invocation extracts fine-grained visual details and improves real-world scene comprehension.
  • Other Benchmarks: DeepEyesV2 consistently outperforms alternative multimodal reasoning models, including text-only multimodal reasoning models, indicating benefits from tool use for mathematical reasoning.The broader evaluation covers mathematical reasoning benchmarks alongside general-purpose and grounded reasoning baselines.
  • Other Benchmarks: DeepEyesV2 achieves consistently higher accuracy across all evaluated search benchmarks than general-purpose and search-enabled comparison models.The search evaluation includes FVQA-test, InfoSeek, MMSearch, and SimpleVQA.
  • Training Analysis: Combining perception, reasoning, and long CoT cold-start data achieves the best overall results, while single-purpose subsets improve mainly their associated capabilities.The ablation attributes this to complementary supervision and the value of long CoT for complex reasoning.
  • Tool-Use Analysis: Reinforcement learning shifts tool use toward numerical operations and synergistic combinations of image manipulation with search.Tool preferences remain task-dependent, including cropping for fine-grained perception and numerical computation for OCR and reasoning-related tasks.
  • Tool-Use Analysis: After reinforcement learning, tool invocation decreases significantly while call-count variance remains high, indicating selective tool use rather than a fixed invocation pattern.The model increasingly solves simple problems directly while retaining complex multi-tool reasoning when needed.

6 Conclusion

The conclusion presents DeepEyesV2 as a two-stage agentic multimodal model trained with curated supervision and reinforcement learning. Experiments and analysis show task-dependent tool use, more complex tool combinations, and strong results across perception, reasoning, and search benchmarks.

  • Conclusion: DeepEyesV2 combines cold-start supervised fine-tuning with reinforcement learning to establish and then adapt tool-use patterns.The model integrates tool invocation into multimodal reasoning across training, dataset design, and evaluation.
  • Conclusion: DeepEyesV2 exhibits task-dependent tool-use behaviors and reinforcement learning enables more complex, context-aware tool combinations.The reported evaluation spans perception, reasoning, and search benchmarks.
  • Conclusion: Experiments demonstrate strong reasoning ability when tool invocation is combined with multimodal reasoning.The conclusion frames this result across perception, mathematical reasoning, and search-oriented tasks.

A.1 Training Data

The training data combines diverse perception, reasoning, search, and long-CoT sources for cold-start supervision, while reinforcement learning uses perception, reasoning, and search data.

  • Cold Start Data: Perception data covers object recognition, visual counting, chart interpretation, and other real-world visual scenarios.The listed sources include V*, ArxivQA, Pixmo Counting, TallyQA, and SeekWorld.
  • Cold Start Data: Reasoning data includes complex multimodal problems and executable code-based problem solving, supplemented by long Chain-of-Thought data.Sources include ReVisual, MathCoder, and Retool.
  • Dataset Composition: Cold-start data is divided into perception, reasoning, search, and Long CoT, whereas reinforcement-learning data contains perception, reasoning, and search.The two datasets therefore differ in their included supervision categories.

A.2 Evaluation Protocol

Evaluation covers real-world understanding, OCR, chart understanding, multimodal reasoning, and online search using benchmark suites and comparisons with general-purpose, grounded, and search-enabled models.

  • Benchmark Coverage: Real-world, OCR, and chart evaluation uses V*, HRBench, MME-RealWorld, TreeBench, OCRBench, Seed-Bench-2-Plus, CharXiv, and ChartQA.Comparisons include open-source general-purpose MLLMs and grounded reasoning models.
  • Benchmark Coverage: Multimodal reasoning evaluation uses MathVista, MathVerse, MathVision, WeMath, and LogicVista against general-purpose, text-only reasoning, and grounded reasoning models.This protocol compares multiple model families rather than a single baseline type.
  • Benchmark Coverage: Online-search evaluation uses FVQA-test, InfoSeek, MMSearch, and SimpleVQA against general-purpose MLLMs and models with incorporated search capability.The comparison set includes Qwen2.5-VL, Gemini 2.5 Pro, and GPT4o.

A.3 Performance Comparison with Proprietary Models

DeepEyesV2 achieves performance comparable to GPT-4o when evaluated against proprietary models using the same prompt.

  • DeepEyesV2 achieves performance comparable to GPT-4o under the same prompting setup.The comparison uses the same prompt as DeepEyesV2 for the proprietary models.

A.4 Tool Usage Pattern Evolution

Tool-use categories change systematically across reinforcement-learning stages, with cropping peaking early while numerical analysis and other tools increase later.

  • The proportion of Mark tools remains largely stable across reinforcement-learning stages.The observed fluctuation is minimal.
  • Crop tool usage rises initially and then declines as training progresses.
  • Numerical analysis and Other tool usage decrease initially before increasing during later training stages.

A.5 Zero-Shot Generalization of Tool Usage

DeepEyesV2 generalizes tool invocation to unseen tools and tasks, using novel image operations, code-based reasoning, and function calls in zero-shot settings.

  • Zero-shot evaluation: TIR-Bench evaluates zero-shot tool invocation because its tasks and required tool categories are absent from DeepEyesV2’s training data.
  • Performance comparison: DeepEyesV2 substantially outperforms Qwen2.5-VL 7B on TIR-Bench after tool integration.The comparison includes the cold-start model and the reinforcement-learning model.
  • New tool visualization: After reinforcement learning, DeepEyesV2 generalizes to unseen OCR, rotation, and maze-solving tools without additional training.The model applies grayscale, dilation, cropping, image rotation, and code-based pathfinding across these tasks.
  • Function call generalization: DeepEyesV2 also generalizes function calling by using a rotation tool for rotated-image OCR.This evaluation requires tool invocation through function calls rather than generated code.
  • Tool taxonomy: The tool taxonomy includes code execution, image search, and text search for image manipulation, visual retrieval, and webpage retrieval.Code execution is further divided into crop, numerical analysis, mark, and other manipulation operations.
  • Error analysis: DeepEyesV2’s zero-shot failures are categorized as tool execution, tool selection, or tool result analysis errors.These categories distinguish operational mistakes, inappropriate tool choices, and misinterpretation of returned outputs.
Loading 2511.05271v4…