Source-linked AI summary
Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch
Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, Shukang Yin, Jiangbo Pei, Wei Shen, Peng Xia, Yi Peng, Tianyidan Xie, Eric Li, Yang Liu, Xuchen Song, Yahui Zhou
TL;DR
Multimodal agents often separate image manipulation from web search, rely on costly reinforcement learning, and lack planning grounded in real tool use. Skywork-R1V4 unifies these capabilities through supervised fine-tuning on fewer than 30,000 consistency-validated trajectories, achieving state-of-the-art benchmark performance without reinforcement learning.
Problem
Existing multimodal agents often separate image manipulation and external search, rely on costly reinforcement learning, and lack planning grounded in real tool-execution trajectories.
Method
Skywork-R1V4 unifies planning, active image manipulation, deep multimodal search, and interleaved visual-retrieval reasoning through supervised fine-tuning on fewer than 30,000 curated trajectories.
Results
Skywork-R1V4 achieves state-of-the-art performance across perception and multimodal search benchmarks, including 66.1 on MMSearch and 67.2 on FVQA, and outperforms Flash on all 11 reported metrics.
Takeaways & Limitations
Carefully curated supervised learning with real tool-grounded trajectories provides an effective and efficient path to multimodal agentic intelligence without reinforcement learning.
Takeaways & Limitations
FVQA evaluation uses a randomly sampled 180-question subset, and efficiency results across three benchmarks may not represent general performance under all conditions.
Abstract
from arXiv · showhide
Despite recent progress in multimodal agentic systems, existing approaches often treat image manipulation and web search as disjoint capabilities, rely heavily on costly reinforcement learning, and lack planning grounded in real tool-execution traces. To address these limitations, we present Skywork-R1V4, a 30B (A3B) parameter multimodal agentic model that unifies multimodal planning, active image manipulation ("thinking with images"), deep multimodal search, and, most critically, interleaved reasoning that dynamically alternates between visual operations and external knowledge retrieval. Trained solely via supervised fine-tuning on fewer than 30,000 high-quality, planning-execution-consistent trajectories and validated through stepwise consistency filtering, Skywork-R1V4 achieves state-of-the-art results across perception and multimodal search benchmarks: it scores 66.1 on MMSearch and 67.2 on FVQA, surpassing Gemini 2.5 Flash on all 11 metrics. Skywork-R1V4 exhibits emergent long-horizon reasoning at inference time, successfully orchestrating more than 10 tool calls to solve complex, multi-step tasks. Our results demonstrate that sophisticated agentic multimodal intelligence can be achieved through carefully curated supervised learning alone, without any reliance on reinforcement learning.
1. Introduction
Skywork-R1V4 addresses gaps in multimodal agents by unifying planning, image manipulation, deep search, and interleaved reasoning through supervised fine-tuning. It achieves state-of-the-art performance across perception and multimodal search benchmarks.
- Existing systems commonly isolate visual manipulation from retrieval, depend on costly reinforcement learning, and train planning without real tool-execution trajectories.These limitations are associated with reduced practicality, reproducibility, fidelity, and generalization in real-world scenarios.
- Skywork-R1V4 unifies multimodal agentic planning, active image manipulation, deep search, and interleaved image-search reasoning.Its interleaved capability dynamically alternates visual operations and external retrieval within one trajectory.
- The model is trained with fewer than 30K curated supervised fine-tuning samples whose reasoning, tool executions, and outcomes undergo consistency checks.The training corpus combines capability-specific, hybrid, and structured planning trajectories while discarding inconsistent or low-fidelity samples.
- 66.1 on MMSearch and 67.2 on FVQA, with Skywork-R1V4 outperforming Flash on all 11 reported metrics.The model also achieves state-of-the-art performance across a wide range of perception and search benchmarks.
2. Method
The method constructs and filters tool-grounded multimodal trajectories, then mixes them for supervised fine-tuning across image reasoning, search, planning, and general VQA. Its data pipeline emphasizes execution consistency, structured dependencies, and diverse visual-search operations.
- 2.1. Think With Image: The pipeline generates image-operation trajectories with sandbox execution and stepwise reasoning–output consistency validation.It supports operations such as cropping, enhancement, zooming, annotation, and pixel-level analysis, iterating validated results until the question is answered.
- 2.2. Multi-modal Search: Search data spans basic queries solvable through reverse image and text search and enhanced queries requiring five or more rounds with cross-validation.Basic search uses FVQA-related data, while enhanced queries are constructed through constrained random walks over a knowledge graph.
- 2.3. Interleaved Reasoning: Interleaved trajectories combine image manipulation and information retrieval within a single reasoning process.The method samples LiveVQA data and generates trajectories intended to alternate visual reasoning with search.
- 2.4. Multi-Modal Agentic Planner: Planner data converts multi-step action trajectories into structured plans with natural-language subtasks, tool names, parameters, and contextual dependencies.Validation and dependency checking produce a corpus coupling reasoning with executable procedural planning.
- Training and Data Curation: The final SFT mix combines the specialized trajectory types with general VQA data, while removing sandbox failures and re-cropping examples that teach erroneous correction patterns.The data pipeline also classifies functionality and reports diverse tool types rather than restricting the corpus to a fixed tool set.
3. Experiments
Skywork-R1V4 improves perception and deep multimodal search over Qwen3-VL 30B(A3B), while also showing faster inference than Gemini models under tested settings.
- Benchmarks and Results: +47.4 on MMSearch, +13.9 on FVQA, and +8.4 on BrowseComp-VL over Qwen3-VL 30B(A3B) demonstrate the strongest gains in deep search tasks.The reported scores are 66.1 versus 18.7, 67.2 versus 53.3, and 38.4 versus 30.0, respectively.
- Benchmarks and Results: Skywork-R1V4 achieves perception gains of +1.5 to +14.4 percentage points across tested settings, including 91.8 on HRBench-4K FSP and 90.4 on V* Attribute.The largest cited perception gain is +14.4 on MME-Real-CN Reasoning.
- Benchmarks and Results: Skywork-R1V4 outperforms Gemini 2.5 Flash on all 11 reported metrics and Gemini 2.5 Pro on 5 metrics, including V* at 88.0 versus 79.1.The comparison is reported for the same 30B-scale architecture as the baseline.
- Visualization of Multimodal Agentic Capabilities: The qualitative examples show coordinated perception, planning, and external knowledge retrieval across Plan Mode and DeepResearch Mode.Figures 4–7 are presented as demonstrations of tool-integrated multimodal reasoning.
- Efficiency: In single-round evaluation, inference is approximately 4× faster than Gemini 2.5 Flash and 15× faster than Gemini 2.5 Pro on MM-Search and FVQA-Test.The reported BCVL speedup is around 5×, with repetitive outputs observed for some single-round BCVL questions.
- Efficiency: Efficiency results use 180 randomly sampled FVQA-Test questions and depend on quantization, hardware, and network latency, limiting generalization across conditions.The reported efficiency comparison covers three benchmarks.
4. Related Work
Prior work separately advances interactive visual reasoning, multimodal search, and agentic planning, but reinforcement-learning overhead and weakly grounded planning remain limitations. Skywork-R1V4 instead uses real tool-execution traces and supervised fine-tuning to learn multimodal planning.
- Multimodal Reasoning and Think with Images: Prior multimodal reasoning methods often treat visual inputs as passive context, while newer “thinking with images” methods actively manipulate visual content.Examples include dynamic cropping and auxiliary visual aids.
- Multi-Modal Search: Multimodal search research extends retrieval beyond internal knowledge, but existing approaches largely rely on reinforcement learning and incur considerable training overhead.The cited progression includes MMSearch-R1 and WebWatcher.
- Multimodal Agentic Planning: Multimodal planning decomposes goals into tool-call sequences, yet reinforcement-learning planners remain costly and difficult to deploy at scale.CogPlanner is cited as a planner that chooses among text search, image search, or no retrieval.
- Multimodal Agentic Planning: Skywork-R1V4 trains with supervised fine-tuning on plans extracted from realistic search-augmented and think-with-images trajectories rather than fabricated abstract supervision.The paper states that this grounding produces more faithful plan distributions and higher downstream planning success rates.
5. Conclusion and Future Directions
Skywork-R1V4 shows that carefully curated, tool-grounded supervised fine-tuning can support executable planning, active image manipulation, deep search, and interleaved reasoning without reinforcement learning. Future work proposes richer tools, memory or predictive planning modules, and hybrid training paradigms.
- Conclusion: High-quality tool-grounded trajectories and strict consistency filtering enable strong multimodal agentic capabilities without reinforcement learning.The model spans executable planning, active image manipulation, deep multimodal search, and interleaved reasoning.
- Conclusion: The approach achieves state-of-the-art perception and complex search performance, including results that outperform larger proprietary systems.The conclusion attributes this outcome to fewer than 30K curated samples with consistency across reasoning, tool execution, and final answers.
- Future Directions: Future directions include richer visual and web tools, memory or predictive modules for long-horizon coherence, and hybrid supervised-learning and adaptation paradigms.Examples include segmentation, depth estimation, structured DOM navigation, and memory mechanisms.
6. Contributions
The listed core contributors are Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, Yang Liu, and Xuchen Song, with additional contributors listed separately.
- Core Contributors: The core contributors are Yifan Zhang, Liang Hu, Haofeng Sun, Peiyu Wang, Yichen Wei, Yang Liu, and Xuchen Song.Yifan Zhang and Liang Hu are marked with an asterisk, while Xuchen Song is marked with a dagger.
- Contributors: Additional contributors are Shukang Yin, Jiangbo Pei, Wei Shen, Peng Xia, Yi Peng, Tianyidan Xie, Eric Li, and James Zhou.
A. Dialogue Modes
Skywork-R1V4 supports distinct General, DeepResearch, and Plan modes that trade off speed, external tool use, and structured execution. Qualitative examples show iterative visual exploration and multimodal search coordinating perception with tool-grounded reasoning.
- General Mode: General Mode relies mainly on internal knowledge and code tools, avoiding heavy external tools for faster multimodal responses.
- DeepResearch Mode: DeepResearch Mode interleaves image analysis, image manipulation, external search, and other tool operations to refine evidence before answering.The mode is designed for autonomous iterative reasoning and progressively supported answers.
- Plan Mode: Plan Mode outputs high-level execution steps and strategies grounded in visual context, with planner tasks illustrated through structured tool sequences.The example decomposes image identification, team lookup, performance search, and win-rate calculation into ordered steps.
- DeepResearch Mode: Iterative cropping and observation progressively narrow a Paris panorama until the model identifies a small white dog as white.The example moves from broad regions to focused park and sidewalk crops before producing the final answer.
- DeepResearch Mode: Multimodal search can combine image and text tools to identify a photograph’s location, while interleaved operations connect visual cues with retrieved context.