Source-linked AI summary
Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, Ziang Yan, Yi Wang, Hongjie Zhang, Yali Wang, Limin Wang
TL;DR
Long-video models often rely on uniform sampling and single-turn inference, making sparse critical evidence difficult to identify amid redundancy. Video-o3 introduces native interleaved clue seeking with task-decoupled attention, trajectory-guided rewards, and Seeker-173K training data. It reports 72.1% on MLVU and 46.5% on Video-Holmes, while acknowledging context-quota and data-scope limitations.
Problem
Existing long-video approaches rely heavily on uniform sampling, single-turn inference, or isolated clue-seeking modules, limiting evidence discovery and multi-clue reasoning.
Method
Video-o3 combines native interleaved tool invocation, Task-Decoupled Attention Masking, Verifiable Trajectory-Guided Reward, and Seeker-173K training trajectories.
Results
Video-o3 reports 72.1% accuracy on MLVU, 47.6% on LVBench, and 46.5% on Video-Holmes, consistently outperforming latest MLLMs in the evaluated domain.
Takeaways & Limitations
The results support native tool invocation as an effective approach for long-form video understanding and multi-hop reasoning.
Takeaways & Limitations
The method can be constrained by context-quota depletion and is not trained for abstract puzzle-style benchmarks requiring specialized plot-reasoning supervision.
Abstract
from arXiv · showhide
Existing multimodal large language models for long-video understanding predominantly rely on uniform sampling and single-turn inference, limiting their ability to identify sparse yet critical evidence amid extensive redundancy. We introduce Video-o3, a novel framework that supports iterative discovery of salient visual clues, fine-grained inspection of key segments, and adaptive termination once sufficient evidence is acquired. Technically, we address two core challenges in interleaved tool invocation. First, to mitigate attention dispersion induced by the heterogeneity of reasoning and tool-calling, we propose Task-Decoupled Attention Masking, which isolates per-step concentration while preserving shared global context. Second, to control context length growth in multi-turn interactions, we introduce a Verifiable Trajectory-Guided Reward that balances exploration coverage with reasoning efficiency. To support training at scale, we further develop a data synthesis pipeline and construct Seeker-173K, comprising 173K high-quality tool-interaction trajectories for effective supervised and reinforcement learning. Extensive experiments show that Video-o3 substantially outperforms state-of-the-art methods, achieving 72.1% accuracy on MLVU and 46.5% on Video-Holmes. These results demonstrate Video-o3's strong multi-hop evidence-seeking and reasoning capabilities, and validate the effectiveness of native tool invocation in long-video scenarios.
1. Introduction
Long-video understanding requires locating sparse, query-relevant moments and reasoning across interdependent clues, but existing approaches often rely on uniform sampling, isolated modules, or fixed search behavior. Video-o3 addresses this with native interleaved tool invocation, task-decoupled attention, trajectory-guided rewards, and large-scale interaction data.
- Long videos require precise localization of query-relevant moments and reasoning across intricate temporal dependencies.
- Existing clue-seeking methods rely on handcrafted heuristics, isolated modules, and fixed search parameters that limit joint reasoning over multiple clues.
- Video-o3 uses native interleaved tool invocation to identify salient clues, inspect targeted video segments, revisit evidence, and decide when to terminate.
- Task-Decoupled Attention Masking separates clue discovery from answer reasoning while preserving shared global context.
- The Verifiable Trajectory-Guided Reward favors accurate localization and less redundant exploration, while Seeker-173K supplies 173k tool-interaction trajectories for training.
- 72.1% accuracy on MLVU, 47.6% on LVBench, and 46.5% on Video-Holmes are reported for Video-o3 across long-video benchmarks.
2. Related work
Related work progresses from uniform sampling and sparse selection toward tool-augmented, multi-turn video reasoning. Native interleaved approaches preserve shared visual and reasoning context, but prior methods do not explicitly model tool-use heterogeneity or provide verifiable tool-use rewards.
- Reasoning paradigms: Conventional reasoning derives answers directly from initial input, whereas decoupled iterative reasoning resets or isolates context between independent calls.
- Reasoning paradigms: Native interleaved invocation preserves visual features and reasoning history within a unified shared context for holistic joint inference.
- MLLMs for Video Understanding: Uniform frame sampling can introduce redundancy and miss critical moments, motivating keyframe and segment selection methods.
- Tool-Augmented MLLMs: Tool-augmented MLLMs extend model capabilities by actively gathering visual information and iteratively collecting evidence through multi-turn tool calls.
- Tool-Augmented MLLMs: Prior native multi-turn approaches do not explicitly address heterogeneous tool-invocation and reasoning patterns or provide verifiable tool-use rewards.
3. Methodology
Video-o3 performs native interleaved multi-turn video reasoning by alternating adaptive clue-seeking with answer reasoning. Task-Decoupled Attention Masking separates these modes, while Verifiable Trajectory-Guided Reward promotes precise localization and concise exploration.
- Overall Architecture: Video-o3 dynamically selects VideoCrop segments and resolution, then terminates retrieval when accumulated evidence is sufficient for multi-clue answer synthesis.The model can alternate clue seeking with direct answer reasoning based on evidence sufficiency.
- Task-Decoupled Attention Masking: Video-o3 addresses attention dispersion caused by interleaved global tokens, local fragments, and reasoning text, which can produce inconsistent final predictions.The masking strategy targets both dispersed attention and fake thinking during answer generation.
- Task-Decoupled Attention Masking: Task-Decoupled Attention Masking restricts clue-seeking attention to global video context and answer reasoning to high-resolution tool observations.The method applies masking to 10% of tool-use training data while preserving holistic synthesis capability.
- Trajectory-Guided Reinforcement Learning: Verifiable Trajectory-Guided Reward combines answer correctness, structural validity, and exploration efficiency to regulate multi-turn tool use.Its trajectory multiplier rewards correct answers with accurate clue localization and fewer redundant turns.
- Trajectory-Guided Reinforcement Learning: The Hybrid Clue Score and Turn Decay Factor discourage irrelevant retrievals and redundant loops by rewarding localization precision and penalizing excessive invocations.The turn penalty applies when actual tool calls exceed the annotated reference limit.
4. Dataset
Seeker-173K addresses the shortage of trajectory-rich data for native interleaved tool invocation through a verified four-stage synthesis pipeline. Its taxonomy trains adaptive tool use, multi-timestamp reasoning, self-reflection, and autonomous planning.
- Dataset Overview: Seeker-173K contains approximately 173k high-quality native multi-turn tool-interaction trajectories for supervised and reinforcement learning.The dataset is designed to cultivate efficient and autonomous tool-use capabilities.
- Data Synthesis Pipeline: The pipeline localizes critical segments, verifies answer sufficiency from crops, generates timestamped tool trajectories, and filters them through logical consistency checks.Human verification is enforced through random sampling across the construction stages.
- Dataset Design: The dataset is stratified by evidence cardinality and visual saliency to support distinct adaptive invocation and complex reasoning behaviors.Adaptive invocation teaches when to bypass tools or use them for subtle clues, while complex reasoning spans disparate timestamps.
- Dataset Design: Supplementary Self-Reflection and Free-Format tasks provide supervision for error correction and autonomous planning.These tasks complement the dataset’s evidence-seeking and multi-hop reasoning trajectories.
5. Experiments
Video-o3 is evaluated across long-video understanding, reasoning, temporal grounding, ablations, interaction efficiency, and transfer to a smaller base model. Results show strong benchmark performance, benefits from its masking, reward, and multi-turn mechanisms, and continued gains with deeper interaction.
- Long Video Understanding: 66.5% on VideoMME follows SFT initialization, improving over the RL-only model’s 66.1%.On MLVU, LVBench, and LongVideoBench, the RL-trained model reaches 71.9%, 47.5%, and 59.3%, respectively, before SFT.
- Video Reasoning: 44.7% on MMR-V improves over Qwen2.5-VL’s 32.4%, while VideoMMMU rises from 50.0% RL-only to 51.7% with SFT.Video-Holmes is also described as requiring intricate multi-hop clue inference.
- Temporal Grounding: 60.7 mIoU on Charades-STA exceeds LongVT’s 27.2 and Qwen2.5-VL’s 43.6 in temporal grounding.The evaluation treats temporal grounding as a capability supporting accurate evidence retrieval for multi-hop reasoning.
- Ablations: Removing all attention masking lowers LongVideoBench accuracy from 56.6% to 55.0%, while partial masks also trail the full strategy.The ablation attributes this pattern to separating clue localization from answer reasoning and reducing interference between them.
- Ablations: Removing VTGR components destabilizes training, weakens efficient tool use, or expands trajectories beyond inference-time caps.The bonus multiplier, Hybrid Clue Score, and Turn Decay Factor regulate different aspects of the reasoning trajectory.
- Interaction Depth: Accuracy consistently increases as the interaction limit grows from 2 to 8 turns, especially on MLVU and VideoMMMU.The study interprets the sustained improvement as evidence that deeper interaction supports decomposition and iterative ambiguity resolution.
- Efficiency: Video-o3 reaches 72.1% MLVU accuracy using 18.7GB GPU memory and 10.2 seconds per sample, gaining 3.4% over VideoChat-R1.The comparison evaluates memory, inference time, visual tokens, response tokens, and accuracy.
6. Conclusion
The paper presents Video-o3 as a native interleaved tool-invocation framework for long-form video understanding. Task-Decoupled Attention Masking, Verifiable Trajectory-Guided Reward, and Seeker-173K support its reported state-of-the-art performance across multiple benchmarks.
- Conclusion: Video-o3 combines native interleaved tool invocation with Task-Decoupled Attention Masking and Verifiable Trajectory-Guided Reward.The masking targets task interference, while the reward balances exploration and efficiency.
- Conclusion: Seeker-173K supports robust end-to-end training, and Video-o3 establishes state-of-the-art performance across multiple benchmarks.The conclusion frames native tool-use paradigms as promising for agentic multimodal language models.
Impact Statement
Video-o3’s impact includes both practical benefits for long-video information extraction and risks involving surveillance, privacy, licensing, and data governance. Its training and inference design supports adaptive evidence gathering across several clue-seeking task types.
- Video-o3 targets efficient information extraction from large-scale video data, with potential applications in digital archives, educational content analysis, and personal media management.
- The framework can be repurposed for unintended surveillance or privacy-intrusive monitoring through fine-grained visual scrutiny and precise temporal grounding.
- The data construction uses public videos and requires attention to source licenses, platform terms, privacy requirements, and derived-annotation safeguards.
- The synthesized task taxonomy covers direct answering and tool invocation for both single-clue and multi-clue evidence settings.
C.2. Dataset statistics
The dataset combines conventional video-understanding data with Seeker-173K and additional self-collected videos, while experiments analyze tool-use rewards, attention masking, and trajectory efficiency.
- Seeker-173K contains roughly 173k instances stratified by the proposed task definitions to instill advanced agentic behaviors.
- The dataset adds 14.8k self-collected YouTube instances targeting single-clue and multi-clue tool-use scenarios.
- Removing the Hybrid Clue Score sharply reduces tool-call frequency and accuracy, while removing Turn Decay slightly raises tool use but lowers accuracy.
- The Hybrid Clue Score improves Charades-STA mIoU from 46.5 to 53.8 without temporal grounding data and from 59.6 to 60.7 with it.
- The Turn Decay Factor reduces average turns from 2.57 to 2.31 and lowers trajectories exceeding six turns from 4.8% to 1.3%.
E.5. Effectiveness of Training Stage
Training-stage analysis shows that native multi-turn tool use requires suitable initialization and reinforcement learning, while free-format answer supervision can supplement sparse trajectory annotations.
- Directly prompting the off-the-shelf Qwen2.5-VL for multi-turn tool use causes significant degradation relative to direct answering.
- SFT cold-start followed by RL delivers optimal performance and enables native interleaved tool invocation for multi-hop clue localization and holistic reasoning.
- Adding 85% free-format Video-QA data improves VideoMME, MLVU, and LVBench by 1.8, 1.7, and 3.1 points, respectively.
- Dense trajectory annotations support fast convergence but are not strictly required for scaling because free-format data encourages autonomous tool-strategy optimization through answer-level feedback.
G. Qualitative Analysis
Qualitative examples illustrate single-round localization, adaptive visual-token allocation, and multi-hop reasoning through iterative tool invocation.
- A single tool invocation can capture transient frames in news segments to extract crucial clues.
- Dynamic quota allocation lets the model adapt visual-token usage to segment length, balancing computational efficiency and detail granularity.
- A multi-hop example demonstrates reasoning over evidence distributed across video segments for complex tasks such as event ordering.
H. Failure Case Analysis
The failure cases show that Video-o3 can exceed its interaction budget during blind clue exploration and can make decisions inconsistent with retrieved evidence or intermediate reasoning.
- Turn-limit failure: Too many incorrect exploration rounds can exhaust the turn limit before the model gathers necessary evidence.This occurs when subtle clues are difficult to locate in long videos.
- Reasoning inconsistency: In one fake-thinking pattern, the model retrieves correct clues but still selects an incorrect answer.The final prediction contradicts the retrieved evidence and intermediate reasoning.
- Reasoning inconsistency: In another pattern, the model commits prematurely after insufficient clue localization instead of invoking tools for additional clips.Both cases indicate disproportionate reliance on overview content.
J. Limitations and Future Work
The paper identifies limits in interaction depth and tool scalability: fixed context budgets may truncate difficult reasoning, while static tool definitions make adding utilities costly.
- Context Window and Interaction Depth: Interaction depth is limited to 8 turns to balance training efficiency with hardware memory constraints.The authors report that this budget covers most current benchmarks empirically.
- Context Window and Interaction Depth: The 8-turn budget may fall short for full-length movies or queries requiring deep multi-hop logical chains.When the context quota is depleted, inference halts and evidence collection may be truncated.
- Flexibility of Tool Scalability: Static predefined tool definitions limit scalability because each added utility requires manually crafted protocols and parsers.The paper proposes Code-as-Action as a future direction for more flexible tool usage.