LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling

Zuhao Yang; Sudong Wang; Kaichen Zhang; Keming Wu; Sicong Leng; Yifan Zhang; Bo Li; Chengwei Qin; Shijian Lu; Xingxuan Li; Lidong Bing

Source-linked AI summary

LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling

Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, Lidong Bing

arXiv:2511.20785v3Nov 25, 2025cs.CV

TL;DR

LongVT targets hallucinations in long-form video reasoning, where decisive evidence is sparse and temporally dispersed. It trains an agentic LMM to interleave temporal cropping and fine-grained inspection with reasoning, supported by VideoSIAH; the framework consistently outperforms strong baselines across four challenging benchmarks. Its recursive history can nevertheless exhaust the underlying model’s context budget on ultra-long videos.

Problem
Long-form videos contain sparse, temporally dispersed evidence, while existing supervision and benchmarks insufficiently support fine-grained temporal grounding and evidence-seeking reasoning.
Method
LongVT uses interleaved Multimodal Chain-of-Tool-Thought with native temporal cropping and a three-stage training strategy supported by VideoSIAH.
Results
LongVT consistently improves long-video understanding and reasoning over existing strong baselines across four challenging benchmarks, with VideoSIAH-Eval performance reaching 42.0 and exceeding the second-best model by 6 points.
Takeaways & Limitations
LongVT demonstrates that active, evidence-seeking temporal inspection can make open-source LMMs more reliable for long-video reasoning.
Takeaways & Limitations
Recursive tool interactions accumulate history tokens and dense visual features, potentially exhausting context budgets and causing training out-of-memory errors or truncation-related degradation.

Abstract

from arXiv · show

Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .

1. Introduction

LongVT addresses hallucination-prone long-video reasoning by having LMMs globally skim videos, retrieve relevant clips, and iteratively inspect evidence through native tool calling. It pairs this agentic framework with VideoSIAH data and reports state-of-the-art long-video reasoning performance.

Long-form videos require sustained comprehension across thousands of frames and contain complex event structures.
Human-inspired global-to-local reasoning first skims a video coarsely, then zooms into promising segments instead of scanning every frame.
iMCoTT interleaves multimodal reasoning with on-demand temporal retrieval, enabling models to propose windows, inspect clips, and revise hypotheses.
VideoSIAH supplies fine-grained segment-in-a-haystack supervision, including 247.9K SFT samples, 1.6K agentic RL samples, and 15.4K RFT samples.
LongVT uses three-stage training to develop precise temporal proposal, dense-frame reasoning, self-correction, and open-ended generalization capabilities.
LongVT combines native tool-augmented reasoning with a scalable data pipeline and decoupled temporal-grounding reward, achieving state-of-the-art long-video reasoning.

3. VideoSIAH: A Fine-Grained Data Suite for Evidence-Sparse Long-Video Reasoning

VideoSIAH is a semi-automatic, human-in-the-loop data suite designed for evidence-sparse long-video reasoning. Its pipeline grounds QA and tool-use traces in coherent temporal segments, filters unsupported answers, and curates duration- and difficulty-aware training splits.

Existing long-video resources often lack supervision for forming, verifying, and revising temporal hypotheses, while multiple-choice benchmarks permit shortcut solutions.
VideoSIAH segments long videos into semantically stable intervals, captions them, and generates temporally grounded reasoning traces through a semi-automatic pipeline.
Text-based and multimodal filtering remove ill-posed, leaked, hallucinated, or visually unsupported QA claims, with annotator feedback refining prompts.
The pipeline generates iMCoTT traces only for cold-start SFT, while RL uses filtered QA pairs.
Multi-round tool-calling traces are sampled with probability determined by video length so longer videos receive more tool-calling rounds.
RL curation balances video durations and retains questions with mixed rollout outcomes, while RFT retains correct answers with temporal-span IoU of at least 0.3.
VideoSIAH includes non-tool and tool-augmented SFT data alongside RL and RFT instances, with the benchmark averaging approximately 1,688-second videos.

4. Training Strategy

LongVT uses a three-stage pipeline to train an agent that combines temporal localization, tool use, and multimodal reasoning for long-video QA. Its reinforcement-learning stage jointly rewards answer accuracy, format compliance, and temporal grounding, while reinforcement fine-tuning stabilizes successful agentic behaviors.

Training Strategy: LongVT’s three stages teach temporal-window proposal, tool invocation, evidence-based reasoning, and stabilization through distilled high-quality RL trajectories.The pipeline comprises cold-start supervised fine-tuning, agentic reinforcement learning, and agentic reinforcement fine-tuning.
Training Strategy: Qwen2.5-VL-7B fails to improve during preliminary RL and eventually collapses, revealing deficiencies in temporal localization and tool-output integration.These findings motivate the cold-start SFT stage before agentic RL.
Agentic Reinforcement Learning: The joint reward combines answer accuracy, format compliance, and temporal grounding precision for sampled tool-using trajectories.The reward is designed to unify answer correctness with temporal alignment rather than optimizing either objective alone.
Agentic Reinforcement Learning: Temporal accuracy uses an LLM judge with fully, partially, and inconsistently correct verdicts mapped to normalized rewards of 1, 0.5, and 0.Temporal localization is scored with IoU, which equals 1 for an exact span match and 0 for no overlap.

5. Experiments

LongVT is evaluated against open-source and proprietary video-centric LMMs across four long-video benchmarks under sparse and dense frame sampling. Its staged training and fine-grained supervision improve long-video reasoning, temporal grounding, and tool use.

Main Results: LongVT-7B-RL slightly surpasses the best existing open-source baseline at 64 frames, while LongVT-7B-RL and LongVT-7B-RFT lead by larger margins with dense sampling.The evaluation compares sparse 64-frame sampling with dense 512- or 768-frame sampling.
Main Results: 42.0 on VideoSIAH-Eval gives LongVT-7B-RFT a 6-point lead over the second-best model on open-ended, fine-grained questions from hours-long videos.The result is reported for the benchmark requiring retrieval of fine-grained visual evidence.
Main Results: LongVT’s best checkpoint comes within roughly four points of GPT-4o on average, narrowing the gap between open-source and proprietary LMMs.The comparison is reported across the evaluated long-video benchmarks.
Ablation Studies: Removing self-curated iMCoTTs during SFT consistently reduces long-form understanding, while removing self-curated QAs during RL quickly lowers accuracy, localization, and systematic tool use.The ablation results identify fine-grained, tool-augmented supervision as important for the learned reasoning behavior.
Ablation Studies: IoU-rewarded training outperforms Recall-rewarded training on temporal grounding because IoU better constrains boundary agreement and penalizes span inflation.Recall can increase by enlarging the predicted span around the ground-truth interval.
Training-Stage Ablations: SFT teaches temporal window selection, tool inspection, and evidence integration; RL refines when and how long to inspect, while RFT stabilizes these behaviors and adds further gains.RFT distills high-reward trajectories back into supervised data, producing the best model on VideoSIAH-Eval.

6. Conclusion

LongVT enables long-video reasoning through interleaved tool-augmented thought and on-demand temporal inspection. Supported by the fine-grained VideoSIAH suite, its three-stage pipeline yields substantial and consistent improvements over strong baselines.

6. Conclusion: LongVT transforms long-video understanding from passive frame consumption into active, evidence-seeking reasoning through multimodal tool-augmented thought.The framework interleaves reasoning with on-demand temporal inspection.
6. Conclusion: VideoSIAH provides large-scale, fine-grained data designed for evidence-sparse long-video reasoning tasks.The data suite supports the proposed training and evaluation framework.
6. Conclusion: The three-stage training pipeline produces substantial and consistent improvements compared with existing strong baselines.The conclusion attributes the reported improvements to the proposed pipeline and supporting data suite.

Supplementary Material

The supplementary material aligns LongVT’s global-to-local strategy with proprietary-model viewing behavior and documents contamination vulnerabilities in existing benchmarks. These findings motivate VideoSIAH-Eval’s open-ended design.

Human-Aligned Watching Strategies: LongVT’s global-to-local strategy mirrors Gemini 2.5 Pro’s coarse-scan then fine-scan workflow for locating and verifying fine-grained video details.Gemini’s proposed process searches broadly for event indicators before isolating the exact moment.
Human-Aligned Watching Strategies: Gemini 2.5 Pro explicitly recommends a coarse scan followed by a fine scan, including scrubbing back 30–60 seconds to verify details.
Benchmark Contamination: Existing benchmarks show severe contamination vulnerabilities, including high performance without visual frames and sensitivity to rearranged answer choices.The contamination study uses “No Visual” and “Rearranged Choices” settings to probe memorization and option-pattern reliance.
Benchmark Contamination: Qwen3-VL scores 0.00 on VideoSIAH-Eval without visual input, contrasting with above-chance no-visual scores on existing multiple-choice benchmarks.
Benchmark Contamination: VideoSIAH-Eval uses fully open-ended QA, avoiding random option-answer mapping and reducing susceptibility to option hacking.The benchmark is presented as a more robust assessment of video-grounded capability.

3. Additional VideoSIAH Details

Additional VideoSIAH details describe the data mixture supporting cold-start SFT and reinforcement learning, alongside the objectives used for token prediction and GRPO optimization. The supplementary benchmark statistics emphasize diverse video reasoning coverage.

Data Construction: The cold-start SFT data mixes general visual reasoning, logical inference, and mathematical problem-solving image-based CoT traces.These traces are used to strengthen fundamental perceptual capabilities through image–video modality synergy.
Benchmark Statistics: VideoSIAH-Eval contains 244 videos and 652 QA pairs spanning diverse domains, with Action Recognition and Temporal Reasoning comprising 17% of queries.
Training Objectives: SFT minimizes negative log-likelihood so the model assigns higher probability to the ground-truth next token.The loss sums token-level conditional log probabilities across a sequence.
Training Objectives: GRPO samples K responses from the behavior policy, computes group-relative advantages using a baseline, and optimizes a length-normalized KL-regularized objective.The reference policy is frozen, and β controls KL strength.

5. Reflection Trajectory: From Verbose Self-Correction to Internalized Tool Usage

Reflection-token trajectories show a three-phase evolution from verbose self-correction toward more efficient tool exploitation during training.

Reflection Trajectory: The model’s reflection-token trajectory begins with verbose self-correction during steps 0–50 as localization accuracy remains insufficient.
Reflection Trajectory: Training is characterized as evolving from exploratory correction toward efficient tool exploitation.
Reflection Trajectory: Reflection density reveals distinct phases in the model’s internal thought process rather than a uniform reasoning pattern.

6. Additional Implementation Details

Implementation details cover the three-stage training infrastructure, tool-enabled evaluation, and supplementary visualizations. The material also identifies context accumulation as a memory bottleneck for recursive tool use.

Tool Execution: The crop video() function is an external executor, while “native” means the tool-invocation policy is internalized through end-to-end training without an external retrieval agent.
Training Infrastructure: SFT uses online stream packing with fixed 51,200-token buffers to reduce padding computation and memory overhead.
Training Infrastructure: RL supports multi-turn multimodal tool-augmented rollouts, using 16 rollouts per prompt and a 16,384-token maximum for new tokens.
Evaluation: Evaluation parses reasoning steps, tool invocations, and final answers, then combines rule-based validation with LLM-as-a-Judge semantic evaluation.

7. Inference Efficiency Analysis

LongVT-7B-RFT achieves strong inference efficiency despite multi-turn tool interactions, using targeted segment sampling rather than encoding entire long videos frame by frame.

Latency comparison: 1329.8 seconds on VideoMMMU and 1509.3 seconds on LVBench are LongVT-7B-RFT’s lowest reported latencies.The comparison covers four long-video benchmarks and uses uniform 64-frame sampling with online inference.
Latency comparison: LongVT-7B-RFT maintains highly competitive inference speeds on VideoMME and VideoSIAH-Eval.The reported efficiency comparison includes representative baselines and final-stage models.
Efficiency mechanism: Its multi-turn agentic framework can outpace single-turn baselines because targeted reasoning avoids unnecessary video processing.The analysis attributes this counter-intuitive efficiency to precision in reasoning and strategic sampling of relevant segments.
Evaluation setup: Inference latency is measured in seconds using LMMs-Eval with online vLLM serving on eight GPUs.Intermediate SFT and RL variants are excluded from the table’s representative comparison.

9. Failure Case Analysis

Failure analysis shows that RL-only training can lose the semantic link between tool outputs and the user’s question, while recursive reasoning remains constrained by context accumulation.

RL-only failure: The RL-only variant invokes a tool but fails to integrate returned evidence, instead reverting to generic video captioning.The failure occurs after inspecting resampled frames and reflects confusion caused by the context shift.
Training implication: SFT cold start is important because it teaches the intended semantics of tool usage and evidence incorporation.The paper connects this requirement to the RL-only model’s inability to interpret tool outputs correctly.
Context limitation: As recursive tool interactions accumulate history tokens and dense visual features, LongVT risks context exhaustion, out-of-memory errors, and truncation-related degradation.This limitation becomes more pronounced when multiple crop calls are needed for ultra-long or infinite video streams.
Future direction: Multi-agent collaboration is proposed as a future direction to decouple context management from reasoning through manager and worker agents.Workers would inspect separate temporal segments and summarize observations for a manager.
Broader implications: LongVT is presented as supporting safer and more trustworthy video analysis through proactive evidence seeking and self-correction.The stated application scope includes surveillance, content moderation, educational analytics, and assistive technologies.

12. Ethical Considerations

LongVT emphasizes reliability and transparency by grounding answers in retrieved evidence and exposing tool-augmented reasoning steps, while advocating privacy safeguards for deployment.

Reliability: On-demand visual verification is designed to reduce fabricated events and context misinterpretation in complex long-form videos.The paper frames this as improving the trustworthiness of video LMM predictions.
Transparency and interpretability: Observable stages—global skimming, tool invocation, evidence retrieval, and self-reflection—support auditing and debugging.Users can trace which video segments informed a model’s conclusion.
Responsible data use: The system is described as not accessing private or surveillance feeds and as introducing no additional personally identifiable information.The authors nevertheless advocate strict privacy standards and ethical guidelines for real-world deployment.