PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Shijian Wang, Jiarui Jin, Xingjian Wang, Linxin Song, Runhao Fu, Hecheng Wang, Zongyuan Ge, Yuan Lu, Xuelian Cheng

arXiv:2510.23473v1cs.CV

TL;DR

Video reasoning lacks the dynamic visual reasoning paradigm developed for images, especially for temporal localization and comprehension. Video-Thinker addresses this gap with intrinsic grounding and captioning trained through Video-Thinker-10K, SFT, and GRPO, achieving state-of-the-art results among 7B-sized MLLMs.

  • Problem

    Video MLLMs struggle to dynamically reason over temporal sequences, while existing approaches do not seamlessly integrate temporal grounding into chain-of-thought.

  • Method

    Video-Thinker trains MLLMs on Video-Thinker-10K with SFT followed by GRPO to autonomously perform grounding and captioning during reasoning without external tools.

  • Results

    43.22% on Video-Holmes, 33.25% on CG-Bench-Reasoning, and 80.69% on VRBench establish new state-of-the-art results among 7B-sized MLLMs.

  • Takeaways & Limitations

    Video-Thinker demonstrates that intrinsic grounding and captioning can support autonomous temporal navigation and video reasoning without external tools.

  • Takeaways & Limitations

    The study evaluates publicly available academic benchmarks and acknowledges that large vision-language models may inherit biases from their training data.

Abstract

from arXiv · show

Recent advances in image reasoning methods, particularly "Thinking with Images", have demonstrated remarkable success in Multimodal Large Language Models (MLLMs); however, this dynamic reasoning paradigm has not yet been extended to video reasoning tasks. In this paper, we propose Video-Thinker, which empowers MLLMs to think with videos by autonomously leveraging their intrinsic "grounding" and "captioning" capabilities to generate reasoning clues throughout the inference process. To spark this capability, we construct Video-Thinker-10K, a curated dataset featuring autonomous tool usage within chain-of-thought reasoning sequences. Our training strategy begins with Supervised Fine-Tuning (SFT) to learn the reasoning format, followed by Group Relative Policy Optimization (GRPO) to strengthen this reasoning capability. Through this approach, Video-Thinker enables MLLMs to autonomously navigate grounding and captioning tasks for video reasoning, eliminating the need for constructing and calling external tools. Extensive experiments demonstrate that Video-Thinker achieves significant performance gains on both in-domain tasks and challenging out-of-domain video reasoning benchmarks, including Video-Holmes, CG-Bench-Reasoning, and VRBench. Our Video-Thinker-7B substantially outperforms existing baselines such as Video-R1 and establishes state-of-the-art performance among 7B-sized MLLMs.

1 INTRODUCTION

Video-Thinker extends dynamic visual reasoning from images to videos by enabling MLLMs to autonomously ground and caption temporal segments during chain-of-thought reasoning. It introduces Video-Thinker-10K and two-stage training to support this capability without external tools.

  • 1 INTRODUCTION: Video reasoning remains difficult because videos contain temporal dependencies, motion, and evolving narratives that require dynamic temporal reasoning.
  • 1 INTRODUCTION: Video-Thinker extends “Thinking with Images” to video by integrating autonomous grounding and captioning into chain-of-thought reasoning.Grounding identifies key frames, while captioning interprets and synthesizes their visual cues.
  • 1 INTRODUCTION: Video-Thinker-10K provides curated video-reasoning samples with selected segments, detailed captions, and structured reasoning traces for synthesizing multimodal cues.
  • 1 INTRODUCTION: SFT teaches the reasoning format before GRPO strengthens autonomous grounding and captioning, enabling temporal navigation without externally constructed tools.
  • 1 INTRODUCTION: The framework establishes new state-of-the-art performance among 7B-sized MLLMs across multiple video reasoning benchmarks.

2 RELATED WORK

Prior video-reasoning approaches largely separate temporal localization from reasoning instead of integrating grounding into chain-of-thought. Video-Thinker addresses this gap by making grounding and captioning integral components of temporal reasoning.

  • 2 RELATED WORK: Existing approaches remain largely limited to temporal localization or standalone video reasoning, without seamless grounding inside chain-of-thought.
  • 2 RELATED WORK: Video-Thinker integrates grounding and captioning into chain-of-thought so MLLMs can attend to, interpret, and analyze relevant temporal segments.

3 THINK WITH VIDEOS: FROM DATA SYNTHESIS TO MODEL TRAINING

Video-Thinker builds a curated dataset and trains models in two stages to generate structured video-reasoning traces using grounding and captioning. SFT teaches the format, while GRPO optimizes autonomous navigation using correctness and format rewards.

  • Data synthesis: Video-Thinker-10K combines video questions, segment-level descriptions, temporal annotations, and structured reasoning traces for video-reasoning training.Traces use <time>, <caption>, and <think> tags to represent localization, visual evidence, and analysis.
  • Data synthesis: The dataset is curated from diverse video sources spanning activities, tutorials, cooking, situated reasoning, and long-form content.The source collection includes ActivityNet, TutorialVQA, YouCook2, STAR, ScaleLong, and LVBench.
  • Data synthesis: Hindsight curation filters generated temporal and captioning content to improve its usefulness for final answers without relying on full manual evaluation.The generated <time> and <caption> content is evaluated as part of the curation process.
  • Training strategy: SFT initializes format-following by training on curated reasoning traces and final answers conditioned on the video, question, and preceding tokens.The objective minimizes negative log-likelihood over the concatenated reasoning trace and answer.
  • Training strategy: GRPO generates multiple reasoning traces per sample and optimizes them with group-relative advantages based on correctness and structured-format rewards.Relative comparisons within each group promote accurate reasoning and adherence to temporal structure.
  • Observed behavior: Video-Thinker exhibits self-reflective behavior by revisiting and refining initial interpretations of grounding and captioning outputs.The paper characterizes these behaviors as “aha moments” involving dynamic internal feedback.

4 EXPERIMENT

Video-Thinker-7B is evaluated across in-domain and out-of-domain video reasoning benchmarks, frame counts, and grounding/captioning tasks. It achieves strong benchmark performance, with GRPO improving generalization over SFT and grounding/captioning results supporting the proposed mechanism.

  • Performance Comparisons and Analysis: 43.22% on Video-Holmes, 33.25% on CG-Bench-Reasoning, and 80.69% on VRBench establish new SOTA results among 7B-sized MLLMs.These results improve over the best baselines by 4.68%, 3.81%, and 11.44%, respectively.
  • Performance Comparisons and Analysis: 11.70%, 8.30%, and 18.29% gains show that GRPO substantially improves out-of-domain generalization over SFT on Video-Holmes, CG-Bench-Reasoning, and VRBench.SFT remains important for acquiring the structured reasoning format, while GRPO provides stronger cross-domain performance.
  • Performance Comparisons and Analysis: Video-Thinker-7B consistently outperforms Qwen2.5-VL-7B and Video-R1-7B across evaluations using 16, 32, and 64 video frames.Increasing frame count generally improves performance, with 64 frames producing optimal results in most cases.
  • In-Depth Analysis of Grounding and Captioning Capabilities: 48.22% mIoU, 79.29% Recall@0.3, and 51.49% Recall@0.5 demonstrate superior video grounding performance over baseline models.The overall average is 59.67%, a 97% relative improvement over the baseline’s 30.23%; Video-R1 is excluded because it could not follow the temporal-annotation prompt.
  • In-Depth Analysis of Grounding and Captioning Capabilities: 17.11% average captioning performance improves on Qwen2.5-VL-7B-Instruct by 31.2% relatively and on Video-R1-7B by 61.0% relatively.Video-Thinker-7B leads on METEOR, ROUGE-L, and BLEU@1.
  • In-Depth Analysis of Grounding and Captioning Capabilities: Oracle experiments show that providing accurate grounding and captioning information significantly enhances Video-R1-7B performance on Video-Holmes.The experiment further tests the importance of these capabilities for video understanding.

5 CONCLUSION AND FUTURE WORK

Video-Thinker extends “Thinking with Images” to video by enabling autonomous grounding and captioning during inference. The Video-Thinker-10K dataset and SFT-plus-GRPO training support reasoning without external tools, while Video-Thinker-7B achieves SOTA among 7B-sized models.

  • Conclusion and Future Work: Video-Thinker extends the “Thinking with Images” paradigm to video reasoning through intrinsic grounding and captioning capabilities.The model generates reasoning clues throughout inference without relying on external tools.
  • Conclusion and Future Work: Video-Thinker-10K and two-stage SFT-plus-GRPO training enable autonomous video reasoning and produce SOTA performance among 7B-sized models.Future directions include larger models, additional intrinsic capabilities, and audio.

ETHICS STATEMENT

The study evaluates multimodal video understanding and reasoning using public benchmarks while addressing data privacy and inherited-bias considerations.

  • Ethics Statement: Experiments use publicly available datasets under their licenses, without collecting, generating, or annotating personally identifiable or sensitive private data.Evaluations were restricted to established academic benchmarks for fair comparison.
  • Ethics Statement: The authors acknowledge that large-scale vision-language models may inherit biases from training data and encourage caution in downstream applications.The statement reports no direct ethical concerns beyond risks inherent in benchmark datasets.

REPRODUCIBILITY STATEMENT

The paper describes its datasets, implementations, training procedure, and evaluation protocol to support reproducibility. Algorithm 1 summarizes data synthesis, supervised training, and GRPO optimization.

  • Reproducibility Statement: Datasets, model implementations, experimental settings, hyperparameters, and evaluation protocols are described in the paper and appendix.Benchmarks and metrics are publicly available, and baselines use released checkpoints or open-source implementations.
  • Reproducibility Statement: Algorithm 1 organizes Video-Thinker training into data synthesis, structured reasoning-trace construction, and GRPO optimization for autonomous navigation.The algorithm takes collected source data and a pretrained MLLM as input and returns a tuned model.

B DATA DISTRIBUTION OVER SOURCE DATASETS IN SECTION 3.1

Figure 5 presents the data distribution of the Video-Thinker-10K dataset.

  • Figure 5 shows how samples are distributed across the Video-Thinker-10K dataset.

C EXPERIMENT CONFIGURATION

The experiments use diverse video datasets and benchmarks spanning short and long videos, instructional content, situated reasoning, narrative understanding, and multimodal reasoning. Baselines include several 7B- and 8B-sized multimodal models and reinforcement-learning video-reasoning systems.

  • Datasets and benchmarks: ActivityNet provides 5,800 long untrimmed videos and 58K bilingual questions for spatio-temporal reasoning and fine-grained comprehension.
  • Datasets and benchmarks: STAR evaluates situated reasoning in daily-life scenarios using 22K short clips and 60K structured questions across four reasoning types.
  • Datasets and benchmarks: ScaleLong isolates four temporal granularities across 269 long videos and 1.7K questions, while LVBench tests long-horizon reasoning on 103 videos and 1.5K questions.
  • Datasets and benchmarks: Video-Holmes, CG-Bench, and VRBench evaluate narrative multi-clue reasoning, evidence-grounded video reasoning, and multi-step reasoning over multilingual narrative videos.
  • Baselines: Compared systems include InternVL, Qwen2.5-VL, VideoChat-R1, Temporal-R1, Time-R1, Open-R1-Video, Video-R1, TW-GRPO, and GRPO-CARE variants.

C.3 EVALUATION METRICS

The evaluation measures temporal localization, segment retrieval, and caption quality using overlap, recall, lexical, semantic, and sequence-level metrics.

  • Temporal localization: mIoU averages the maximum temporal-segment IoU for each ground-truth segment, measuring localization alignment, boundary precision, and temporal coverage.
  • Temporal localization: Recall@K measures the fraction of ground-truth spans matched by predictions with IoU at least K, commonly using K=0.3 and K=0.5.
  • Caption quality: BLEU@1 measures unigram precision and captures surface-level lexical overlap between generated and reference captions.
  • Caption quality: METEOR combines precision, recall, synonymy, stemming, paraphrase matching, and a fragmentation penalty to assess caption completeness and coherence.
  • Caption quality: ROUGE-L uses the longest common subsequence to compare candidate and reference sequences through recall, precision, and an F1-like score.

D PROMPTS

The prompts require models to identify relevant video segments, describe visual evidence, reason across those observations, and return structured answers. Dataset-specific question-generation prompts enforce multi-segment synthesis, analytical or perceptual reasoning, constrained answer options, and JSON output.

  • D.1 TRAINING AND EVALUATION: The training-and-evaluation prompt requires time ranges, visual-clue captions, analysis, natural thought expressions, and a final answer in tagged fields.
  • D.2 VIDEO CAPTION GENERATION: The caption-generation prompt requests factual, objective descriptions of visual evidence relevant to the question, without reasoning or analysis.
  • D.3 QA GENERATION: ActivityNet question generation uses background captions and chronologically ordered event descriptions to create multiple-choice questions requiring cross-segment synthesis.
  • D.3 QA GENERATION: YouCook2 prompts target reasoning about cooking causes, techniques, ingredient interactions, sequential logic, detailed perception, or summarization across cooking steps.
  • D.3 QA GENERATION: Across QA-generation prompts, questions must avoid explicit references to segments, use only supplied visual information, and test multi-event reasoning or sophisticated perception.
  • D.3 QA GENERATION: Answer-generation prompts require 4–6 options, one definitive answer, sophisticated distractors, and options derivable from or contradicted by the event descriptions.
  • D.3 QA GENERATION: The required output format is a valid JSON object containing exactly question, options, and answer keys.
  • D.3 QA GENERATION: TutorialVQA prompts similarly require reasoning across chronological visual events, selecting analytical or perceptual questions according to the relationships among events.

E EXPERIMENTAL VERIFICATION OF GROUNDING AND CAPTIONING CAPABILITIES

The experiments evaluate how grounding and captioning affect video reasoning, with their combination producing the strongest reported accuracy. Additional tables document performance across training steps and learning rates.

  • The evaluation compares four configurations using Video-R1-7B on Video-Holmes, with accuracy as the primary metric.The configurations test the contribution of temporal grounding and captioning information to reasoning performance.
  • Table 5 reports Video-Thinker performance across different training steps, while Table 6 reports performance across different learning rates.These tables provide the experimental setup for the training-step and learning-rate analyses.
  • 63% accuracy with both grounding and captioning exceeded captioning alone at 56% and grounding alone at 53%, versus 37% without either.The results support complementary benefits from temporal focus and contextual understanding.

F ABLATION STUDIES

The ablations identify training settings associated with stronger performance and illustrate Video-Thinker’s iterative reasoning outputs across video benchmarks. They also show that Video-R1-7B could not follow the required temporal-annotation format for the evaluation.

  • Impact of Training Steps: 58.35% average performance at 2500 GRPO steps was optimal, while further training degraded several benchmarks, especially out of domain.The result suggests a balance between learning and generalization at 2500 steps.
  • Impact of Learning Rate: A 5e-6 learning rate produced optimal performance and outperformed the Qwen2.5-VL-7B-Instruct and Video-R1-7B baselines across the evaluated benchmarks.Performance degraded dramatically at 1e-5, indicating instability and poor convergence at the highest tested rate.
  • Qualitative Examples: Supplementary examples across Video-Holmes, VRBench, and CG-Bench show iterative reasoning and error correction in Video-Thinker-7B outputs.The examples are described as evidence of a dynamic internal feedback mechanism rather than simple pattern matching.
  • Evaluation Constraint: Video-R1-7B was excluded from the temporal-annotation evaluation because it could not generate grounding content within the required <time></time> tags.The cited example illustrates the instruction-following failure underlying that exclusion.
Loading 2510.23473v1…