Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Introduction

• Video reasoning is challenging because temporal dependencies, motion, and evolving narratives require reasoning over sequences that MLLMs struggle to manipulate dynamically.The paper contrasts its emergent capabilities with pre-designed chain-of-thought methods and reports achieving effective video reasoning with 10K examples, versus datasets exceeding 160K samples in prior general visual reasoning work (Feng et al., 2025).Source: §1 · p. 2
• Video-Thinker enables MLLMs to reason over video through grounding, which localizes key frames, and captioning, which interprets and synthesizes their visual cues.These operations support autonomous analysis of specific temporal segments through structured chain-of-thought reasoning, rather than treating a video as a monolithic input or invoking external handcrafted tools.Source: §1 · p. 2
• Video-Thinker-10K provides 10K curated examples with key video segments, visual-clue captions, and structured reasoning traces.The traces use <time> for temporal localization, <caption> for visual-cue extraction, and <think> for reasoning that synthesizes the cues.Source: §1 · p. 3
• Video-Thinker achieves state-of-the-art performance among 7B-sized MLLMs on Video-Holmes, CG-Bench-Reasoning, and VRBench.The paper presents this as performance across challenging out-of-domain video reasoning benchmarks.Source: Figure 1 · p. 1, §1 · p. 3
• Training combines SFT on curated reasoning traces with GRPO using only the final answer as the outcome reward.The two-stage approach establishes format-following before reinforcement learning strengthens intrinsic grounding and captioning capabilities.Source: §1 · p. 3

• Deepeyes (Zheng et al., 2025) and Visual-ARFT (Liu et al., 2025) develop image reasoning with visual tools, CoT, task planning, and GRPO-based tool use.Deepeyes uses end-to-end reinforcement learning for autonomous visual-tool invocation, while Visual-ARFT uses Python-based image-processing operators.Source: §2 · p. 3
• Video-R1 (Feng et al., 2025), VideoChat-R1 (Li et al., 2025c), and Temporal-R1 (Li et al., 2025b) use reinforcement learning to improve video reasoning or spatiotemporal localization.Video-R1 promotes implicit temporal and spatial reasoning; VideoChat-R1 strengthens localization while preserving conversational proficiency; Temporal-R1 uses temporal-grounding rewards and variance-aware data selection.Source: §2 · p. 3
• Unlike approaches limited to temporal localization or standalone video reasoning, Video-Thinker integrates grounding and captioning into CoT to navigate and analyze relevant temporal segments.These capabilities let MLLMs attend to, interpret, and analyze temporal content throughout video tasks.Source: §2 · pp. 3–4

3.1 DATA SYNTHESIS VIA HINDSIGHT-CURATION REASONING

• Video-Thinker-10K combines caption-labeled sources for generating multi-segment reasoning questions with QA-labeled sources for generating answer-conditioned segment captions.The six sources are ActivityNet, TutorialVQA, YouCook2, STAR, ScaleLong, and LVBench.Source: §3.1 · p. 4
• After filtering corrupted videos and those with fewer than 64 frames, the pipeline synthesizes tagged temporal reasoning traces and retries samples up to three times when answer validation fails.DeepSeek-V3 generates traces from the ground-truth answer, captions, and temporal annotations; Qwen2.5-VL-7B-Instruct checks whether the time and caption content supports the correct answer.Source: §3.1 · pp. 4–5

3.2 TRAINING STRATEGY OF VIDEO-THINKER

• SFT cold-starts Video-Thinker on the <time>, <caption>, and <think> format by training it to predict the reasoning trace and final answer.The objective minimizes their concatenated sequence’s negative log-likelihood conditioned on the video, question, and preceding tokens.Source: §3.2 · p. 5
• GRPO further optimizes autonomous grounding and captioning by comparing multiple candidate reasoning traces per sample using rewards for correctness and format adherence.The format reward checks adherence to the structured <time>, <caption>, and <think> tags.Source: §3.2 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• In-domain tests use held-out 1:9 test-training splits from five datasets, while out-of-domain evaluation uses Video-Holmes, CG-Bench-Reasoning, and VRBench.The five in-domain datasets are ActivityNet, LVBench, ScaleLong, Star, and YouCook2; TutorialVQA has no corresponding test set because its training set contains only 76 samples.Source: §4.1 · pp. 6–7

4.2 PERFORMANCE COMPARISONS AND ANALYSIS

• Video-Thinker-7B outperforms Qwen2.5-VL-7B and Video-R1-7B at 16, 32, and 64 inference frames, while 64 frames give the best results in most cases.Increasing frame count improves performance across most benchmarks and all evaluated models.Source: Table 2 · p. 8, §4.2 · p. 8
• Video-Thinker-7B sets a new 7B MLLM SOTA, scoring 43.22% on Video-Holmes (+4.68%), 33.25% on CG-Bench-Reasoning (+3.81%), and 80.69% on VRBench (+11.44%).Each improvement is over the best baseline.Source: §4.2 · p. 8
• GRPO substantially improves out-of-domain generalization over SFT, supporting a two-stage approach that uses SFT for format acquisition and GRPO for performance and generalization.GRPO raises results from 31.52% to 43.22% on Video-Holmes, 24.95% to 33.25% on CG-Bench-Reasoning, and 62.40% to 80.69% on VRBench; SFT alone underperforms most baselines and falls below the base model on several benchmarks.Source: §4.2 · p. 8

4.3 IN-DEPTH ANALYSIS OF GROUNDING AND CAPTIONING CAPABILITIES

• Video-Thinker-7B achieves 48.22% mIoU versus Qwen2.5-VL-7B’s 27.47%, and scores 79.29% Recall@0.3, 51.49% Recall@0.5, and 59.67% average versus 30.23%.Models output question-relevant time segments, evaluated against annotations with mIoU and Recall@K; Video-R1 is excluded because it could not follow the temporal-annotation prompt.Source: Table 3 · p. 9, §4.3 · p. 9
• Video-Thinker-7B leads captioning with 15.34% BLEU@1, 15.87% METEOR, and 20.11% ROUGE-L, averaging 17.11%.Its overall performance is 31.2% relatively higher than Qwen2.5-VL-7B-Instruct and 61.0% higher than Video-R1-7B.Source: Table 3 · p. 9, §4.3 · p. 9
• Providing Video-R1-7B with ground-truth grounding and captioning annotations significantly enhances its Video-Holmes performance.These oracle experiments test whether accurate grounding and captioning information benefits video understanding.Source: §4.3 · p. 9

5 CONCLUSION AND FUTURE WORK

• Video-Thinker-7B establishes state-of-the-art performance among 7B-sized models.The approach extends “Thinking with Images” to video reasoning, enabling MLLMs to autonomously use intrinsic grounding and captioning without external tools.Source: §5 · p. 10
• Video-Thinker-10K and a two-stage SFT–GRPO strategy train MLLMs to generate reasoning clues throughout inference using intrinsic grounding and captioning.This enables video reasoning without relying on external tools.Source: §5 · p. 10
• Future work could scale Video-Thinker to larger models, add intrinsic capabilities beyond grounding and captioning, or incorporate modalities such as audio.Source: §5 · p. 10

ETHICS STATEMENT

• The authors used public datasets under their licenses and usage terms, collected no PII or sensitive private data, and report no direct ethical concerns.The stated concerns include misuse of personal data, harmful content, and bias amplification beyond what is inherent in the benchmark datasets.Source: ETHICS STATEMENT · p. 10
• The authors acknowledge that vision-language models may inherit training-data biases, restrict evaluations to established academic benchmarks, and urge care about downstream social implications.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Benchmarks and metrics are standard and public, while baselines use released checkpoints or publicly accessible open-source implementations.The paper also says it documents hyperparameters, training details, and evaluation protocols to promote reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10

A OVERALL ALGORITHM OF VIDEO-THINKER

• The algorithm takes a collected dataset and a pretrained MLLM, then returns the model trained with Video-Thinker.Source: A OVERALL ALGORITHM OF VIDEO-THINKER · p. 15
• Phase 1 synthesizes missing visual captions and reasoning questions, curates structured reasoning traces, and constructs the Video-Thinker-10K dataset.Source: A OVERALL ALGORITHM OF VIDEO-THINKER · p. 15
• Phase 2 applies SFT to each Video-Thinker-10K sample by minimizing the SFT loss specified in Equation 1.Source: A OVERALL ALGORITHM OF VIDEO-THINKER · p. 15
• In Phase 3, the current policy generates G reasoning traces; the algorithm also refers to formatting in Equation 3 and std({r(j)}) in Equation 4.Source: A OVERALL ALGORITHM OF VIDEO-THINKER · p. 15
• Phase 3 optimizes the GRPO objective with clipped importance sampling and returns the MLLM with tuned parameters.Source: A OVERALL ALGORITHM OF VIDEO-THINKER · p. 15

B DATA DISTRIBUTION OVER SOURCE DATASETS IN SECTION 3.1

• Figure 5 presents the data distribution of the Video-Thinker-10K dataset.The caption does not provide the distribution’s underlying values.Source: Figure 5 · p. 15

C.1 DATASETS AND BENCHMARKS

• ActivityNet (Caba Heilbron et al., 2015), STAR (Wu et al., 2024), and ScaleLong (Ma et al., 2025a) cover spatiotemporal, situated, and multi-scale temporal video reasoning.ActivityNet includes long untrimmed videos and bilingual QA pairs; STAR uses structured questions and situational hyper-graphs; ScaleLong aligns questions with four temporal granularities.Source: §C.1 · pp. 15–16
• YouCook2 (Zhou et al., 2018a), LVBench (Wang et al., 2024), and Video-Holmes (Cheng et al., 2025) evaluate procedural understanding, long-horizon reasoning, and narrative-driven reasoning.YouCook2 provides temporal annotations and imperative descriptions; LVBench adds clue-length annotations; Video-Holmes tests multi-clue integration, causal inference, and social-relation reasoning.Source: §C.1 · p. 16
• CG-Bench (Chen et al., 2024a) and VRBench (Yu et al., 2025b) assess clue-grounded video reasoning and multi-step reasoning across narrative videos.The study uses CG-Bench’s reasoning section; VRBench evaluates both reasoning process and outcome.Source: §C.1 · p. 16

C.2 BASELINE MODELS

• InternVL-2.5-8B (Chen et al., 2024b), InternVL-3-8B (Zhu et al., 2025), and Qwen2.5-VL-7B (Bai et al., 2025) are described as multimodal baselines with video, reasoning, or grounding capabilities.The passages highlight InternVL’s training and perception enhancements and Qwen2.5-VL’s dynamic temporal modeling and structured outputs.Source: §C.2 · p. 16
• Qwen2.5-VL-Omni-7B (Xu et al., 2025), Temporal-R1-7B (Li et al., 2025b), Time-R1-7B (Wang et al., 2025b), Open-R1-Video-7B, and Video-R1 (Feng et al., 2025) cover multimodal interaction and video reasoning.The passages describe Thinker-Talker streaming, temporal localization rewards, future-event prediction, and GRPO-driven video reasoning.Source: §C.2 · p. 16
• TW-GRPO-7B (Dang et al., 2025), GRPO-CARE-7B, and VideoChat-R1-7B (Li et al., 2025c) focus on denser reasoning, coherence-aware rewards, and temporally grounded dialogue.Their described approaches use token-wise weighting and soft rewards, coherence-aware reward design, and structured interactive video reasoning, respectively.Source: §C.2 · p. 17

C.3 EVALUATION METRICS

• mIoU averages each ground-truth segment’s maximum IoU across predictions, capturing temporal localization through both boundary precision and span coverage.The section describes IoU as overlap between temporal segments and mIoU as the average of these per-instance maxima across the test set.Source: §C.3 · p. 17
• Recall@K measures the fraction of ground-truth spans matched at IoU thresholds K = 0.3 or 0.5, representing coarse or fine alignment, respectively.It complements mIoU by reporting retrieval success at different overlap-quality thresholds.Source: §C.3 · p. 17
• BLEU@1 measures unigram precision for caption words but does not capture semantic adequacy or fluency beyond exact token matches.Its score ranges from 0 to 1, with higher scores indicating stronger lexical overlap.Source: §C.3 · p. 17
• METEOR combines unigram precision and recall with synonym, stemming, and paraphrase matches, then applies a penalty for disordered matches.Its 0-to-1 score favors captions that are semantically complete and linguistically coherent.Source: §C.3 · p. 17
• ROUGE-L uses longest-common-subsequence matches to assess precision, recall, and F1-like correspondence, emphasizing sequence structure and token order.Unlike BLEU@1’s exact n-gram focus, it measures global sequence-level correspondence.Source: §C.3 · pp. 17–18

D.1 TRAINING AND EVALUATION

• The training and evaluation system prompt asks the model to timestamp important segments, describe visual clues, reason about their significance, and use natural internal dialogue or verification when useful.It specifies <time>, <caption>, and <answer> tags and allows direct conclusions for straightforward questions.Source: §D.1 · p. 18
• The corresponding question prompt requests careful video analysis in <time>, <caption>, and <think> tags, then a single option letter inside <answer> tags.Source: §D.1 · p. 18

D.2 VIDEO CAPTION GENERATION

• The caption-generation prompt asks for concise, factual descriptions of visual evidence relevant to the question and answer, without reasoning or reference to the video itself.Its inputs are the question and answer, and it emphasizes essential objects, actions, events, and details.Source: §D.2 · p. 18

D.3 QA GENERATION

• The ActivityNet QA template uses background context and chronological event descriptions to generate a multiple-choice question requiring synthesis across multiple segments.It directs the generator toward reasoning questions when events connect, or complex perceptual questions when they do not.Source: §D.3 · pp. 18–19
• ActivityNet questions must rely strictly on the supplied descriptions, require analysis across segments, and provide 4–6 options with one definitively correct answer in a JSON object.The prompt also calls for sophisticated distractors and prohibits questions answerable from one event alone.Source: §D.3 · p. 19
• The YouCook2 variant draws on chronological cooking-step descriptions for questions about cooking logic or detailed perception, while TutorialVQA adds the video title and full transcript to its step descriptions.Both templates require synthesis across multiple visual segments; the TutorialVQA prompt also supports reasoning about causes, intentions, motivations, or sequence.Source: §D.3 · pp. 19–20

E EXPERIMENTAL VERIFICATION OF GROUNDING AND CAPTIONING CAPABILITIES

• The Video-Holmes evaluation used Video-R1-7B across base inference, grounding-only, captioning-only, and combined-input configurations, measuring accuracy.Grounding supplied question-relevant temporal segments, while captioning supplied descriptions of the full video.Source: E EXPERIMENTAL VERIFICATION OF… · pp. 20–21
• 63% accuracy with grounding and captioning was best, versus 56% with captioning and 53% with grounding, each from a 37% baseline.The authors attribute the combined improvement to complementary benefits: temporal focus from grounding and comprehensive context from captioning.Source: E EXPERIMENTAL VERIFICATION OF… · p. 21

F ABLATION STUDIES

• At 2500 GRPO steps, Video-Thinker reached its best average score of 58.35%, while further training degraded performance on several benchmarks, especially out-of-domain tasks.The authors interpret the peak as balancing learning with avoiding overfitting to the training distribution.Source: F ABLATION STUDIES · p. 21
• At a 5e-6 learning rate, Video-Thinker achieved optimal performance and outperformed Qwen2.5-VL-7B-Instruct and Video-R1-7B; a 1e-5 rate caused dramatic degradation.The authors attribute the high-rate drop to training instability and poor convergence.Source: F ABLATION STUDIES · pp. 21–22

G CASES

• Supplementary examples across Video-Holmes, VRBench, and CG-Bench demonstrate Video-Thinker-7B’s iterative reasoning and error-correction behavior.The authors describe this self-correction as a dynamic internal feedback mechanism rather than simple pattern matching.Source: Figure 6 · p. 22, G CASES · p. 23, Figure 7 · p. 23

H USE OF LLMS

• The authors used publicly available LLMs only for stylistic and linguistic improvements to English writing, comparable to grammar or spell checking.They state that LLMs did not generate research insights, data, or analyses.Source: H USE OF LLMS · p. 24
• The authors state that ideas, methods, experiments, and result interpretation were conceived and written without LLM involvement, and that scientific claims remain their responsibility.LLM assistance was confined to language improvements.Source: H USE OF LLMS · p. 24
• Figure 14 illustrates Video-R1-7B failing to follow instructions to generate temporal grounding content within <time></time> tags, explaining its exclusion from that evaluation.The figure links the failure to the paper’s stated rationale in Section 4.3.Source: Figure 14 · p. 31