PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoAuto-R1: Video Auto Reasoning via Thinking Once, Answering Twice

Shuming Liu, Mingchen Zhuge, Changsheng Zhao, Jun Chen, Lemeng Wu, Zechun Liu, Chenchen Zhu, Zhipeng Cai, Chong Zhou, Haozhe Liu, Ernie Chang, Saksham Suri, Hongyu Xu, Qi Qian, Wei Wen, Balakrishnan Varadarajan, Zhuang Liu, Hu Xu, Florian Bordes, Raghuraman Krishnamoorthi, Bernard Ghanem, Vikas Chandra, Yunyang Xiong

arXiv:2601.05175v2cs.CV

TL;DR

VideoAuto-R1 addresses whether explicit chain-of-thought reasoning is always necessary for video understanding, where direct answering can match or exceed CoT. It trains initial and reviewed answers together, then uses initial-answer confidence to decide whether to continue reasoning, achieving state-of-the-art accuracy while reducing average response length from 149 to 44 tokens.

  • Problem

    Video understanding lacks clear evidence about when explicit reasoning is necessary, because direct answering can match or outperform CoT while visual tasks often require limited symbolic reasoning.

  • Method

    VideoAuto-R1 trains an initial answer and a reviewed answer with verifiable rewards, then uses the initial answer's confidence to trigger early exit or continued CoT reasoning.

  • Results

    VideoAuto-R1 achieves state-of-the-art accuracy across video QA and temporal grounding while reducing average response length from 149 to 44 tokens.

  • Takeaways & Limitations

    Explicit reasoning is activated selectively: less often for perception-oriented tasks and more often for reasoning-intensive tasks.

  • Takeaways & Limitations

    The confidence-based early-exit rule is applied only at test time and is not explicitly shaped during training.

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning has emerged as a powerful tool for multimodal large language models on video understanding tasks. However, its necessity and advantages over direct answering remain underexplored. In this paper, we first demonstrate that for RL-trained video models, direct answering often matches or even surpasses CoT performance, despite CoT producing step-by-step analyses at a higher computational cost. Motivated by this, we propose VideoAuto-R1, a video understanding framework that adopts a reason-when-necessary strategy. During training, our approach follows a Thinking Once, Answering Twice paradigm: the model first generates an initial answer, then performs reasoning, and finally outputs a reviewed answer. Both answers are supervised via verifiable rewards. During inference, the model uses the confidence score of the initial answer to determine whether to proceed with reasoning. Across video QA and grounding benchmarks, VideoAuto-R1 achieves state-of-the-art accuracy with significantly improved efficiency, reducing the average response length by ~3.3x, e.g., from 149 to just 44 tokens. Moreover, we observe a low rate of thinking-mode activation on perception-oriented tasks, but a higher rate on reasoning-intensive tasks. This suggests that explicit language-based reasoning is generally beneficial but not always necessary.

1 Introduction

VideoAuto-R1 addresses whether video understanding always needs explicit reasoning by combining direct and reviewed answers with confidence-based early exit. It targets strong accuracy while reducing unnecessary reasoning and response length.

  • Motivation: Direct answering often matches or outperforms CoT for RL-trained video reasoning models, while CoT consistently helps only on explicitly multi-step benchmarks.Long reasoning traces can introduce overthinking and degrade performance.
  • Method: VideoAuto-R1 trains with an answer →think →answer template, supervising both the initial and reviewed answers with verifiable rewards.The final answer receives greater weight to encourage refinement or confirmation.
  • Method: At inference, a length-normalized mean log probability of the initial answer determines whether decoding exits early or continues into reasoning and review.The confidence threshold avoids requiring an additional mode-switch token or head.
  • Results: 25% activation on MVBench and 51% on VideoMMMU show that thinking is used less on perception-oriented tasks and more on reasoning-intensive tasks.These rates reflect adaptive mode selection across benchmark types.
  • Results: VideoAuto-R1 achieves state-of-the-art accuracy across video QA and temporal grounding while improving efficiency.The framework reduces average response length from 149 to 44 tokens while preserving accuracy.
  • Method: The framework eliminates the need for per-sample think/no-think labels by training both answers to be correct.This produces an adaptive reasoning model without explicit mode supervision.

2 Related Work

Prior video reasoning methods commonly enforce always-thinking behavior, despite evidence that CoT is not universally beneficial for perceptual tasks. VideoAuto-R1 instead combines direct and CoT answers during training with confidence-based switching at inference.

  • CoT limitations: CoT improves robustness on compositional and symbol-intensive tasks but can cause overthinking on primarily perceptual or intuitive tasks.This motivates applying explicit reasoning selectively in video understanding.
  • Video reasoning models: Prior video methods extend reinforcement learning and explicit reasoning to video QA, temporal grounding, relational reasoning, and long-video narrative tasks.These approaches broaden video reasoning beyond standard question answering.
  • Video reasoning models: Thinking with frames uses progressive perception by selecting salient frames or segments before revisiting them at higher resolution or frame rate.The approach interleaves video and text reasoning to improve answer formation.
  • Adaptive reasoning: Many prior video methods enforce always-thinking, whereas direct answering often matches CoT on perception-oriented QA tasks.This contrast motivates reserving CoT for cases where it provides tangible gains.
  • Adaptive reasoning: Auto-thinking methods generally learn a switching policy with supervised fine-tuning or reinforcement learning to decide when to invoke CoT.AdaptThink additionally emphasizes balanced sampling between thinking and no-thinking examples.
  • VideoAuto-R1: VideoAuto-R1 trains both direct and CoT answers instead of binary mode labels, then uses the first answer's mean log probability to decide whether to continue.This removes the need for think/no-think labels, switch tokens, or cold-start SFT.

3 Preliminaries

The paper examines when video chain-of-thought is necessary and finds that direct inference often matches or outperforms CoT while using fewer tokens. This motivates an auto-thinking approach that selectively invokes reasoning for difficult video inputs.

  • The framework builds on GRPO, which uses group-normalized, rule-based verifiable rewards for scalable reinforcement-learning training.Rewards can include answer accuracy, temporal IoU, or format correctness.
  • The paper evaluates QA, temporal grounding, and grounding QA, and conducts reinforcement learning directly without a cold-start SFT stage.
  • The central question is when video chain-of-thought is necessary and how it compares with direct answering.
  • Direct inference often matches or outperforms CoT on several benchmarks while generating significantly fewer tokens.Consistent CoT gains are primarily observed on Video-MMMU.
  • CoT provides a tangible advantage for math- or physics-oriented videos requiring multi-step deduction.These cases typically contain symbolic inputs and relatively clean visual signals.
  • CoT is often redundant for perception-oriented queries, increasing latency and inference cost without changing the conclusion.

4 VideoAuto-R1

VideoAuto-R1 trains models to produce an initial answer, reasoning trace, and reviewed answer, then uses confidence-based early exit to decide whether reasoning is necessary. Its reward design emphasizes the reviewed answer while preserving fast direct answers and discouraging unreliable guesses.

  • 4.1 Thinking Once, Answering Twice: VideoAuto-R1 uses an answer →think →answer template to jointly train initial and reviewed answers.The format contains two boxed answers and one free-form rationale.
  • 4.1 Thinking Once, Answering Twice: A fallback string lets the model defer difficult initial answers and continue to reasoning without producing a spurious guess.The fallback preserves the output grammar and makes early exit unambiguous.
  • 4.1 Thinking Once, Answering Twice: Training decouples when to think from how to think, avoiding per-sample mode labels, switch tokens, and specialized heads.The distinction between direct and thinking modes is made at test time through confidence-based early exit.
  • 4.2 Training: Dual-Answer Reward with GRPO: The dual-answer reward supervises both answers while assigning a larger weight to the reviewed answer.The reward also penalizes cases where the first answer is correct but the second is incorrect.
  • 4.2 Training: Dual-Answer Reward with GRPO: The fallback bonus rewards a designated deferral followed by a correct final answer, especially for mathematically or symbolically complex problems.
  • 4.2 Training: Dual-Answer Reward with GRPO: During training, the reviewed-answer task reward typically exceeds the initial-answer task reward.This is presented as evidence that explicit reasoning benefits more challenging instances while retaining fast first answers when appropriate.
  • 4.3 Inference: Confidence-Based Early Exit: Inference computes a length-normalized confidence score for the first answer and exits when it exceeds a threshold.Otherwise, the model generates the rationale and second answer; the threshold controls the accuracy–efficiency trade-off.

5 Experiments

Across video QA and temporal grounding, VideoAuto-R1 combines strong accuracy with adaptive reasoning and shorter responses. It routes more difficult reasoning inputs to CoT while often early-exiting on perception-oriented or grounding tasks.

  • VideoAuto-R1 achieves state-of-the-art video QA results, including 58.6% on VideoMMMU and 39.4% pairwise accuracy on MVP.
  • 44 tokens is VideoAuto-R1’s average response length, compared with 386 tokens for Video-R1, while think-mode activation rises from 25% on MVBench to 51% on Video-MMMU.
  • On temporal grounding, the initial boxed prediction is sufficient for accurate localization, so subsequent CoT mainly adds explanation without improving localization.
  • VideoAuto-R1 improves temporal grounding mIoU from 52.9% to 60.0% on Charades-STA and from 26.9% to 47.6% on ActivityNet.
  • Compared with RL with thinking, VideoAuto-R1 reaches +3.9% on VideoMMMU and +1.3% on VideoMME while reducing average response length from 149 to 44 tokens.
  • Increasing the early-exit threshold from 0.86 to 0.98 raises VideoMMMU accuracy from 57.5% to 58.7% and think ratio from 29% to 55%.
  • On VideoMME, accuracy remains essentially unchanged across thresholds while think ratio increases, motivating τ = 0.97 as a default.
  • A qualitative example shows reasoning revising an incorrect initial prediction D to the correct reviewed answer C through probability and integration.

6 Conclusion

VideoAuto-R1 is an adaptive video reasoning framework designed to reason only when necessary. Its results across perception, reasoning, and temporal grounding benchmarks support the thinking-once, answering-twice approach, with potential applications beyond video understanding.

  • VideoAuto-R1 is an adaptive framework that reasons only when necessary.
  • The thinking-once, answering-twice scheme addresses the observation that long CoT does not reliably improve video understanding and can degrade accuracy through overthinking.
  • Experiments across perception, reasoning, and temporal grounding benchmarks consistently validate the model’s advantages.
  • The paper presents VideoAuto-R1 as an alternative to the standard reasoning framework and notes potential applications beyond video understanding.

A Training Data

The training data combines text, image, and video samples, with filtering designed to remove invalid, overly easy, and overly hard examples. Combining modalities produces the best overall performance, while filtering improves results with fewer samples.

  • Text reasoning uses DAPO-Math, image reasoning uses ViRL and ThinkLite-Hard, and video QA draws from Video-R1, TVBench, STI-Bench, and MMR-VBench.
  • The training dataset contains text, image, and video data totaling 83K samples.
  • The filtering pipeline removes invalid ground truth and evaluates 8 generated responses per remaining sample.
  • Samples with uniformly correct or uniformly incorrect responses are excluded as too easy or too hard for QA-based data.
  • Combining text, image, and video data yields the best overall performance under both filtered and unfiltered settings.
  • Filtering overly easy and hard samples consistently improves results while reducing dataset size.

B Reward Designs

VideoAuto-R1 trains an answer–think–answer pipeline with verifiable task and format rewards, emphasizing the reviewed answer while supporting multiple video task types.

  • Reward components: The reward combines task correctness with a binary format check requiring two boxed answers separated by one think block.Task rewards cover question answering, temporal grounding, and grounding QA.
  • Dual-answer weighting: The dual-answer design assigns separate coefficients to initial and reviewed answers, with the reviewed answer weighted more heavily.The intended ratio is w1 < w2, prioritizing final-answer reliability.
  • Fallback handling: A fallback string signals that the model recognizes difficulty and defers reasoning rather than making an incorrect guess.The fallback bonus distinguishes this signal from an ordinary wrong prediction.
  • Training formats: Training prompts compare reinforcement learning without thinking against standard reasoning prompts and the answer–think–answer format.The training curves track task rewards for both initial and reviewed answers.

D Training Curve

Training improves both initial and reviewed answers, while inference uses confidence-based early exit to retain direct answers when reliable and continue reasoning otherwise.

  • Training curves: The reviewed answer consistently achieves higher task reward than the initial answer for both evaluated backbones.The stable post-convergence gap indicates that intermediate reasoning supports revision rather than simple resampling.
  • Training curves: Both answer rewards rise rapidly early in training before increasing more slowly toward convergence.The authors associate this pattern with coarse structure being learned before finer reasoning capabilities.
  • Backbone capacity: Qwen3-VL-8B consistently outperforms Qwen2.5-VL-7B for both answers throughout training.The larger backbone maintains a higher reward margin after convergence.
  • Inference strategy: Inference generates an initial answer, computes its confidence, and either exits early or continues with reasoning and a reviewed answer.The confidence score is based on the length-normalized mean log probability of initial-answer tokens.
  • Inference strategy: Early termination detects the opening think tag during greedy decoding, reducing decoding overhead when the initial answer is accepted.The initial answer typically contains only a few tokens, enabling low-overhead confidence computation.
  • Frame analysis: Increasing frames from 64 to 256 improves VideoMME accuracy from 63.1% to 66.0% and LongVideoBench accuracy from 59.7% to 60.9%.VideoMMMU shows weaker dependence on frame count and slightly decreases with additional frames.
  • Frame analysis: With 64 frames, VideoAuto-R1 improves VideoMME from 63.1% to 64.6% and MMVU from 66.2% to 69.7% over the Qwen baseline.The gains are reported across both low- and high-frame regimes.

F.2 Analysis on Temporal Grounding Benchmarks

Temporal grounding gains little from additional language reasoning: initial answers are usually sufficient, so VideoAuto-R1 generally exits early on grounding tasks.

  • Inference strategies: Grounding results compare first-boxed, second-boxed, and confidence-based auto strategies, with early exit used to save computation.The first boxed answer is typically sufficient on grounding benchmarks.
  • Answer comparison: Temporal grounding shows almost no gap between initial and reviewed answers, including identical mIoU on ActivityNet and NExT-GQA.NExT-GQA grounding QA accuracy also remains unchanged.
  • Reasoning analysis: Grounding traces are shorter than QA traces because localization can map an event directly to a time span without multi-step deduction.The authors also note that the missing SFT stage limits explicit refinement of predicted segments.
  • Conclusion: Explicit reasoning provides limited additional benefit over direct localization on temporal grounding benchmarks.Accordingly, the reported grounding results use direct answering for VideoAuto-R1.
  • Training choice: The training framework omits chain-of-thought SFT and proceeds directly to reinforcement learning.The paper motivates this choice by the cost and noise of collecting high-quality reasoning traces.
  • Training choice: Pure SFT on Video-R1-CoT data substantially hurts performance across the three evaluated benchmarks.The authors therefore directly incentivize reasoning through GRPO-style reinforcement learning.

G Limitations

The paper identifies limitations in confidence training, language-only reasoning, benchmark scope, and the scarcity of video data requiring indispensable multi-step reasoning.

  • Confidence calibration: The early-exit distinction is made only at test time and does not explicitly shape the model’s confidence distribution during training.The authors propose jointly optimizing accuracy and calibrated confidence.
  • Multimodal reasoning: Language-only chain-of-thought yields limited gains on perception-oriented QA and temporal grounding compared with direct answering.Future work could revisit video segments or visual features during reasoning.
  • Evaluation scope: Existing video reasoning benchmarks remain limited in scope and difficulty, often using short clips and perception-oriented questions.The paper calls for benchmarks stressing long-range temporal dependencies, compositional logic, and counterfactual reasoning.
  • Data limitations: Truly must-think video data requiring indispensable multi-step reasoning remains scarce.Suggested examples include multi-event causal chains, temporal puzzles, and physically challenging scenarios.
  • Future directions: Constructing large-scale, high-quality deep-reasoning video datasets is identified as an important future direction.The paper also highlights advanced reasoning patterns for grounding as an open direction.

H Qualitative Examples

The qualitative examples contrast cases where direct answering is more reliable with cases where chain-of-thought reasoning helps, and illustrate confidence-based early exit across task types.

  • In a perception-oriented failure case, direct answering is correct while CoT reasoning hallucinates visual details and produces an incorrect answer.The errors can arise from a single misperception or flawed reasoning step.
  • For a science problem involving equations, chain-of-thought performs step-by-step deductions and reaches the correct numerical result, unlike direct answering alone.The example is drawn from VideoMMMU.
  • On temporal grounding tasks, reasoning is typically straightforward and initial and reviewed answers are often identical, supporting early exit without further reasoning.The traces often only identify when an action begins and ends.
  • Over 99% confidence in initial answers on relatively simple perception-oriented questions triggers early exit while maintaining strong accuracy and improving inference efficiency.The qualitative examples describe consistently accurate initial responses on these benchmarks.
  • Lower initial confidence on reasoning-intensive questions triggers reasoning, whose traces are longer and contain more detailed deduction steps.The examples contrast these traces with those from perception-oriented tasks.
Loading 2601.05175v2…