PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Chain-of-Thought Reasoning for Videos

Yiwu Zhong, Zi-Yuan Hu, Yin Li, Liwei Wang

arXiv:2512.09616v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Video CoT models face overhead from long reasoning traces, redundant visual tokens, and annotation-heavy training. This paper tests concise reasoning with compressed visual tokens through RL-based post-training. Its framework improves inference efficiency while maintaining competitive accuracy across benchmarks, and its conclusions remain bounded by current video benchmarks.

  • Problem

    Video CoT can require many redundant visual tokens and long reasoning sequences, while CoT training uses costly annotations and prolonged SFT-plus-RL cycles.

    Source: §1 · p. 1

  • Method

    The framework uses GRPO post-training with visual-token compression and generates brief reasoning traces from compressed tokens at inference.

    Source: §1 · p. 2

  • Results

    Across benchmarks, the framework improves inference efficiency while maintaining competitive accuracy; the final model achieves the best results over Video-R1 on all benchmarks, including +5.7 on VideoMME.

    Source: §5.3 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The findings suggest that effective video reasoning can use concise traces over compressed visual tokens rather than relying on long, human-like CoT and large visual-token sets.

    Source: §1 · p. 2, §6 · p. 8

    The conclusions are based on empirical evidence rather than theoretical guarantees and are constrained by current video benchmarks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning has been highly successful in solving complex tasks in natural language processing, and recent multimodal large language models (MLLMs) have extended this paradigm to video reasoning. However, these models typically build on lengthy reasoning chains and large numbers of input visual tokens. Motivated by empirical observations from our benchmark study, we hypothesize that concise reasoning combined with a reduced set of visual tokens can be sufficient for effective video reasoning. To evaluate this hypothesis, we design and validate an efficient post-training and inference framework that enhances a video MLLM's reasoning capability. Our framework enables models to operate on compressed visual tokens and generate brief reasoning traces prior to answering. The resulting models achieve substantially improved inference efficiency, deliver competitive performance across diverse benchmarks, and avoid reliance on manual CoT annotations or supervised fine-tuning. Collectively, our results suggest that long, human-like CoT reasoning may not be necessary for general video reasoning, and that concise reasoning can be both effective and efficient. Our code will be released at https://github.com/LaVi-Lab/Rethink_CoT_Video.

1. Introduction

The introduction argues that video MLLM reasoning incurs substantial training and inference overhead, while CoT’s benchmark gains are limited and concise reasoning alone can hurt performance. It proposes RL post-training with token compression to support brief reasoning on fewer visual tokens, while bounding its conclusions to current video benchmarks.

1. Introduction

  • Video CoT adds inference costs from thousands of redundant visual tokens and long reasoning sequences, while annotation-heavy SFT and RL over long traces prolong training.Figure 1 contrasts dense prefilling and lengthy decoding with sparse prefilling and concise decoding.Source: §1 · p. 1, Figure 1 · p. 1
  • The paper frames overhead as input-token prefilling and output-token decoding costs, then benchmarks MLLMs with and without CoT across general, long-form, and complex video tasks.Token compression is proposed to reduce the redundant visual-token input.Source: §1 · pp. 1–2
  • Adding CoT yields only modest gains over direct answering, while concise prompting and token compression cause larger performance drops for reasoning modes than for direct answering.The authors observe that CoT traces often include “pondering” patterns that contribute little to reasoning, and conjecture that pre-trained models are poorly aligned with concise reasoning despite having the needed knowledge.Source: §1 · p. 2
  • The framework applies GRPO directly to a pre-trained video MLLM, combines it with visual-token compression, and uses compressed tokens with brief reasoning traces at inference.The authors report reduced inference overhead, improved accuracy across benchmarks, and no need for costly CoT annotations or SFT.Source: §1 · p. 2
  • The resulting approach challenges the assumption that video reasoning requires long, human-like CoT, offering improved inference efficiency, competitive performance, and RL-only post-training.The authors describe the work as an alternative approach rather than a new method.Source: §1 · p. 2
  • The work is grounded in empirical evidence rather than theoretical guarantees, and its conclusions are constrained by the scope of current video benchmarks.The authors suggest possible generalization beyond these datasets but do not establish it.Source: §1 · p. 2

2. Related Work

Related work covers multimodal models and reasoning, including CoT methods that commonly use annotation-based SFT followed by RL. The paper contrasts that pipeline with direct GRPO training and incorporates token compression to reduce computation.

2. Related Work

  • Multimodal research spans image [6] [89], video [28] [88], and long-video understanding [69], alongside unified image-video models [1] [25] [58] [90].Multimodal CoT methods generate intermediate reasoning steps for visual understanding [81].Source: §2 · p. 2
  • Video reasoning methods commonly use two-stage post-training: generate CoT annotations for SFT, then apply GRPO-based RL [12] [18] [20] [71].Examples include Video-R1’s temporal-aware reward [12] and Video-RFT’s semantic-consistency reward.Source: §2 · p. 2
  • The paper challenges annotation-based SFT followed by GRPO, reporting that direct GRPO training with concise reasoning can elicit strong multimodal reasoning without CoT annotation or CoT SFT.The authors position this as an alternative to the prevailing CoT post-training paradigm.Source: §2 · p. 2
  • Token-compression methods reduce processed tokens and computation; the paper incorporates them to improve training and inference efficiency, especially for long videos.Some methods operate without retraining, including AIM, which merges similar visual tokens and prunes uninformative ones.Source: §2 · p. 3

3. Benchmarks and Analysis

The analysis finds that conventional CoT incurs substantial training and inference overhead while yielding only modest benchmark gains, and that pretrained concise reasoning and training-free token compression have important performance trade-offs.

3.2. Analysis

  • 11.9 vs. 0.47 and 1.71 inference runtime for CoT versus direct-answer and concise-reason modes, while CoT post-training takes over 30 hours on four GPUs.The runtime comparison is attributed to long-sequence decoding; the training figure measures runtime on four A800-SXM4-80GB GPUs.Source: Figure 3 · p. 3, §3.2 · p. 4

3.1. Background

  • Video inference prefills visual and text tokens before autoregressively decoding; long CoT traces increase decoding cost, while direct answers minimize it but lack explainability.Concise reasoning reduces generated tokens for faster inference while preserving response accuracy, as described in the analysis background.Source: §3.1 · p. 3

3.2. Analysis

  • CoT improves the pretrained model on some benchmarks but not others; the pretrained model performs favorably and sometimes outperforms it, including on VideoMME and EgoSchema.The table caption notes that CoT’s improvements require CoT annotations, supervised fine-tuning, and lengthy decoding.Source: Table 1 · p. 4, §3.2 · p. 4
  • Training-free token compression reduces prefilling workload, but concise reasoning experiences larger accuracy drops than direct-answer or CoT modes.The authors conjecture that concise reasoning is more vulnerable to this unseen input intervention because the model is not optimized for that decoding mode.Source: Table 2 · p. 4, §3.2 · p. 5
  • The analysis compares a pretrained model such as Qwen2.5-VL with a CoT model such as Video-R1 across general, long-video, and complex-video benchmarks.The benchmark suite includes VideoMME and MVBench; MLVU, LVBench, LongVideoBench, and Egoschema; and VideoHolmes, Video-TT, and MMVU.Source: §3.2 · p. 4
  • Pretrained concise reasoning reduces decoding computation, but its performance is clearly below direct answering.The authors conjecture that post-training could better align the model with concise reasoning.Source: §3.2 · p. 4
  • The analysis motivates a framework that removes supervised fine-tuning and its annotations, generates concise reasoning, and uses trainable token compression to reduce computation overhead.The section’s analysis is presented as motivation for the final model design.Source: §3.2 · p. 4, Figure 4 · p. 5

4. Methods

The framework pairs concise reasoning with compressed visual tokens, using direct RL post-training rather than CoT-supervised fine-tuning. It compresses inputs for training and inference and generates concise reasoning at decoding.

4. Methods

  • The method’s central hypothesis is that RL-optimized concise reasoning can work effectively with compressed visual tokens for video reasoning.Its three design choices are direct RL post-training, compression during prefilling, and concise reasoning at decoding.Source: §4 · p. 5

4.1. Post-Training with GRPO

  • Directly fine-tune the pre-trained model with GRPO, bypassing the CoT-annotation and SFT stages used in prior two-stage pipelines.GRPO compares groups of sampled responses and uses normalized rewards to favor higher-scoring responses; a KL term regularizes deviation from the reference policy.Source: §4.1 · pp. 5–6

4.2. Prefill with Token Compression

  • Token merging and pruning reduce visual-token counts, and physically discarding unimportant tokens cuts memory use and runtime.Unlike attention-mask-only compression, physical pruning removes tokens; the reduced computation also permits more video frames at comparable cost.Source: §4.2 · p. 6

4.3. Decode with Concise Reasoning Mode

  • Concise reasoning provides an explanation for the final answer while maintaining reasonable decoding overhead.The framework uses this decoding mode during training and inference.Source: §4.3 · p. 6

5. Experiments and Results

The final model achieves the best reported results across the video benchmarks while using token compression and concise reasoning. Experiments also find that GRPO supports concise reasoning and compression, and visualizations illustrate how denser frame coverage can retain details missed by sparse sampling.

5.1. Avoid Overthinking

  • GRPO improves the pre-trained model over direct-answer and concise-reason modes, surpasses the heavily post-trained CoT model on VideoMME and VideoHolmes, and needs neither CoT annotations nor SFT.The reported comparisons include 55.4 vs. 54.9 on VideoMME and 40.0 vs. 39.4 on VideoHolmes.Source: §5.1 · p. 6
  • After GRPO, concise reasoning can outperform direct answering on MVBench, MLVU, and VideoHolmes, mitigating the pre-trained model’s mode gap.Reported concise-reason vs. direct-answer scores include 65.2 vs. 64.0 on MVBench, 58.4 vs. 57.7 on MLVU, and 40.0 vs. 38.6 on VideoHolmes.Source: §5.1 · pp. 6–7

5.3. Results of Final Model

  • +5.7 on VideoMME, +8.1 on MLVU, and +2.2 on VideoHolmes over Video-R1 [12]; the final model achieves the best results across benchmarks.The model uses token compression during training and inference and concise reasoning trained with GRPO.Source: Table 5 · p. 7, §5.3 · p. 7

5.2. Look Less, See Further

  • GRPO reduces the accuracy drop from inference-time token compression, with the reported performance drop shrinking from –4.3 to –2.2.The paper reports that this compatibility allows more input frames and improved accuracy.Source: §5.2 · p. 7

5. Experiments and Results

  • The experiments evaluate the final model against pre-trained direct-answer and concise-reason modes and the CoT model, using a matched prefilling computation budget.The final model uses 96 input frames because token compression reduces visual tokens per frame.Source: §5.3 · p. 7

5.4. Ablation Study

  • Training with token compression improves over applying compression only at inference, while expanded frame coverage yields gains of +5.9 on MLVU and +4.2 on LVBench.The ablation covers token compression and the number of video frames; compression permits more frames within a comparable prefilling budget.Source: §5.4 · pp. 7–8, Table 6 · p. 7

5.5. Visualization

  • In the visualized examples, verbose CoT can overcomplicate reasoning, while a key frame missed by sparse sampling reveals the correct answer under denser coverage.The comparison is between Video-R1 CoT reasoning and the final model’s concise reasoning.Source: §5.5 · p. 8, Figure 5 · p. 8

6. Conclusion

The paper concludes that concise reasoning on compressed video tokens can support effective video reasoning, with improved inference efficiency and competitive accuracy across diverse tasks, without CoT annotations or heavy SFT.

6. Conclusion

  • The framework validates concise reasoning on compressed video tokens as an effective alternative, improving inference efficiency while maintaining competitive accuracy without CoT annotations or heavy SFT.The conclusion challenges the assumption that long, human-like reasoning traces and large sets of video tokens are necessary for strong performance.Source: §6 · p. 8
Loading 2512.09616v1…