Source-linked AI summary
TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning
Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang
TL;DR
Existing video reasoning research offers limited evidence for small-scale models and is constrained by scarce reasoning-intensive video data. This paper introduces TinyLLaVA-Video-R1, applying reinforcement learning to the traceably trained TinyLLaVA-Video on general Video-QA data. The model improves reasoning and thinking abilities and exhibits emergent “aha moments,” while the authors identify directions for stronger data and reinforcement-learning algorithms.
Problem
Video reasoning remains underdeveloped, while large-scale foundations and reasoning-intensive data create barriers for researchers with limited computational resources.
Method
TinyLLaVA-Video-R1 applies reinforcement learning to the traceably trained TinyLLaVA-Video using general Video-QA data.
Results
TinyLLaVA-Video-R1 significantly improves reasoning and thinking abilities and exhibits emergent “aha moments” involving retrospection and reflection.
Takeaways & Limitations
Small-scale models can demonstrate interpretable video reasoning and self-checking after reinforcement learning on weakly reasoned general video data.
Takeaways & Limitations
The current model is trained only on general video question-answering data, and its GRPO training algorithm has notable limitations.
Abstract
from arXiv · showhide
Recently, improving the reasoning ability of large multimodal models (LMMs) through reinforcement learning has made great progress. However, most existing works are based on highly reasoning-intensive datasets such as mathematics and code, and researchers generally choose large-scale models as the foundation. We argue that exploring small-scale models' reasoning capabilities remains valuable for researchers with limited computational resources. Moreover, enabling models to explain their reasoning processes on general question-answering datasets is equally meaningful. Therefore, we present the small-scale video reasoning model TinyLLaVA-Video-R1. Based on TinyLLaVA-Video, a traceably trained video understanding model with no more than 4B parameters, it not only demonstrates significantly improved reasoning and thinking capabilities after using reinforcement learning on general Video-QA datasets, but also exhibits the emergent characteristic of "aha moments". Furthermore, we share a series of experimental findings, aiming to provide practical insights for future exploration of video reasoning (thinking) abilities in small-scale models. It is available at https://github.com/ZhangXJ199/TinyLLaVA-Video-R1.
1 Introduction
Video reasoning research has progressed unevenly, with limited advances for video models and unresolved questions about whether small models can reason effectively. TinyLLaVA-Video-R1 addresses this gap by applying reinforcement learning to a traceably trained small-scale video model.
- Video reasoning models have not made significant progress because highly reasoning-intensive video data remain scarce.
- Open-R1-Video performs unsatisfactorily on general Video-QA benchmarks, while subsequent work argues that small-scale models struggle to produce effective reasoning processes.
- The model is presented as a small-scale alternative for studying video reasoning under limited computational resources.
- TinyLLaVA-Video-R1 applies reinforcement learning to TinyLLaVA-Video and improves reasoning and thinking abilities while exhibiting emergent “aha moments”.
2 Related Work
Prior multimodal reasoning work has mainly emphasized language, image, or large-scale models, while small-scale video reasoning remains comparatively underdeveloped. The related challenges include limited data, short responses, incoherent reasoning chains, and constrained performance gains.
- Reinforcement-learning methods such as GRPO use rule-based rewards to reduce computational costs and have been associated with emergent “aha moments”.
- Multimodal reasoning studies have expanded into image and video domains, but many use base models with 7B or more parameters.
- Small-scale multimodal research has focused primarily on images, leaving video reasoning significantly underdeveloped.
- Small-scale video reasoning models face scarce high-quality datasets, limited response length, incoherent reasoning chains, and constrained performance improvement.
3 Methods
The method applies GRPO to a traceably trained small-scale video model, using general Video-QA data and tailored rewards to encourage formatted, accurate reasoning. It also evaluates reasoning behavior through benchmark comparisons and observed reflection and backtracking.
- Group Relative Policy Optimization: GRPO generates multiple responses per question, normalizes their rewards into advantages, and optimizes the policy with a clipped objective.The current and old policies, clipping parameter, and KL coefficient define the optimization objective.
- Training Data and Template: Training uses 5,496 multiple-choice NextQA samples covering videos lasting 0 to 30 seconds.The dataset is perception-oriented, but training aims to elicit explicit reasoning rather than answers alone.
- Training Data and Template: Responses must place reasoning in <think></think> tags and the final option in <answer></answer> tags, with rewards enforcing this format.The four tags may appear only once in a response.
- Reward Rules: The format reward combines a base reward with a continuous reasoning-length reward, capped at 1 when r0 = r1 = 0.5.The length reward is computed from the reasoning length relative to a maximum length.
- Reward Rules: The total reward penalizes incorrect answers so longer reasoning is rewarded only when correct and penalized when incorrect.Correct responses receive accuracy plus format reward, while incorrect formatted responses receive a penalty based on the format and accuracy rewards.
4 Experiments
Experiments evaluate TinyLLaVA-Video-R1 across video benchmarks and training behaviors, showing stronger reasoning, stable growth in response length and rewards, and emergent self-verification. Ablations find that cold-start data and refined rewards help stabilize and improve small-model reasoning, while some objective changes reduce reasoning content.
- Evaluation: TinyLLaVA-Video-R1 is evaluated on MVBench, VideoMME, MLVU, and MMVU, covering varied video domains and durations.The evaluation uses four commonly used video understanding and reasoning benchmarks.
- Main Results: Compared with TinyLLaVA-Video-SFT, TinyLLaVA-Video-R1 performs better across multiple benchmarks and improves reasoning over TinyLLaVA-Video-ColdStart.The model also follows the required response format relative to the base model.
- Main Results and Aha Moment: During training, response length and rewards increase stably, while the model analyzes video content, evaluates options step-by-step, and produces interpretable reasoning.The model’s reasoning is presented as an advantage over systems that output answers without reasoning.
- Main Results and Aha Moment: TinyLLaVA-Video-R1 exhibits an “aha moment” through self-verification, retrospection, and reflection even with weakly reasoned general video data.The model revisits and verifies its initial reasoning after completing a round of thought.
- Impact of Cold-Start Data: Without length reward, the small model can learn shortcuts that omit reasoning, whereas 16 human-annotated cold-start CoT samples prevent this behavior and accelerate format compliance.The authors therefore regard cold starting as necessary for stabilizing reasoning training in small-scale models.
- Reward and Objective Ablations: Continuous length reward increases output length but can induce meaningless reasoning; adding an answer-correctness penalty improves responses while retaining growth in length and rewards.Removing KL divergence and additional objective terms can improve performance but reduce response length and cause the model to omit answer analysis.
5 Conclusion and Future Work
TinyLLaVA-Video-R1 improves reasoning and thinking capabilities while exhibiting “aha moments.” Future work targets higher-quality reasoning data and improved reinforcement-learning algorithms.
- TinyLLaVA-Video-R1 enhances reasoning and thinking capabilities and exhibits the emergent characteristic of “aha moments.”
- Figure 6 compares the original model with variants removing KL divergence or replacing GRPO with Dr. GRPO across four video benchmarks.
- The study reports experimental findings intended to guide future exploration of reasoning abilities in small-scale video models.
- Future Work: Current training uses only general video question-answering data, motivating future use of higher-quality video reasoning data.
- Future Work: The GRPO-based training approach has notable limitations, motivating refinement of reinforcement-learning algorithms for video reasoning.