PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoning

Sicheng Tao, Jungang Li, Yibo Yan, Junyan Zhang, Yubo Gao, Hanqian Li, ShuHang Xun, Yuxuan Fan, Hong Chen, Jianxiang He, Xuming Hu

arXiv:2509.21113v2cs.CV

TL;DR

Video reasoning remains challenging for MLLMs, and fine-grained temporal reasoning supervision is scarce. MOSS-ChatV trains on annotated state-prediction traces with a rule-based DTW process reward. It improves performance across video benchmarks, reaching 87.2% accuracy on MOSS-Video (test).

  • Problem

    Video reasoning remains challenging because models must connect visual observations with temporal dynamics and causal dependencies, while fine-grained temporal reasoning supervision is scarce.

    Source: §1 · p. 1

  • Method

    MOSS-ChatV uses annotated MOSS-Video state-prediction traces and a rule-based split-align DTW process reward to fine-tune with GRPO without a learned reward model.

    Source: §1 · p. 2

  • Results

    MOSS-ChatV achieves state-of-the-art results on MVBench, VideoMME, RTVBench, and MOSS-Video test, improves TempCompass over Qwen2.5-VL, and reaches 87.2% accuracy on MOSS-Video (test).

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    Results support video state prediction as a useful proxy for reasoning, with gains extending to general video benchmarks.

    Source: §1 · p. 1, §4.1 · p. 7

Abstract

from arXiv · show

Video reasoning has emerged as a critical capability for multimodal large language models (MLLMs), requiring models to move beyond static perception toward coherent understanding of temporal dynamics in complex scenes. Yet existing MLLMs often exhibit process inconsistency, where intermediate reasoning drifts from video dynamics even when the final answer is correct, undermining interpretability and robustness. To address this issue, we introduce MOSS-ChatV, a reinforcement learning framework with a Dynamic Time Warping (DTW)-based process reward. This rule-based reward aligns reasoning traces with temporally grounded references, enabling efficient process supervision without auxiliary reward models. We further identify dynamic state prediction as a key measure of video reasoning and construct MOSS-Video, a benchmark with annotated reasoning traces, where the training split is used to fine-tune MOSS-ChatV and the held-out split is reserved for evaluation. MOSS-ChatV achieves 87.2\% on MOSS-Video (test) and improves performance on general video benchmarks such as MVBench and MMVU. The framework consistently yields gains across different architectures, including Qwen2.5-VL and Phi-2, confirming its broad applicability. Evaluations with GPT-4o-as-judge further show that MOSS-ChatV produces more consistent and stable reasoning traces.

1 INTRODUCTION

Video reasoning remains challenging because it requires connecting observations with temporal dynamics and causal dependencies, while fine-grained temporal reasoning supervision is scarce. MOSS-ChatV combines annotated state-prediction data with a rule-based process reward to align reasoning with temporal references.

1 INTRODUCTION

  • Existing MLLMs struggle with video reasoning, and fine-grained temporal reasoning supervision is scarce despite videos’ dense temporal signals.Models must recognize present states and infer future trajectories from context and world knowledge.Source: §1 · p. 1
  • MOSS-Video provides annotated video state-prediction reasoning traces with separate training and held-out test splits for process-supervised learning and evaluation.The paper treats temporal state prediction as a proxy for reasoning ability.Source: §1 · pp. 1–2
  • MOSS-ChatV captures finer-grained video states and extrapolates them accurately, as illustrated by recognizing a surfer’s crouched position before a maneuver.Figure 1 contrasts correctly reasoned key points with reasoning errors across models.Source: Figure 1 · p. 2
  • Process Reasoning Reward uses a two-stage split-align strategy and subsequence Dynamic Time Warping to align generated and reference processes without a learned reward model.MOSS-ChatV is fine-tuned with GRPO using the MOSS-Video training split.Source: §1 · p. 2
  • 87.2% accuracy on MOSS-Video (test) surpasses GPT-4o; MOSS-ChatV also improves MVBench and MMVU and yields gains across Qwen2.5-VL and TinyLLaVA-Video.GPT-4o judging further finds more consistent and stable reasoning traces.Source: §1 · p. 2

2 PRELIMINARY

Video state prediction combines object identification, state inference, and a temporally grounded explanation. GRPO optimizes sampled responses using relative rewards, while accuracy and format rewards are complemented by process-level supervision for temporal logic.

2.1 VIDEO STATE PREDICTION AND REASONING

  • Video state prediction requires identifying a target object, inferring its current or imminent state, and giving a temporally grounded explanation.Source: §2.1 · p. 3

2.2 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

  • GRPO samples a group of candidate responses, compares them through scalar rewards, and updates the policy without a learned value function.Source: §2.2 · p. 3
  • The GRPO objective favors higher-advantage responses with importance weighting and regularizes policy drift from a fixed reference policy.The KL term, scaled by β, controls policy drift.Source: §2.2 · p. 3
  • Accuracy and format rewards are complemented by process-level reward to align intermediate reasoning with reference temporal processes.Accuracy reward is binary for multiple-choice or short-answer tasks, while format reward checks adherence to a required schema.Source: §2.2 · p. 3

3 PROCESS REASONING REWARD

MOSS-ChatV scores generated reasoning against reference traces using step segmentation and subsequence DTW, then combines the process score with accuracy and format rewards. The section also introduces MOSS-Video, a state-prediction dataset with annotated reasoning traces and held-out evaluation data.

  • The Process Reasoning Reward uses reference annotations as a gold-standard reasoning process and avoids large model-based evaluators through efficient algorithms.Source: §3 · p. 4
  • Reasoning texts are segmented into steps, then SDTW matches the entire reference sequence to its best-matching subsequence in generated reasoning using ROUGE-based pairwise distances.ROUGE-1 and ROUGE-2 capture n-gram overlap, while ROUGE-L measures within-step structural similarity.Source: §3 · p. 4
  • SDTW tolerates exploratory generated segments outside the optimal alignment and offers tunable jump steps to balance flexibility with alignment precision.Source: §3 · p. 5
  • The process reward is the exponential of negative α times the minimum cumulative SDTW distance, with α controlling reward sensitivity to distance.Source: §3 · p. 6
  • Each response’s total reward sums its process, accuracy, and format rewards, and the process signal encourages temporal coherence and ordered reasoning steps.Source: §3 · p. 6
  • MOSS-Video derives from ShareGPT4Video and provides annotated state-prediction reasoning traces, with 11,654 training samples from 1,218 videos and 2,836 held-out test samples from 479 videos.The dataset comparison describes MOSS-Video as uniquely supporting state prediction with explicit reasoning annotations.Source: §3 · p. 6, Table 1 · p. 6

4 EXPERIMENT

The experiments evaluate MOSS-ChatV across video benchmarks and architectures, then test how frame counts, DTW alignment, process supervision, and reasoning quality relate to its performance.

4 EXPERIMENT

  • The evaluation fine-tunes Qwen2.5VL on MOSS-Video training data and tests across benchmarks for temporal reasoning, action recognition, causal inference, and narrative comprehension.Generalization is also tested with TinyLLaVA-Video, using Phi2 and SigLIP.Source: §4 · p. 6

4.1 RESULTS

  • MOSS-ChatV achieves state-of-the-art performance on MVBench, VideoMME, RTVBench, and MOSS-Video test, and improves over Qwen2.5-VL on TempCompass.The comparison uses 32-frame inputs.Source: §4.1 · p. 6, Table 2 · p. 7
  • MOSS-Video test results indicate that reasoning capabilities contribute positively to video prediction, and Video-R1 improves despite not being trained on MOSS-Video Train.Neither Qwen2.5-VL nor Video-R1 was trained on that split.Source: §4.1 · p. 7
  • Increasing input frames enhances state-prediction performance, while MOSS-ChatV likely reaches peak accuracy with fewer frames.The authors attribute this likely difference to more efficient information extraction and reasoning.Source: §4.1 · p. 7, Figure 3 · p. 7

4.2 SDTW VS. DTW

  • Naive DTW can reward drastically shortened reasoning because minimizing its distance can produce trivial outputs, whereas SDTW treats annotations as a minimal rather than absolute gold standard.This permits valid model explorations beyond the shortest annotated reasoning chain without relying on the naive distance behavior.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 ABLATION STUDY

  • Across benchmarks, full MOSS-ChatV outperforms its no-PRR and supervised-fine-tuning variants; no-PRR also outperforms SFT.Removing process supervision degrades temporal reasoning performance.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 MLLM AS A JUDGE FOR REASONING QUALITY EVALUATION

  • Against Video-R1, MOSS-ChatV has comparable Reasoning-Answer Consistency but higher information density, logical coherence, and video relevance; it also exceeds no-PRR in consistency.The evaluation uses GPT-4o as a judge to assess reasoning and answers across four dimensions.Source: §4.4 · pp. 8–9, Table 4 · p. 8

5 RELATED WORK

Prior video-LLM work improves dialogue, long-video processing, and general video understanding through instruction tuning and token compression. Video reasoning work applies RL, while this paper distinguishes itself by rewarding the reasoning path directly.

5.1 ADVANCED VIDEO-LLM

  • VideoChat-GPT targets video dialogue, description, and reasoning with video-specific instruction data, while other models use visual-token compression for long videos.Large-scale video-text pretraining followed by instruction tuning is another common approach to diverse video understanding tasks.Source: §5.1 · p. 9

5.2 REASONING AND REINFORCEMENT LEARNING IN VIDEO-LLMS

  • Video-R1 and related approaches use RL rewards for temporal grounding or understanding, whereas PRR provides more granular supervision on the reasoning path itself.The cited video-RL examples include Videochat-R1, TimeZero, TinyLLaVA-Video-R1, and Video-R1.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion presents video state prediction and video reasoning as mutually reinforcing, and reports gains from process-supervised reinforcement fine-tuning. MOSS-ChatV improves video analysis with single-task training data while maintaining stable reasoning quality.

6 CONCLUSION

  • The paper reports that video state prediction and video reasoning mutually reinforce one another.This relationship motivates MOSS-Video as a dataset for training and evaluating video state prediction.Source: §6 · p. 9
  • Comparative and ablation experiments support the effectiveness of rule-based Process Reasoning Reward for reinforcement fine-tuning of video models.The conclusion also reports state-of-the-art results under low-quality video inputs.Source: §6 · p. 9
  • Using single-task training data alone, MOSS-ChatV yields holistic video-analysis improvements while maintaining stable reasoning quality.The conclusion calls for greater attention to video-context reasoning capability.Source: §6 · p. 9

7 ETHICS STATEMENT

The ethics statement reports that the work uses synthetic or public datasets and involves no direct human subjects.

7 ETHICS STATEMENT

  • The study uses synthetic or publicly available datasets and contains no sensitive or personally identifiable information.It reports no direct human subjects.Source: §7 · p. 10
  • The authors report conducting methods and experiments in accordance with applicable laws and research-integrity practices.They also report no conflicts of interest or undue influence from external sponsorship.Source: §7 · p. 10
  • The statement reports no privacy, security, discrimination, bias, or fairness concerns and no harmful insights or applications.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

The paper describes steps intended to support reproducibility, including detailed reporting of experimental settings and plans to release source code upon acceptance.

8 REPRODUCIBILITY STATEMENT

  • Experimental settings, including datasets and model designs, are described in Section 4.The authors state these descriptions are intended to support reproducibility.Source: §8 · p. 10
  • The authors say source code will be made publicly available upon acceptance.Source: §8 · p. 10
  • The paper states that steps were taken to ensure reproducibility of its results.Source: §8 · p. 10

9 LLM USAGE STATEMENT

LLMs were used only for writing and presentation support, not for the research activities or interpretation of results.

9 LLM USAGE STATEMENT

  • LLMs were used exclusively to assist with writing, editing, and LaTeX formatting.Source: §9 · p. 10
  • Their role was limited to improving clarity, grammar, and overall presentation.Source: §9 · p. 10
  • LLMs had no impact on experiment design, data processing, analysis, or interpretation of results.Source: §9 · p. 10

A DETAILS OF MOSS-VIDEO

MOSS-Video is constructed from ShareGPT4Video annotations using parallel coarse- and fine-grained pipelines, which are integrated into a temporal model for generating questions about future object states.

A DETAILS OF MOSS-VIDEO

  • GPT4-o processes ShareGPT4Video annotations in parallel pipelines to capture coarse- and fine-grained object states.The coarse pipeline produces 〈Object, State, Timestamp〉 triplets, while the fine pipeline adds a textual description at a precise moment.Source: A DETAILS OF MOSS-VIDEO · p. 13
  • GPT4-o integrates both annotation streams into a unified temporal model of object dynamics.Source: A DETAILS OF MOSS-VIDEO · p. 13
  • The temporal model is used to automatically generate question–answer pairs probing predicted future object states.Source: A DETAILS OF MOSS-VIDEO · p. 13
  • Figure A.1 presents an example of MOSS-Video.Source: Figure A.1 · p. 13

B MLLM-AS-A-JUDGE FOR REASPONSE QUALITY

The section describes GPT-4-o evaluation of video-question-answering reasoning and response quality through consistency, repetitiveness, coherence and accuracy, and video relevance dimensions.

B MLLM-AS-A-JUDGE FOR REASPONSE QUALITY

Loading 2509.21113v2…