Source-linked AI summary
Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling
Yuan Wang, Ouxiang Li, Yulong Xu, Borui Liao, Jiajun Liang, Jinghan Li, Meng Wang, Xintao Wang, Pengfei Wan, Kuien Liu, Xiang Wang
TL;DR
Video reward models must align with human preferences while generalizing across diverse and out-of-distribution videos, but existing discriminative and generative approaches expose different limitations. DeScore decouples CoT reasoning from scalar scoring through a dedicated query-token regression module and two-stage training, achieving stronger generalization and efficiency while improving generated video quality.
Problem
Video reward models need accurate human-preference alignment and OOD generalization, while discriminative models lack explicit reasoning and generative models couple reasoning with scoring.
Method
DeScore uses a decoupled think-then-score architecture with MLLM-generated CoT, a learnable query token, a regression head, and two-stage training with masking and dual objectives.
Results
DeScore consistently outperforms discriminative and generative baselines on in-domain and OOD benchmarks, uses substantially less training data, and improves generated video quality.
Takeaways & Limitations
Decoupling reasoning and scoring is supported as a promising approach for stable, efficient, and generalizable video reward modeling.
Takeaways & Limitations
DeScore primarily evaluates faithfulness to input text prompts and may be less effective for motion implausibility and visual artifacts.
Abstract
from arXiv · showhide
Recent advances in generative video models are increasingly driven by post-training and test-time scaling, both of which critically depend on the quality of video reward models (RMs). An ideal reward model should predict accurate rewards that align with human preferences across diverse scenarios. However, existing paradigms face a fundamental dilemma: \textit{Discriminative RMs} regress rewards directly on features extracted by multimodal large language models (MLLMs) without explicit reasoning, making them prone to shortcut learning and heavily reliant on massive data scaling for generalization. In contrast, \textit{Generative RMs} with Chain-of-Thought (CoT) reasoning exhibit superior interpretability and generalization potential, as they leverage fine-grained semantic supervision to internalize the rationales behind human preferences. However, they suffer from inherent optimization bottlenecks due to the coupling of reasoning and scoring within a single autoregressive inference chain. To harness the generalization benefits of CoT reasoning while mitigating the training instability of coupled reasoning and scoring, we introduce DeScore, a training-efficient and generalizable video reward model. DeScore employs a decoupled ``think-then-score'' paradigm: an MLLM first generates an explicit CoT, followed by a dedicated discriminative scoring module consisting of a learnable query token and a regression head that predicts the final reward. DeScore is optimized via a two-stage framework: (1) a discriminative cold start incorporating a random mask mechanism to ensure robust scoring capabilities, and (2) a dual-objective reinforcement learning stage that independently refines CoT reasoning quality and calibrates the final reward, ensuring that higher-quality reasoning directly translates to superior model performance.
1 Introduction
Video reward models must align with human preferences across diverse videos while generalizing beyond their training distribution. DeScore addresses the trade-off between discriminative stability and generative CoT interpretability by decoupling reasoning from scalar scoring.
- Motivation: Discriminative reward models regress scalar rewards from MLLM features but can shortcut semantic understanding and require massive data scaling for OOD generalization.Their regression objectives provide stable optimization, yet coarse preference labels may not capture fine-grained human judgment.
- Motivation: Generative reward models use CoT reasoning to provide fine-grained semantic supervision and internalize why one video is preferred over another.However, their reward is generated as a token sequence rather than optimized as an explicit scalar.
- Motivation: Coupling reasoning and scoring in one autoregressive chain creates optimization bottlenecks, including indirect reward-value optimization and difficult credit assignment.These limitations increase reliance on SFT and RL methods that optimize discrete token probabilities rather than the reward value directly.
- DeScore: DeScore separates CoT generation from final reward prediction through a learnable query token and regression head, enabling direct scalar optimization with a margin-based loss.The decoupled scoring module bypasses credit assignment across the entire reasoning sequence while retaining explicit reasoning.
- Training: DeScore uses discriminative cold start with random CoT masking, followed by dual-objective RL that separates reasoning refinement from reward calibration.Random masking encourages scoring from both raw multimodal inputs and generated reasoning tokens.
- Results: Extensive experiments show that DeScore outperforms SOTA baselines, with stronger OOD generalization and higher training efficiency, while improving generated video quality.The reported gains extend DeScore’s usefulness to video-generation post-training.
2 Related Work
Video reward modeling includes discriminative and generative paradigms, while DeScore combines explicit reasoning with a separate scoring module and staged optimization. Figure 2 depicts how inference and training implement this decoupling.
- Video Reward Model: Discriminative methods regress scalar rewards from MLLM features using objectives such as MSE or BT loss, but lack explicit reasoning.The absence of reasoning makes them prone to shortcut learning and dependent on large-scale data for generalization.
- Video Reward Model: Generative methods formulate reward modeling as next-token prediction, with CoT-based variants generating reasoning before the final reward.CoT provides semantic supervision for modeling the rationale behind human preferences.
- Reinforcement Learning: GRPO estimates advantages from relative rewards across multiple responses without requiring a separate critic model.The method is used broadly for reinforcement-learning optimization in multimodal language-model tasks.
- DeScore Framework: Figure 2 shows DeScore generating CoT first, then projecting a learnable query token’s hidden state through a regression head into the video reward.Its training combines BT-based cold start with random CoT masking and a dual-objective RL stage using GRPO for CoT rollouts and BT for reward calibration.
- Reinforcement Learning: GRPO can suffer weakened gradient signals and increased variance when response groups are uniformly correct or incorrect, while gradient variance grows with sequence length.These limitations are inherited by CoT-based video reward models when reasoning and scoring share one sampling chain.
3 Method
DeScore trains video reward models by separating explicit CoT reasoning from scalar scoring, using a discriminative scoring module and staged optimization. Its inference procedure generates reasoning first, then predicts a reward from the combined multimodal inputs and CoT.
- Discriminative Cold Start: The discriminative cold-start stage jointly fine-tunes the MLLM backbone and scoring module with Bradley-Terry loss to align predicted rewards with human preferences.Winning and losing samples are scored separately, and the BT objective trains their relative ordering.
- Scoring Module: DeScore appends a learnable [Reward] query token to multimodal inputs and CoT, then maps its hidden state through a regression head to produce scalar reward s.The reward token aggregates information from the video, prompt, and generated or pre-collected reasoning.
- Discriminative Cold Start: Randomly masking CoT during training forces DeScore to score from both reasoning and raw multimodal inputs rather than relying solely on CoT.When masked, the reward is computed from the video and prompt alone, strengthening grounding in visual evidence.
- Dual-Objective Reinforcement Learning: The second stage applies GRPO to refine CoT reasoning while using a composite reward with format, quality, and length components.The length component uses the generated CoT length, while format and quality rewards encourage structured and accurate sub-dimension scoring.
- Dual-Objective Reinforcement Learning: An auxiliary BT loss calibrates final scalar rewards during reinforcement learning, and the combined objective keeps reward regression stable instead of dominated by the sampling chain.The total loss combines GRPO and auxiliary BT objectives with a balancing coefficient α.
- Inference: At inference, DeScore autoregressively generates CoT and then feeds the resulting sequence through [Reward] and the regression head to obtain s.This think-then-score procedure decouples scoring from reasoning while retaining information from both modalities and CoT.
4 Experiments
Experiments show that DeScore generalizes across in-domain and OOD benchmarks, uses less training data, and improves video generation quality. Ablations attribute these gains to CoT, random masking, and dual-objective training.
- High Generalization: DeScore achieves the strongest overall performance across in-domain and OOD video reward benchmarks.On VideoGen-Bench, it reaches 0.768 Acc w/o Tie, versus 0.722 for VideoAlign and 0.582 for the best generative baseline.
- Training Efficiency: DeScore reduces training data by 76% while outperforming state-of-the-art reward models across evaluated benchmarks.This establishes a superior performance-to-data tradeoff relative to existing models.
- Stage 1 Components: CoT improves accuracy across benchmarks, while random masking further raises in-domain accuracy from 0.615 to 0.656.Random masking also improves GenAI from 0.636 to 0.685 and VideoGen-Bench from 0.654 to 0.672.
- Stage 2 Components: Adding the auxiliary BT loss prevents the accuracy drop caused by optimizing only GRPO during the RL stage.VideoGen-Bench accuracy declines from 0.768 to 0.648 with GRPO alone, while dual-objective training calibrates reward outputs.
- Reward Modeling Paradigms: Both discriminative and generative variants underperform DeScore, while BT loss produces smoother preference-accuracy improvement than GRPO.The discriminative variant also requires substantially more training data to reach comparable performance.
- Improving Video Generation: DeScore consistently improves subject consistency, background consistency, aesthetic quality, image quality, and dynamic degree under two post-training frameworks.The frameworks are Longcat-GRPO and Flow-DPO evaluated on VBench.
5 Conclusion
DeScore decouples CoT reasoning from reward prediction through a two-stage Think-then-Score framework. It outperforms discriminative and generative baselines on in-domain and OOD benchmarks with less training data, and improves generated video quality in post-training and test-time scaling.
- Conclusion: DeScore decouples CoT reasoning from final reward prediction to retain interpretability while avoiding coupled optimization instability.Its two-stage training framework targets stable and efficient optimization.
- Conclusion: DeScore consistently outperforms discriminative and generative baselines on in-domain and OOD benchmarks while requiring substantially less training data.The conclusion identifies decoupled reasoning and scoring as a training-efficient and generalizable paradigm.
- Conclusion: Applying DeScore to post-training and test-time scaling further improves generated video quality.The conclusion presents this as an additional benefit of the reward model.
A Analysis on Optimization Direction
Generative reward models optimize discrete score tokens, so their objectives ignore ordinal distance and provide no numerical direction for correcting score errors. In contrast, BT loss optimizes continuous rewards with deterministic gradients that adapt to margin difficulty.
- Generative reward models represent final rewards as discrete tokens and optimize them with cross-entropy, treating reward values as orthogonal classes.A prediction of 5 as 4 can receive a penalty similar to predicting 1.
- Policy-gradient optimization suppresses suboptimal score-token probabilities without indicating whether the predicted score should increase or decrease.
- BT loss models chosen and rejected rewards as continuous scalars and optimizes their difference through LBT = −log σ(sw − sl).
- BT gradients provide deterministic push-and-pull forces proportional to margin error, concentrating updates on hard cases and decaying on easy cases.This behavior functions as a built-in curriculum learning mechanism.
B Analysis on Gradient Variance
The analysis attributes GRPO instability to autoregressive response length: cumulative score variance aggregates across token steps, and the resulting gradient variance scales with trajectory length. Clipping can limit magnitude but introduces optimization bias rather than removing this growth.
- An autoregressive MLLM samples token sequences from multimodal inputs, and GRPO evaluates groups of responses through policy-gradient contributions.The sampled response and its token-level log-probability terms form the optimization trajectory.
- The cumulative score function is built from per-step score functions that form a martingale difference sequence under autoregressive generation.This structure makes cross-step covariances vanish and decomposes cumulative variance into per-step variances.
- Response-length-dependent GRPO variance causes pronounced fluctuations in human-preference accuracy during training.Clipping importance weights partially controls gradient magnitude but creates additional optimization bias, yielding a bias-variance tradeoff rather than eliminating Ω(T) growth.
C Details on Data Collection
The data pipeline captions diverse real-world videos, uses those captions to prompt multiple video-generation models, and integrates the resulting preference data into two-stage training. DeScore’s evaluation instruction structures video-text alignment judgments through explicit reasoning stages.
- Real-world videos spanning subjects, dynamics, environments, styles, and camera movements are captioned to construct diverse text prompts.The captions are used to challenge text-to-video models to preserve semantic alignment.
- The resulting preference data is incorporated into DeScore’s two-stage training paradigm, with CoT data generated and filtered for stage-specific objectives.The cold-start stage prioritizes activating the scoring module rather than enforcing reasoning quality.
- DeScore evaluation uses a structured instruction that guides the model to reason about whether a generated video matches its text prompt.The process first establishes expected visual elements from the prompt before evaluating the video.
- Figure 5 presents the user instruction used for DeScore during training and inference.
E Detailed Implementation
DeScore uses Qwen3-VL-8B with LoRA-based discriminative cold start followed by dual-objective reinforcement learning combining GRPO and BT losses.
- DeScore uses Qwen3-VL-8B as its backbone and fine-tunes it with LoRA at rank 64 during discriminative cold start.Cold start training runs for two epochs with AdamW, learning rate 2 × 10^-6, weight decay 0.01, and batch size 32.
- The subsequent reinforcement-learning stage optimizes a dual objective comprising GRPO and BT losses.
F Detailed Evaluation Setting
The evaluation compares DeScore with discriminative and generative reward-model baselines on an in-domain preference dataset and two OOD benchmark suites.
- Baselines: The baseline set includes discriminative models VideoScore and VideoAlign, plus generative models VisionReward, UnifiedReward, UnifiedReward-Thinking, and VideoScore2.The generative baselines either directly generate reward tokens or generate CoT before the final reward.
- Evaluation Benchmarks: Evaluation uses 1,469 held-out in-domain preference pairs that remain unseen during training.The pairs come from the original data source and are used for comparison experiments.
- Evaluation Benchmarks: Generalization is evaluated on the OOD suites GenAI and VideoGen-Bench.GenAI comprises 1.9k pairs in the described evaluation setting.
G Training Details of Ablation Study
The ablation study compares a discriminative variant with DeScore’s explicit CoT design and examines how random masking affects reward-query attention during training.
- Discriminative Version: The discriminative variant removes CoT and predicts rewards from multimodal inputs using a regression head.It appends the learnable query token directly after the multimodal inputs and uses the same dataset and cold-start configuration as DeScore.
- Random Masking: Random masking encourages the reward query to attend jointly to multimodal inputs and CoT instead of relying only on reasoning tokens.The attention visualization considers the top 150 tokens receiving the highest attention from the final reward query.
- GRPO Training: The GRPO training configuration uses AdamW with weight decay 0.01, learning rate 1 × 10^-6, rollout group size G = 8, and rollout batch size 256.The supplied training description identifies these settings for the GRPO stage.
H Additional Experiments
Additional experiments test DeScore’s effect on video generation, explain its attention behavior, and state its current scope and reproducibility provisions.
- Attention Visualization: Random masking makes the final reward query attend to both multimodal inputs and CoT, contributing to better reward performance.The attention pattern is presented as evidence for jointly leveraging the two information sources.
- Improving Video Generation: DeScore improves generated-video quality when integrated with both Longcat-GRPO and Flow-DPO on Wan-2.1-1.3B, evaluated on VBench.The reported improvements span subject fidelity, camera motion, spatial relationships, and dynamics.
- Limitation and Future Work: DeScore primarily evaluates faithfulness to the input text prompt and may be less effective for motion implausibility and visual artifacts.Future work extends the decoupled paradigm toward multidimensional video reward modeling.
- Evaluation Scope: The study uses videos generated by open-source and closed-source models and evaluates on publicly available benchmarks.The supplied passage describes the data sources and public evaluation setting without identifying human-subject data collection.
- Reproducibility: The authors plan to provide code and describe datasets, training configurations, and hyperparameters in the main text and supplementary materials.These provisions are intended to support reproducibility.