PaperlayerView exact revision on arXiv

Source-linked AI summary

Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling

Yuan Wang, Ouxiang Li, Yulong Xu, Borui Liao, Jiajun Liang, Jinghan Li, Meng Wang, Xintao Wang, Pengfei Wan, Kuien Liu, Xiang Wang

arXiv:2605.05922v2cs.CV

TL;DR

Video reward modeling seeks human-aligned rewards that generalize across scenarios, but coupling reasoning and scoring creates optimization instability. DeScore decouples chain-of-thought reasoning from reward prediction and consistently outperforms existing baselines on in-domain and OOD benchmarks while using less training data.

  • Problem

    Video reward modeling needs human-aligned rewards that generalize across scenarios, while coupling chain-of-thought reasoning and scoring creates optimization instability.

    Source: §1 · pp. 2–3

  • Method

    DeScore separates chain-of-thought reasoning from reward prediction in a two-stage framework with discriminative cold-start training and dual-objective reinforcement learning.

    Source: §1 · p. 3

  • Results

    DeScore consistently outperforms discriminative and generative baselines on in-domain and OOD benchmarks while reducing training data by 76%.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    Applied to post-training and test-time scaling, DeScore improves generated video quality.

    Source: §5 · p. 10

    DeScore primarily evaluates prompt faithfulness and may be less effective at capturing motion implausibility and visual artifacts.

    Source: I Limitation and Future Work · p. 17

Abstract

from arXiv · show

Recent advances in generative video models are increasingly driven by post-training and test-time scaling, both of which critically depend on the quality of video reward models (RMs). An ideal reward model should predict accurate rewards that align with human preferences across diverse scenarios. However, existing paradigms face a fundamental dilemma: \textit{Discriminative RMs} regress rewards directly on features extracted by multimodal large language models (MLLMs) without explicit reasoning, making them prone to shortcut learning and heavily reliant on massive data scaling for generalization. In contrast, \textit{Generative RMs} with Chain-of-Thought (CoT) reasoning exhibit superior interpretability and generalization potential, as they leverage fine-grained semantic supervision to internalize the rationales behind human preferences. However, they suffer from inherent optimization bottlenecks due to the coupling of reasoning and scoring within a single autoregressive inference chain. To harness the generalization benefits of CoT reasoning while mitigating the training instability of coupled reasoning and scoring, we introduce DeScore, a training-efficient and generalizable video reward model. DeScore employs a decoupled ``think-then-score'' paradigm: an MLLM first generates an explicit CoT, followed by a dedicated discriminative scoring module consisting of a learnable query token and a regression head that predicts the final reward. DeScore is optimized via a two-stage framework: (1) a discriminative cold start incorporating a random mask mechanism to ensure robust scoring capabilities, and (2) a dual-objective reinforcement learning stage that independently refines CoT reasoning quality and calibrates the final reward, ensuring that higher-quality reasoning directly translates to superior model performance.

1 Introduction

Video reward models set the optimization ceiling for generative video and must align with human preferences across diverse scenarios and generalize out of distribution. DeScore separates CoT reasoning from scalar reward scoring to retain reasoning benefits while addressing coupled optimization bottlenecks.

1 Introduction

  • Discriminative video RMs [10] [20] regress rewards from MLLM features, risking shortcut learning [47] from coarse preference labels and requiring costly data scaling for OOD adaptability.The passage notes that regression losses provide stable optimization signals, but do not explicitly supervise fine-grained semantic distinctions.Source: §1 · p. 2
  • Generative models [42] [9] [41] [39] [40] can add CoT [42] [9] [41] [40], providing fine-grained semantic supervision that helps models learn preference rationales and generalize.The introduction contrasts learning why a video is preferred with merely fitting a ranking.Source: §1 · p. 2
  • Token-sequence scoring lacks direct reward-value gradients versus BT, while coupled reasoning and scoring complicate credit assignment and produce high-variance RL gradients [7] [31] [48] [11] [46].A single rollout reward makes it difficult to distinguish weak intermediate reasoning from an inaccurate final reward token.Source: §1 · p. 2
  • DeScore separates CoT reasoning from reward prediction, using a learnable query token and regression head to enable direct, stable scalar-reward optimization.The scoring module can use a margin-based loss such as BT, bypassing credit assignment across the reasoning sequence.Source: §1 · p. 3
  • DeScore uses a discriminative cold start with random CoT masking, then dual-objective RL to refine reasoning with GRPO and calibrate scores with BT loss.Random masking encourages scoring from both raw inputs and generated CoT rather than letting either dominate.Source: §1 · p. 3
  • DeScore outperforms state-of-the-art baselines in OOD generalization and training efficiency, and improves generated video quality during post-training.The introduction reports these outcomes without specifying quantitative results.Source: §1 · p. 3

2 Related Work

Related work contrasts discriminative video reward models’ shortcut-learning and data-scaling weaknesses with generative CoT models’ interpretability and generalization, but notes instability from coupled reasoning and scoring. GRPO’s optimization bottlenecks motivate more efficient video reward modeling.

Video Reward Model

  • Discriminative video RMs regress rewards from MLLM features with MSE or BT loss, but lack of reasoning leaves them prone to shortcut learning and dependent on large-scale data.The passage cites discriminative methods [10] [20], objectives, and shortcut learning [45].Source: §2 · p. 3
  • Generative RMs progressed from direct score generation [41] to SFT-then-RL CoT [39] [9] [40] [36] [31], improving interpretability and generalization but retaining coupled-inference instability.Token-probability methods [42] instead depend on reference videos or pairwise comparisons, limiting practical applicability.Source: §2 · p. 3
  • DeScore is positioned as a Think-then-Score approach that decouples reasoning from scoring to achieve robust preference alignment and training efficiency.Source: §2 · p. 3

Reinforcement Learning

  • Recent RL advances improve MLLM visual reasoning [13] [2] [14] [49] [32] [38], with GRPO estimating relative-response advantages without a separate critic [31] [7] [30].Source: §2 · p. 3
  • Uniformly correct or incorrect GRPO response groups weaken gradients [46], while variance grows with sequence length [11], destabilizing coupled CoT video RMs [9] [39] [36].These inherited bottlenecks motivate moving beyond standard GRPO toward more efficient video reward modeling.Source: §2 · p. 4

3 Method

DeScore separates video reasoning from scalar reward prediction: a query-token scoring module follows CoT, and two-stage training refines scoring and reasoning separately. At inference, it uses a think-then-score procedure.

3.1 Data Collection

  • The dataset contains 22K training pairs and 1469 in-domain evaluation pairs, human-compared across object, dynamics, environment, style, and camera movement.Stage-specific CoT annotations come from Qwen3-VL-8B for cold start and Gemini-2.5-Pro for RL, with consistency-based filtering against human preferences.Source: §3.1 · p. 4

3.2 Reward Model Learning

  • A discriminative cold start trains DeScore’s [Reward] query and regression head on video, prompt, and CoT, while random CoT masking preserves raw-input grounding.The query token’s hidden state is projected to a scalar reward, with preference pairs trained using Bradley-Terry loss.Source: §3.2 · pp. 4–5
  • The dual-objective RL stage uses GRPO to refine CoT reasoning and auxiliary BT loss to calibrate rewards, countering reward drift.The combined objective balances these losses so reward prediction remains grounded in stable regression rather than a coupled sampling chain.Source: §3.2 · pp. 5–6

3.3 Inference

  • At inference, DeScore first generates CoT for a video and prompt, then appends [Reward] and regresses its hidden state to a scalar score.This decoupled procedure uses CoT for interpretability and generalization while shielding training from coupled-chain optimization instability.Source: §3.3 · p. 6

4 Experiments

DeScore leads preference benchmarks with substantially less training data; ablations highlight contributions from CoT, random masking, and dual-objective RL, while post-training experiments show improved video generation quality.

4.2 Main Results

  • On GenAI-Bench, DeScore scores 0.765 Acc w/o Tie; on VideoGen-Bench, it scores 0.768 versus VideoAlign’s 0.722 and the best generative baseline’s 0.582, leading overall.The prose reports that DeScore outperforms discriminative and generative methods in-domain and across OOD settings, with the GenAI exception that UnifiedReward-Thinking is slightly higher on Acc w/ Tie.Source: Table 1 · p. 7, §4.2 · p. 7
  • DeScore outperforms state-of-the-art reward models across evaluated benchmarks while reducing training data by 76%.The reported performance-to-data tradeoff is attributed to fine-grained semantic rationales and is presented as evidence of improved sample efficiency.Source: Figure 3 · p. 7, §4.2 · p. 7

4.1 Experimental Setups.

  • Evaluation covers 1,469 in-domain pairs and OOD sets GenAI (1.9k pairs) and VideoGen-Bench (26.5k pairs).DeScore uses Qwen3-VL-8B, a LoRA discriminative cold start, then GRPO with auxiliary BT loss; videos are processed at 2 fps.Source: §4.1 · pp. 6–7

4.2 Main Results

  • Qualitatively, DeScore tolerates imperfect CoT and differentiates scores even when generative models produce identical reward tokens.The comparison includes camera-motion and dynamic-motion scenarios; UnifiedReward-Thinking often fails when its generated CoT is low quality because scoring is coupled to reasoning.Source: §4.2 · p. 8

4.3 Ablation Study

  • Adding CoT raises in-domain accuracy from 0.588 to 0.615; random masking then raises it to 0.656 and improves both OOD benchmarks.CoT also increases Acc w/o Tie by 5.1% on GenAI and 6.5% on VideoGen-Bench; random masking moves GenAI from 0.636 to 0.685 and VideoGen-Bench from 0.654 to 0.672.Source: §4.3 · p. 9
  • On VideoGen-Bench, training with only GRPO lowers accuracy from 0.768 to 0.648, while auxiliary BT loss calibrates rewards alongside improved CoT.Without cold-start initialization, dual-objective RL still performs respectably across benchmarks, but slightly below the default setting; the passage attributes the GRPO-only drop to neglected scoring.Source: Table 2 · p. 8, §4.3 · p. 9
  • Both discriminative and generative variants underperform DeScore across benchmarks; BT training improves preference accuracy more smoothly than GRPO.The discriminative variant relies more heavily on data scaling, while GRPO shows pronounced fluctuations in preference accuracy.Source: §4.3 · p. 9

4.4 Improving Video Generation

  • DeScore consistently improves VBench generation quality under both Longcat-GRPO and Flow-DPO post-training frameworks.Reported gains span subject consistency, background consistency, aesthetic quality, image quality, and dynamic degree.Source: Table 3 · p. 9, §4.4 · p. 9

5 Conclusion

DeScore decouples Chain-of-Thought reasoning from reward prediction in a two-stage training framework. It stably and efficiently outperforms discriminative and generative baselines on in-domain and OOD benchmarks with less training data, and improves generated video quality in post-training and test-time scaling.

5 Conclusion

  • DeScore outperforms discriminative and generative baselines on in-domain and OOD benchmarks with substantially less training data, and improves generated video quality in post-training and test-time scaling.The conclusion attributes stable and efficient optimization to DeScore’s two-stage training framework.Source: §5 · p. 10
  • DeScore decouples Chain-of-Thought reasoning from final reward prediction, retaining explicit reasoning’s interpretability and generalization benefits while avoiding coupled reasoning-and-scoring optimization instability.The paper presents decoupled reasoning and scoring as a training-efficient, generalizable paradigm for video reward modeling.Source: §5 · p. 10

A Analysis on Optimization Direction

Generative reward optimization treats score tokens categorically, losing ordinal distance and directional guidance. The Bradley–Terry loss instead uses continuous rewards to provide deterministic, adaptive optimization signals.

A Analysis on Optimization Direction

  • Categorical score-token optimization ignores ordinal distance, while policy gradients suppress suboptimal tokens without indicating whether the score should rise or fall.This leaves the model to explore the discrete token space without clear guidance for distinguishing quality levels.Source: A Analysis on Optimization Direction · p. 13
  • The Bradley–Terry loss models chosen and rejected items with continuous scalar rewards and optimizes their difference using −log σ(s_w−s_l).Source: A Analysis on Optimization Direction · p. 13
  • Its gradient provides a deterministic push-pull force proportional to margin error, emphasizing hard pairs and decaying for easy pairs as an adaptive curriculum.The gradient is substantial when s_w ≈ s_l and decays when s_w ≫ s_l.Source: A Analysis on Optimization Direction · p. 13

B Analysis on Gradient Variance

The analysis shows that GRPO gradient variance grows with response length because per-step score contributions accumulate. Clipping can partially control gradient magnitude but introduces optimization bias.

B Analysis on Gradient Variance

  • The analysis models an autoregressive MLLM policy sampling token sequences and simplifies a group of GRPO responses to one sampling chain.The gradient contribution uses the sampled response’s group-normalized advantage.Source: B Analysis on Gradient Variance · p. 13

Theorem: GRPO Gradient Variance Scales as Ω(T)

  • The per-step score function forms a martingale difference sequence, so cross-covariances vanish and cumulative-score variance sums per-step variances.This variance analysis is established independently of any reward signal.Source: Theorem: GRPO Gradient Variance Scales… · pp. 13–14
  • GRPO gradient variance scales as Ω(T) with response length, while clipping only partially controls gradient magnitude and introduces optimization bias.The coupled-estimator proof approximates independence between the group-normalized advantage and each token score gradient, and uses E[Â^2] = 1 under GRPO z-score normalization.Source: Theorem: GRPO Gradient Variance Scales… · p. 14

C Details on Data Collection · D User Instruction

The sections describe how DeScore’s preference data and CoTs are collected and filtered, then specify the structured instruction used to judge prompt–video alignment. The instruction decodes the prompt before inspecting the video and feeds the resulting rationale to DeScore’s scoring module.

C Details on Data Collection

  • The dataset contains 22K training video pairs and 1,469 in-domain benchmark pairs, generated from captions of diverse real-world videos using seven T2V models and labeled by human preference.Annotators assess alignment across object, dynamics, environment, style, and camera movement.Source: C Details on Data Collection · p. 14
  • Cold-start CoTs come from Qwen3-VL-8B [2] and are consistency-filtered to retain pairs whose CoT-derived preference matches human labels; this stage prioritizes activating scoring over reasoning quality.Source: C Details on Data Collection · p. 14
  • For dual-objective RL, Gemini-2.5-Pro generates fine-grained CoTs with sub-dimension scores and a final preference; preference-accuracy filtering selects data for GRPO reasoning refinement.The sub-dimensions cover subjects, dynamics, environment, style, and camera movement.Source: C Details on Data Collection · p. 14

D User Instruction

  • For in-domain and OOD alignment evaluation, DeScore uses its specific instruction, while compared SOTA reward models use their original holistic video-quality prompts.Source: D User Instruction · p. 15
  • The instruction casts the model as a professional video-text alignment judge and requires plain-text Chain-of-Thought reasoning before the final answer.It directs the judge to assess whether the generated video matches the user’s text prompt.Source: D User Instruction · p. 15
  • The reasoning first decodes the text prompt into a gold-standard visual expectation before inspecting the video, then scores subject, dynamics, camera, environment, and style.The final CoT rationale is processed by DeScore’s dedicated scoring module to produce a scalar alignment reward.Source: D User Instruction · p. 15

E Detailed Implementation

DeScore is implemented with Qwen3-VL-8B [2], a LoRA-based discriminative cold start, and dual-objective GRPO/BT refinement.

E Detailed Implementation

  • DeScore uses Qwen3-VL-8B [2] for LoRA cold-start fine-tuning and dual-objective GRPO/BT optimization, with video processed at 2 fps on 8 A800 GPUs.Cold start uses LoRA rank 64 for two epochs with AdamW, a learning rate of 2 × 10−6, weight decay 0.01, and batch size 32. RL uses BT-loss coefficient 0.005, GRPO coefficient 1.0, learning rate 1 × 10−6, group size G = 8, 65 steps, rollout batch size 128, and mini-batch size 32.Source: E Detailed Implementation · p. 16

F Detailed Evaluation Setting

DeScore is evaluated against discriminative and generative video reward models on held-out in-domain and out-of-distribution preference benchmarks. Evaluation uses preference accuracy, with text-alignment accuracy reported specifically for VideoGen-Bench.

Baseline

  • Baselines include discriminative VideoScore [9] and VideoAlign [20], plus generative VisionReward, UnifiedReward [41], UnifiedReward-Thinking [39], and VideoScore2 [9].VideoScore and VideoAlign use MSE and BT losses, respectively; VisionReward and UnifiedReward generate reward tokens, while UnifiedReward-Thinking and VideoScore2 generate CoT before the reward. UnifiedReward-Thinking is pair-wise, identifying the superior video in a pair.Source: F Detailed Evaluation Setting · p. 16

Evaluation Benchmarks

  • Evaluation covers 1,469 held-out in-domain pairs and OOD suites GenAI (1.9k pairs) and VideoGen-Bench (26.5k pairs).GenAI contains early-stage videos from 10 T2V models across 508 prompts, typically at ∼320×512 resolution and 2s–2.5s duration; VideoGen-Bench spans 12 open- and closed-source models across 420 prompts, up to 576×1024 and 4s–6s, representing current state-of-the-art videos.Source: F Detailed Evaluation Setting · p. 16

Evaluation Metrics

  • Preference accuracy is reported with and without ties; VideoGen-Bench also reports Text Alignment (TA) preference accuracy.Acc w/o Tie selects the higher point-wise score, while Acc w/ Tie classifies pairs as ties when the absolute score difference is below a predefined threshold. Baseline comparisons use task-specific scores: VideoAlign’s TA sub-score, text-to-video alignment scores for UnifiedReward and VideoScore2, and overall scores for the remaining models.Source: F Detailed Evaluation Setting · p. 16

G Training Details of Ablation Study

The ablation section specifies training recipes for discriminative and generative reward models and illustrates how random masking changes reward-token attention.

Discriminative Version

  • The discriminative baseline uses Qwen3-VL-8B [2], places its query token after multimodal inputs without CoT, and matches DeScore’s cold-start data and configuration.It is fine-tuned with LoRA (rank 64) for two epochs using AdamW, learning rate 2 × 10−6, weight decay 0.01, and batch size 32.Source: G Training Details of Ablation Study · p. 16

Generative Version

  • The generative version uses pairwise SFT followed by GRPO, with CoT data generated by Gemini-2.5-Pro and additional data for robust convergence.GRPO uses AdamW with weight decay 0.01 and learning rate 1 × 10−6, rollout group size G = 8, rollout batch size 256, and update mini-batch size 64.Source: G Training Details of Ablation Study · pp. 16–17

Random Masking

  • Random masking broadens the [Reward] token’s attention to multimodal inputs, preventing the final reward prediction from relying solely on CoT.Figure 6 visualizes the 150 tokens most attended to by the [Reward] token.Source: Figure 6 · p. 17

H Additional Experiments … K Reproducibility statement

The additional experiments show that DeScore’s random masking encourages reward scoring to attend to both multimodal inputs and chain-of-thought, while integrating DeScore into two post-training paradigms improves generated video quality. The paper also notes limitations in motion and artifact assessment and provides ethics and reproducibility statements.

H.1 Visualization

  • With random masking, the final reward query attends to both multimodal inputs and CoT rather than relying solely on reasoning tokens.The paper visualizes the 150 tokens receiving the highest attention from the reward query and links this behavior to better reward performance.Source: §H.1 · p. 17

H.2 Improving Video Generation

  • DeScore consistently improves generated-video quality under both Longcat-GRPO and Flow-DPO post-training.The comparison is evaluated on VBench; no specific metric values are provided in the passages.Source: §H.2 · p. 17
  • The evaluation uses Wan-2.1-1.3B, 10K prompts, and VBench, with Flow-DPO pairs periodically updated from reward scores.Longcat-GRPO follows its official technical-report setting.Source: §H.2 · p. 17
  • Qualitative examples show improved subject fidelity, camera motion, spatial relationships, dynamics, and prompt-semantic matching with DeScore post-training.The examples compare the baseline Wan-2.1-1.3B with Flow-DPO and Longcat-GRPO equipped with DeScore.Source: §H.2 · p. 17, Figure 7 · p. 18

I Limitation and Future Work

  • DeScore primarily assesses faithfulness to the input text prompt and may be less effective at capturing motion implausibility and visual artifacts.Future work aims to extend the decoupled paradigm to multidimensional video reward modeling.Source: I Limitation and Future Work · p. 17

J Ethics statement

  • The work uses videos generated by open- and closed-source models, evaluates on public benchmarks, collects no human-subject data, and anticipates no direct, immediate negative societal impacts.The named benchmarks include GenAI, VideoGen-Bench, and VBench.Source: J Ethics statement · p. 17, Figure 7 · p. 18

K Reproducibility statement

  • The authors report detailed dataset, training-configuration, and hyperparameter descriptions in the paper and supplement, and say they will provide code.Source: K Reproducibility statement · p. 18

Related papers

Loading 2605.05922v2…