PaperlayerView exact revision on arXiv

Source-linked AI summary

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

Zhenchen Tang, Yang Li, Songlin Yang, Bo Peng, Xiaotong Zhao, Shuai Li, Haotian Fan, Alan Zhao, Jing Dong

arXiv:2609.22947v2cs.CVcs.AI

TL;DR

Directly scoring subjective video quality lacks explicit criteria and can produce unstable reward scales. RewardVerse inserts a dynamic, query-adaptive rubric between the query and scorer, with RGPO training the scorer and rubric generator in two stages. The authors report strong pointwise and pairwise results and a useful downstream reward with only 30 preference pairs per dimension.

  • Problem

    Existing video reward models directly map subjective, multidimensional video quality to scalar scores without explicit criteria, and their scales can collapse or shift across prompts.

    Source: §1 · p. 1

  • Method

    RewardVerse scores videos against query-adaptive rubrics and uses two-stage RGPO to warm up the scorer with seed rubrics, then jointly optimize rubric generation and scorer alignment.

    Source: §1 · p. 2

  • Results

    With only 30 preference pairs per dimension, RewardVerse achieves high pointwise correlation on EvalVerse, strong pairwise agreement on VGRB, and improves downstream video generation while being less prone to reward hacking.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The reported results support using scalar-drift-mitigating video reward models as reinforcement-learning signals for video generation.

    Source: §6 · p. 10

    Rubric-guided scoring alone leaves static rubrics unable to adapt to different queries and may still produce scores that deviate from human ratings.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.

1 INTRODUCTION

Direct scalar scoring lacks explicit criteria for subjective, multidimensional video quality, leaving reward scales unstable across prompts. RewardVerse inserts a dynamic rubric between query and scorer, and RGPO trains the rubric generator and scorer together.

1 INTRODUCTION

  • Direct scalar scoring lacks explicit evaluation criteria for subjective, multidimensional video quality, so scoring scales can collapse or shift across prompts and contexts.The authors motivate explicit criteria by noting that professional annotators decompose evaluations before making judgments.Source: §1 · p. 1
  • A dynamic rubric turns the query into evaluation themes, weights, and scoring tips that the scorer applies independently, anchoring scores and mitigating scalar drift.The authors describe this as the rubric-as-reward paradigm for video reward modeling.Source: §1 · p. 2, Figure 1 · p. 2
  • RGPO first warms up the scorer with self-evolving seed rubrics, then jointly optimizes query-adaptive rubrics and scorer alignment with human ratings.The framework is reported to learn from only 30 preference pairs per dimension.Source: §1 · p. 2
  • RewardVerse achieves state-of-the-art evaluation performance and provides a stable, interpretable reward for downstream video-generation optimization.The paper reports evaluations on the multi-dimensional EvalVerse-adapted dataset and external benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior work uses rubrics or decomposed criteria in open-ended generation, while video reward models commonly score videos directly. RewardVerse instead learns query-conditioned rubrics jointly with its scorer.

2 RELATED WORK

  • Prior rubric methods use explicit criteria in language and image-generation tasks, whereas RewardVerse makes its rubric a learned intermediate representation generated from each query without the candidate video.Its rubric themes, weights, and scoring tips are jointly optimized with the scorer.Source: §2 · p. 3
  • Video reward models commonly use discriminative regression or generative reasoning to output scalar scores directly.The passage names VideoScore and VideoReward as discriminative examples, and UnifiedReward and VideoScore2 as generative examples.Source: §2 · p. 3
  • The paper says direct video scoring lacks explicit anchors and can drift across prompts, while RewardVerse's structured rubric bottleneck is designed to mitigate drift and provide a stable downstream reward.The related-work discussion links this stability to data efficiency and use in video-generation models.Source: §2 · p. 3

3 THE PHENOMENON OF SCALAR DRIFT IN VIDEO EVALUATION

The paper analyzes scalar drift in direct video scoring and tests whether rubric guidance and scoring format address it. Rubrics expand score variation, but static criteria and residual disagreement with human ratings motivate joint optimization.

3.1 DEFINING SCALAR DRIFT

  • Scalar drift comprises pointwise score-range collapse and inconsistent scores or pairwise decisions under context shifts such as paraphrasing or reordering.The resulting narrow score gaps can leave downstream RL with near-flat gradients.Source: §3.1 · p. 3

3.2 EMPIRICAL STUDY: RUBRIC IS NECESSARY BUT NOT SUFFICIENT

  • Only V3, combining rubrics with soft-logits, effectively spreads predictions, yet its distribution still deviates from human ratings, motivating RGPO.Figure 2 contrasts pointwise score distributions across four variants; the complete RGPO training further improves score alignment and correlation with human ratings.Source: Figure 2 · p. 4, §3.2 · p. 4
  • The study compares rubric presence and decoding format across four variants on 240 held-out pointwise videos with human labels.It investigates direct-scoring behavior, rubric guidance, and limitations that motivate joint policy optimization.Source: §3 · p. 3, §3.2 · p. 4
  • Adding a rubric with soft-logits expands Qwen2.5-VL-7B's scoring standard deviation by 82.6%, from 0.258 to 0.471.With natural text generation, rubrics increase Qwen2.5-VL-7B's standard deviation from 0.203 to 0.275 and reduce Gemini-3.1-Pro's from 1.500 to 1.158.Source: §3.2 · p. 4
  • Static rubrics cannot adapt themes, tips, and weights to individual queries, and rubric-guided scores may still diverge from human ratings.The authors also state that workable pointwise rewards must preserve population-level scale resolution and robustness to phrasing and ordering.Source: §3.1 · p. 3, §3.2 · p. 4

4 METHODOLOGY

RewardVerse inserts a query-adaptive rubric between the evaluation query and scorer, then uses two-stage RGPO to train the shared pipeline while aligning scores with human ratings.

4.1 REWARDVERSE PIPELINE AND FORMULATION

  • The generator maps each query to themes, weights, and execution tips without seeing the video; the scorer evaluates videos independently against that rubric.Queries pair a target dimension with the video-generation prompt.Source: §4.1 · p. 5
  • For each rubric theme, the scorer reads expected values over rating tokens 1–5 and aggregates theme scores by their weights.The generator and scorer share a backbone but use role-specific prompts.Source: §4.1 · pp. 5–6

4.2 STAGE 1: SEED-GUIDED SCORER WARM-UP

  • Thirty preference pairs per dimension drive an offline propose–verify–revise loop; verified rubrics are deduplicated and reduced to five representative entries.Stage 1 uses fixed seed rubrics because learning the generator and scorer from scratch is underdetermined.Source: §4.2 · p. 6
  • Stage 1 combines GRPO preference training with format rewards and human-margin calibration, aligning relative score differences rather than absolute scores.The calibration target is the human-annotated margin; the paper motivates relative calibration as less prone to overfitting in the low-data setting.Source: §4.2 · pp. 6–7

4.3 STAGE 2: JOINT POLICY OPTIMIZATION VIA RGPO

  • The Stage-2 rubric reward scores separation, format, and seed-theme alignment; scheduled calibration supports query-adaptive tips and weights after generator warm-up.The scheduled calibration coefficient is described as improving training stability under shared parameters.Source: §4.3 · p. 7
  • In Stage 2, the generator receives rubric-level policy-gradient updates, while the scorer receives no direct update from that reward and instead uses calibration loss.The generated rubric is fixed text context for the scorer loss.Source: §4.3 · p. 7
  • Table 1 reports per-dimension PLCC and SRCC; its average covers only the ten dimensions supported by all compared reward models.Source: Table 1 · p. 8

5 EXPERIMENTAL EVALUATION

The evaluation covers pointwise scoring on 16 EvalVerse dimensions, transfer to VGRB pairwise preferences, RGPO ablations, and downstream video-generation fine-tuning. RewardVerse performs strongly across both evaluation types and improves target quality while preserving other measured capabilities.

5.1 BENCHMARKS AND BASELINES

  • The evaluation uses a 16-dimension EvalVerse pointwise test set and VGRB, an external pairwise benchmark of over 26.5K video pairs, including seen VQ and unseen TA splits.The model is trained on Qwen2.5-VL-7B and compared with six state-of-the-art video reward models.Source: §5.1 · p. 8

5.2 POINTWISE EVALUATION ON EVALVERSE

  • RewardVerse achieved the highest PLCC in 14 of 16 EvalVerse dimensions, including Logic (0.750 vs. 0.593) and Action (0.566 vs. 0.390).The reported comparisons are against the next-best result on each named dimension.Source: §5.2 · p. 8

5.3 PAIRWISE TRANSFER ON VGRB

  • On VGRB, RewardVerse beat VisionReward by 7.1 Acc w/ Tie and 7.0 Acc w/o Tie percentage points on seen VQ, and reached 0.471 accuracy with ties and 0.623 without ties on unseen TA.VideoReward is shown as an oracle upper bound, not a comparable baseline.Source: Table 2 · p. 9, §5.3 · p. 9

5.4 ABLATION STUDIES AND DOWNSTREAM EVALUATION

  • Under a 480-pair budget, removing Stage 1 dropped PLCC by 0.099 and SRCC by 0.061; skipping Stage 2 left a 0.074 PLCC gap, while rubric-free training remained below full RGPO.The reported ablation findings concern scorer warm-up, query-adaptive rubrics, and the rubric intermediate representation.Source: §5.4 · p. 9, Table 3 · p. 9
  • Wan-2.2-A14B fine-tuning with RewardVerse raised Imaging Quality from 0.640 to 0.653, VBench-Quality from 0.808 to 0.809, and VBench-Text from 0.428 to 0.446.The comparison is for GRPO fine-tuning on Visual Quality.Source: §5.4 · p. 9

6 CONCLUSION

The conclusion presents RewardVerse and RGPO as a rubric-guided approach to video reward modeling. It reports reduced scalar drift and strong evaluation results with 30 preference pairs per dimension.

6 CONCLUSION

  • RewardVerse inserts a dynamic rubric into pointwise video reward modeling, while RGPO provides its training algorithm.Source: §6 · p. 9
  • Inserting a dynamic rubric improves score resolution, and RGPO joint optimization further alleviates the remaining scalar drift.The conclusion identifies seed-guided warm-up and joint optimization as essential stages.Source: §6 · p. 10
  • With 30 preference pairs per dimension, RewardVerse achieves high pointwise correlation on EvalVerse, strong pairwise agreement on VGRB, and improves downstream video generation while being less prone to reward hacking.Source: §6 · p. 10

A APPENDIX

The appendix points readers to the project page for code and models.

A APPENDIX

  • Code, models, and the project page are available at https://github.com/2kxx/RewardVerse.Source: A APPENDIX · p. 13

A.1.1 FULL 4-VARIANT ABLATION FOR QWEN2.5-VL

The four-variant ablation reports pair accuracy on the subset of video pairs that receive distinct scores; this can overstate real-world separability.

A.1.1 FULL 4-VARIANT ABLATION FOR QWEN2.5-VL

  • V1 and V4 produce distinct scores for only a small portion of pairs, so pair accuracy on the decided subset overstates real-world separability.Source: §A.1.1 · p. 13

A.1.2 LARGE-SCALE EVALUATION OF GEMINI-3.1-PRO ON 1K TEST SET

On the complete 1K pointwise test set, adding rubrics to Gemini-3.1-Pro improved correlation with human ratings, while calibration remained imperfect. The score distribution moved closer to human rating variance, but the model continued to underestimate ratings.

A.1.2 LARGE-SCALE EVALUATION OF GEMINI-3.1-PRO ON 1K TEST SET

  • The evaluation used natural-language float outputs because the closed-source API restricts access to token-level logits, comparing V1 without rubrics against V4 with rubrics.Table 5 covers PLCC, SRCC, standard deviation, mean scores, biases, and discrete rating distributions across 16 evaluation dimensions.Source: §A.1.2 · p. 13, Table 5 · p. 13
  • PLCC increased by +0.103 (21.0% relative) and SRCC by +0.084 (17.5% relative) from V1 to V4 on the 1K test set.The authors report that rubric-guided sub-dimensions align more closely with human multi-dimensional preferences.Source: §A.1.2 · p. 13, Table 5 · p. 13
  • Gemini-3.1-Pro remained negatively biased: mean bias shifted from −0.398 in V1 to −0.584 in V4.The authors note that zero-shot static rubrics alone do not eliminate systematic score-scale biases across foundation models.Source: §A.1.2 · pp. 13–14
  • The uncalibrated standard deviation fell from 1.500 in V1 to 1.158 in V4, bringing rating variance closer to the human baseline despite worsened calibration.Figure 4 reports this distribution comparison over the complete 1K pointwise test set.Source: Figure 4 · p. 14

A.1.3 SCALAR DRIFT ACROSS EXISTING REWARD MODELS

Across six reward models on a shared 10-dimensional test subset, score compression or distribution shifts from human ratings were common. Q-Scorer had a broader distribution, but it still did not match the human distribution.

A.1.3 SCALAR DRIFT ACROSS EXISTING REWARD MODELS

  • Most of the six reward models showed narrow score ranges or distributions shifted from human ratings, indicating that compression or scale shift was common.Predictions were remapped to [1, 5] only to visualize distributions; this remapping was not part of the models' native output ranges.Source: §A.1.3 · p. 14
  • VideoReward and VideoScore-v1.1 concentrated mainly at low scores, while VisionReward and VideoScore-v2 placed many predictions near the upper end.UnifiedReward also produced several narrow score peaks.Source: §A.1.3 · p. 14
  • Q-Scorer produced the only relatively broad and continuous distribution, but its mean and overall distribution still differed from human ratings.Figure 5 overlays model predictions and human labels on the shared 10-dimensional subset.Source: §A.1.3 · p. 14

A.1.4 PER-DIM BREAKDOWN

This appendix section documents RewardVerse’s evaluation behavior and scoring protocol, including implementation, inference cost, and limitations, and clarifies how rubric-based scoring relates to CoT. It also outlines future directions for the framework.

A.1.4 PER-DIM BREAKDOWN

  • +0.771 pooled bias characterized deployed V3, with 15 of 16 dimensions positively biased; pacing (+1.68), consistency (+1.64), and character (+1.48) had the largest gaps, while rhythm was −0.23.The analysis distinguishes overestimation on higher-level temporal and consistency dimensions from minimal bias on low-level sensory dimensions.Source: Figure 6 · p. 15, §A.1.4 · p. 15

A.1.5 PROTOCOL SELECTION: HOW TO USE THE RUBRIC

  • PR4 uniquely combined 0% format failures, continuous scalar outputs, and 100% decided pairs; PR2 and PR3 failed at 46.7% and 53.3%, while PR1 lacked scalar rewards.PR1's perfect pairwise accuracy did not make it suitable as an RL reward signal, and it was sensitive to input ordering.Source: Table 6 · p. 16, §A.1.5 · p. 16

A.1.6 STABILITY UNDER PROMPT PERTURBATION AND PAIR SWAP

  • V3 expanded score standard deviation from 0.258 to 0.471 (82.6%) with comparable normalized prompt variation (0.18 vs. 0.20); rubric injection cut pair-swap flips from 0.792 to 0.562.The residual 56.2% flip rate remained high, leaving pairwise A/B comparison vulnerable to position bias and unsuitable as an RL reward signal.Source: §A.1.6 · pp. 16–17, Table 8 · p. 17

A.1.7 SCALAR DRIFT BEFORE AND AFTER RGPO

  • +0.610 to +0.164: RGPO reduced mean bias, while PLCC/SRCC rose from 0.240/0.222 to 0.505/0.458 and prediction standard deviation increased from 0.457 to 1.332.Absolute calibration bias also fell on 10 of 16 dimensions. The before-and-after comparison uses the same dynamic-rubric protocol.Source: Table 9 · p. 17, §A.1.7 · p. 17

A.2 EVALVERSE 16-DIMENSIONAL TAXONOMY; A.4.1 PROMPT TEMPLATES; A.4.2 REWARD MODEL TRAINING SETTINGS

  • RewardVerse uses 16 dimension-conditioned EvalVerse criteria and 480 preference pairs; training warms up the scorer with a fixed seed rubric, then jointly optimizes dynamic rubric generation and scoring.Rubrics use weighted, observable evaluation tips; both training stages run for one epoch.Source: §A.2 · pp. 17–18, §A.4.1 · p. 20, §A.4.2 · p. 21

A.3.2 DOWNSTREAM RESULTS

  • RewardVerse raised Imaging Quality from 0.640 to 0.653 and Human Action from 0.900 to 0.950, while VideoReward lowered VBench-Quality from 0.808 to 0.804 and VBench-Text from 0.428 to 0.392.With RewardVerse, other reported capabilities remained close to the base model; the authors interpret this as less susceptibility to reward hacking than VideoReward.Source: §A.3.2 · p. 19, Table 13 · p. 20

A.4.3 QUALITATIVE STUDY: INSIDE THE LEARNED RUBRICS; A.5 INFERENCE-TIME COST

  • Visual Quality tips stay abstract while Consistency tips ground in prompt entities; V3 adds about 18% inference overhead (2.0 vs. 1.7 s/video) over rubric-free V2, and query-based rubrics are reusable across videos.Source: §A.4.3 · pp. 21–22, §A.5 · pp. 22–23

A.6 LIMITATIONS AND FUTURE WORK

  • RewardVerse remains limited by backbone capability, lacks direct theme-level calibration, and has been tested with a 7B reward backbone, 30 pairs per dimension, and one 14B video generator.The authors call for stronger backbones, larger training sets, and more generation models to test whether gains persist at scale.Source: §A.6 · p. 23
  • RewardVerse generates rubrics from the evaluation dimension and prompt without video access, defining criteria before scoring, unlike CoT methods that summarize observed evidence (Zhao et al., 2025; He et al., 2025; Wang et al., 2026b).The authors describe rubric-guided CoT scoring as a potential combination of the approaches.Source: §A.6 · p. 23
  • Future work will explore stronger MLLM backbones, direct theme-level calibration, larger-scale training, downstream RL experiments, and rubric-guided CoT scoring.Source: §A.6 · p. 23

Related papers

Loading 2609.22947v2…