Source-linked AI summary
Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
Nai-Xin Zhai, Weihua Cheng, Dexu Yu, Yikai Gu, Hanwen Du, Junchen Fu, Chenxi Huang, Yingwei Song, Liyuan Lillian Ma, Yang Ran, Youhua Li, Yongxin Ni
TL;DR
Video generation alignment is limited by noisy preference data, scalar reward representations, and policy objectives that may miss global preference structure. The paper addresses these issues with calibrated supervision, multidimensional reward distributions, and Wasserstein-based GRPO. Experiments report more reliable reward signals and greater consistency between generated videos and human preferences.
Problem
Human-aligned video generation is challenged by noisy and biased preference data, scalar reward models, and locally constrained KL-based policy optimization.
Method
The framework calibrates annotations with elite-guided filtering, models quality as a multidimensional reward distribution, and applies Wasserstein alignment in reward modeling and GRPO.
Results
The framework significantly improves reward-signal reliability and the consistency of generated videos with human preferences.
Takeaways & Limitations
Preference-aware alignment can be addressed jointly at the data, reward-representation, and policy-optimization levels.
Takeaways & Limitations
High-complexity samples can still produce substantial inter-rater disagreement and remain unsuitable for model training.
Abstract
from arXiv · showhide
Video generation is central to AI-powered content creation. Aligning generated videos with human preferences is a key criterion for evaluating generation quality. Despite significant progress in visual quality, three key challenges remain. First, the reliability of reward signals is constrained by the quality of human preference data, which is often affected by subjective noise and bias. Second, standard scalar reward models collapse multi-aspect human preferences into a single value, leading to the loss of dynamic trade-offs across multiple preference dimensions. Third, in policy optimization, the widely adopted KL divergence imposes primarily local constraints and may fail to capture the global structure of human preferences. To address these challenges, we propose a unified preference-aware learning framework for video generation. First, we introduce elite-guided filtering to calibrate preference data and construct reliable supervision for reward model training. We then model video quality as a multidimensional reward distribution to capture the uncertainty inherent in human preferences, and use the Wasserstein distance to align the learned reward distribution with the empirical human preference distribution. Finally, we introduce Wasserstein-based distributional alignment into GRPO, guiding policy optimization to better match the global structure of human preferences over videos. Experiments on reward modeling and video generation demonstrate that our approach improves the reliability of reward signals and the perceptual consistency of generated videos. Our code is available at https://github.com/alignhs26/ahs.
1 Introduction
Video generation quality has improved, but aligning outputs with multidimensional human preferences remains a critical bottleneck. The proposed framework addresses unreliable preference data, scalar reward representations, and locally constrained policy optimization through calibrated supervision and distributional alignment.
- Motivation: Video generation has improved in temporal coherence and perceptual fidelity, shifting attention toward alignment with human preferences.Human judgment is multidimensional and context-sensitive, making preference alignment a critical bottleneck for generated video quality.
- Challenges: Existing alignment methods face unreliable human preference data affected by subjective noise and bias.This can make learned rewards deviate from the accurate preference distribution.
- Challenges: Scalar reward models compress interrelated preferences such as semantic plausibility, visual naturalness, and temporal coherence into one value.This representation fails to capture multidimensional preference structure.
- Challenges: KL-based policy optimization imposes local denoising-step constraints and may miss temporal dependencies and semantic evolution across steps.The paper associates this limitation with motion-conservative outputs that fail to capture complex dynamic interactions.
- Proposed Framework: The framework calibrates preference data, models quality as a multidimensional reward distribution, and aligns policy optimization with human preferences.Its three-level design targets data reliability, reward representation, and policy alignment.
2 Related Work
Related work improves video alignment through calibrated feedback, multidimensional reward modeling, and preference optimization, while video generation itself has advanced through diffusion and flow-matching models. The framework builds on these directions by combining calibration, reward modeling, and Wasserstein-based policy optimization.
- Data Calibration: Data-calibration research addresses costly and inconsistent human annotations through selective use of high-quality data and correction of noisy supervision.Automated surrogates such as reward models have been explored for this purpose.
- Human Feedback Alignment: Visual-generation alignment studies collect human preferences, train reward models, and optimize generative models.VideoScore uses multidimensional ratings, while VisionReward models fine-grained preferences including composition, motion, and prompt consistency.
- Video Generation: Video generation has advanced through diffusion and flow-matching models, but outputs can remain misaligned with semantic intent and user judgments.Reported mismatches include semantic misinterpretation, visual deviation from descriptions, and disagreement with perceived quality.
- Framework Overview: The framework combines elite-guided calibration, VLM-based reward modeling, and Wasserstein-based GRPO in a three-stage pipeline.These stages respectively calibrate data, fit a reward model, and fine-tune the video generator.
- Open Alignment Challenge: Human preference data remain pivotal because alignment depends on accurately capturing and modeling subjective standards.This dependency applies across diffusion and flow-matching optimization settings.
3 Method
The method calibrates noisy preference annotations, represents video quality as multidimensional reward distributions, and applies Wasserstein alignment during reward modeling and policy optimization. These stages expand reliable supervision and align generated-video behavior with global preference structure.
- Elite-Guided Preference Calibration: Elite-Guided Preference Calibration selects high-consistency annotations, trains an elite scorer, and calibrates lower-consistency samples to expand reliable supervision.Samples enter the high-quality subset when ratings agree or contain at most one outlier within one point of the modal rating.
- Elite-Guided Preference Calibration: The calibrated low-quality samples are merged with original high-quality samples into an augmented supervision set for reward modeling.This preserves expert-consensus structure while expanding the scale of consistent preference supervision.
- Preference-Aware Reward Modeling: Preference-Aware Reward Modeling represents video quality as a probability distribution over a K-dimensional score space instead of a single scalar.The empirical target is represented by δ_yi, while the model predicts a conditional distribution over multidimensional preference scores.
- Preference-Aware Reward Modeling: The reward-model objective combines mean squared error with a Wasserstein-distance term aligning predicted and empirical preference distributions.The coefficient λ > 0 controls the relative importance of Wasserstein alignment.
- Preference-Guided Policy Optimization: Wasserstein-based GRPO replaces KL divergence in policy optimization with a global behavioral alignment constraint between the output and reference policies.The objective combines the clipped GRPO surrogate with a Wasserstein penalty, while group-relative advantages normalize trajectory rewards.
- Preference-Guided Policy Optimization: GRPO samples candidate trajectories, computes new-to-old policy importance ratios, clips timestep updates, and averages objectives across timesteps and groups.Group-relative advantages preserve relative preference relationships among candidates while reducing reward-scale variance.
4 Experiments
Experiments evaluate calibrated reward modeling and preference-guided policy adaptation across datasets, models, metrics, and ablations. Calibration improves reward reliability and generated-video quality, while Wasserstein alignment strengthens performance across reward modeling and policy optimization.
- Experimental Setting: The study uses Video-Bench for multi-dimensional reward modeling and 50,000 VidProM prompts for text-conditional video policy training.Video-Bench contains 8,799 video samples rated by four annotators across nine dimensions.
- Experimental Setting: Calibration isolates high-consistency annotations, trains an elite scorer, and reweights or filters remaining samples for reward-model training.The reward model fine-tunes Qwen2-VL-2B with LoRA, while policy adaptation uses three open-source video generators.
- Main Results: 60.61 with ties and 83.33 without ties on Action Consistency, and 71.72 with ties and 87.88 without ties on Motion Effectiveness, are achieved with the full calibrated dataset.These pairwise-accuracy results indicate improved evaluation of both action consistency and motion effectiveness over raw-data training.
- Main Results: 2.1% for Wan2.1, 2.9% for CogVideoX, and 3.2% for ModelScope are the reported total-score increases after preference-guided optimization.CogVideoX also improves from 86.53 to 90.25 in Motion Smoothness and from 31.33 to 31.99 in Dynamic Degree, with a small Subject Consistency decrease.
- Ablation Study: 3.39 points on Motion Effectiveness and the increase from 58.98 to 60.61 on Action Consistency result from adding Wasserstein loss to reward modeling.The loss constrains transport between predicted reward and human-preference distributions rather than only local deviations.
- Ablation Study: 69.58 with Wasserstein regularization exceeds 67.97 without regularization and 68.02 with KL regularization in policy optimization.Wasserstein achieves stronger results across motion and scene-coherence metrics, including Motion Smoothness 89.32, Spatial Relation 63.23, and Background Consistency 92.55, although KL is slightly better on Dynamic Degree.
- Further Analysis: 28.8% of Action samples and 42.9% of Motion samples are noisy, while replacing raw rewards with calibrated rewards consistently improves both FlowGRPO and DanceGRPO.The analysis links substantial annotation noise with the need for calibration and shows that the calibrated reward signal is not optimizer-specific.
5 Conclusion
The framework aligns video generation with human perceptual judgments through calibrated preference data, multidimensional reward distributions, and Wasserstein-based policy alignment. Experiments show improved reward-signal reliability and consistency between generated videos and human preferences.
- The framework uses elite-guided filtering to calibrate preference data for more reliable alignment with human perceptual judgments.
- It represents human preference through a multidimensional reward distribution rather than a single scalar value.
- Wasserstein-based distributional alignment is integrated into GRPO to match the global structure of human preference.
- The framework improves reward-signal reliability and the consistency of human preference in generated videos.