PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Video Generation with Human Feedback

Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang, Wanli Ouyang

arXiv:2501.13918v2cs.CVcs.AIcs.GRcs.LG

TL;DR

Current video generators still exhibit unstable motion, imperfect text-video alignment, and limited alignment with human preferences, especially when adapting alignment methods to flow-based models. The paper builds a multidimensional human-preference pipeline with VideoReward and three flow-based alignment algorithms; VideoReward outperforms existing reward models, while Flow-DPO performs best among alignment methods and Flow-NRG supports custom objective weighting.

  • Problem

    Current video generation systems face unstable motion, imperfect text-video alignment, and insufficient alignment with human preferences, while adapting existing alignment methods to flow-based models remains problematic.

  • Method

    The paper constructs a 182k-annotation multidimensional preference dataset, develops VideoReward, and introduces Flow-DPO, Flow-RWR, and Flow-NRG from a KL-regularized reinforcement-learning perspective.

  • Results

    VideoReward outperforms all other models on VideoGen-RewardBench, while Flow-DPO with a fixed KL term outperforms the other alignment methods.

  • Takeaways & Limitations

    Flow-NRG enables users to customize the weights of multiple alignment objectives during inference, supporting personalized video quality preferences.

  • Takeaways & Limitations

    The VideoReward model still carries a potential risk of reward hacking, with human assessments indicating a marked decrease in video quality.

Abstract

from arXiv · show

Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.

1 Introduction

Modern video generators still exhibit unstable motion, imperfect text-video alignment, and weak alignment with human preferences. The paper addresses these issues with modern preference data, VideoReward, and three flow-based alignment algorithms.

  • Motivation: Existing video preference datasets target earlier short, low-resolution models, limiting reward reliability for modern video generation.Such data may miss fine spatial detail and long-range dynamics while producing noisy, biased, and exploitable supervision.
  • Motivation: Flow-based models predict velocity rather than noise, creating new adaptation questions for DPO and RWR alignment methods.A prior flow-matching DPO attempt degraded quality relative to an unaligned baseline.
  • Contributions: The authors construct 182k human-labeled examples from 12 modern text-to-video models across visual quality, motion quality, and text alignment.The dataset is built from 16k high-quality prompts and its dimensions can be aggregated into a total human-preference reward.
  • Contributions: VideoReward is a multi-dimensional reward model whose annotation and design choices are systematically studied, alongside a 26.5k-pair VideoGen-RewardBench benchmark.The benchmark contains preference labels for diverse prompts and videos generated by modern video generation models.
  • Contributions: The unified flow-alignment framework introduces Flow-DPO and Flow-RWR for training, plus Flow-NRG for inference-time reward guidance.Flow-DPO uses a fixed KL term and outperforms other methods, while Flow-NRG supports custom objective weightings during inference.

2 Related Work

Prior video reward models use human preference data and VLM-based scoring, but their design choices and applicability to modern video generation remain insufficiently explored.

  • Reward Modeling: Existing video reward models largely focus on short, low-quality pre-Sora videos and lack rigorous evaluation of design choices.The paper responds by targeting modern video generation and exploring broader reward-modeling strategies.

3 VideoReward

VideoReward addresses the mismatch between legacy video-preference data and modern text-to-video models by collecting large-scale, multidimensional human annotations and systematically studying reward-model design choices. The study compares annotation and preference-learning paradigms, including Bradley-Terry models with explicit tie handling.

  • Human Preference Data Collection: Modern video-preference datasets are needed because legacy collections use short, low-resolution clips that may miss fine spatial detail and long-range dynamics.Such datasets may also over-weight artifacts that modern text-to-video systems have already reduced.
  • Human Preference Data Collection: The dataset uses 12 T2V models to generate 108k videos from 16k prompts, forming 182k annotated triplets for modern video-preference modeling.Prompts span diverse categories, while annotators provide pairwise preferences across Visual Quality, Motion Quality, and Text Alignment, plus individual 1–5 Likert scores.
  • Reward Modeling: VideoReward investigates VLM reward-model choices including annotation paradigms, token positioning, and multi-dimensional score design.The approach uses Qwen2-VL-2B and targets cleaner, more reliable reward signals for downstream alignment.
  • Reward Modeling: Pairwise Bradley-Terry training remains consistently more accurate than pointwise score regression as the dataset grows.Pairwise annotations capture subtle relative differences even when two videos receive identical pointwise scores.
  • Reward Modeling: BTT models explicit ties, clustering tied pairs near zero while preserving larger score margins for decisive wins and losses.The model uses θ > 1 to control tie tendency and is empirically trained with θ = 5.0.

4 Video Alignment

This section develops three alignment methods for rectified-flow video generation from a KL-regularized reinforcement-learning objective: Flow-DPO and Flow-RWR for training, and Flow-NRG for inference. It also identifies timestep-dependent KL weighting as harmful for Flow-DPO and derives reward guidance by shifting the velocity field.

  • Overview: Three methods align flow-based video models: Flow-DPO and Flow-RWR train the model, while Flow-NRG guides inference.All three are derived under a unified objective that maximizes reward with KL regularization.
  • Unified RL Objective: The KL-regularized RLHF objective maximizes expected reward while constraining divergence from a reference model with coefficient β.The objective learns pθ(x0 | y) relative to pref(x0 | y).
  • Flow-DPO: Flow-DPO adapts direct preference optimization to rectified flow by training predicted velocities toward preferred data and away from less-preferred data.Its preference signal depends on differences between predicted and reference errors.
  • Flow-DPO: The derived schedule βt = β(1 − t)^2 degrades rectified-flow alignment, causing reward hacking and visual artifacts; constant β improves stability across reward dimensions.The scheduled penalty vanishes as t approaches 1 and reaches β at t = 0.
  • Flow-RWR: Flow-RWR formulates reward-weighted regression for rectified-flow models as velocity regression and omits the (1 − t)^2 factor for better performance.The method is motivated by KL-regularized reward maximization and expectation-maximization.
  • Noisy Reward Guidance: Flow-NRG applies reward guidance directly to noisy videos by shifting the rectified-flow velocity field during denoising.The guidance strength is controlled by w, and the resulting sampling distribution matches the reward-reweighted target form.

5 Experiments

Experiments evaluate reward models and flow-based alignment across benchmarks, dimensions, and human assessments. VideoReward generalizes across T2V generations, while constant-β Flow-DPO performs strongly without reward hacking and reward guidance supports custom objective trade-offs.

  • Reward Model Evaluation: VideoReward outperforms existing models on VideoGen-RewardBench and remains comparable on GenAI-Bench despite disjoint training data.VideoScore, LiFT, and VisionReward show weaker performance on modern T2V videos or limited pairwise discrimination.
  • Evaluation Setup: Evaluation combines pairwise reward-model win rates, Vbench scores, and human assessments across Vbench, VideoGen-Eval, and TA-Hard prompts.Each human-evaluated sample is reviewed by two annotators, with a third resolving disagreements.
  • Multi-dimensional Alignment: Flow-DPO with constant β significantly improves over the pretrained model across visual quality, motion quality, and text alignment, outperforming SFT and Flow-RWR.The timestep-dependent β variant underperforms the pretrained model on text alignment despite high visual and motion reward win rates.
  • Single-dimensional Alignment: Flow-DPO with constant β achieves the best performance across all datasets in single-dimensional text alignment.The table caption reports that it does so without reward hacking.
  • Reward Guidance: Reward guidance lets users steer generation toward custom trade-offs by selecting weights for multiple alignment objectives during inference.The method combines dimension rewards into a weighted sum and backpropagates guidance to the noised latent with strength w = 100.
  • β Ablation: A constant β consistently outperforms the timestep-dependent variant across tested β values.The authors attribute this to uneven training caused by varying β across timesteps when T2V models share weights across noise levels.

6 Conclusion

The paper contributes a large-scale preference dataset, a benchmark and multi-dimensional reward model, and three alignment algorithms tailored to flow-based video generation.

  • 6 Conclusion: The work presents 182k human annotations covering visual quality, motion quality, and text alignment for modern video generation models.It also introduces VideoReward and VideoGen-RewardBench, then proposes three flow-based alignment algorithms from a unified reinforcement learning perspective.

7 Limitations & Future Work

The paper identifies quality degradation and reward hacking risks, limits validation to text-to-video tasks, and outlines directions for more robust training and reward modeling.

  • Limitations: Excessive Flow-DPO training significantly deteriorates model quality despite improving alignment on specific dimensions.LoRA training was used to prevent this decline.
  • Future Work: The algorithms have been validated on text-to-video tasks, while image-to-video and other conditional generation tasks remain future extensions.The scope boundary is stated as a direction for future validation.
  • Limitations: VideoReward retains a potential reward-hacking risk because its differentiable reward function can be manipulated despite incorporating tie annotations.The authors suggest uncertainty estimates and increased data augmentation as possible directions for more robust reward models.

B Details of the Derivation

The derivation formulates reward guidance for rectified-flow models as sampling from a reward-tilted probability path. It motivates Flow-DPO and Flow-NRG implementations while identifying the practical assumption behind the guidance guarantee.

  • Velocity and score relations: The derivation connects conditional velocity fields, marginal velocity fields, and score functions along Gaussian paths.The appendix introduces the Gaussian-path setting and relates generating velocity to ∇log pt(xt|y).
  • Reward-guided probability paths: Reward guidance changes the target from pt(xt|y) to a reward-tilted distribution ˜pt(xt|y).The tilted path combines the original model distribution with exp(r(xt, y)) through guidance weighting.
  • Guarantee and assumption: Reward Guided Flows are guaranteed to sample from ˜q(·|y) at t = 1 when the tilted probability path is close to the corresponding marginal path.The text explicitly qualifies this guarantee by requiring closeness between the constructed path and the target marginal path.
  • Guarantee and assumption: The extent to which the path-closeness assumption holds in practice is unclear.This is stated as a limitation of the theoretical guarantee rather than as an established empirical failure.
  • Reward-guided probability paths: The guided velocity adds a reward-gradient term to the original rectified-flow velocity field.The displayed formulation expresses guidance as a modification of vt(xt|y) using ∇r(xt, y).
  • Implementation: The Flow-DPO implementation compares model and reference velocity errors on preferred and non-preferred latents using a regularized logistic loss.The pseudo-code samples timesteps and noise, computes four velocity errors, forms preferred-versus-rejected differences, and applies sigmoid loss.
  • Implementation: The pseudo-code applies reward guidance during inference by differentiating reward with respect to latents and updating the predicted velocity.The implementation weights multi-dimensional rewards, computes reward gradients, adjusts v_pred, and advances the latents.

D Architecture of Internal Video Diffusion Model

The internal video generator is a 1B-parameter Transformer-based latent diffusion model trained with Flow Matching. It uses a 3D VAE and combines spatial, temporal, cross-attention, and feed-forward processing.

  • Model architecture: The internal text-to-video foundation model is a 1B-parameter Transformer-based latent diffusion model trained under Flow Matching.Its video data are encoded into a latent space before diffusion modeling.
  • Model architecture: A 3D VAE encodes video data into latents for the Transformer-based video diffusion model.The architecture is explicitly described as latent rather than pixel-space video generation.
  • Model architecture: Each Transformer block sequences 2D spatial attention, 3D self-attention, cross-attention, and FFN modules.These modules provide spatial, temporal, conditioning, and feed-forward processing within each block.

E Ablation Study of Our Reward Model

The reward-model ablations examine annotation format, reward-model type, noisy-latent training, and dataset characteristics. Pairwise modeling and noisy-latent training are supported by the reported comparisons.

  • Reward-model design: The Bradley-Terry reward model slightly outperforms the regression-style variant, consistent with the advantages attributed to pairwise annotations.A Bradley-Terry-With-Tied variant is also included in the comparison.
  • Noisy-latent reward training: 74.6 MQ win rate is achieved when reward guidance is trained with noised latents on TA-Hard prompts.The clean-latent model achieves only 38.6 and underperforms the unguided baseline for intermediate noised latents.
  • Dataset characteristics: The training dataset contains 182k samples and includes recent T2V models at higher resolutions and longer durations than earlier reward-model datasets.The earlier datasets are described as mostly ≤480p and 2s, whereas this dataset includes ≥720p and 3–6s clips.
  • Annotation process: Each annotated sample is evaluated by three independent annotators using trained annotators, scoring guidelines, and reference examples.Training and validation annotations include pairwise preferences across the stated evaluation dimensions.

H.1 Evaluation Benchmarks

The evaluation uses separate benchmarks for earlier and modern text-to-video models, with reward models compared through overall and dimension-specific preference accuracy. The proposed reward model is trained on pairwise data and evaluated with tie-aware metrics.

  • Evaluation benchmarks: GenAI-Bench evaluates reward models on pre-SOTA-era T2V models, while VideoGen-RewardBench targets modern T2V models.The two benchmarks therefore cover different generations of video-generation systems.
  • GenAI-Bench: GenAI-Bench contains 10 T2V models, 508 prompts, and 1.9k preference pairs, mostly involving 2s–2.5s videos around 320x512 resolution.Its videos predominantly originate from earlier-generation models.
  • VideoGen-RewardBench: VideoGen-RewardBench is constructed from VideoGen-Eval to cover recent closed-source and open-source T2V models with high-quality data.The benchmark is manually constructed to provide broad coverage of recent advances and third-party evaluation data.
  • Compared baselines: VideoScore, LiFT, and VisionReward use different pointwise, judge-based, or checklist-based reward-modeling procedures for comparison.Their reported scores are converted or averaged to obtain overall and dimension-specific accuracies.
  • Reward-model evaluation: The proposed reward model uses QWen2-VL-2B, pairwise data, and Bradley-Terry loss, with overall accuracy averaged across three dimensions.Evaluation on VideoGen-RewardBench samples videos at 2 FPS and 448×448 resolution.
  • Reward-model evaluation: Ties-included and ties-excluded accuracy measure preference classification with and without samples labeled as ties.Pointwise baselines are converted into pairwise reward relationships before accuracy is calculated.

J Additional Qualitative Results

The section presents additional qualitative comparisons between videos generated by the original model and the Flow-DPO aligned model. It also specifies independent 0-to-10 evaluation criteria for visual quality, motion quality, and text alignment.

  • Additional Qualitative Results: Additional qualitative results compare videos generated by the original model with videos generated by the Flow-DPO aligned model.These comparisons are shown in Figures 11 and 12.
  • Additional Qualitative Results: Example prompts include animated sneakers and a talking basketball playing basketball, a person watching a sunset by a lake, and a boy petting a Border Collie.The examples cover interactions among animated objects, a reflective landscape scene, and human-animal interaction.
  • Evaluation Protocol: Each of the three categories receives an independent rating from 0 to 10, where 0 is worst and 10 is best.The evaluation separately reports Visual Quality, Motion Quality, and Text Alignment rewards.
  • Visual Quality: Visual Quality assesses reasonableness, clarity, detail richness, aesthetic and creativity, and safety.Safety concerns include harmful or inappropriate political, violent, or adult material, which receives the lowest possible satisfaction score.
  • Motion Quality: Motion Quality assesses stability, naturalness, aesthetic quality, fusion, clarity of motion, and amplitude.The rubric considers frame continuity, physically realistic movement, smooth transitions, artifact-free blending, motion clarity, and the amount of movement.
  • Text Alignment: Text Alignment assesses subject, motion, environment, style, and camera movement relevance to the textual prompt.The evaluator checks whether subjects, actions, scenes, styles, and camera behavior match the prompt.
Loading 2501.13918v2…