Improving Video Generation with Human Feedback

• Older preference datasets [19] [73] [47] [76] may miss fine detail and long-range dynamics, while under-explored reward-model designs can yield noisy, biased, exploitable supervision.Source: §1 · p. 1
• The dataset contains 182k annotations from 16k high-quality prompts rendered by 12 representative text-to-video models across visual quality, motion quality, and text alignment.Source: §1 · p. 2
• VideoReward is a multi-dimensional video reward model whose design choices and annotation strategies are systematically studied, with dimensions aggregatable into a total human-preference reward.Source: §1 · p. 2
• The unified KL-regularized reinforcement-learning framework introduces Flow-DPO and Flow-RWR for training, plus Flow-NRG for inference-time alignment.Source: §1 · pp. 2–3
• Flow-NRG guides noisy videos during denoising and lets users weight multiple objectives at inference, avoiding extensive retraining for personalized requirements.Source: §1 · p. 2
• VideoGen-RewardBench adds 26.5k preference-labeled video pairs from diverse prompts and videos generated by modern video diffusion models.Source: §1 · p. 3
• Flow-DPO’s fixed KL term outperforms other methods and avoids the timestep-dependent KL overfitting that makes a simple Diffusion-DPO extension vulnerable to reward hacking.The paper reports robust performance across all dimensions for Flow-DPO.Source: §1 · pp. 2–3

2 Related Work

• Most video reward models target short, low-quality pre-Sora videos and lack rigorous evaluation of design choices.The paper contrasts these limitations with its focus on modern video generation and broader reward-modeling strategies.Source: §2 · p. 3
• Reward models train CLIP-based models on human preferences [75] [77] [31] [86] [41], while newer VLM approaches predict multidimensional scores using regression heads.Learning paradigms include point-wise regression [19] [76] and pair-wise comparison via Bradley-Terry loss.Source: §2 · p. 3
• Image-generation alignment includes reward backpropagation [58] [77], RWR [36], DPO [61] [71], PPO [2], and training-free methods [80].Concurrent work extends DPO/RWR to diffusion-based video models using older generation models or image-level rewards [17] [47] [76] [85] [73]; this paper extends DPO to flow-based video generation as Flow-DPO.Source: §2 · p. 3

3.1 Human Preference Data Collection

• 12 T2V models generated 108k videos from 16k prompts, forming 182k prompt-and-two-video annotated triplets.The dataset targets modern models because legacy preferences from earlier, shorter, lower-resolution generations may not reflect current user preferences.Source: Table 1 · p. 3, §3.1 · pp. 3–4

3.2 Reward Modeling

• BT reward models consistently outperform score regression as training data grows, because pairwise preferences capture subtle relative quality differences.Accuracy is averaged across Visual Quality, Motion Quality, and Text Alignment on validation data.Source: Figure 2 · p. 4, §3.2 · p. 4
• BTT clusters tied pairs near zero while preserving large margins for decisive wins and losses, yielding more reliable feedback than vanilla BT.The model uses θ = 5.0 to control tie probability.Source: Figure 3 · p. 4, §3.2 · p. 5

3.1 Human Preference Data Collection

• Annotators recorded pairwise wins, ties, and losses for Visual Quality, Motion Quality, and Text Alignment, plus 1–5 video-level Likert scores; 13 000 triplets form validation.Validation prompts never appear in training.Source: §3.1 · p. 4

3.2 Reward Modeling

• Dimension-specific tokens separate context-independent visual and motion scoring from prompt-aware text alignment, preventing context leakage in a Qwen2-VL-2B reward model.The [VQ] and [MQ] tokens follow the video, while [TA] follows the full prompt; a shared linear layer maps their final-layer embeddings to scores.Source: §3.2 · pp. 4–5

4.1 Preliminaries

• The preliminaries define rectified flow by interpolating between data and noise, then frame RLHF as maximizing reward while regularizing divergence from the reference model.Source: §4.1 · p. 5

4.2 Flow-DPO

• Flow-DPO moves predicted velocities toward preferred videos and away from less-preferred ones using human preference pairs.The derived βt = β(1 − t)^2 schedule caused reward hacking and visual artifacts, so the method instead uses constant β for more stable training and improved alignment across reward dimensions.Source: §4.2 · p. 6

4.3 Flow-RWR

• Inspired by Reward-weighted Regression (RWR) in diffusion models [36] [17], the authors propose an expectation-maximization-based counterpart for flow models, formulated for rectified-flow models as reward-weighted velocity regression; as in Flow-DPO, they omit the (1 − t)^2 factor for better performance.Source: §4.3 · p. 6

4.4 Noisy Reward Guidance

• Flow-NRG shifts the velocity field by a reward-gradient term weighted by guidance strength, enabling reward guidance during inference.To reduce the cost of pixel-space gradients, it trains a time-dependent reward model on identically noised preference pairs in latent space, reusing early pretrained-backbone layers and applying guidance at each inference step except t = 1.Source: §4.4 · p. 7

5.1 Reward Learning

• VideoGen-RewardBench adds 26.5k human-annotated triplets to evaluate modern T2V models, while GenAI-Bench evaluates reward models on earlier-generation outputs.Evaluation uses pairwise accuracy, including ties-included and ties-excluded reporting; VideoGen-RewardBench reports VQ, MQ, and TA accuracy, while GenAI-Bench reports overall accuracy.Source: §5.1 · pp. 7–8

5.2 Video Alignment

• Flow-DPO with constant β improves over the pretrained model on VQ, MQ, and TA, outperforming SFT and Flow-RWR without the reward hacking seen with timestep-dependent β.The timestep-dependent variant has high VQ/MQ reward win rates but underperforms the pretrained model on TA; human studies confirm the multi-dimensional findings.Source: Table 3 · p. 8, §5.2 · p. 9
• Flow-DPO with constant β achieves the best performance across all datasets on the single-dimensional text-alignment task.The comparison is reported for TA; the table caption identifies constant-β Flow-DPO as best without reward hacking.Source: Table 4 · p. 8, §5.2 · p. 9

5.1 Reward Learning

• VideoReward outperforms the other reward models on VideoGen-RewardBench and remains comparable on GenAI-Bench despite training on a disjoint dataset.The paper attributes weaker modern-video performance by VideoScore, LiFT, and VisionReward to limited generalization or difficulty judging improved fidelity and motion smoothness.Source: §5.1 · p. 8

5.2 Video Alignment

• Alignment experiments fine-tune a rectified-flow Transformer with LoRA, use VideoReward for training-based reward signals, and compare aligned models with pretrained outputs using automatic and human assessments.SFT uses chosen data; the evaluation uses matched random seeds, two annotators per sample with a third resolving disagreements, and prompts from Vbench, VideoGen-Eval, and TA-Hard.Source: §5.2 · pp. 8–9
• Constant β consistently outperforms timestep-dependent β for TA across tested β values, possibly because varying β causes uneven training with shared weights across noise levels.The timestep-dependent schedule is β_t = β(1 − t)^2 [18].Source: Figure 6 · p. 9, §5.2 · p. 9
• Reward guidance combines dimension rewards with user-chosen weights and backpropagates gradients to noisy latents, enabling custom trade-offs.The reported guidance strength is w = 100.Source: §5.2 · p. 9, Table 5 · p. 9

6 Conclusion

• The authors present a preference dataset containing 182k human annotations on visual quality, motion quality, and text alignment for modern video generation models.Source: §6 · p. 10
• They introduce VideoReward, a multi-dimensional video reward model, and establish VideoGen-RewardBench for more accurate and fair evaluation.Source: §6 · p. 10
• From a unified reinforcement learning perspective, they propose three alignment algorithms tailored to flow-based video generation and demonstrate their effectiveness in practice.Source: §6 · p. 10

7 Limitations & Future Work

• Excessive Flow-DPO training degraded video quality despite dimension-specific alignment gains; future work includes joint supervised data, other conditional tasks, robust rewards, and PPO.LoRA was used to prevent the decline; reward hacking remains possible because differentiable rewards can be manipulated even with tie annotations.Source: §7 · p. 10

Appendix of Improving Video Generation with Human Feedback

• Appendix B gives derivations and proofs for Flow-DPO, Flow-RWR, and Flow-NRG; C provides pseudo-code for Flow-DPO and Flow-NRG.D describes the internal video diffusion model architecture, while I lists reward-modeling and alignment hyperparameters and K gives the reward-model input template.Source: Appendix of Improving Video Generation… · p. 16
• Appendices E–H and J–L cover reward-model ablations, dataset statistics, annotation guidelines, evaluation, visual comparisons, and prompts.H includes benchmark comparisons, evaluation procedures, and metrics; J compares the original and Flow-DPO-aligned models, while L includes TA-Hard prompts.Source: Appendix of Improving Video Generation… · p. 16

A Related Works

• Prior evaluation moved from FID [20] and CLIP scores [60] toward human-preference-trained CLIP models [75] [77] [31] [86] [41] and VLM reward models [1] [72].VLM reward models commonly replace the token-classification head with a regression head to predict multidimensional scores.Source: A Related Works · p. 16
• Video reward learning uses point-wise scores or labels [19] [76], pairwise preferences [74], or VLM-as-a-judge [73] [39] [44], while this work targets modern video models and broader reward-model strategies.The passage notes that pairwise comparison is largely unexplored for video reward models and that prior models mostly address pre-Sora systems with shorter, lower-quality videos.Source: A Related Works · p. 16
• Image-alignment methods include reward backpropagation [58] [9] [77] [59], RWR [54] [36] [17], DPO [61] [71] [42] [14] [78] [83] [47] [85] [17], PPO [65] [2] [87] [7], and training-free methods [80] [68].Unlike prior reward guidance on noised images, Flow-NRG trains noise-aware rewards for gradients; Flow-DPO applies DPO [61] [71] to video flow matching, and constant KL constraints improve certain tasks over timestep-dependent constraints.Source: A Related Works · p. 17
• Concurrent video-alignment methods include unified offline DPO/RWR [17], pair reweighting in VideoDPO [47], VisionReward [76], OnlineVPO [85], and synthetic-data RWR in LIFT [73].These methods use diffusion-based models, whereas this work targets flow-based generation; fixing Flow-DPO's timestep-dependent β resolves suboptimal performance on certain tasks, and Flow-NRG trains noise-aware rewards on noised videos unlike SPO [42].Source: A Related Works · p. 17

B.1 Relation beween Velocity and Noise

• Lemma B.1 relates the L2 error of noise prediction to the L2 error of the predicted velocity field for real-data and noise samples.Source: §B.1 · p. 17
• The proof sets up the time-dependent rectified flow and writes its marginal and conditional velocity fields before substituting between them.Source: §B.1 · pp. 17–18

B.2 Reward as Classifier Guidance

• Reward guidance samples from a tilted path proportional to p_t(x_t|y)^(1-w)[exp(r(x_t,y))]^w, combining the conditional path and reward.The derivation expresses this as a weighted blend of their log densities.Source: §B.2 · p. 18
• The stated sampling guarantee depends on the tilted path being close to a marginal probability path, an assumption the appendix says is unclear in practice.Source: §B.2 · p. 18
• With reward scaling alone, the guided path is proportional to p_t(x_t|y)[exp(r(x_t,y))]^w, replacing the original sampling target.Source: §B.2 · p. 19

C Pseudo-code of Flow-DPO and Flow-NRG

• Flow-DPO code samples a timestep and shared noise for preferred and non-preferred latents, compares model and frozen-reference velocity errors, then applies a sigmoid loss.The code identifies the preferred and non-preferred inputs as latents and the conditioning as text.Source: C Pseudo-code of Flow-DPO and Flow-NRG · p. 20
• Flow-NRG computes a reward-weighted gradient with respect to latents and uses it to adjust the predicted velocity before updating the latents.The pseudocode also includes classifier-free guidance and scales the reward-gradient adjustment by time and a guidance factor.Source: C Pseudo-code of Flow-DPO and Flow-NRG · p. 20

• The internal text-to-video foundation model is a 1B-parameter Transformer-based latent diffusion model [53].Source: D Architecture of Internal Video… · p. 21
• A 3D VAE encodes video into latent space, and a Transformer-based video diffusion model operates on that representation.Source: D Architecture of Internal Video… · p. 21
• Each Transformer block sequences 2D spatial attention, 3D self-attention, cross-attention, and FFN modules.Source: D Architecture of Internal Video… · p. 21
• The model is trained under the Flow Matching framework.Source: D Architecture of Internal Video… · p. 21

E Ablation Study of Our Reward Model

• The Bradley-Terry model slightly outperforms regression, Bradley-Terry-With-Tied improves ties-included accuracy, and separate attribute tokens further improve performance.Bradley-Terry-With-Tied matches Bradley-Terry on ties-excluded accuracy; separate tokens represent reward attributes individually rather than sharing a last token.Source: E Ablation Study of Our Reward Model · p. 21, Table 6 · p. 21
• Noised-latent training yields a 74.6 MQ win rate, versus 38.6 for clean-latent training, which underperforms the unguided baseline.Directly predicting x0 from xt and backpropagating via r(x0, y), following Yu et al. [81], produces unusable videos without meaningful content.Source: E Ablation Study of Our Reward Model · p. 21

F Statistics of Training Data

• The training dataset contains 182k samples, and includes recent T2V models with ≥720p resolution and 3–6s clips, unlike prior datasets’ mostly ≤480p, 2s videos.The passage identifies VideoScore [19] and LIFT [73] as examples of prior reward models; dataset statistics are presented in Table 7 and Figure 8.Source: F Statistics of Training Data · p. 22, Table 7 · p. 22, Figure 8 · p. 22

G Details of Human Annotation

• Three independent annotators evaluate each sample, using detailed guidelines, training sessions, and reference examples to clarify the scoring criteria.An additional reviewer resolves disagreements, and the reviewer’s evaluation determines the final label.Source: G Details of Human Annotation · p. 22
• Training and validation annotations combine pairwise preferences with pointwise VQ, MQ, and TA scores; VideoGen-RewardBench uses pairwise preferences for those dimensions and Overall Quality.The benchmark’s annotation protocol adds Overall Quality and uses pairwise preferences only.Source: G Details of Human Annotation · p. 22
• Annotators flag unsafe content, and videos identified as unsafe are excluded from the training and evaluation dataset.Source: G Details of Human Annotation · p. 22

H.1 Evaluation Benchmarks

• GenAI-Bench covers earlier-generation models with 10 models, 508 prompts, and 1.9k pairs; Figure 10 contrasts its model coverage with VideoGen-RewardBench and training sets.GenAI-Bench videos are typically lower resolution (around 320x512) and shorter (2s–2.5s), whereas the modern benchmark targets state-of-the-art models.Source: §H.1 · pp. 23–24, Figure 10 · p. 24, Table 9 · p. 24
• VideoGen-RewardBench is the primary modern-model benchmark, with 12 models, 420 prompts, and 26.5k annotated pairs spanning four preference dimensions.Built from VideoGen-Eval, it covers higher-resolution, longer videos and annotates Visual Quality, Motion Quality, Text Alignment, and Overall Quality.Source: §H.1 · p. 24

H.2 Comparison Methods

• A random-score baseline controls for metric and benchmark-distribution effects, with expected ties-excluded accuracy of 1/2 and ties-included accuracy of max(1/3, tie-label proportion).For each prompt and video pair, it samples both scores from a standard normal distribution.Source: §H.2 · p. 25
• VideoScore uses pointwise MSE preference scores, while LiFT uses instruction-tuned discrete scores; VideoScore’s earlier-model training favors GenAI-Bench, and LiFT ties are randomized for tie-excluded accuracy.VideoScore averages five dimensions for evaluation; LiFT scores Video Fidelity, Motion Smoothness, and Semantic Consistency on a 1–3 scale.Source: §H.2 · p. 25
• VisionReward converts 64 binary checklist judgments into weighted scores and performs well on GenAI-Bench with reasonable VideoGen-RewardBench capability.Its training includes pre-SOTA-era and recent open-source models.Source: §H.2 · p. 25
• The paper’s reward model uses QWen2-VL-2B with pairwise BTT training and averages normalized scores; GenAI-Bench overall scoring uses predicted Text Alignment scores only.For VideoGen-RewardBench, it samples at 2 FPS and 448×448 resolution; for GenAI-Bench, it samples at 2 FPS and 256×256 resolution.Source: §H.2 · p. 25

H.3 Evaluation Metrics

• Evaluation reports ties-included and ties-excluded accuracy, excluding ties in the latter and selecting the best three-class accuracy over tie thresholds in the former.The tie-calibration procedure follows Deutsch et al. [11].Source: §H.3 · p. 25

I Hyperparameters

• Alignment experiments apply LoRA to transformer linear layers because full-parameter fine-tuning can degrade performance or cause model collapse.Tables 10 and 11 present alignment-algorithm and reward-model hyperparameters, respectively.Source: I Hyperparameters · p. 26, Table 10 · p. 26, Table 11 · p. 26

J Additional Qualitative Results

• Figures 11 and 12 provide additional visual comparisons of videos generated by the original and Flow-DPO aligned models.The passages identify the comparisons but do not state a qualitative winner.Source: J Additional Qualitative Results · p. 26, Figure 11 · p. 27, Figure 12 · p. 28

K Input Template for Reward Model

• The reward-model template requests independent 0–10 ratings for Visual Quality, Motion Quality, and Text Alignment.Each category is scored separately, with 0 the worst and 10 the best.Source: K Input Template for Reward Model · p. 29
• Visual Quality covers reasonableness, clarity, detail, aesthetics and creativity, and safety, emphasizing static video factors.Harmful or inappropriate content is assigned the lowest image-quality and satisfaction score.Source: K Input Template for Reward Model · p. 29
• Motion Quality evaluates stability, naturalness, smoothness, fusion with the background, motion clarity, and amplitude.Largely static videos or videos with little movement should receive a low motion-quality score.Source: K Input Template for Reward Model · p. 29
• Text Alignment assesses whether subjects, actions, environments, style, and camera movement match the textual prompt.Some stylistic adaptation of real-world scenes is acceptable.Source: K Input Template for Reward Model · p. 29

L Prompt Subset of TA-Hard

• The TA-Hard subset includes varied animal scenarios, from a rabbit-and-turtle race to dolphins and sea turtles exploring a coral reef.Other prompts pair animals in activities such as soccer, stargazing, singing, and playing a piano duet.Source: L Prompt Subset of TA-Hard · p. 30
• Other prompts combine humans, robots, and fantasy characters in coordinated activities, including basketball, archery, talent shows, and cooking.Examples include a robot and a human with a cybernetic arm playing basketball and a knight and wizard practicing archery.Source: L Prompt Subset of TA-Hard · p. 30
• The subset also features animated objects and fantastical figures in scenarios such as gift-wrapping, hopscotch, alchemy, and potion-making.Examples include anthropomorphic sneakers and a basketball playing one-on-one, and a plague doctor mixing remedies in an apothecary.Source: L Prompt Subset of TA-Hard · p. 30