PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoDPO: Omni-Preference Alignment for Video Diffusion Generation

Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, Qifeng Chen

arXiv:2412.14167v1cs.CVcs.AIcs.LG

TL;DR

Video diffusion models can produce outputs that miss user preferences in visual quality and text–video alignment. VideoDPO adapts DPO with OmniScore-based preference data and reweighting, and reports improvements in both dimensions. The paper concludes that preference scoring and alignment should account for both visual quality and semantic alignment.

  • Problem

    Video diffusion generations can fall short of user preferences in visual quality and text–video semantic alignment.

    Source: §1 · p. 1

  • Method

    VideoDPO adapts DPO for video diffusion using OmniScore for visual quality and semantic alignment, automatic preference-pair generation, and score-based pair reweighting.

    Source: §1 · pp. 1–2

  • Results

    The authors report improvements in visual quality and semantic alignment across experiments on three open-source text-to-video models and multiple metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that video preference alignment should evaluate both visual quality and semantic alignment rather than treating either dimension alone as sufficient.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Recent progress in generative diffusion models has greatly advanced text-to-video generation. While text-to-video models trained on large-scale, diverse datasets can produce varied outputs, these generations often deviate from user preferences, highlighting the need for preference alignment on pre-trained models. Although Direct Preference Optimization (DPO) has demonstrated significant improvements in language and image generation, we pioneer its adaptation to video diffusion models and propose a VideoDPO pipeline by making several key adjustments. Unlike previous image alignment methods that focus solely on either (i) visual quality or (ii) semantic alignment between text and videos, we comprehensively consider both dimensions and construct a preference score accordingly, which we term the OmniScore. We design a pipeline to automatically collect preference pair data based on the proposed OmniScore and discover that re-weighting these pairs based on the score significantly impacts overall preference alignment. Our experiments demonstrate substantial improvements in both visual quality and semantic alignment, ensuring that no preference aspect is neglected. Code and data will be shared at https://videodpo.github.io/.

1. Introduction

Video diffusion outputs can miss user preferences in visual quality and text–video alignment. VideoDPO adapts DPO with multidimensional scoring, automatically generated preference pairs, and score-based reweighting, and reports improvements on both dimensions.

1. Introduction

  • Video diffusion models can fall short of user preferences in both generation quality and text–video semantic alignment.The cited passages link these issues to low-quality videos and mismatches between text descriptions and video content.Source: §1 · p. 1
  • OmniScore evaluates both visual quality and semantic alignment because these dimensions and visual-quality facets have low correlation.The authors argue that addressing one aspect does not inherently capture the others.Source: §1 · p. 1
  • The pipeline automatically generates preference pairs from multiple videos sampled for a prompt, avoiding reliance on human annotation.The highest- and lowest-scoring videos are selected as the preferred and negative samples.Source: §1 · p. 1
  • OmniScore-Based Re-Weighting uses the score frequency distribution to prioritize influential samples during training.The paper motivates higher weights for pairs with larger quality differences.Source: §1 · p. 1
  • VideoDPO reports significant performance improvements with high visual fidelity and precise semantic alignment, illustrated for VideoCrafter2 and T2VTurbo.The figure caption identifies VideoCrafter2 in the first row and T2VTurbo in the second.Source: §1 · p. 2, Figure 1 · p. 2
  • The paper presents VideoDPO as the first adaptation of DPO to video diffusion models and evaluates it on three open-source text-to-video models across multiple metrics.The experiments report improvements in visual quality and semantic alignment.Source: §1 · p. 2

2. Related Work

Prior text-to-video systems use diffusion and other generation approaches, but video quality can remain below user expectations. Related work also describes preference-alignment methods and video-evaluation benchmarks.

2.1. Text-to-Video Diffusion Models

  • Diffusion is a predominant approach to video generation, alongside non-diffusion frameworks; VideoCrafter uses a 1.4-billion-parameter U-Net, while Open-Sora and CogVideoX use DiT backbones.The text also notes parameter-efficient tuning, data-centric work, and human preference alignment as approaches to improve video quality, which can remain below user expectations.Source: §2.1 · p. 2

2.2. RLHF and RLAIF

  • DPO is a reward-model-free diffusion alignment method, but the cited work says DPO-based methods had rarely been tested on text-to-video diffusion.The passage also discusses RLHF's reward-model and reinforcement-learning stages, and video methods using reward models or reward gradients.Source: §2.2 · p. 2
  • The VideoDPO pipeline diagram shows multidimensional OmniScore ratings, highest-versus-lowest score pair selection, a score-frequency histogram, and pair reweighting during training.The caption says distinctive pairs with lower sampling probabilities are typically assigned higher weights.Source: Figure 2 · p. 3
  • Video-generation benchmarks use pretrained score models on curated human-designed prompts, while other benchmarks target specific tasks such as story, chronology, or motion quality.The passage names evaluation metrics including IS, FID, FVD, and CLIPSIM.Source: §2.2 · p. 3

3. Preliminaries

The preliminaries describe diffusion generation as iterative denoising and introduce DPO as preference alignment using positive and negative samples. They also outline the denoising model’s training objective.

3.1. Diffusion Models

  • Diffusion models generate samples by progressively denoising initial noise through a Markov chain.The forward process adds noise according to a variance schedule; α_t represents the proportion of original data retained.Source: §3.1 · p. 3
  • The denoising model is trained to predict the noise added to the original sample by minimizing the loss between the ground-truth and predicted noise.The cumulative product of α_t is also defined for the training formulation.Source: §3.1 · p. 3

3.2. Direct Preference Optimization

  • DPO aligns generative models with human preferences by training on positively and negatively labeled sample pairs.Its loss compares the losses for positive and negative samples, encouraging outputs closer to preferences.Source: §3.2 · p. 3

4. VideoDPO

VideoDPO scores generated videos for both visual quality and text-video alignment, then uses those scores to select preference pairs and reweight training. The pipeline prioritizes pairs with clearer distinctions.

4.1. OmniScore

  • OmniScore combines visual quality and text-video semantic alignment, assessing frame-level fidelity and aesthetics, temporal consistency and motion, and prompt alignment.The paper describes these dimensions as a holistic basis for preference-pair generation.Source: §4.1 · pp. 3–4

4.2. Score-Ranked Preference Data Generation

  • For each prompt, the pipeline scores generated videos with OmniScore and selects the highest-scoring video as the winner and the lowest-scoring as the negative sample.The construction process uses VidProm, a dataset of human-written text-to-video prompts.Source: §4.2 · p. 4

4.3. OmniScore-Based Data Re-Weighting

  • Pairs with clearer score distinctions receive greater training emphasis, addressing cases where winning and negative samples have nearly identical scores.The pair weight scales the DPO loss to encourage learning from clearer distinctions.Source: §4.3 · pp. 4–5
  • The reweighting scheme estimates video probabilities from OmniScore histogram frequencies and uses the pair probability to set each pair’s weight.The pair probability is defined from the winning and losing samples’ probabilities; increasing α raises the weight of lower-probability pairs.Source: §4.3 · p. 4

5. Experiment

VideoDPO is evaluated across text-to-video models using benchmark and human-preference metrics, with analyses of OmniScore, preference-pair construction, and reweighting. Results and ablations report gains across alignment dimensions and examine trade-offs in data selection and training.

5.1. Experiment Setup

  • Experiments compare VideoDPO against VC2, T2V-Turbo, CogVideo, and VADER using VBench, HPS (V), and PickScore.The reported setup trains for 3,000 steps on 10,000 prompts with four generated videos per prompt.Source: §5.1 · p. 5

5.3. Aligning Video Diffusion Models

  • All three tested models achieve their best VBench performance after VideoDPO, with gains on HPS (V) or PickScore; VideoDPO also surpasses VADER in semantic and visual quality.The reported evaluation uses VBench, HPS (V), and PickScore.Source: Table 1 · p. 5, §5.3 · p. 5

5.2. Dataset Analysis

  • OmniScore analysis examines per-prompt score ranges, overall scores, preference-pair score differences, and correlations across scoring dimensions.The score correlations are quantified with Pearson correlation coefficients.Source: §5.2 · p. 5, Figure 3 · p. 6

5.3. Aligning Video Diffusion Models

  • Qualitative comparisons show fewer visual artifacts and improved prompt accuracy within frames, alongside more stable text, objects, and colors across frames.Examples include improved vase and dog appearance, more accurate boats and character relationships, and steadier signs, giraffes, and pandas.Source: §5.3 · p. 6, Figure 4 · p. 7, Figure 5 · p. 8

5.4. Analysis

  • OmniScore reaches 81.93% on VBench, exceeding semantic-only scoring at 80.20% and aesthetic-only scoring at 79.65%.The paper presents this comparison as evidence for combining preference dimensions.Source: §5.4 · p. 6
  • Best-vs-Worst pairing performs best among the tested strategies, while removing less-distinctive pairs worsens performance, possibly by reducing prompt diversity.Best-vs-Worst forms one pair from the highest- and lowest-scoring videos for each prompt.Source: §5.4 · pp. 6–7
  • Reweighting improves performance for all tested nonzero α values; larger N also improves performance but increases data-generation cost.The optimal balance between performance gains and generation cost as N grows remains open.Source: §5.4 · p. 7

N VBench (%) HPS (V) PickScore Total Q S

The ablations examine how the number of generated videos affects performance and data-generation cost, alongside reweighting and supervised fine-tuning comparisons.

Reweighting

Supervised fine-tuning

  • VideoDPO shows a clear advantage over supervised fine-tuning for VC2 and CogVideo; the paper suggests negative samples reduce lower-quality outputs.Supervised fine-tuning uses only winning samples.Source: N VBench (%) HPS (V) PickScore Total Q S · p. 8

N

  • Performance improves as the number of generated videos N increases, but data-generation cost also rises and the optimal balance remains open.Larger N tends to produce more distinctive samples.Source: N VBench (%) HPS (V) PickScore Total Q S · p. 8

6. Conclusion

VideoDPO is a pipeline for aligning video diffusion models that combines comprehensive scoring with automatically constructed and prioritized preference data. Experiments report improvements in visual quality and semantic alignment for state-of-the-art text-to-video models.

6. Conclusion

  • VideoDPO proposes a pipeline for aligning video diffusion models.Its stated components include OmniScore and a data reweighting strategy for automatically constructing and prioritizing preference data.Source: §6 · pp. 8–9
  • The pipeline combines comprehensive scoring with a data reweighting strategy to construct and prioritize preference data for alignment training.Source: §6 · p. 9
  • Experiments show improved visual quality and semantic alignment for state-of-the-art text-to-video models.Source: §6 · p. 9

Supplementary Material

The supplementary material provides further scoring details, additional analyses and experiments, and more qualitative results.

Section A

Section B

  • Section B compares single- and multi-dimensional settings and aggregation methods, and examines the effect of training-data scale.Source: Supplementary Material · p. 12

Section C

A. OmniScore Implementation.

OmniScore evaluates generated videos for preference learning by combining visual-quality measures with text-video semantic alignment. Its component weights favor visual dimensions over semantic alignment.

A. OmniScore Implementation.

  • OmniScore combines six visual-quality dimensions with ViCLIP-based text-video semantic alignment.The visual dimensions are motion smoothness, temporal flickering, subject consistency, imaging quality, aesthetic quality, and dynamic degree.Source: A. OmniScore Implementation · p. 12
  • OmniScore adapts referenced evaluation models and weights to assess individual video samples for preference learning.Source: A. OmniScore Implementation · p. 12
  • Each of six visual dimensions receives weight 4, while text-video semantic alignment receives weight 1.Subject consistency is scaled from 0.1462 to 1.0, temporal flickering from 0.6293 to 1.0, motion smoothness from 0.706 to 0.9975, and overall consistency from 0.0 to 0.364.Source: A. OmniScore Implementation · p. 12

B. Additional Analysis

Additional analyses compare OmniScore with alternative score and pair-selection strategies, and examine the effect of training prompt-data scale.

B. Additional Analysis

  • 78.26% on VBench-Total: combining four dimension-specific sets of 10,000 preference pairs significantly lowered performance relative to OmniScore-based selection.The combined set contained 40,000 pairs in total.Source: B. Additional Analysis · p. 12
  • OmniScore achieved the best performance compared with single-dimensional reward scores.Source: B. Additional Analysis · p. 12
  • Using half or 25% of the prompt data caused a significant drop across all metrics; the associated table reports scores for different dataset sizes.The authors attribute the gains from more prompt data to improved generalization across a broader range of prompts.Source: B. Additional Analysis · p. 12, Table 4 · p. 12

C. Additional Qualitative Results

Additional qualitative results compare inter-frame and intra-frame alignment before and after learning, with improvements reported across a wide range of prompts.

C. Additional Qualitative Results

Loading 2412.14167v1…