OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization

1. Introduction

• Image-level reward models may misjudge video quality because they cannot perceive temporal dynamics, while ReFL’s decoded-video scoring and gradient propagation consume substantial memory and are difficult to scale.The paper also notes that ReFL is prone to reward hacking.Source: §1 · p. 2
• OnlineVPO combines video-quality assessment feedback with online preference generation and curriculum reference updates to improve preference optimization effectiveness and scalability.The framework is presented as a video-specific alternative to image-level feedback and ReFL-style tuning.Source: §1 · p. 2

1.1. Diffusion-based Video Generation

• Video diffusion models retain challenges including poor image quality, frame flicker, and subject inconsistency despite advances in longer-video generation.DiT-based models can generate videos up to one minute, but these quality issues persist.Source: §1.1 · p. 2

1.2. Preference Learning from Human Feedback

• Prior video-preference methods use image-level rewards or self-supervised models not specifically trained to assess video quality, especially temporal dynamics.InstructVideo uses HPSv2 [48], while VADER [35] also explores V-JEPA [7].Source: §1.2 · p. 3
• The paper identifies modality-misaligned feedback and limited, inefficient ReFL-based tuning as gaps, and presents a systematic exploration of DPO for video generation.It also notes that concurrent VideoDPO [31] explores DPO adaptation but retains inefficient offline optimization.Source: §1 · p. 2, §1.2 · p. 3

2.1. Preliminary

• ReFL maximizes generated-image preference scores, whereas DPO reparameterizes the reward and optimizes preferences between preferred and unpreferred responses.The DPO formulation uses policy and reference models, with preference pairs drawn from dataset D.Source: §2.1 · p. 3

2.2. Investigation on Video Preference Feedback

• Image-level preference models perform poorly at distinguishing human-preferred videos, while VQA models offer a low-cost alternative with decent alignment to human feedback.The feedback study compares reward rankings against human-selected preferences; the authors propose VQA models because large-scale manually annotated video-preference data is labor-intensive and expensive.Source: §2.2 · pp. 3–4, Table 2 · p. 4

2.3. Online Video Preference Optimization

• OnlineVPO combines on-the-fly preference generation with a curriculum reference update for online preference optimization of video diffusion models.The framework uses VQA scores to select preferred and unpreferred videos generated by the current policy, and periodically updates the reference model with the aligned policy.Source: §2.3 · pp. 4–5
• Online preference generation addresses the mismatch between a changing policy and DPO's fixed, pre-collected preference pairs.At each optimization step, the policy samples multiple videos and the VQA model selects the highest- and lowest-scoring samples as the preference pair.Source: §2.3 · p. 5
• Curriculum reference updates periodically replace the fixed reference with the newly aligned policy to balance exploration, optimization efficiency, and training stability.The authors report that optimization efficiency slows after the policy surpasses the fixed reference's preference alignment.Source: §2.3 · p. 5

3.1. Experiment Setup

• Experiments use OpenSora v1.2 and VideoCrafter-v2, fine-tuned on WebVid-10M, with VideoScore as the default feedback model and VBench for evaluation.Online generation uses six candidate videos; OpenSora samples 34 frames at 240p, while VideoCrafter samples 16 frames at 512×320.Source: §3.1 · p. 5

3.2. Compared with State-of-the-art Methods

• OnlineVPO consistently improves VBench performance over existing methods on OpenSora and VideoCrafter-v2, especially subject consistency and temporal flicker, and outperforms competitors in human evaluations.Visual comparisons also report clearer frames and greater temporal consistency than competing methods.Source: Table 3 · p. 6, §3.2 · p. 6, Figure 6 · p. 8

3.3. Ablation Study

• VideoScore feedback improves performance across video-quality dimensions; temporal-consistency feedback performs best, while averaging dimensions can underperform when their signals conflict.Temporal-consistency optimization also improves other dimensions, whereas feedback from other dimensions does not show the same cross-dimensional benefit.Source: §3.3 · p. 7, Table 4 · p. 8
• Online preference learning outperforms offline learning, whose fixed pairs cannot adapt to evolving generations; OnlineVPO rewards rise steadily, unlike ReFL’s rise followed by decline.The authors associate ReFL’s later reward decline with reward hacking and its unbounded optimization design.Source: §3.3 · pp. 7–8, Figure 7 · p. 8
• OnlineVPO retains 25% of available GPU memory at 720p with 68 frames, while ReFL supports at most 240p with 68 frames or 360p with 17 frames.The comparison indicates greater scalability for high-resolution, longer-video optimization.Source: §3.3 · p. 8

• Experiments on open-source video generation models demonstrate the superiority of OnlineVPO.Source: §4 · p. 8
• OnlineVPO uses video quality assessment models as an alternative to human feedback in video preference optimization.Source: §4 · p. 8
• The paper presents the first systematic exploration of DPO in video generation and introduces OnlineVPO as an efficient preference optimization method.Source: §4 · p. 8