Video Diffusion Alignment via Reward Gradients

1 Introduction

• Broad web-scale training can yield videos with dull colors, suboptimal camera angles, and inadequate text-video alignment, motivating task-specific adaptation.Specialized goals can include close adherence to a script, temporal consistency, and desirable camera perspectives.Source: §1 · p. 2
• Collecting target-video datasets for supervised fine-tuning is costly and laborious, and must be repeated for each new video task.The introduction frames this as a reason to seek another alignment signal.Source: §1 · p. 2
• Prior reward-model approaches use either dense reward gradients or scalar feedback with policy gradients; direct gradients can reduce reward-query needs.The introduction notes that gradient-free methods remain dominant in text-to-image settings, where gradient-efficiency gains are less significant.Source: §1 · p. 2
• The reward-gradient advantage over policy-gradient approaches grows as generation moves from images to videos; VADER is also reported to outperform DPO/DDPO and generalize to unseen prompts.The authors attribute the growing gap to dense feedback across spatial and temporal dimensions, versus one scalar for an entire video.Source: §1 · pp. 2–3
• VADER adapts text-to-video and image-to-video diffusion models using pretrained vision-model rewards across varied alignment tasks.The tested rewards include aesthetics, image-text alignment, object detection, video-action classification, and video masked autoencoding.Source: §1 · pp. 2–3

2 Related Work

• Likelihood-based fine-tuning requires preferred-behavior datasets, whereas reward or preference signals can be easier to collect and may come from pretrained vision models.The cited examples include reward-weighted likelihood and PPO for reward-based fine-tuning.Source: §2 · p. 3
• Policy-gradient methods and DPO provide scalar feedback for an entire video, while prior diffusion work found direct reward gradients more sample-efficient.DPO avoids an explicit reward model but, as described here, does not resolve policy-gradient sample inefficiency.Source: §2 · p. 3
• Video diffusion alignment remains challenging, and naively applying image-alignment techniques can incur significant memory overhead.The paper proposes using widely available image or video discriminative models as rewards without assuming access to a video dataset.Source: §2 · p. 3
• InstructVideo also aligns video diffusion using human preferences, but requires video datasets that are difficult to obtain for each task and scale across many tasks.Source: §2 · p. 3

3 Background

• The forward process progressively adds noise to a data sample over T steps, with the cumulative product of α_i determining how much original signal remains at timestep t.Source: §3 · pp. 3–4
• Conditional generation adds c as an input to the denoising model.Source: §3 · p. 4
• The training objective minimizes squared distance between predicted and actual noise over image-condition pairs, aligning with a variational lower bound for log p(x|c).Source: §3 · p. 4
• The learned reverse process estimates added noise and progressively denoises a noisy sample; generation starts from Gaussian noise and iteratively applies that process.Source: §3 · p. 4

Reward-gradient updates

• VADER maximizes a reward objective by backpropagating gradients from differentiable reward functions to update video diffusion model weights.The method allows different denoising schedules and limits gradient backpropagation to K steps, where K < T, to reduce memory use and prevent overoptimization.Source: §4 · pp. 4–5

Text similarity rewards

• For text alignment, VADER can average image-context similarity across frames or reward whole-video similarity, which permits frames to deviate from the prompt and represent motion.Experiments use HPS v2 and PickScore for image-text alignment, and VideoMAE action-classification probabilities for video-text similarity.Source: §4 · p. 5

Image and video metric rewards

• Image-only rewards average per-frame metrics, while video-level rewards score the generated sequence with a model that evaluates multiple frames.The image-based experiments use an aesthetic reward model and an object detector; the video-level formulation uses a video metric model.Source: §4 · pp. 5–6

Long-horizon consistent generation

• For longer clips, VADER makes Stable Video Diffusion autoregressive, extends its context length by 3×, and uses V-JEPA rewards to encourage temporal consistency.Passing the model's own last generated frame as input causes distribution shift; training with the video metric produces temporally and spatially coherent clips.Source: §4 · pp. 6–7

Reducing memory overhead

• VADER reduces memory use with LoRA, mixed precision, gradient checkpointing, CPU graph offloading, one-timestep backpropagation, and subsampled-frame decoding.The codebase supports training on a single GPU with 16GB VRAM.Source: §4 · p. 7

Resolution comparison

• After 100 optimization steps, the performance gap between VADER and DDPO significantly increases as output resolution increases.Figure 4 reports reward after 100 steps at increasing video resolutions.Source: Figure 4 · p. 7

5 Results

• The evaluation spans text-to-video and image-to-video models, with reward models for aesthetics, image-text alignment, object removal, action classification, and temporal consistency.The compared baselines include DDPO and Diffusion-DPO.Source: §5 · p. 8

5.1 Sample and Computational Efficiency

• Within 12 GPU-hours, VADER starts converging while DPO and DDPO show little improvement, and it is more sample- and compute-efficient through dense reward gradients.Figure 7 compares reward queries in its upper half and GPU-hours in its lower half; the baselines backpropagate scalar feedback.Source: Figure 7 · p. 10, §5.1 · p. 10

5.2 Generalization Ability

• VADER achieves the best rewards on both metrics for test prompts that do not overlap with training prompts.The evaluation uses HPSv2 Action Prompts for T2V training and Activity Prompts for testing, and Labrador and Maltese categories for I2V training and testing, respectively.Source: Table 1 · p. 10, §5.2 · p. 11

5.3 Human Evaluation

• In side-by-side tests with 100 responses per comparison, participants preferred VADER over baseline methods for text alignment and video quality.Table 2 reports the human evaluation for the HPS reward model on image-text alignment.Source: §5.3 · p. 11, Table 2 · p. 11

5.4 Qualitative Visualization

• Across reward types, visualizations show improved prompt alignment, more artistic outputs, book replacement, and clearer action depiction after fine-tuning.Examples include correcting raccoon and fox prompt mismatches, removing books with alternative objects, and making a person eating donuts more recognizable; Figures 6 and 8 show additional model and action results.Source: §5.4 · p. 11, Figure 8 · p. 12
• With autoregressive SVD generation extended to three times its context length, V-JEPA reward fine-tuning reduces the observed spatial and temporal discrepancies.Without VADER, accumulated errors deform a teddy bear or affect a rocket in motion.Source: §5.4 · p. 12, Figure 9 · p. 12

• VADER fine-tunes pre-trained video diffusion models via reward gradients as a sample- and compute-efficient framework.Source: §6 · p. 12
• VADER uses reward functions evaluated on images or videos to fine-tune video diffusion models.Source: §6 · p. 12
• VADER is agnostic to conditioning and works with both text-to-video and image-to-video diffusion models.Source: §6 · p. 12