PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Reinforcement Learning for Diffusion Models via Velocity Matching

Jaemoo Choi, Wei Guo, Yuchen Zhu, Arash Vahdat, Molei Tao, Julius Berner, Yongxin Chen

arXiv:2608.23664v1cs.CVcs.LG

TL;DR

Diffusion reward fine-tuning is complicated by intractable endpoint likelihoods and costly likelihood-based approximations. The paper proposes trajectory-free reward-based velocity matching, which directly updates the velocity field and, across diffusion tasks, matches or exceeds trajectory-based methods at lower cost while highlighting reward design for video motion.

  • Problem

    Diffusion models lack tractable generated-sample likelihoods, forcing existing reward fine-tuning methods to estimate likelihoods through stochastic trajectories or ELBO surrogates.

  • Method

    RVM directly applies signed reward weighting to velocity targets, with an optional anchor velocity controlling drift from a reference model.

  • Results

    Across text-to-image, text-to-video, and image-to-video tasks, velocity-based methods match or outperform trajectory-based baselines at substantially reduced training cost; RVM reaches 84.13 VBench Overall on Wan2.1-T2V-1.3B.

  • Takeaways & Limitations

    Reward and anchor design matter more than precise velocity-loss variants, while dynamic-tracking rewards improve video motion and overall VBench performance.

  • Takeaways & Limitations

    Trajectory-based methods remain costly because they require stochastic rollouts, trajectory storage, and can produce high-variance updates.

Abstract

from arXiv · show

Reward fine-tuning is becoming an important tool for adapting diffusion models to human preferences and task-specific objectives, but existing methods largely inherit policy-gradient machinery from large language models. Unlike autoregressive models, diffusion models do not provide tractable likelihoods for generated samples. As a result, current approaches either construct trajectory likelihoods from stochastic denoising transitions or approximate endpoint likelihoods with evidence lower bound, introducing additional computation and algorithmic complexity. We demonstrate that this likelihood-based machinery is not necessary for effective diffusion reward fine-tuning. We propose reward-based velocity matching (RVM), a simple trajectory-free update that acts directly on the velocity field. RVM reinforces directions associated with high-reward generations, suppresses those with low reward, and involves an optional anchor term controlling drift from a reference velocity. Notably, it provides a general framework that recovers recent fine-tuning methods, including RAM and DiffusionNFT, as special cases. Across various large-scale diffusion models reward fine-tuning tasks, RVM is competitive with or outperforms trajectory-based policy-gradient methods under substantially reduced training cost. We further find that, once the velocity update is simplified, the particular loss variant matters less than reward and anchor design. For video generation, standard preference rewards can favor visually clean but nearly static outputs; introducing a new dynamic-tracking reward that substantially improve motions while improving overall VBench performance. These results suggest that scalable reward fine-tuning for diffusion models is better posed in the native velocity representation than as likelihood-based policy optimization.

1 INTRODUCTION

Diffusion reward fine-tuning typically relies on likelihood-based policy-gradient machinery that is costly or indirect. RVM instead updates the native velocity field directly, unifies prior objectives, and performs competitively across image and video settings.

  • Motivation: Diffusion models lack tractable endpoint likelihoods, motivating trajectory-based or ELBO-based approximations for reward fine-tuning.Trajectory methods use stochastic denoising transitions, while ELBO methods approximate final-sample likelihoods.
  • RVM: RVM applies reward feedback directly to the native velocity-matching representation through a trajectory-independent update.Generated samples are independently noised to a single intermediate state and regressed toward their velocity targets.
  • RVM: Positive reward weights reinforce sample-associated velocity directions, negative weights suppress them, and an optional anchor controls drift from a reference velocity.The anchor also helps retain desirable reference-model properties.
  • Unification: RVM generalizes RAM and DiffusionNFT as special cases, shifting emphasis toward reward transformation, anchoring, and sample generation.The framework treats these design choices as more important than the precise velocity-loss algebra.
  • Experiments: Across text-to-image, text-to-video, and image-to-video settings, velocity-based methods match or outperform trajectory-based baselines at substantially lower training cost.On Wan2.1-1.3B, the RVM recipe uses a 16-step deterministic ODE rollout and achieves the best VBench Overall score at a fraction of trajectory-based training cost.
  • Video reward design: Generic preference and visual-quality rewards can produce nearly static videos, whereas the dynamic-tracking reward improves motion and overall VBench performance.The DT reward explicitly encourages meaningful motion through optical flow.

2 BACKGROUND

Diffusion and flow models learn velocity fields from interpolated data and noise, but reward fine-tuning must approximate unavailable likelihood ratios. Existing trajectory and ELBO approaches motivate direct reward-weighted velocity matching.

  • Diffusion and flow models: Diffusion and flow models interpolate clean data with Gaussian noise and learn a velocity field through weighted regression.Sampling integrates the learned velocity field through an ODE or uses an equivalent SDE sampler.
  • Policy-gradient objectives: Policy-gradient reward fine-tuning maximizes generation reward with KL regularization toward a pretrained reference policy.REINFORCE reweights sample log-likelihoods by reward, while GRPO adds group-normalized rewards and clipped likelihood ratios.
  • Likelihood estimation: Diffusion models cannot directly form the required marginal likelihood ratio, so existing methods estimate it from denoising transitions or final-sample ELBOs.Trajectory methods use per-step transition probabilities; ELBO methods use a likelihood surrogate from independently noised states.
  • Trajectory-based approaches: Trajectory-based objectives reinforce stochastic one-step denoising directions that may be noisy and not directly aligned with the clean endpoint.The target direction can vary across rollouts because intermediate states include sampler noise.
  • Direct velocity matching: ELBO-based policy-gradient updates ultimately reduce to reward-weighted velocity regression at independently noised samples.This observation motivates formulating reward fine-tuning directly as velocity matching rather than beginning with likelihood estimation.

3 DESIGNING SIMPLE AND EFFICIENT DIFFUSION FINE-TUNING

RVM simplifies diffusion reward fine-tuning by directly matching reward-weighted velocity directions while optionally anchoring the model to a reference velocity. Its formulation unifies RAM and DiffusionNFT, indicating that reward and anchor design matter more than the precise velocity-loss algebra.

  • 3.1 SIMPLIFIED AND VELOCITY-BASED OBJECTIVES: RVM uses reward-weighted velocity matching and an optional anchor term, without policy ratios or likelihood interpretation.The reward term pulls predictions toward high-reward sample directions, while the anchor limits policy drift.
  • 3.1 SIMPLIFIED AND VELOCITY-BASED OBJECTIVES: Positive reward weights reinforce desirable velocity directions, whereas negative weights suppress directions associated with undesirable samples.The reward acts as a signed preference signal for the velocity direction generated by each sample.
  • 3.1 SIMPLIFIED AND VELOCITY-BASED OBJECTIVES: RVM uses an endpoint-defined velocity target from a single noised state, providing a more direct and potentially lower-variance signal than trajectory-based losses.Trajectory-based methods instead reinforce stochastic one-step transitions sampled during denoising.
  • 3.2 CONNECTION BETWEEN THE OBJECTIVES: RAM and DiffusionNFT are special cases of RVM under specific sampling, reward, anchor, and anchor-strength choices.RAM is update-equivalent to on-policy RVM with a frozen reference anchor, while DiffusionNFT uses an EMA-updated anchor.
  • 3.2 CONNECTION BETWEEN THE OBJECTIVES: Because these methods share the same update structure, their main differences lie in reward transformation and anchor design rather than basic velocity-regression direction.The paper also identifies the anchor velocity and reward as distinguishing design choices.
  • 3.2 CONNECTION BETWEEN THE OBJECTIVES: Dropping ELBO-based likelihood-ratio terms and time-dependent weighting reduces the policy-gradient objective exactly to RVM.The resulting gradient matches RVM with the reference anchor at unit strength.

4 EXPERIMENTS

Experiments across text-to-image, text-to-video, and image-to-video models evaluate RVM against trajectory-based, ELBO-based, and velocity-based alternatives. RVM matches or improves quality while reducing training cost, with reward design—especially dynamic tracking—driving video performance.

  • Experimental setup: RVM is evaluated across Stable Diffusion 3.5, Wan2.1-T2V-1.3B, and SkyReels-I2V using separate training rewards and evaluation metrics.The study compares pretrained models, trajectory-based RL methods, and trajectory-independent velocity-based methods.
  • Main results: 84.13 VBench Overall: RVM outperforms trajectory-based methods on Wan2.1-T2V-1.3B, while reaching 86.27 Overall on SkyReels-I2V.On Wan2.1-T2V-1.3B, RVM also reaches 75.00 dynamic degree versus 55–58 for trajectory-based methods; FlowGRPO reduces SkyReels-I2V from 81.68 to 77.97.
  • Main results: RVM, RAM, and DiffusionNFT achieve similar performance, indicating that the particular velocity-matching loss variant matters less than remaining design choices.The shared anchored velocity-matching structure shifts attention toward reward transformation, anchoring, and sampling.
  • Main results: 525 GPU-hours: RVM costs 2.2× less than FlowGRPO and 11.8× less than DanceGRPO/TaRoS for a full Wan2.1-T2V-1.3B run.The savings arise from fewer function evaluations per rollout and fewer training iterations under standard protocols.
  • Further analysis: RVM remains robust under aggressive step reduction, losing only 0.43 VBench points at 5 steps and 1.83 points at 4 steps.The base model drops from 83.10 at 16 steps to 73.62 at 5 steps and 64.39 at 4 steps.

5 CONCLUSION

The paper presents RVM as a trajectory-free reward fine-tuning method that unifies velocity-based objectives and scales effectively to video. It finds that reward design, especially for motion, matters more than the precise velocity-loss functional.

  • RVM unifies recent velocity-based objectives, including RAM and DiffusionNFT, at the level of velocity-field updates.The framework provides a common view of these methods despite their different motivations.
  • Experiments show velocity-matching losses remain scalable and effective for video fine-tuning, where rollouts are expensive.
  • Reward design is more important than the specific loss functional for improving motion and visual quality.The conclusion emphasizes carefully shaped rewards as crucial for translating reward optimization into generation improvements.
  • Extending the framework to autoregressive video generation is identified as a natural future direction.

IMPACT STATEMENT

The paper positions diffusion-model reward fine-tuning within broader reinforcement-learning work while noting its potential for harmful and offensive content generation. It reviews trajectory-based, simplified velocity-based, and video-generation approaches relevant to this setting.

  • The authors caution that video diffusion fine-tuning may be abused to create harmful and offensive content.
  • RL fine-tuning for language models includes PPO, GRPO, and related policy-objective variants.
  • Trajectory-based diffusion RL treats denoising or flow trajectories as policy trajectories and estimates likelihoods along them.
  • Simplified diffusion fine-tuning methods replace explicit trajectory-level likelihood estimation with velocity- or score-based objectives.
  • Recent work applies RL-style fine-tuning to video diffusion and autoregressive video generators.

B.2 CONNECTIONS BETWEEN THE LOSS FUNCTIONALS

The paper shows that several diffusion fine-tuning objectives share a reward-weighted velocity-regression structure. This connection explains how likelihood-based policy-gradient formulations reduce to velocity updates and how trajectory-based methods differ in their regression targets.

  • Connections between the loss functionals: RAM matches RVM when the anchor is the frozen reference velocity, with unit anchor strength.
  • Connections between the loss functionals: DiffusionNFT matches RVM using an EMA velocity anchor, a transformed reward, and reward-dependent anchor strength.
  • Policy-gradient-based loss functionals: ELBO-based policy-gradient objectives reduce to velocity regression because only the squared velocity-error term contributes a gradient.
  • Policy-gradient-based loss functionals: Trajectory-based methods require likelihood estimation from denoising transitions, whereas velocity-based forms avoid log-probabilities and likelihood ratios.
  • Trajectory-based objectives as velocity regression: Trajectory-based updates are reward-weighted velocity regressions toward a stochastic one-step direction rather than the clean-sample velocity target used by RVM.In the ODE limit, the trajectory target becomes the realized one-step direction; noise can keep it from moving toward the clean endpoint.

C.1 EXPERIMENT SETUP

The experiments fine-tune frozen diffusion backbones with LoRA using deterministic few-step ODE sampling and single-state reward-weighted velocity regression. Evaluation also tests reduced-step schedules matched between the base and trained models.

  • Fine-tuning uses LoRA while keeping the pretrained backbone frozen.
  • Samples are generated with a deterministic 16-step DPM-Solver-2 flow ODE at guidance scale 1.0, without CFG or policy-ratio clipping.
  • The loss is computed as reward-weighted velocity regression at a single noised state per generation.
  • Few-step evaluation subsamples the trained noise grid to four- and five-step schedules, with the base model evaluated under identical schedules.

C.2 TEXT-TO-IMAGE (OCR) TASK

The OCR task fine-tunes Stable Diffusion 3.5-M with LoRA using CFG-free deterministic sampling and a multi-component reward mixture. Evaluation measures OCR accuracy and held-out preference and image-quality metrics.

  • Task setup: Stable Diffusion 3.5-M is fine-tuned with LoRA at 512 × 512 resolution for OCR text rendering.The protocol follows Choi et al. (2026).
  • Task setup: Training uses a CFG-free deterministic 10-step DPM-Solver-2 flow ODE with guidance scale 1.0.Each epoch samples 48 prompts with K=24 images per prompt before one accumulated-batch update.
  • Reward design: The reward is an equal-weight sum of OCR accuracy, PickScore, CLIPScore, and HPSv2.1.PickScore is rescaled by 1/26 inside the scorer.
  • Evaluation: Evaluation reports OCR accuracy alongside held-out PickScore, ClipScore, HPSv2.1, Aesthetic score, and ImageReward.Images are sampled with 40 steps during evaluation.

D TEXT-TO-IMAGE EXPERIMENTAL RESULTS

Additional text-to-image experiments evaluate RVM on DrawBench for PickScore fine-tuning and document the default RVM configurations. The reported result says RVM remains competitive on this task.

  • PickScore task: RVM remains competitive on the DrawBench PickScore fine-tuning task for SD3.5-M.The model is trained for 700 epochs on a mixture of PickScore, CLIPScore, and HPSv2.1.
  • Default settings: Table 5 lists default RVM hyperparameters for each base model, with shared optimizer, DPM-2 ODE sampler, and CFG-free settings.The video runs share one reward mixture, while the OCR run uses a separate reward.
  • Evaluation: Table 6 reports DrawBench PickScore results for a multi-reward fine-tuning task, with training rewards marked in gray and best column entries bolded.Cited numbers from Choi et al. (2026) are marked with daggers.

E.1 PER-REWARD TRAINING CURVES

Per-reward training curves show that velocity-matching objectives improve each tracked reward component, whereas FlowGRPO stagnates or degrades. The curves separate aggregate behavior into individual reward dimensions.

  • Training curves: Velocity-matching objectives improve every tracked reward component throughout training.The components include VideoAlign text alignment and motion quality, HPSv3, and VBench dynamic degree.
  • Training curves: FlowGRPO stagnates or degrades on the individual reward curves, consistent with the aggregate results in Table 1.Figure 8 compares these trajectories across training.
  • Figure encoding: Each panel in Figure 8 shows one reward component across training iterations, with dashed lines marking the untrained base.

E.2 FULL PER-DIMENSION VBENCH RESULTS

Full-dimension evaluations show strong aggregate performance for RVM on Wan2.1-T2V-1.3B and similar overall scores among velocity-matching methods on SkyReels-I2V. The supplementary figures provide additional qualitative comparisons under the reported protocols.

  • Wan2.1-T2V-1.3B: RVM attains the best Quality and Overall aggregates on Wan2.1-T2V-1.3B.Its largest margins are in dynamic degree, aesthetic quality, and imaging quality.
  • Wan2.1-T2V-1.3B: The CFG-guided base retains the best Semantic aggregate on Wan2.1-T2V-1.3B.
  • SkyReels-I2V: The three velocity-matching methods achieve similar Overall scores on SkyReels-I2V, while RVM is the only fine-tuned method that raises dynamic degree.
  • Qualitative comparisons: Additional figures compare RVM with base and competing methods across sampled frames for Wan2.1-T2V-1.3B and SkyReels-I2V.The comparisons use the protocols of Tables 1 and 2.
Loading 2608.23664v1…