PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffusionNFT: Online Diffusion Reinforcement with Forward Process

Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, Ming-Yu Liu

arXiv:2509.16117v2cs.LGcs.AIcs.CV

TL;DR

Diffusion-model RL is hindered by intractable likelihoods and limitations of reverse-process training. DiffusionNFT instead performs online policy optimization on the forward process through flow matching, contrasting positive and negative generations, and reports up to 25× higher efficiency than FlowGRPO while operating without CFG.

  • Problem

    Diffusion-model reinforcement learning is difficult because likelihoods are intractable and reverse-process methods face forward inconsistency, solver restrictions, and complicated CFG integration.

  • Method

    DiffusionNFT performs policy optimization directly on the forward diffusion process through flow matching, contrasting reward-split positive and negative generations within a supervised objective.

  • Results

    DiffusionNFT achieves up to 25× higher efficiency than FlowGRPO and outperforms CFG baselines across diverse in-domain and out-of-domain rewards.

  • Takeaways & Limitations

    DiffusionNFT provides a CFG-free, native off-policy approach that supports arbitrary black-box solvers and requires only clean images rather than full sampling trajectories.

  • Takeaways & Limitations

    Reverse-process GRPO-style methods remain limited by forward inconsistency, solver restrictions, and complicated two-model CFG optimization.

Abstract

from arXiv · show

Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due to intractable likelihoods. Recent works discretize the reverse sampling process to enable GRPO-style training, yet they inherit fundamental drawbacks, including solver restrictions, forward-reverse inconsistency, and complicated integration with classifier-free guidance (CFG). We introduce Diffusion Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that optimizes diffusion models directly on the forward process via flow matching. DiffusionNFT contrasts positive and negative generations to define an implicit policy improvement direction, naturally incorporating reinforcement signals into the supervised learning objective. This formulation enables training with arbitrary black-box solvers, eliminates the need for likelihood estimation, and requires only clean images rather than sampling trajectories for policy optimization. DiffusionNFT is up to $25\times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO achieves 0.95 with over 5k steps and additional CFG employment. By leveraging multiple reward models, DiffusionNFT significantly boosts the performance of SD3.5-Medium in every benchmark tested.

1 INTRODUCTION

Diffusion-model RL is difficult because likelihoods are intractable and reverse-process approaches impose consistency, solver, and CFG limitations. DiffusionNFT instead optimizes the forward process through flow matching, using reward-split positive and negative generations while retaining practical training flexibility.

  • Motivation: Diffusion-model RL remains challenging because their likelihoods are not directly computable, unlike autoregressive models.Existing likelihood estimates rely on costly probabilistic ODE or variational SDE bounds.
  • Limitations: Reverse-process GRPO-style training risks forward inconsistency, restricts sampling to first-order SDE solvers, and complicates CFG optimization.The approach can break adherence to the forward diffusion process and requires a two-model CFG optimization scheme.
  • Research question: DiffusionNFT asks whether reinforcement can be performed on the forward process rather than the reverse process.This framing follows from the single forward process and multiple possible reverse samplers in diffusion policies.
  • Method: DiffusionNFT optimizes the forward diffusion process through flow matching by contrasting reward-split positive and negative generations, without modifying sampling.The contrast defines an implicit policy-improvement direction toward the positive policy.
  • Practical benefits: DiffusionNFT supports arbitrary black-box solvers, needs only clean images instead of full sampling trajectories, and remains compatible with standard diffusion training.It is also described as a native off-policy algorithm that decouples training and sampling policies without importance sampling.
  • Results: DiffusionNFT improves GenEval from 0.24 to 0.98 within 1k steps, while FlowGRPO reaches 0.95 after over 5k steps with additional CFG.Across four tested tasks, it shows 3× to 25× efficiency and better final scores than FlowGRPO.

2 BACKGROUND

Diffusion models learn data distributions by adding noise through a forward process and learning to reverse it. Flow matching parameterizes the velocity field for this process, while policy-gradient methods discretize reverse sampling into tractable multi-step decisions.

  • Diffusion models: Diffusion models gradually perturb clean data with Gaussian noise through a forward process, then learn to reverse that process for generation.The clean-data distribution is denoted x_0 ∼ π_0 = p_data.
  • Diffusion models: The forward noising process has a closed-form transition kernel mapping clean data to noisy states.The supplied passage identifies this kernel as π_t|0(x_t|x_0).
  • Flow models: Flow matching trains a velocity parameterization v_θ(x_t, t) to predict the trajectory tangent using a schedule-defined velocity target.The target is v = α̇_t x_0 + σ̇_t ε under the stated notation.
  • Flow models: The learned velocity field defines an ODE, and Euler discretization provides an effective solver equivalent to DDIM.The ODE is written as dx_t/dt = v_θ(x_t, t).
  • Flow models: Rectified flow is a special case with α_t = 1 − t and σ_t = t, yielding the simplified target v = ε − x_0.These schedule choices simplify the general velocity target.
  • RL for diffusion: Policy-gradient methods apply to diffusion models by discretizing reverse sampling into a multi-step Markov decision process.This makes adjacent-step transitions tractable for algorithms such as PPO and GRPO.
  • RL for diffusion: FlowGRPO introduces stochasticity through an SDE because deterministic ODE sampling otherwise hinders direct GRPO application.Its Euler discretization produces tractable Gaussian transition kernels between adjacent steps.

3 DIFFUSION REINFORCEMENT VIA NEGATIVE-AWARE FINETUNING

DiffusionNFT performs diffusion policy optimization on the forward process by contrasting positive and negative generations within a supervised flow-matching framework. The method targets forward consistency while supporting flexible sampling and likelihood-free, off-policy training.

  • Negative-Aware Policy Optimization: RFT trains solely on positive data, whereas DiffusionNFT additionally uses negative data because positive-only fine-tuning cannot effectively leverage it.The paper notes that positive-only fine-tuning can lead to collapse.
  • Negative-Aware Policy Optimization: DiffusionNFT defines a contrastive improvement direction from positive and negative generated samples and optimizes the forward diffusion model toward it.The method uses reinforcement signals within the flow-matching objective rather than conventional policy gradients.
  • Advantages: Forward-process training preserves forward consistency by maintaining the coupling between noisy states and clean images under the diffusion process.The paper characterizes this consistency as adherence of the underlying density to the Fokker–Planck equation.
  • Advantages: DiffusionNFT supports arbitrary black-box solvers and requires only clean images with rewards, eliminating the need to store complete sampling trajectories.Training and data sampling are decoupled, enabling off-policy use without importance sampling.
  • Negative-Aware Policy Optimization: DiffusionNFT directly integrates reinforcement guidance into a single target policy through implicit parameterization instead of learning a separate guidance model.The target combines the old policy with a guidance direction, while dual implicit positive and negative policies support supervised optimization.
  • Practical Implementation: CFG-free optimization initializes from the conditional model alone and can rapidly surpass the CFG baseline despite lower initial performance.The paper presents reinforcement learning as learning or substituting CFG functionality during post-training.

4 EXPERIMENTS

The experiments evaluate DiffusionNFT through multi-reward training, FlowGRPO head-to-head comparisons, and ablations of key design choices. DiffusionNFT achieves strong CFG-free performance, improves reward optimization efficiency, and reveals the importance of negative signals, sampler choice, weighting, soft updates, and guidance strength.

  • Experimental Setup: DiffusionNFT’s evaluation covers multi-reward joint training, single-reward comparison with FlowGRPO, and ablations of key design choices.The experiments use SD3.5-Medium and multiple reward models, with additional studies of samplers, weighting, soft updates, and guidance strength.
  • Multi-Reward Joint Training: The final CFG-free model surpasses CFG, matches FlowGRPO on in-domain and out-of-domain metrics, and outperforms larger CFG-based models.The larger comparison models are SD3.5-L with 8B parameters and FLUX.1-Dev with 12B parameters.
  • Head-to-Head Comparison: 3× to 25× greater wall-clock efficiency is achieved against FlowGRPO in single-reward head-to-head comparisons.DiffusionNFT achieves a GenEval score of 0.98 within approximately 1k iterations.
  • Ablation Studies: Removing the negative policy loss causes rewards to collapse almost instantly during online training.The result contrasts with observations in LLMs, where rejection fine-tuning remains a strong baseline.
  • Ablation Studies: ODE samplers outperform SDE samplers, with second-order ODE slightly better than first-order ODE on GenEval and comparable on PickScore.The advantage is especially pronounced on PickScore, which is described as noise-sensitive.
  • Ablation Studies: Higher flow-matching-loss weight at larger t improves stability, while inverse weighting can cause collapse; gradually increasing soft-update η balances speed and stability.Fully on-policy updates progress quickly but destabilize training, whereas overly off-policy updates slow convergence.
  • Ablation Studies: Guidance strength β near 1 performs stably, while β = 0.1 can produce faster reward increases.The guidance parameter governs a trade-off between stability and convergence speed.

5 RELATED WORK

Related work addresses diffusion-model reinforcement learning through likelihood-free, likelihood-based, and score-based approaches. These methods differ in reward requirements, data assumptions, guidance mechanisms, and reliance on reverse-process samplers.

  • Overview: Diffusion RL is challenging because exact model likelihoods are difficult to compute, although likelihoods are central to many RL algorithms.Existing approaches therefore either approximate likelihoods or avoid likelihood-based optimization.
  • Likelihood-Free Methods: Reward Backpropagation supports likelihood-free optimization but requires differentiable rewards and is limited to low-noise timesteps by memory and gradient costs.Unrolling long denoising chains creates memory costs and gradient explosion.
  • Likelihood-Free Methods: Reward-Weighted Regression is an offline fine-tuning method that lacks a negative policy objective for penalizing low-reward generations.Policy-guidance methods instead combine multiple models for guided sampling, complicating online optimization.
  • Likelihood-Based Methods: Diffusion-DPO adapts DPO to paired human-preference data but requires additional likelihood and loss approximations compared with autoregressive models.DDO avoids paired data by using high-quality data as positive signals and self-generated samples as negative signals.
  • Sampler-Based Methods: MixGRPO improves efficiency by mixing SDE and ODE samplers, but coupling and forward inconsistency remain.This work is situated among approaches addressing SDE-sampler efficiency bottlenecks.

6 CONCLUSION

DiffusionNFT performs online reinforcement learning directly on the diffusion forward process, contrasting positive and negative generations within the standard diffusion objective. It achieves up to 25× higher efficiency than FlowGRPO and outperforms CFG baselines across diverse rewards.

  • 6 CONCLUSION: DiffusionNFT directly operates on the forward process for online reinforcement learning of diffusion models.The method formulates policy improvement as a contrast between positive and negative generations and integrates reinforcement signals into the standard diffusion objective.
  • 6 CONCLUSION: 25× higher efficiency than FlowGRPO is achieved while producing a single model that outperforms CFG baselines across diverse in-domain and out-of-domain rewards.The reported comparison combines efficiency with performance across multiple reward settings.
  • 6 CONCLUSION: DiffusionNFT positions the forward process as a foundation for unifying supervised and reinforcement learning in diffusion.The authors describe this direction as promising for scalable, efficient, and theoretically principled diffusion reinforcement learning.

A PROOF OF THEOREMS

The appendix establishes how the positive, negative, and old diffusion policies relate through distribution and posterior splits. It then characterizes the improvement direction and the optimal velocity predictor induced by the training objective.

  • Distribution and posterior splits: The distribution triplet π+, π−, and πold is analyzed through a distribution split and a posterior split.The posterior relationship is obtained using Bayes’ Rule and preserves the decomposition between positive and negative components.
  • Improvement direction: Theorem A.3 states that the directional differences among v+, v−, and vold are parallel.This theorem supplies the appendix’s improvement-direction characterization for the three associated diffusion models.
  • Reinforcement guidance optimization: The implicit positive policy is formed from vold and vθ, while the implicit negative policy uses the corresponding opposing combination.The theorem defines these policies as (1 − β)vold + βvθ and (1 + β)vold − βvθ, respectively.
  • Reinforcement guidance optimization: The optimal solution satisfies vθ∗(xt, c, t) = vold(xt, c, t) + 2.This conclusion follows from substituting the split relationships into the training objective.

B THEORETICAL DISCUSSIONS

The theoretical discussion relates flow models to diffusion SDEs under the rectified-flow schedule and clarifies how reverse-process noise choices interpolate between ODE and maximum-variance SDE behavior. It also compares discretization choices and their sampling implications.

  • Flow and diffusion SDEs: Flow models are a special case of diffusion models under the rectified schedule αt = 1−t and σt = t.This connection allows results for diffusion SDEs to apply to flow models.
  • Flow and diffusion SDEs: The forward SDE coefficients f(t) and g(t) are matched to the forward transition kernel through αt and σt.For rectified flow, the discussion gives f(t) = −1/(1−t) and g^2(t) = 2t/(1−t).
  • Flow and diffusion SDEs: λt = 0 corresponds to ODE sampling, while λt = 1 corresponds to the maximum variance SDE.The generalized reverse SDE can therefore be viewed as adding Langevin dynamics to the diffusion ODE.
  • Sampler comparisons: DanceGRPO uses a fixed variance across timesteps, which is described as less effective on image models but more stable on video models.This contrasts with the timestep-dependent variance choice discussed for FlowGRPO.
  • Sampler comparisons: FlowGRPO and DanceGRPO use Euler discretization of the flow SDE, whereas DDIM-style discretization is described as avoiding boundary singularities and potentially reducing sampling errors.The authors report no notable advantage from replacing the SDE sampler with stochastic DDIM.

B.2 HIGH-ORDER FLOW ODE SAMPLER

This section describes high-order flow ODE sampling and analyzes reverse-process diffusion RL through the FlowGRPO objective. The discussion connects multistep solver choices with advantage-weighted velocity updates and the trade-off between gradient variance and backpropagation length.

  • B.2 HIGH-ORDER FLOW ODE SAMPLER: The implemented second-order ODE sampler uses a multistep DPM-Solver method and half the log signal-to-noise ratio for time discretization.For three consecutive timesteps, previously obtained states are used to update the current state.
  • B.2 HIGH-ORDER FLOW ODE SAMPLER: Second-order sampling is infeasible at the boundaries, so the first and last steps degrade to the first-order Euler solver.The boundary issue arises because λt becomes infinite at t = 0 or t = 1.
  • FlowGRPO analysis: FlowGRPO’s reverse sampling step produces xs = l(s,t)xt − m(s,t)vθ(xt,t) + n(s,t)ϵ, with coefficients determined by s, t, and the sampler.The step uses Gaussian noise ϵ sampled from N(0, I).
  • FlowGRPO analysis: MixGRPO branches from a shared xt using multiple noises, producing multiple clean samples and corresponding advantages for on-policy optimization.On-policy GRPO minimizes negative advantage-weighted log likelihoods for the branched samples.
  • FlowGRPO analysis: FlowGRPO aligns the velocity field with advantage-weighted noise, while timestep and sampler choices affect the weighting across sampling steps.The analysis presents this as a sampler-agnostic interpretation of the objective’s gradient.
  • FlowGRPO analysis: GRPO optimizes the reverse transition t → s while treating the remaining trajectory s → 0 as gradient-free.Compared with direct backpropagation methods, this introduces higher estimation variance but avoids backpropagating through the remaining sampling process.

C EXPERIMENT DETAILS

The experiments use matched FlowGRPO-style configurations, timestep-aligned forward noising, and a multi-stage multi-reward training scheme. Single-reward comparisons fix sampling steps for fairness and adjust soft updates for OCR stability.

  • The setup matches FlowGRPO on groups per epoch, group size, LoRA configuration, and learning rate.It uses 48 groups per epoch, group size 24, LoRA α = 64 and r = 32, and learning rate 3e −4.
  • Forward noising and loss computation are performed on the corresponding sampling timesteps for each collected clean image.Data collection uses a 2nd-order ODE sampler with adaptive time weighting enabled by default.
  • Single-reward comparisons fix sampling steps at 10 to ensure fairness with FlowGRPO.The default settings use β = 1 and η_i = min(0.001i, 0.5), with a more conservative OCR soft update capped at η_max = 0.999.
  • Multi-reward training uses a multi-stage scheme spanning Pick-a-Pic, GenEval, and OCR datasets with overlapping reward categories.The stages combine PickScore, CLIPScore, HPSv2.1, GenEval, and OCR rewards in the configurations described.

D ADDITIONAL RESULTS

Additional results include single-reward evaluation caveats and qualitative comparisons against the base model and FlowGRPO across GenEval, OCR, and DrawBench prompts.

  • Training exclusively on OCR reward impairs generalization to other metrics.For OCR-trained models, CFG is enabled when evaluating non-OCR rewards.
  • Qualitative comparisons include the base model, FlowGRPO, and the multi-reward optimized model.These comparisons are provided in Figures 11, 12, and 13.
  • Figure 11 compares FlowGRPO and DiffusionNFT on GenEval prompts.
  • Figure 12 compares FlowGRPO and DiffusionNFT on OCR prompts.
  • Figure 13 compares FlowGRPO and DiffusionNFT on DrawBench prompts.
Loading 2509.16117v2…