PaperlayerView exact revision on arXiv

Source-linked AI summary

V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think

Bingda Tang, Yuhui Zhang, Xiaohan Wang, Jiayuan Mao, Ludwig Schmidt, Serena Yeung-Levy

arXiv:2604.23380v1cs.LGcs.CV

TL;DR

Aligning denoising generative models with rewards is difficult because their likelihoods are intractable and existing approaches trade efficiency for performance. V-GRPO stabilizes ELBO-based policy optimization with GRPO, achieving state-of-the-art text-to-image performance while speeding training over key baselines.

  • Problem

    Aligning denoising generative models is important, but ELBO-based likelihood surrogates have historically underperformed visually and naive GRPO implementations suffer instability and poor convergence.

  • Method

    V-GRPO integrates ELBO-based surrogates with GRPO, using surrogate-variance reduction and gradient-step regularization for stable online reinforcement learning.

  • Results

    V-GRPO achieves state-of-the-art performance, runs 2× faster than MixGRPO, matches DiffusionNFT, and delivers a 3× speedup in multi-stage multi-reward settings.

  • Takeaways & Limitations

    Properly stabilized ELBO-based methods can outperform complex MDP approaches and provide a competitive default for post-training denoising generative models.

  • Takeaways & Limitations

    The approach rests on the hypothesis that excessive variance in ELBO-based surrogates causes training failure.

Abstract

from arXiv · show

Aligning denoising generative models with human preferences or verifiable rewards remains a key challenge. While policy-gradient online reinforcement learning (RL) offers a principled post-training framework, its direct application is hindered by the intractable likelihoods of these models. Prior work therefore either optimizes an induced Markov decision process (MDP) over sampling trajectories, which is stable but inefficient, or uses likelihood surrogates based on the diffusion evidence lower bound (ELBO), which have so far underperformed on visual generation. Our key insight is that the ELBO-based approach can, in fact, be made both stable and efficient. By reducing surrogate variance and controlling gradient steps, we show that this approach can beat MDP-based methods. To this end, we introduce Variational GRPO (V-GRPO), a method that integrates ELBO-based surrogates with the Group Relative Policy Optimization (GRPO) algorithm, alongside a set of simple yet essential techniques. Our method is easy to implement, aligns with pretraining objectives, and avoids the limitations of MDP-based methods. V-GRPO achieves state-of-the-art performance in text-to-image synthesis, while delivering a $2\times$ speedup over MixGRPO and a $3\times$ speedup over DiffusionNFT.

1. Introduction

Online RL could align denoising generative models with human preferences or verifiable rewards, but exact likelihoods are generally intractable and prior MDP or ELBO-based approaches have important limitations. V-GRPO addresses these issues by stabilizing ELBO-based GRPO training, achieving state-of-the-art text-to-image performance with substantial speedups.

  • Online RL is motivated by the need to align pretrained denoising generative models with human preferences or verifiable rewards.
  • Exact likelihood requirements make policy-gradient methods difficult to apply to denoising generative models.
  • MDP formulations enable policy gradients through tractable Gaussian reverse kernels but converge slowly, restrict solvers, and tightly couple optimization with rollout transitions.
  • ELBO-based likelihood surrogates had underperformed in visual generation, but the paper argues this gap is not fundamental.
  • V-GRPO integrates ELBO-based surrogates with GRPO and uses surrogate variance reduction plus gradient-step regularization for stable, efficient training.The method is easy to implement, aligns with pretraining, and avoids MDP limitations.
  • 2× faster than MixGRPO, V-GRPO achieves state-of-the-art multi-reward text-to-image performance; it also matches DiffusionNFT with a 3× speedup in multi-stage multi-reward synthesis.

2. Related Work

Prior work applies RL to denoising generative models through offline ELBO-based surrogates or online MDP formulations, each with important limitations. V-GRPO revisits online ELBO optimization with simple techniques that enable state-of-the-art performance and improved training efficiency.

  • Offline methods: Offline RL methods use ELBO-based likelihood surrogates, but distributional shift and limited static-dataset coverage constrain them.Examples include reward-weighted regression (RWR) and Direct Preference Optimization (DPO).
  • Online methods: Online methods typically model stochastic sampling as an MDP and optimize reverse transition kernels over induced state-action spaces with policy gradients.Recent extensions include GRPO-based variants, flow-matching models, and algorithmic designs addressing theoretical and practical limitations.
  • ELBO-based online methods: DDPO and FPO found ELBO-based surrogates underperforming on visual generation, but V-GRPO shows this limitation is not fundamental.The paper attributes the improvement to simple, effective techniques that unlock the approach’s potential.
  • ELBO-based online methods: V-GRPO achieves state-of-the-art performance with significantly improved training efficiency, while AWM concurrently explores ELBO-based surrogates.Both methods revisit ELBO-based online optimization for denoising generative models.
  • Alternative likelihood-free methods: DiffusionNFT avoids likelihood approximation by contrasting positive and negative policies, but maintaining two model-weight sets introduces additional overhead.It also builds on pretraining objectives, similarly to V-GRPO.

3. Preliminaries

Denoising generative models transform data into tractable noise through forward processes and learn approximate reverse dynamics using weighted regression and iterative sampling. GRPO provides group-relative advantage estimation, while V-GRPO treats multi-step generation as an atomic action using a marginal likelihood surrogate.

  • Denoising generative models: Denoising generative models define a forward process that gradually transforms data x ∼ π0 into noise ϵ ∼ π1, typically N(0, I).The models discussed include diffusion and flow matching models.
  • Denoising generative models: Reverse dynamics are approximated by minimizing a suitably weighted regression objective with method-specific schedules and targets.Common targets include x-prediction, ϵ-prediction, and v-prediction, where v = ϵ − x.
  • Sampling: First-order sampling iteratively denoises from an initial noise sample over a discretized time schedule.SDE samplers add Gaussian noise, whereas ODE samplers are deterministic with σ_ti ≡ 0.
  • Group Relative Policy Optimization: GRPO removes PPO’s value function and estimates each output’s advantage by normalizing its reward relative to the group mean and standard deviation.The behavior policy generates a group of outputs for each input.
  • V-GRPO: V-GRPO treats multi-step generation as an atomic action and directly optimizes the final output’s marginal likelihood surrogate rather than averaging per-step losses.MDP-based approaches model generation as individual actions across trajectory transitions.

4. Approach

V-GRPO replaces intractable denoising-model log-likelihoods with tractable, weighted diffusion-ELBO surrogates inside GRPO. It stabilizes these surrogates through variance reduction and gradient-step regulation, improving training while retaining compatibility with denoising generative models.

  • ELBO-based surrogates: V-GRPO substitutes model log-likelihoods in GRPO with pretraining objectives interpreted as weighted diffusion ELBOs, yielding tractable policy-gradient surrogates.The importance ratio is computed as exp(−Lw(θ | oi, c) + Lw(θold | oi, c)).
  • Model compatibility: The generalized formulation covers common denoising objectives and remains compatible with models such as rectified flow.The approach relies on the equivalence of marginal-density evolution under a fixed forward process.
  • Variance analysis: Naive ELBO-based GRPO is unstable because timestep-dependent surrogate variance produces high-variance gradients whose noise can overwhelm reward signals.Gradient norms scale with surrogate magnitude, propagating unstable losses into training dynamics.
  • Variance reduction: Three variance-reduction techniques lower surrogate and gradient variability, including group-shared timestep-noise pairs, x-prediction, and self-normalization.Sharing timestep-noise pairs across outputs for each prompt removes a major source of intra-group variance; x-prediction and self-normalization regulate weighting and gradient scales.
  • Variance reduction: 0.230 → 0.128: applying the techniques reduces the coefficient of variation of surrogate magnitude and improves training stability and performance.The mean within-group CV decreases from 0.170 to 0.038, while the quadratic-fit coefficient of determination decreases from 0.406 to 0.328.
  • Gradient-step regulation: Importance-ratio clipping generally stabilizes training, while KL penalties preserve earlier capabilities but may not suppress loss spikes; soft-clipping advantages stabilizes on-policy training but can underperform on coarse rewards.Soft-clipping also improves stability with reduced sampling steps, yet can underperform prior methods on GenEval.

5. Experiments

Experiments evaluate V-GRPO on FLUX.1-dev and SD 3.5 M using baseline-aligned prompts, rewards, and training configurations. V-GRPO outperforms FLUX.1-dev baselines, matches DiffusionNFT on SD 3.5 M with fewer updates, and depends on variance-reduction and gradient-control techniques.

  • Main experiments: 300 iterations with four rewards train V-GRPO on FLUX.1-dev, where it outperforms all baselines across every reward metric and converges 2× faster than MixGRPO.The reward ensemble comprises HPSv2.1, PickScore, ImageReward, and UnifiedReward.
  • Main experiments: 580 gradient updates across five curriculum stages yield SD 3.5 M performance matching DiffusionNFT while requiring roughly three times fewer gradient steps and markedly lower NFE.The curriculum uses GenEval, OCR, HPSv2.1, PickScore, and CLIPScore.
  • Main experiments: V-GRPO also achieves competitive performance in single-reward settings.These results are reported in Table 6.
  • Ablations: On FLUX.1-dev, removing group-shared timestep-noise pairs or stratified timestep sampling destabilizes training, while omitting adaptive loss weighting slightly reduces performance.The naive baseline is severely unstable, and all variance-reduction techniques are jointly required for optimal results.
  • Ablations: SD 3.5 M is more robust to ELBO-based training: individual variance-reduction ablations have limited effects, although the techniques remain collectively beneficial.The naive baseline still shows severe Stage-1 instability, but no individual component is critical.
  • Ablations: A KL penalty preserves prior capabilities during SD 3.5 M Stage-5 training, whereas importance-ratio clipping degrades GenEval from 0.92 →0.87; reducing NMC from 4 to 2 prevents convergence.Increasing NMC to 8 provides only marginal gains, and advantage soft-clipping improves stability in specified training settings but underperforms importance-ratio clipping.

6. Conclusion

V-GRPO integrates ELBO-based surrogates into GRPO for online RL of denoising generative models, using variance reduction and gradient-step control to achieve state-of-the-art performance with substantial baseline speedups.

  • 6. Conclusion: V-GRPO integrates ELBO-based surrogates into GRPO for online reinforcement learning of denoising generative models.The method uses simple techniques to reduce surrogate variance and control gradient steps.
  • 6. Conclusion: V-GRPO achieves state-of-the-art performance with substantial speedup over baselines.The authors propose establishing ELBO-based methods as a new default while motivating further research into robustness and scalability.

A. Additional Implementation Details

The implementation closely follows the baseline methods, with deviations restricted to the key techniques described in Sections 4.4 and 4.5.

  • The implementation adheres closely to the baseline methods.
  • Deviations are limited to the key techniques described in Section 4.4.
  • The remaining deviations are limited to the key techniques described in Section 4.5.

A.1. FLUX.1-dev Experiments

The FLUX.1-dev experiments follow MixGRPO’s prompt and multi-reward setup, using controlled optimization, rollout, clipping, and evaluation configurations across 16- and 25-step sampling. Ablations use 25 sampling steps while keeping other settings unchanged.

  • Main experiments: Experiments use HPDv2 prompts and an ensemble of HPSv2.1, PickScore, ImageReward, and UnifiedReward, with multi-reward advantages averaged across individual reward advantages.PickScore is normalized as in MixGRPO.
  • Main experiments: 300 training iterations use AdamW at 1×10^-5 learning rate and 1×10^-4 weight decay, with 4 gradient steps, batch size 8, group size 12, and no EMA.These settings define the optimization schedule and batch configuration.
  • Main experiments: NMC = 4; importance-ratio clipping is 6 × 10^-3 for 25 rollout steps and 1 × 10^-2 for 16 rollout steps, with advantages soft-clipped to 2 and no KL penalty.The clipping settings differ by rollout length, while soft clipping and the absence of KL regularization apply to both configurations.
  • Main experiments: Rollouts use 720×720 resolution with 16 or 25 sampling steps, while evaluation uses 50 steps at 1024 × 1024 and MixGRPO hybrid sampling with pmix = 0.8.The trained model handles the first pmixT steps, and the original base model completes the remainder.
  • Ablation studies: Ablation studies use 25 sampling steps, with all other configurations matching the main experiments.The ablation settings are otherwise consistent with the main experimental setup.

A.2. SD 3.5 M Experiments

SD 3.5 M experiments use a DiffusionNFT-style multi-stage curriculum with LoRA optimization, reward aggregation, and controlled advantage clipping. Stages 1 and 3 train on Pick-a-Pic with an HPSv2.1, PickScore, and CLIPScore ensemble, while evaluation uses 40 sampling steps at 512 × 512 and includes out-of-domain metrics.

  • Multi-stage curriculum: The training curriculum aggregates multiple rewards by averaging them before computing multi-reward advantages.This curriculum follows the multi-stage training approach adopted from DiffusionNFT.
  • Optimization: LoRA optimization uses r = 32, α = 64, AdamW, a learning rate of 3×10−4, weight decay of 1×10−4, global batch size 48, and group size 24.The per-step batch and group configuration matches DiffusionNFT.
  • Multi-stage curriculum: Stages 1 and 3 train on Pick-a-Pic with HPSv2.1, PickScore, and CLIPScore, using 150 iterations, NMC = 4, and advantage soft-clipping to 3.Each iteration performs 1 gradient step, with no importance ratio clipping or KL penalty applied.
  • Evaluation: 40 sampling steps are used for both rollout and evaluation at 512 × 512 resolution, with CLIPScore, UnifiedReward, and Aesthetics added as out-of-domain evaluation metrics.These metrics supplement the reward functions used during training.
  • Efficiency: 3× speedup over DiffusionNFT is achieved in gradient steps, while requiring fewer function evaluations per step on average.DiffusionNFT’s per-stage step counts are approximate because of early stopping, whereas this method’s counts are exact.

B. Additional Results

Additional results show that V-GRPO improves qualitative alignment, coherence, and style, while ablations identify stable prediction parameterizations and collectively useful variance-reduction techniques.

  • Quantitative comparisons: Quantitative comparisons of single-reward experiments on SD 3.5 M are reported in Table 6.The supplied passage identifies the comparison but does not provide the numerical results.
  • Prediction parameterization: ϵ-prediction causes severe training collapse, while v-prediction remains stable but converges slightly slower than x-prediction.These findings concern the adaptive loss-weighting technique.
  • Surrogate variance reduction: Variance-reduction techniques are collectively beneficial, although no single component is individually critical.The ablation is conducted on SD 3.5 M.
  • Qualitative comparisons: V-GRPO achieves superior alignment, coherence, and style in qualitative comparisons on FLUX.1-dev and SD 3.5 M.The FLUX.1-dev results also demonstrate strong world knowledge in one example.
Loading 2604.23380v1…