PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing Diffusion Models by Direct Group Preference Optimization

Yihong Luo, Tianyang Hu, Jing Tang

arXiv:2510.08425v1cs.LGcs.CV

TL;DR

Diffusion models’ efficient deterministic ODE samplers conflict with GRPO’s stochastic-policy requirement. DGPO directly learns group-level preferences, and experiments report faster training with superior performance across in-domain and out-of-domain metrics.

  • Problem

    GRPO requires stochastic policies, whereas diffusion models predominantly use deterministic ODE samplers, complicating reinforcement-learning post-training.

    Source: §1 · p. 1

  • Method

    DGPO directly optimizes preferences between positive and negative sample groups, without relying on a policy-gradient framework or stochastic policy.

    Source: §1 · p. 2

  • Results

    Around 20× faster training than Flow-GRPO accompanies superior in-domain and out-of-domain performance; GenEval training is nearly 30× faster as the base score rises from 63% to 97%.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    DGPO’s direct group-preference approach supports efficient ODE-based sampling and avoids full-trajectory training while retaining strong evaluation performance.

    Source: §1 · pp. 2–3

    The work focuses on text-to-image synthesis; adapting it to text-to-video synthesis remains future work.

    Source: C LIMITATIONS AND FUTURE WORKS · p. 14

Abstract

from arXiv · show

While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at https://github.com/Luo-Yihong/DGPO.

1 HKUST 2 CUHK (SZ) 3 HKUST (GZ)

Figure 1 contrasts DGPO’s training speed on GenEval with its performance on out-of-domain metrics: it reports near-30-times-faster training than Flow-GRPO while maintaining strong out-of-domain performance.

1 HKUST 2 CUHK (SZ) 3 HKUST (GZ)

  • Near 30 times faster training than Flow-GRPO is reported for improving GenEval score.This is the left-figure comparison.Source: Figure 1 · p. 1
  • The left figure compares training speed while improving GenEval score.Source: Figure 1 · p. 1
  • Strong performance on other out-of-domain metrics is maintained alongside the improvement.This is shown in the right figure.Source: Figure 1 · p. 1

1 INTRODUCTION

Diffusion models commonly use deterministic ODE samplers, while GRPO requires a stochastic policy, creating a mismatch for reinforcement-learning post-training. DGPO instead optimizes group-level preferences directly, and the reported experiments show faster training with stronger in-domain and out-of-domain performance.

1 INTRODUCTION

  • Reinforcement-learning post-training of diffusion models lags language models, leaving a gap in methods for preference and quality-metric alignment.Source: §1 · p. 1
  • GRPO’s stochastic-policy requirement conflicts with diffusion models’ prevalent deterministic ODE samplers, which balance sample quality and cost.Source: §1 · pp. 1–2
  • Qualitative comparisons report more accurate instruction following and better visual quality than Flow-GRPO, whose image quality degrades despite accurate instruction following.The comparisons use the same initial noise for all images.Source: Figure 2 · p. 2
  • Prior SDE-based work induces a stochastic policy, but incurs less efficient, lower-quality rollouts, weak noise-driven learning signals, and costly full-trajectory training.Source: §1 · p. 2
  • DGPO directly learns preferences between positive and negative sample groups, using efficient ODE rollouts instead of a policy-gradient framework.For each prompt, it generates a group, partitions the samples, and maximizes the likelihood of the group-wise preferences.Source: §1 · p. 2
  • DGPO uses high-fidelity ODE rollouts, group-level preference guidance, and avoids full-trajectory training to improve learning and reduce iteration cost.Source: §1 · pp. 2–3
  • Around 20× faster training than Flow-GRPO accompanies superior performance on in-domain and out-of-domain metrics.On GenEval, training is nearly 30× faster and the base model’s score rises from 63% to 97%.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries introduce diffusion generation and reward-based alignment, then describe policy-based GRPO and preference-based DPO approaches. They establish the sampling and optimization context for the paper’s method.

2 PRELIMINARIES

  • Diffusion models progressively add Gaussian noise over timesteps, then learn a reverse process through denoising.The forward process uses a noise schedule, and the network is learned with a denoising objective.Source: §2 · p. 3
  • Reward modeling represents preferences between better and worse samples using reward differences, with a reward network trained by maximum likelihood.Source: §2 · p. 3
  • RLHF optimizes a conditional sample distribution for reward while regularizing it toward a reference distribution.Source: §2 · p. 3
  • GRPO samples output groups per prompt and normalizes rewards into relative advantages, but its policy learning requires stochastic transitions.Diffusion implementations use SDE sampling rather than more efficient deterministic ODE sampling, and whole-trajectory training adds cost.Source: §2 · p. 3
  • Diffusion DPO avoids requiring a stochastic policy but is restricted to pairwise samples by an intractable partition function.Source: §2 · p. 4

3 METHOD

DGPO directly learns group-level preferences while avoiding the stochastic-policy requirement and intractable-partition obstacle associated with prior approaches. Its method generates and scores online sample groups, then uses preference-based weighting to train the diffusion model.

3 METHOD

  • DGPO optimizes group-level preferences directly, dispensing with the policy-gradient framework and the need for an inefficient stochastic policy.It uses fine-grained relative reward information within groups without requiring an intractable partition function.Source: §3 · p. 4
  • At each iteration, an online model generates samples for a condition, and a reward function scores them to form positive and negative groups.The online model may use current parameters or an exponential moving average of earlier parameters.Source: §3 · p. 4

3.1 DIRECT GROUP PREFERENCE OPTIMIZATION

  • DGPO fits Bradley-Terry group preferences by parameterizing each group’s reward as a weighted sum of its individual sample rewards.The objective uses relative preferences between positive and negative groups to retain fine-grained sample-level information.Source: §3.1 · p. 4
  • The training objective approximates an expensive inversion-chain expectation with forward diffusion and avoids training across the entire sampling trajectory.The paper attributes reduced per-iteration cost and compatibility with deterministic ODE rollouts to this objective.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 ADVANTAGE-BASED WEIGHT DESIGN

  • Because the partition function is intractable, DGPO uses advantage-based weights that emphasize samples whose rewards deviate more from the group average.The method partitions samples by advantage and uses normalized advantages to set the weights.Source: §3.1 · p. 5, §3.2 · pp. 5–6
  • When using few-step online samples, DGPO clips sampled timesteps to [tmin, T] to mitigate performance degradation from poor sample quality.The authors say this can prevent overfitting to artifacts such as blurriness.Source: §3.2 · p. 6

4 EXPERIMENTS

Experiments evaluate DGPO on compositional image generation, visual text rendering, and human-preference alignment. DGPO outperforms baselines on the reported tasks and out-of-domain metrics, trains much faster than Flow-GRPO, and its ablations examine rollout type, timestep clipping, offline training, and comparison with Diffusion DPO.

4.1 EXPERIMENTAL SETUP

  • Evaluation post-trains SD3.5-M on compositional generation, visual text rendering, and human-preference alignment, with four independent image-quality metrics on DrawBench for out-of-domain evaluation.The four out-of-domain metrics are Aesthetic Score, DeQA, ImageReward, and UnifiedReward.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • DGPO achieves state-of-the-art GenEval performance, surpasses GPT-4o and Flow-GRPO, and improves visual-text and human-preference targets while maintaining out-of-domain metrics.The target-task improvements extend beyond compositional image generation to visual text rendering and human preference alignment.Source: §4.2 · p. 8
  • DGPO follows instructions more accurately than the base model and Flow-GRPO, while Flow-GRPO’s image quality degrades more severely.These are qualitative comparisons of methods trained with GenEval’s signal.Source: §4.2 · p. 8
  • Around 20 times faster overall training than Flow-GRPO is reported for DGPO.The authors attribute the speed to avoiding stochastic-policy training and full-trajectory training.Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • Without timestep clipping, OCR Accuracy falls from 0.96 to 0.95 while visual quality seriously degrades.Instruction following remains accurate without the strategy, according to the figure caption.Source: §4.3 · p. 8, Figure 4 · p. 8
  • ODE rollout outperforms SDE rollout in both convergence speed and ultimate metrics.The authors suggest prior SDE use may reflect a policy-gradient requirement rather than a need for more diverse training samples.Source: §4.3 · pp. 8–9
  • Offline DGPO boosts performance over the baseline, but performs notably worse than online DGPO.The offline dataset is generated using the reference model.Source: §4.3 · p. 9
  • DGPO notably outperforms Diffusion DPO in both online and offline settings, consistent with leveraging fine-grained group-relative information.Diffusion DPO avoids stochastic policies but cannot leverage each sample’s fine-grained reward signal.Source: §4.3 · p. 9, Figure 5 · p. 9

5 RELATED WORKS

Prior diffusion-model alignment work includes curated-data fine-tuning, explicit reward maximization, and implicit preference optimization. DGPO builds on group-relative information while differing from policy-gradient GRPO methods and pairwise preference approaches.

5 RELATED WORKS

  • Diffusion alignment work spans curated image-prompt fine-tuning, explicit reward maximization, and implicit reward methods using raw preference data.The explicit-reward category includes multi-step generation evaluation and policy-gradient learning; Diffusion-DPO and Diffusion-KTO are examples of implicit methods.Source: §5 · p. 9
  • A concurrent method adds group information to Diffusion DPO through all within-group pairwise comparisons, whereas DGPO uses one group-level reward with maximum-likelihood learning.Source: §5 · p. 9
  • GRPO adaptations for diffusion models show promising scalability and performance but rely on stochastic policies and inefficient SDE rollouts.DGPO instead directly optimizes group preferences to use group-relative information without stochastic policies.Source: §5 · p. 9

6 CONCLUSION

DGPO is an online reinforcement-learning method for diffusion-model post-training that preserves group-relative preference learning without stochastic policies. It enables efficient ODE sampling and avoids model-agnostic exploration noise and trajectory-based training, with reported speed and performance gains.

6 CONCLUSION

  • DGPO is an online reinforcement-learning method for diffusion-model post-training that preserves group-relative preference information while dispensing with stochastic policies.The paper frames group-relative preference information, rather than policy-gradient formulation, as the key strength to retain.Source: §6 · p. 9
  • DGPO achieves around 20× speedup in overall training time compared to Flow-GRPO, and experiments show it consistently outperforms baseline methods across both in-domain and out-of-domain evaluation metrics.Source: §6 · p. 10

A EXPERIMENT DETAILS

The experiments evaluate compositional image generation, visual text rendering, and human-preference alignment, alongside out-of-domain image-quality and preference metrics. The setup uses grouped sampling, LoRA fine-tuning, and specified rollout and training settings.

A EXPERIMENT DETAILS

  • The study evaluates compositional generation on six GenEval tasks, including object counting, spatial relations, and attribute binding.The evaluation uses complex compositional prompts.Source: A EXPERIMENT DETAILS · p. 13
  • Visual text-rendering evaluation tests models’ ability to render exact strings in sign images and assesses text fidelity, defining Ne as minimum edit distance and Nref as the character count within prompt quotation marks.Source: A EXPERIMENT DETAILS · p. 13
  • Human-preference alignment uses PickScore as a reward signal to assess visual quality and text-image alignment.PickScore is trained on large-scale human-preference data.Source: A EXPERIMENT DETAILS · p. 13
  • Training uses groups of 24 samples, 10-step Flow-DPM-Solver rollouts, rank-32 LoRA, and 512-resolution experiments on A100 GPUs.β is 100; the reference model uses identity updates for 200 steps, then EMA with decay 0.3, and text is dropped with probability 0.05.Source: A EXPERIMENT DETAILS · p. 13
  • Out-of-domain assessment uses aesthetic score, DeQA, ImageReward, and UnifiedReward to evaluate visual appeal, image quality, and human preference.The metrics respectively address CLIP-based aesthetic scoring, imperfections, text-image coherence and visual fidelity, and multimodal understanding and generation.Source: A EXPERIMENT DETAILS · p. 13

B ADDITIONAL QUALITATIVE COMPARISON

Additional qualitative samples are provided in Figures 6 and 7.

B ADDITIONAL QUALITATIVE COMPARISON

C LIMITATIONS AND FUTURE WORKS

The work focuses on text-to-image synthesis and identifies text-to-video adaptation as future work. Additional qualitative comparisons use OCR Accuracy or PickScore as the training signal, with images generated from the same initial noise.

C LIMITATIONS AND FUTURE WORKS

  • The work focuses on text-to-image synthesis; adapting it to text-to-video is left for future work.Source: C LIMITATIONS AND FUTURE WORKS · p. 14
  • Figure 6 compares DGPO with competing methods using OCR Accuracy as the training signal, with all images generated from the same initial noise.Source: Figure 6 · p. 14
  • Figure 7 compares DGPO with competing methods using PickScore as the training signal, with all images generated from the same initial noise.Source: Figure 7 · p. 15

Related papers

Loading 2510.08425v1…