PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Grained GRPO for Precise Preference Alignment in Flow Models

Yujie Zhou, Pengyang Ling, Jiazi Bu, Yibin Wang, Yuhang Zang, Jiaqi Wang, Li Niu, Guangtao Zhai

arXiv:2510.01982v3cs.LGcs.CV

TL;DR

Flow-based GRPO methods face sparse reward feedback and fixed denoising granularity, limiting evaluation of sampling directions. Granular-GRPO combines single-step stochastic sampling with multi-granularity advantage integration; experiments report superior performance across diverse reward conditions.

  • Problem

    Flow-based GRPO methods provide sparse reward feedback and tie denoising directions to fixed granularities, limiting precise and comprehensive evaluation of those directions.

    Source: §1 · p. 2

  • Method

    Granular-GRPO confines stochastic sampling to one denoising step and integrates advantages from images produced at multiple denoising granularities.

    Source: §1 · pp. 2–3

  • Results

    Granular-GRPO consistently achieves superior performance across diverse reward conditions in the reported experiments.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The framework provides a more precise and comprehensive evaluation of SDE denoising directions for flow-based GRPO training.

    Source: §1 · p. 2

    The tri-granular schedule introduces additional denoising step counts and relative overhead.

    Source: §11 · p. 12

Abstract

from arXiv · show

The incorporation of online reinforcement learning (RL) into diffusion and flow-based generative models has recently gained attention as a powerful paradigm for aligning model behavior with human preferences. By leveraging stochastic sampling via Stochastic Differential Equations (SDEs) during the denoising phase, these models can explore a variety of denoising trajectories, enhancing the exploratory capacity of RL. However, despite their ability to discover potentially high-reward samples, current approaches often struggle to effectively align with preferences due to the sparsity and narrowness of reward feedback. To overcome this limitation, we introduce a novel framework called Granular-GRPO (G$^2$RPO), which enables fine-grained and comprehensive evaluation of sampling directions in the RL training of flow models. Specifically, we propose a Singular Stochastic Sampling mechanism that supports step-wise stochastic exploration while ensuring strong correlation between injected noise and reward signals, enabling more accurate credit assignment to each SDE perturbation. Additionally, to mitigate the bias introduced by fixed-granularity denoising, we design a Multi-Granularity Advantage Integration module that aggregates advantages computed across multiple diffusion scales, resulting in a more robust and holistic assessment of sampling trajectories. Extensive experiments on various reward models, including both in-domain and out-of-domain settings, demonstrate that our G$^2$RPO outperforms existing flow-based GRPO baselines, highlighting its effectiveness and generalization capability.

1. Introduction

Flow-based GRPO can explore denoising directions, but sparse reward feedback and fixed denoising granularity limit their evaluation. Granular-GRPO addresses these issues with single-step stochastic sampling and advantage integration across granularities.

1. Introduction

  • GRPO optimizes policies using group comparisons and avoids a separate value model, offering flexibility and scalability for preference alignment.The introduction situates GRPO within RLHF approaches including PPO and DPO.Source: §1 · p. 2
  • Existing flow-based GRPO assigns terminal reward uniformly across SDE steps, obscuring which perturbation produced it and making individual-step optimization imprecise.Fixed association between denoising directions and a single number of denoising steps also prevents thorough group comparisons across granularities.Source: §1 · p. 2
  • Singular Stochastic Sampling applies SDE sampling at one step and deterministic ODE sampling elsewhere, concentrating stochasticity to correlate reward with injected noise.The paper presents this as a way to support more stable optimization by strengthening the connection between a reward and a perturbation.Source: §1 · p. 2
  • Multi-Granularity Advantage Integration assigns group directions different denoising-step counts and fuses their image rewards into a unified advantage estimate.This evaluates directions across multiple denoising granularities rather than binding each direction to one fixed subsequent trajectory length.Source: §1 · pp. 2–3
  • Granular-GRPO combines single-step stochastic sampling with multi-granularity advantage integration to provide more precise and comprehensive evaluation of SDE-sampled denoising directions.The framework confines stochasticity to individual sampling steps and integrates advantages from denoised images at multiple granularities.Source: §1 · pp. 2–3

2. Related Work

Related work traces preference alignment from online RL for language models to diffusion- and flow-model methods. Flow-model approaches use ODE-to-SDE sampling for exploration, but sparse rewards and fixed denoising granularity limit evaluation of sampling directions.

2.1. Alignment for Large Language Models

  • Online RLHF for language models moved beyond supervised fine-tuning, using reward models trained from pairwise comparisons to guide policy optimization.PPO is described as computationally intensive, while value-free GRPO optimizes relative preferences within a sample group.Source: §2.1 · p. 3
  • GRPO avoids a separate value model and uses relative group preferences, providing a robust policy-improvement signal when absolute rewards are difficult to define or noisy.The passage notes its adoption by DeepSeek-R1 [9].Source: §2.1 · p. 3

2.2. Alignment for Flow Models

  • Diffusion-model alignment includes PPO-based DDPO [3] and ReFL [39], while Diffusion-DPO [34] uses paired preferences without collecting new samples during training.The cited discussion associates the latter approach with distribution shift.Source: §2.2 · p. 3
  • DanceGRPO [40] and Flow-GRPO [20] convert ODE sampling to SDE sampling for group denoising directions, but sparse rewards and fixed granularity constrain direction evaluation.MixGRPO [17] uses hybrid ODE-SDE sampling for improved efficiency while maintaining comparable performance.Source: §2.2 · p. 3

3. Granular-GRPO

Granular-GRPO evaluates sampled denoising directions through single-step stochastic exploration and multi-granularity denoising. It combines these choices to provide more precise and comprehensive reward assessments for flow-based GRPO.

3.1. Preliminary

  • Flow-based GRPO models denoising as a multi-step MDP and convert deterministic ODE sampling to an equivalent SDE formulation for grouped stochastic exploration.The SDE conversion is described as preserving the ODE sampler's marginal distribution.Source: §3.1 · p. 3

3.2. Singular Stochastic Sampling

  • Training restricts optimized timesteps to M, the first half of denoising, prioritizing early steps with larger SDE exploration spaces and reducing computation.The paper states that early steps significantly determine the denoising chain's overall direction, whereas later-stage GRPO improvements are relatively minor.Source: §3.2 · p. 4
  • Single-step SDE sampling makes group reward variation depend on the distinct denoising directions, enabling step-aware advantages tied to the injected perturbation.The method selects a timestep in M and uses a common ODE-derived starting point for the group before sampling distinct directions.Source: §3.2 · p. 5

3.3. Multi-Granularity Advantage Integration

  • Denoising granularity changes image details and textures, producing reward-score discrepancies that can alter group advantages and even the optimization direction.The paper identifies these differences as evidence that single-granularity group evaluation is insufficient.Source: §3.3 · p. 5, Figure 3 · p. 6
  • Multi-Granularity Advantage Integration denoises each sampled direction at multiple intervals and integrates the resulting advantages into a comprehensive evaluation.Integer interval factors λ define the denoising granularities; larger factors produce coarser denoising.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate G2RPO using shared training and sampling settings, compare it with flow-based GRPO baselines, and test multi-granularity denoising and robustness to fewer inference steps.

4.1. Implementation Details

  • Experiments use 103,700 HPSv2 prompts for training, 400 for testing, and Flux.1-dev; evaluations include five reward models, with joint HPS-v2.1 and CLIP Score training as the primary setting.Sampling uses groups of 12 images, 16 steps, and granularities Λ = {1, 2, 3}; comparison methods share the sampling configuration.Source: §4.1 · pp. 6–7

4.2. Main Results

  • 6.52% relative improvement over DanceGRPO with HPS-v2.1 training; with joint HPS-v2.1 and CLIP Score training, G2RPO outperforms baselines on in-domain and out-of-domain rewards.The HPS-v2.1-only setting can compromise other out-of-domain metrics through model hacking.Source: Table 1 · p. 7, §4.2 · p. 8

4.3. Ablation Study

  • Increasing the number of selected denoising granularities makes sampling-direction evaluation more comprehensive and improves performance across in-domain and out-of-domain reward models.The ablation compares different denoising granularities.Source: §4.3 · p. 8, Table 2 · p. 8

4.4. Further exploration of varying inference steps

  • With jointly trained HPS-v2.1 and CLIP, G2RPO retains significant improvements across in-domain and out-of-domain rewards when inference uses 20 or 10 steps.The authors attribute robustness to MGAI's mixed assessment of denoising samples at different granularities.Source: §4.4 · p. 8, Table 3 · p. 8

5. Conclusion

G2RPO targets precise evaluation of flow-based GRPO denoising directions through single-step stochasticity and multi-granularity assessment. Experiments report superior performance across diverse reward conditions.

  • G2RPO localizes stochasticity to one denoising step and integrates advantages from images at different granularities to evaluate SDE denoising directions.The conclusion attributes denser, more precise reward signals and more robust, higher-quality alignment to this design.Source: §5 · p. 8

6. Additional Implementation Details

The experiments use a consistent hyperparameter configuration.

  • The same hyperparameter configuration is maintained across all experiments.Table 4 presents the configuration.Source: §6 · p. 11, Table 4 · p. 11

7. Additional Quantitative Evaluation

Additional evaluation on UniGenBench++ examines reward-model alignment across diverse prompt themes and fine-grained criteria. The results describe trade-offs from single-granularity alignment and the role of MGAI.

  • Single-granularity alignment improves Attribute, Relation, and Text metrics, but overfitting reward-model biases narrows outputs and degrades Style and Layout.The passage identifies this result for Singular Stochastic Sampling without MGAI.Source: §7 · p. 11
  • MGAI selects samples with advantages at both coarse structural and fine textural scales, which the authors say mitigates domain-collapse risk.The module provides a more comprehensive evaluation of sampling directions within each group.Source: §7 · p. 11
  • G2RPO achieves substantial overall performance improvements on UniGenBench++.The benchmark covers 10 dimensions, including semantic evaluation, image quality, and text alignment.Source: §7 · p. 11

8. Visual Ablation Study of MGAI

The visual ablation links single-granularity denoising to biased group-wise advantage estimates and structural fragility, while MGAI reweights samples across coarse and fine scales. The text reports that multi-dimensional reward metrics capture gains beyond those reflected by uni-dimensional metrics.

8. Visual Ablation Study of MGAI

  • Without MGAI, fixed-step single-granularity denoising can bias group-wise advantage estimates toward fine details over coarse structural coherence.Higher-reward samples may consequently show distorted textures or global misalignment.Source: §8 · p. 11
  • Single-granularity generators tend to produce excessive textural detail alongside structural fragility.Source: §8 · p. 11
  • MGAI reweights a sample only when it surpasses peers at both coarse and fine scales, targeting detail fidelity and global layout jointly.Source: §8 · p. 11
  • Unified Reward and UniGenBench++ capture gains in prompt adherence, detail accuracy, and overall quality, while HPSV2.1, CLIP Score, and Pick Score alone cannot fully capture MGAI's holistic gains.Source: §8 · p. 11

9. Additional Qualitative Comparisons

The section presents qualitative comparisons of G2RPO with existing flow-based GRPO methods and identifies superior visual fidelity and text-image alignment. It also includes a quantitative comparison on UniGenBench.

9. Additional Qualitative Comparisons

  • G2RPO achieves superior visual fidelity and text-image alignment in qualitative comparisons with existing flow-based GRPO methods [17] [40].Source: §9 · p. 11
  • Figures 6, 7, and 8 show the qualitative comparisons with existing flow-based GRPO methods [17] [40].Source: §9 · p. 11
  • Table 5 presents a quantitative comparison on UniGenBench [36].The caption identifies bold scores as best and underlined scores as second-best.Source: Table 5 · p. 12

10. Gallery of G2RPO

The section provides additional G2RPO image samples generated from prompts randomly sampled from UniGenBench++.

10. Gallery of G2RPO

  • Figures 9, 10, and 11 provide additional G2RPO samples using prompts randomly sampled from UniGenBench++ [36].Source: §10 · p. 12

11. Limitation

Multi-granularity sampling adds training-time sampling cost, quantified through denoising-step counts and an example overhead. The text says this cost is incurred only once and does not change inference latency.

11. Limitation

  • Multi-granularity sampling incurs additional sampling time.Source: §11 · p. 12
  • For M = {16, 15, ..., 9}, the tri-granular schedule uses 184 total steps with approximately 45.7% relative overhead.Source: §11 · p. 12
  • The added MGAI training-time cost is incurred only once and leaves inference latency unchanged.Source: §11 · p. 12
Loading 2510.01982v3…