PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow-GRPO: Training Flow Matching Models via Online RL

Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, Wanli Ouyang

arXiv:2505.05470v5cs.CVcs.AI

TL;DR

Flow matching models are strong image generators but struggle with complex composition and text rendering, while online RL has not been effectively integrated with their deterministic sampling. Flow-GRPO converts ODE sampling to equivalent SDE sampling and shortens training denoising trajectories while retaining full inference steps. It improves compositional generation, visual text rendering, and human-preference alignment with little reward hacking and no noticeable image-quality or diversity degradation.

  • Problem

    Flow-matching models struggle with complex composition and text rendering, while online RL remains underexplored for these deterministic generative models.

  • Method

    Flow-GRPO integrates GRPO into flow matching using ODE-to-SDE conversion for stochastic exploration and Denoising Reduction for shorter training trajectories.

  • Results

    Flow-GRPO improves SD3.5-M GenEval accuracy from 63% to 95% and visual text-rendering accuracy from 59% to 92%, while improving human-preference alignment across tasks.

  • Takeaways & Limitations

    Flow-GRPO provides a general framework for efficient online RL optimization of flow-based text-to-image models with minimal reward hacking.

  • Takeaways & Limitations

    The work focuses on text-to-image tasks; extending Flow-GRPO to video requires better reward design, multi-objective balancing, and scalable training pipelines.

Abstract

from arXiv · show

We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.

1 Introduction

Flow-GRPO addresses the difficulty of applying online RL to deterministic, costly flow-matching generation by introducing stochastic sampling and shorter training trajectories. Across text-to-image tasks, it improves compositional and text-rendering performance while preserving image quality and limiting reward hacking.

  • Motivation: Flow-matching models struggle with complex composition and text rendering, while online RL for these models remains underexplored.The challenges include multiple objects, attributes, relationships, and visual text generation.
  • Motivation: Deterministic ODE sampling conflicts with RL exploration, and many denoising steps make online data collection inefficient.These constraints are especially problematic for large generative models.
  • Approach: Flow-GRPO converts deterministic ODE sampling into equivalent SDE sampling and reduces denoising steps during training while retaining the full inference schedule.The ODE-to-SDE strategy introduces randomness while preserving marginal distributions; Denoising Reduction accelerates sample generation.
  • Results: 63% to 95%: Flow-GRPO improves SD3.5-M accuracy on GenEval and outperforms GPT-4o.GenEval evaluates object counts, colors, spatial relations, and other compositional properties.
  • Results: 59% to 92%: Flow-GRPO improves SD3.5-M accuracy on visual text rendering.The improvement is reported alongside gains across text-to-image tasks.
  • Results: The KL constraint prevents reward hacking by matching high reward while preserving image quality and diversity, albeit with longer training.KL regularization is reported as empirically distinct from early stopping.

2 Related Work

Related work establishes flow matching as an efficient deterministic generation framework and reviews existing approaches for RL and text-to-image alignment. Flow-GRPO builds on these lines by applying value-free online policy optimization to flow matching.

  • Online RL: Online RL has improved LLM reasoning with policy-gradient methods including PPO and GRPO.GRPO removes the need for a value network and is therefore more memory efficient.
  • Diffusion and Flow Matching: Flow matching learns a continuous-time velocity field, enabling efficient deterministic sampling with few ODE steps and competitive FID.This efficiency has supported its adoption in recent image and video generation models.
  • Alignment for T2I: T2I alignment methods include differentiable rewards, RWR, DPO variants, PPO-style policy gradients, and training-free techniques.These approaches have improved aesthetics and semantic consistency, primarily through human-preference alignment.

3 Preliminaries

The paper formulates flow matching and its denoising trajectory, then represents iterative denoising as a finite-horizon MDP. This formulation defines states, actions, deterministic transitions, initialization, and terminal rewards.

  • Flow Matching: Flow matching uses data and noise samples to define noised states within the Rectified Flow framework.Advanced image and video models adopt this continuous-time formulation.
  • Flow Matching: A transformer is trained to regress the velocity field vθ(xt, t) using the Flow Matching objective.The model directly predicts the velocity governing the continuous transformation.
  • Denoising as an MDP: The iterative denoising process is formulated as an MDP with state st=(c,t,xt), action at=xt−1, and policy π(at|st)=pθ(xt−1|xt,c).The state includes the conditioning information, timestep, and current sample.
  • Denoising as an MDP: The MDP transition is deterministic, mapping each state and denoised action to the next conditioning, timestep, and sample state.The transition is represented using Dirac delta distributions.
  • Denoising as an MDP: The reward is provided only at the final denoising step and is otherwise zero.The terminal reward is r(x0,c).

4 Flow-GRPO

Flow-GRPO adapts GRPO to flow matching by converting deterministic ODE sampling into stochastic SDE sampling while preserving marginal distributions. Denoising Reduction then accelerates online RL data collection by using fewer training steps while retaining the original inference schedule.

  • GRPO on Flow Matching: Flow-GRPO integrates GRPO with flow matching by treating the denoising process as an MDP and optimizing a regularized policy objective.GRPO estimates advantages from normalized rewards across groups of sampled images and trajectories.
  • From ODE to SDE: Deterministic ODE sampling conflicts with GRPO because it limits stochastic exploration and makes transition-probability computation expensive.Reduced randomness lowers training efficiency, while deterministic sampling provides no randomness beyond the initial seed.
  • From ODE to SDE: The ODE-to-SDE strategy converts the flow ODE into an equivalent reverse-time SDE that preserves the model’s marginal distribution at every timestep.The SDE introduces Wiener-process noise, with σ_t controlling generation stochasticity; Euler–Maruyama discretization produces the update rule.
  • From ODE to SDE: The resulting transition policy is isotropic Gaussian, enabling closed-form KL divergence computation against the reference policy.The noise level is controlled by a scalar hyper-parameter a.
  • Denoising Reduction: Denoising Reduction uses T = 10 training steps while retaining SD3.5-M’s original T = 40 inference steps, accelerating online RL sampling without sacrificing performance.The strategy collects low-quality but informative trajectories for the GRPO loss, which updates the model online.

5 Experiments

Flow-GRPO is evaluated on compositional generation, visual text rendering, and human preference alignment, alongside image quality, preference, and reward-hacking analyses. Across these evaluations, it improves task performance while largely preserving quality and diversity, with KL regularization and sampling choices affecting stability and efficiency.

  • Main Results: Flow-GRPO improves compositional generation, visual text rendering, and human preference alignment while evaluating image quality and preference scores on DrawBench.The experiments cover three text-to-image tasks and assess whether task-specific reward optimization compromises broader image quality or preference measures.
  • Main Results: Flow-GRPO outperforms SFT, Flow-DPO, and their online variants on the compositional generation task.Offline methods use a fixed pretrained data-collection model, whereas online variants update that model every 40 steps.
  • Reward Hacking: Removing KL regularization reduces image quality and preference scores for compositional and text-rendering tasks, while causing visual-diversity collapse in human preference alignment.Properly tuned KL preserves quality and diversity while achieving similar task-specific gains.
  • Denoising Reduction: Reducing data-collection timesteps from 40 to 10 produces over a 4× speedup across all three tasks without affecting final reward.Reducing to 5 timesteps does not consistently improve speed and sometimes slows training, so 10 is used subsequently.
  • Group Size: Group sizes G = 12 and G = 6 cause unstable training and eventual collapse, whereas G = 24 remains stable.The analysis attributes smaller-group instability to inaccurate advantage estimates and increased variance.

6 Conclusion

Flow-GRPO integrates online policy-gradient RL into flow matching models through stochastic sampling and reduced training denoising. It improves several text-to-image tasks while preserving image quality or diversity, though broader video applications and reward design remain future work.

  • Flow-GRPO is presented as the first method to integrate online policy-gradient RL into flow matching models.
  • ODE-to-SDE conversion introduces stochasticity for RL while preserving the original model’s marginal distribution.
  • Denoising-step reduction during training improves RL efficiency without noticeably compromising image quality or diversity.
  • Flow-GRPO significantly improves compositional generation, text rendering, and human preference alignment, with minimal reward hacking.
  • Applying Flow-GRPO to video generation remains future work because reward design, multiple-objective balancing, scalability, and reward hacking require further study.

A Mathematical Derivations for Stochastic Sampling using Flow Models

The derivation converts deterministic flow-model dynamics into stochastic forward and reverse SDEs whose marginals match the original ODE. The resulting process can be simulated for stochastic sampling using the flow model’s velocity field and an explicitly controlled diffusion coefficient.

  • The derivation adapts flow models to an SDE so pθ(xt−1 | xt, c) can be computed during forward sampling.
  • The stochastic counterpart is constructed so its marginal probability distribution matches that of the deterministic ODE.
  • The forward SDE’s diffusion coefficient σt controls the level of stochasticity during sampling.
  • The corresponding reverse-time SDE provides the stochastic sampling process used in place of deterministic ODE sampling.
  • Once the score function is available, the process can be simulated directly because flow matching links the score implicitly to the velocity field.
  • Euler–Maruyama discretization yields the numerical update rule for simulating the stochastic process.

B.3 Hyperparameters Specification

The experiments use fixed GRPO settings across tasks except for the KL ratio, with separate sampling and evaluation timestep budgets. Training uses LoRA and 24 NVIDIA A800 GPUs.

  • GRPO uses sampling timestep T = 10 and evaluation timestep T = 40, with group size G = 24 and noise level a = 0.7.
  • The KL ratio β is 0.04 for GenEval and Text Rendering and 0.01 for Pickscore.
  • Experiments use 512 resolution and LoRA with α = 64 and r = 32.
  • Training uses 24 NVIDIA A800 GPUs, with GPU-hour details reported in Appendix C.5 learning curves.

C.1 Flow-GRPO vs. Other Alignment Methods

Flow-GRPO outperforms the compared alignment methods across the reported evaluations, while online updates and stochastic initial noise support its training behavior. The comparisons use matched group sizes and training prompts where appropriate.

  • Flow-GRPO consistently outperforms SFT, Flow-RWR, Flow-DPO, and their online variants by a significant margin.
  • Offline methods use a fixed pretrained data-collection model, whereas online variants update that model every 40 steps.
  • Online DPO surpasses offline DPO, while excessively small β values can cause online DPO training collapse.
  • Figure 8 uses training prompts on the x-axis because methods such as DPO use different tuned batch sizes from Flow-GRPO.
  • DDPO’s reward increases more slowly and later collapses, whereas Flow-GRPO trains stably and continues improving over time.
  • With different initial noise, Flow-GRPO consistently achieves high rewards during training compared with using the same initial noise.

C.4 Additional Results on FLUX.1-Dev

Flow-GRPO extends evaluation beyond SD3.5-M to FLUX.1-Dev and examines reward optimization, KL regularization, and qualitative training behavior. The results emphasize steadily increasing reward without noticeable hacking and improved capability with preserved image quality when KL is used.

  • Additional Results on FLUX.1-Dev: FLUX.1-Dev with Flow-GRPO is evaluated using PickScore, with reward increasing steadily without noticeable reward hacking.The comparison uses DrawBench results for the base and fine-tuned models.
  • KL Regularization: KL regularization can reach the same high reward as the KL-free version while maintaining image quality, but requires longer training.The learning-curve comparison covers three tasks and distinguishes KL regularization from early stopping.
  • Qualitative Comparisons: Flow-GRPO with KL improves GenEval, OCR, and PickScore capabilities while maintaining image quality and minimizing reward hacking.Removing KL substantially degrades image quality and diversity in the qualitative comparisons.
  • Training Dynamics: Generated samples are visualized at regular training intervals using a fixed 40-step ODE schedule to show progressive quality and task alignment improvements.The visualizations correspond to GenEval, OCR, and PickScore optimization.

D Training Sample Visualization with Denoising Reduction

The denoising-reduction analysis compares ODE and SDE sampling across inference schedules and visualizes qualitative outcomes across reward settings and training iterations. Although very short SDE schedules introduce artifacts, they can accelerate optimization because relative preferences still provide useful reward signals.

  • Denoising Reduction: Figure 19 compares ODE and SDE samples using 40-step, 10-step, and 5-step schedules to illustrate denoising reduction.The four settings include ODE with 40 steps, SDE with 40 steps, SDE with 10 steps, and SDE with 5 steps.
  • KL Regularization: Learning curves compare optimization with and without KL, showing that KL slows early training while suppressing reward hacking.The figure presents learning curves for three tasks.
  • Denoising Reduction: 40-step ODE and SDE sampling produce visually indistinguishable images, while 10-step and 5-step SDE sampling introduces color drift and blurred fine details.The shorter schedules nevertheless accelerate optimization because Flow-GRPO uses relative preferences to extract a useful reward signal.
  • Qualitative Comparisons: Qualitative comparisons cover GenEval, OCR, and PickScore versions of SD3.5-M enhanced with Flow-GRPO.Additional figures compare the base model with Flow-GRPO-enhanced models for each reward setting.
Loading 2505.05470v5…