PaperlayerView exact revision on arXiv

Source-linked AI summary

MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

Junzhe Li, Yutao Cui, Tao Huang, Weijie Kong, Yiming Cheng, Chuxuan Zeng, Yinping Ma, Chun Fan, Miles Yang, Zhao Zhong, Liefeng Bo

arXiv:2507.21802v7cs.AIcs.CV

TL;DR

Flow-based GRPO methods improve human-preference alignment but are inefficient because they sample and optimize across all denoising steps. MixGRPO uses a mixed ODE-SDE process with a sliding optimization window, while MixGRPO-Flash further accelerates training with comparable performance.

  • Problem

    Flow-GRPO and DanceGRPO optimize complete stochastic denoising sequences, causing prohibitive computational overhead and unfocused optimization.

  • Method

    MixGRPO applies SDE sampling and GRPO optimization within a sliding window while using ODE sampling outside it, enabling higher-order solvers beyond the window.

  • Results

    MixGRPO consistently improves performance and reduces training overhead across multiple frameworks, while MixGRPO-Flash provides a faster comparable-performance variant.

  • Takeaways & Limitations

    The framework supports more focused coarse-to-fine optimization and substantially lower training cost for flow-based GRPO alignment.

  • Takeaways & Limitations

    MixGRPO remains subject to reward-model limitations, including potential reward hacking, and its scheduler relies on predefined settings.

Abstract

from arXiv · show

Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose $\textbf{MixGRPO}$, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed $\textbf{MixGRPO-Flash}$, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

1 Introduction

MixGRPO confines stochastic optimization to a sliding SDE window within a mixed ODE-SDE framework, reducing overhead while focusing updates. It outperforms DanceGRPO in alignment and training efficiency, with MixGRPO-Flash providing further acceleration.

  • Motivation: Flow-GRPO and DanceGRPO optimize complete stochastic denoising sequences, creating prohibitive computational overhead and unfocused optimization.Their rollouts use SDE sampling at every denoising step, framing the process as a stochastic MDP for GRPO optimization.
  • MixGRPO: MixGRPO confines policy optimization to a sliding SDE sub-interval that advances from low-SNR exploration toward high-SNR refinement.The schedule prioritizes early global structure before progressively refining local details.
  • Results: MixGRPO improves ImageReward from 1.088 to 1.629, exceeding DanceGRPO’s 1.436, while producing better semantic fidelity, aesthetics, and fewer distortions.On FLUX, it also reduces training time by nearly 50% versus DanceGRPO’s official setting.
  • Efficiency: The mixed ODE-SDE framework enables high-order ODE solvers for deterministic sampling outside the optimized interval, accelerating training with negligible performance degradation.Deterministic sampling of πθold can be accelerated because timesteps outside the window are not optimized.
  • Results: MixGRPO consistently achieves performance gains while reducing training overhead across multiple frameworks, and MixGRPO-Flash offers a faster comparable-performance variant.The paper presents MixGRPO-Flash as a further efficiency improvement enabled by the hybrid framework.

2 Related Work

Prior flow-based GRPO methods use broad SDE exploration but optimize many denoising steps. MixGRPO formulates a mixed SDE-ODE MDP to select fewer stochastic steps and improve efficiency.

  • Existing methods: FlowGRPO and DanceGRPO use SDE-induced stochasticity for broader exploration but remain bottlenecked by optimization across many denoising steps.Global-SDE sampling creates redundant effective MDP horizons and inefficient optimization.
  • MixGRPO: MixGRPO combines mixed ODE-SDE sampling with a sliding-window strategy to reduce the number of optimized timesteps and improve training efficiency.The methods comparison specifically contrasts this design with DanceGRPO.
  • MixGRPO: The mixed formulation exploits SDE-ODE equivalence in probability flow to establish a more efficient MDP for optimizing selected stochastic timesteps.This connects stochastic exploration with the use of higher-order ODE sampling methods.

3 Method

MixGRPO combines SDE sampling and GRPO optimization within a sliding window with ODE sampling outside it, shortening the optimized MDP. A progressive window schedule and higher-order solvers reduce overhead while preserving alignment quality.

  • Mixed ODE-SDE Sampling: MixGRPO restricts SDE sampling and policy optimization to a sliding interval while using ODE sampling outside it.The interval shifts along the denoising direction during training, shortening the optimized trajectory to a subset of timesteps.
  • Policy Optimization: MixGRPO’s GRPO objective sums reward and KL-regularization terms only over timesteps in S.This concentrates optimization on the selected stochastic sub-interval rather than the complete denoising sequence.
  • Mixed ODE-SDE Sampling: The mixed process uses SDE dynamics inside S and deterministic ODE dynamics otherwise.Under stated regularity assumptions, it preserves the corresponding probability-flow ODE’s time-marginal dynamics up to discretization and score-approximation error.
  • Sliding-Window Scheduler: The sliding-window curriculum advances from low-SNR, high-stochasticity steps toward high-SNR, more deterministic steps for refinement.The window is controlled by its size, shift interval, and stride, with a fixed progressive-constant scheduler used as the default.
  • Trade-off Between Overhead and Performance: Higher-order ODE solvers accelerate post-window sampling, while excessive acceleration can reduce image quality and corrupt reward computation.The authors report that 2nd-order DPMSolver++ provides significant acceleration while maintaining alignment with human preferences.

4 Experiments

Experiments evaluate MixGRPO across datasets, reward models, alignment baselines, model scales, sampling strategies, and solver settings. The results show improved alignment and efficiency, with MixGRPO-Flash providing additional acceleration while preserving image quality.

  • MixGRPO achieves stronger performance than DanceGRPO while optimizing only 4 timesteps instead of 14 randomly selected from 25.This reduces per-iteration GRPO overhead.
  • MixGRPO-Flash reduces sampling time from 291.284s to 83.278s while maintaining strong image quality as overhead changes.
  • MixGRPO consistently outperforms DanceGRPO and the FLUX baseline on both in-domain and out-of-domain reward metrics.The evaluation uses aesthetics, semantic-consistency, Pick Score, ImageReward, and Unified Reward.
  • MixGRPO maintains strong alignment performance across other methods, larger-scale image generation, and video-generation extensions.Experiments include SD3.5-M and the 80B-parameter HunyuanImage-3.0 backbone trained on 512 GPUs.
  • Progressive sliding-window movement is consistently strong, while the second-order midpoint ODE solver is the preferred acceleration setting.Progressive-Constant shows strong overall and out-of-domain performance, and Table 8 identifies the second-order midpoint method as optimal for human-preference metrics.
  • CPS consistently improves all reward metrics relative to standard SDE sampling.The paper attributes this to reduced sampling artifacts and more reliable optimization signals.

5 Limitation

MixGRPO does not remove limitations originating from imperfect reward models, and its current scheduler still depends on predefined hyperparameters.

  • GRPO’s performance ceiling remains tied to reward-model capability, with weaker models making reward hacking more likely during later training.The authors state that MixGRPO targets faster convergence and stronger performance under imperfect reward models rather than eliminating these limitations.
  • The sliding-window scheduler relies on predefined w, τ, and s values, motivating adaptive scheduling based on online signals as future work.

6 Conclusion

The paper presents MixGRPO as a hybrid ODE-SDE framework that confines optimization to a dynamic stochastic interval. MixGRPO improves alignment while reducing overhead, and MixGRPO-Flash exposes a performance–cost trade-off through faster sampling.

  • MixGRPO confines GRPO optimization to a dynamic stochastic interval managed by a sliding window.The schedule moves from broad exploration toward fine-grained refinement.
  • MixGRPO achieves superior performance in single-reward and multi-reward settings while substantially reducing overhead.
  • MixGRPO-Flash offers a flexible trade-off between performance and computational cost.

7 Related Work

Prior work introduced reinforcement-learning and preference-optimization approaches for aligning generative models, while diffusion research developed complementary SDE and ODE sampling formulations. MixGRPO builds on their unification by interleaving SDE and ODE sampling to shorten the effective optimization horizon.

  • Early reinforcement-learning methods optimized diffusion-model score functions with policy gradients to improve human-preference alignment.
  • Offline DPO enabled text-to-image models to learn directly from human-preference pairs without explicit reward modeling.
  • DDPM used slow SDE-based sampling, while DDIM introduced deterministic ODE-based sampling with substantially fewer steps.
  • Theoretical equivalence between flow matching and diffusion SDE/ODE formulations supports interleaved sampling that confines stochasticity to RL-optimization intervals.The paper states that this shortens the effective MDP horizon and reduces training overhead.

8 Proof of Convergence for Mixed ODE-SDE Sampling

The proof compares mixed ODE-SDE sampling with ODE-only sampling through distribution evolution. It concludes that their evolution within the SDE interval is consistent.

  • Distribution-evolution setup: The proof defines the SDE interval as S = [t_l, t_r) and compares probability-distribution evolution under mixed sampling with ODE sampling.With the same initial Gaussian noise distribution, the preceding ODE interval evolves identically; equivalence within S preserves the subsequent ODE evolution.
  • SDE and Fokker–Planck dynamics: Within S, the SDE has drift f(x,t) − g^2(t)∇_x log q_t(x) and diffusion g(t)dw.Its marginal density follows the corresponding Fokker–Planck equation.
  • Equivalence derivation: Using the Laplace-operator and score-function identities, the SDE Fokker–Planck equation is rewritten in probability-flow form.The resulting drift is identified as f_ODE(x,t) = f(x,t) − 1/2 g^2(t)∇_x log q_t(x).
  • Conclusion: The resulting equation is the Fokker–Planck equation of the ODE, so SDE and ODE sampling have consistent distribution evolution within S.This establishes the claimed convergence equivalence at the distribution level, subject to the derivation’s assumptions and approximation setting.

9 DPM-Solver++ for Recitified Flow

This section adapts DPM-Solver++ to Rectified Flow by relating velocity prediction to x_0 prediction and deriving a higher-order transfer update. The resulting solver is used outside the optimization window in MixGRPO-Flash.

  • Notation: Discrete sampling steps i are distinguished from continuous times t_i across a total of T sampling steps.This notation separates the solver’s discrete update index from its continuous-time condition.
  • Rectified Flow parameterization: Rectified Flow uses x_{t_i} = t_i x_1 + (1 − t_i)x_0 and models the displacement v_{t_i} = x_1 − x_0.These relations imply x_0 = x_{t_i} − v_{t_i}t_i.
  • Prediction conversion: A neural network converts the Rectified Flow velocity estimate into an x_0-prediction through x_θ(x_i,t_i,c) = x_i − v_θ(x_i,t_i,c)t_i.This connects the RF model output to the prediction format expected by DPM-Solver++.
  • Higher-order solver: DPM-Solver++ forms a second-order multistep correction D_i from the current and previous model-dependent states.The correction uses h_i = λ_{t_i} − λ_{t_{i−1}}, where λ_{t_i} is the log-SNR.
  • Integration into MixGRPO-Flash: MixGRPO-Flash uses first-order ODE sampling before the window, SDE sampling inside it, and DPM-Solver++ sampling after it.Policy optimization is performed only over timesteps in the sliding window.

10 MixGRPO-Flash Algorithm

MixGRPO-Flash accelerates sampling outside the sliding window with DPM-Solver++ and a compression rate. Its progressive window movement yields an average speedup bounded by the frozen-window speedup.

  • ODE compression: MixGRPO-Flash reduces post-window ODE sampling to (T − l − w)r̃ steps, giving total sampling steps T̃ = l + w + (T − l − w)r̃.The compression applies only to ODE sampling after the window.
  • Frozen-window variant: For the frozen-window variant MixGRPO-Flash*, the left boundary remains l = 0.This fixed placement determines the stated theoretical speedup expression.
  • Progressive scheduling: Progressive window movement produces an average speedup S = T / E_l[w + l + ceil((T − w − l)r̃)].The paper states this is less than the frozen-window expression T / [w + (T − w)r̃].

11 Implementation Details

The implementation trains FLUX.1-dev and SD3.5-M with specified GPU, sampling, optimization, and precision settings. FLUX.1-dev uses 25 sampling steps and generates 12 images per prompt.

  • Sampling and GRPO settings: FLUX.1-dev uses T = 25 sampling steps, 12 images per prompt, advantage clipping to [−5, 5], and four gradient updates per iteration.The four updates result from three-step gradient accumulation.
  • Optimization settings: The optimizer configuration uses AdamW with learning rate 1 × 10^-5 and weight decay 1 × 10^-4.Training uses bf16 mixed precision while retaining master weights in fp32.
  • Model configurations: The implementation also reports training on SD3.5-M with 24 NVIDIA GPUs.The supplied configuration passage begins the SD3.5-M setup but does not provide its complete details.

12 Multi-step MDP GRPO vs. Alignment with Single-step Generators

MixGRPO scales alignment to industrial-size models while reducing computational burden through selective optimization, with stable gains across datasets, hyperparameters, and video generation. Its efficiency–alignment trade-off supports practical scaling, though some acceleration choices can degrade image quality.

  • Efficiency Trade-offs: Accelerating both ODE segments in MixGRPO-Flash degrades high-frequency detail and increases colour saturation, with degradation worsening during training.The Post configuration preserves detail and medium colour saturation better than the Dual configuration.
  • Industrial-Scale Scaling: MixGRPO focuses reinforcement-learning updates within a sliding window, concentrating gradients and reducing the effective optimization search space.This design is described as resolving gradient conflicts that affect full-trajectory reinforcement learning in extremely large models.
  • Text-to-Video Extension: MixGRPO improves stability and convergence across video-generation metrics, producing monotonic gains in HPSv3, Motion Quality, and Visual Quality.The method is reported to significantly outperform Flow-GRPO, whose gains are marginal and degrade at some stages.
  • Hyperparameter Sensitivity: τ = 25 is consistently optimal across in-domain and out-of-domain evaluations, whereas τ = 30 causes a sharp performance decline below the τ = 15 baseline.The reported interpretation is that overly slow window movement over-optimizes certain timesteps and can diverge from reward-model preferences.
  • Hyperparameter Robustness: MixGRPO consistently outperforms DanceGRPO and FlowGRPO across reasonable window-size and stride settings, indicating robustness to these choices.Performance curves remain relatively flat within a reasonable window-size range, including 2 ≤ w ≤ 6.
  • Hyperparameter Sensitivity: w = 4 offers the best performance–overhead trade-off, while s = 1 provides the strongest overall generalizability and benefits from repeated optimization of high-SNR timesteps.The repeated optimization of later window timesteps is associated with improved performance in the reported analysis.
  • Reward-Hacking Mitigation: Fixing intra-group initial noise improves HPS-v2.1, Pick Score, and ImageReward while mitigating reward hacking, and p_mix = 80% is an effective hybrid-inference setting.Hybrid inference applies the GRPO-trained model to the initial high-SNR denoising steps and the original model afterward.
Loading 2507.21802v7…