Source-linked AI summary
E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Models
Shengjun Zhang, Zhang Zhang, Chensheng Dai, Yueqi Duan
TL;DR
Existing flow-model GRPO methods face sparse and ambiguous rewards because stochasticity accumulates across denoising steps, while low-entropy steps produce undistinguished roll-outs. E-GRPO merges consecutive low-entropy SDE steps, uses ODE sampling elsewhere, and computes group-relative advantages within consolidated steps; experiments report consistent gains over prior methods, with a practical limitation of residual reward hacking on challenging prompts.
Problem
Existing GRPO-based flow-model methods optimize across denoising timesteps, producing sparse and ambiguous reward signals, while low-entropy steps yield undistinguished roll-outs.
Method
E-GRPO consolidates consecutive low-entropy SDE steps into one effective SDE step, applies ODE sampling to other steps, and computes group-relative advantages within consolidated steps.
Results
E-GRPO consistently outperforms prior methods across single-reward and multi-reward settings; with HPS-v2.1, it surpasses DanceGRPO by 10.8% on HPS and by 4.4% on PickScore.
Takeaways & Limitations
Entropy-guided optimization concentrates updates on informative denoising steps and supports broader, more reliable alignment across reward models.
Takeaways & Limitations
Reward hacking still occurs on some challenging prompts, including cases where models introduce unintended human faces or humanoid shapes.
Abstract
from arXiv · showhide
Recent reinforcement learning has enhanced the flow matching models on human preference alignment. While stochastic sampling enables the exploration of denoising directions, existing methods which optimize over multiple denoising steps suffer from sparse and ambiguous reward signals. We observe that the high entropy steps enable more efficient and effective exploration while the low entropy steps result in undistinguished roll-outs. To this end, we propose E-GRPO, an entropy aware Group Relative Policy Optimization to increase the entropy of SDE sampling steps. Since the integration of stochastic differential equations suffer from ambiguous reward signals due to stochasticity from multiple steps, we specifically merge consecutive low entropy steps to formulate one high entropy step for SDE sampling, while applying ODE sampling on other steps. Building upon this, we introduce multi-step group normalized advantage, which computes group-relative advantages within samples sharing the same consolidated SDE denoising step. Experimental results on different reward settings have demonstrated the effectiveness of our methods.
1. Introduction
Existing GRPO-based flow-model methods optimize across denoising timesteps despite sparse and ambiguous rewards. E-GRPO focuses stochastic exploration on high-entropy steps, consolidates low-entropy steps, and uses group-normalized advantages for more reliable alignment.
- Motivation: High-entropy timesteps produce diverse samples and distinguishable reward variations, whereas low-entropy timesteps yield nearly undistinguishable roll-outs.The latter resemble perturbations from adding 10% random noise to the final image.
- Motivation: Existing GRPO-based flow-model methods optimize across multiple denoising timesteps, creating sparse and ambiguous reward signals that hinder alignment.Prior work combines SDE and ODE sampling or uses tree-based structures to reduce sampling steps.
- E-GRPO: E-GRPO consolidates multiple low-entropy SDE steps into one effective SDE step while keeping remaining steps deterministic through ODE sampling.This preserves exploration where it is informative while avoiding cumulative stochasticity across low-entropy steps.
- E-GRPO: Multi-step group normalized advantage computes group-relative advantages among samples sharing the same consolidated SDE step.The method is designed to provide denser and more trustworthy reward signals for trajectory alignment.
- Evaluation: E-GRPO is evaluated under single-reward and multi-reward settings, with in-domain and out-of-domain evaluations, and is reported to outperform prior methods.The contribution summary characterizes the approach as entropy-guided stochastic optimization for flow models.
2. Related Works
Prior alignment methods extend reinforcement-learning and preference-optimization ideas to visual generation, but differ in their sampling formulation and credit-assignment strategy. Recent flow-model approaches use stochastic trajectories, finer timestep granularity, or temporal weighting to improve policy updates.
- RL Alignment for Image Generation: PPO-based alignment uses a value model, whereas DPO-derived methods extend direct preference optimization to diffusion models.The passage notes that diffusion-DPO-style methods generate no new samples during training and therefore suffer from distribution shifting.
- RL Alignment for Image Generation: DanceGRPO and Flow-GRPO reformulate deterministic ODE sampling as stochastic SDE trajectories for GRPO-style policy updates in visual domains.This connects stochastic sampling with reinforcement-learning optimization for image generation.
- RL Alignment for Image Generation: GranularGRPO refines timestep granularity for denser credit assignment, while TempFlow-GRPO introduces temporally aware weighting to address uniform optimization.
3. Methods
E-GRPO uses entropy analysis to retain stochastic exploration at informative denoising steps while consolidating consecutive low-entropy steps into single SDE transitions. It then computes group-relative advantages within shared merged-step groups and optimizes the resulting clipped objective.
- GRPO for flow models: Flow-based GRPO frames reverse SDE sampling as an MDP whose actions are denoising transitions and whose terminal reward evaluates the generated image.The policy uses group-normalized advantages, while the KL-regularization term is omitted from the objective.
- Entropy analysis: E-GRPO classifies timesteps using an adaptive entropy threshold τ, treating steps with entropy at least τ as high-entropy and the remainder as low-entropy.The entropy expression is derived for reverse SDE steps, with the entropy of a merged timestep increasing with the number of merged steps.
- Entropy-driven step merging: Consecutive low-entropy SDE steps are consolidated into one equivalent SDE transition, while other steps use ODE sampling to preserve stochastic exploration where it is informative.The merged interval preserves the total diffusion effect and avoids cumulative stochasticity that can make reward attribution ambiguous.
- Entropy-driven step merging: The adaptive merge length l is selected so the merged-step entropy just exceeds τ, avoiding excessively large entropy and unreliable optimization directions under limited exploration.This strategy replaces a uniform merge length with a timestep-specific choice.
- Multi-step group normalized advantage: For each active timestep, E-GRPO generates trajectories sharing the same consolidated timestep set and computes advantages within that merge group.The merge-grouped advantages align reward signals with the consolidated SDE steps before the clipped surrogate objective updates the policy.
4. Experiments
Experiments evaluate E-GRPO across reward settings, visualization comparisons, training dynamics, and step-merging ablations. The method achieves stronger preference-alignment results, improves training behavior, and benefits from entropy-aware merging and high-entropy-step optimization.
- Experimental Settings: Experiments use the HPD dataset with approximately 103,000 training prompts and 400 test prompts, using FLUX.1-dev as the backbone model.Evaluation includes representative reward models for human-preference alignment.
- Main Experiments: 10.8% higher HPS than DanceGRPO under the single HPS-v2.1 reward establishes new state-of-the-art performance.The comparison includes FLUX.1-dev, DanceGRPO, MixGRPO, BranchGRPO, TempFlowGRPO, and GranularGRPO.
- Main Experiments: 32.4% higher ImageReward and 4.4% higher PickScore than DanceGRPO under joint HPS-v2.1 and CLIP rewards improve out-of-domain performance.The joint-reward setting maintains state-of-the-art in-domain HPS performance and addresses reward hacking associated with HPS-only training.
- Qualitative Results: E-GRPO produces more semantically grounded and visually coherent images than baseline methods in qualitative comparisons.Examples show better integration of object structure, human-like attributes, expressive faces, and material texture.
- Training Dynamics: E-GRPO shows faster early reward growth, smoother convergence, and a higher final reward than prior work.The training curves support more efficient and reliable optimization behavior.
- Ablation Studies: Entropy-aware adaptive merging consistently outperforms naive 2-step, 4-step, and 6-step merging across almost all evaluation metrics.Table 2 reports higher HPS, CLIP, PickScore, and ImageScore for the proposed strategy, while high-entropy early-step training achieves higher alignment with lower computational cost.
5. Conclusion
The work identifies high-entropy denoising steps as the informative source of exploration and proposes E-GRPO to address sparse, ambiguous rewards in flow-model GRPO. Its entropy-aware step merging and multi-step group normalized advantage provide more reliable reward signals for preference alignment.
- E-GRPO targets sparse and ambiguous reward signals caused by uniform optimization across denoising timesteps.
- High-entropy timesteps support meaningful exploration and preference alignment, whereas low-entropy timesteps produce undistinguished rollouts.
- The method merges consecutive low-entropy SDE steps into single high-entropy SDE steps while retaining ODE sampling elsewhere.
- Multi-step group normalized advantage computes relative advantages among samples sharing a consolidated step, producing denser and more reliable reward signals.
6. Ablation Study on the Entropy Threshold τ
The entropy threshold τ controls how consecutive low-entropy steps are merged. Increasing τ improves HPS until excessive merging becomes too coarse; τ = 2.2 provides the best reported balance.
- The adaptive threshold τ separates high-entropy and low-entropy timesteps during E-GRPO sampling.
- Consecutive low-entropy steps are merged until their combined entropy reaches τ, making the threshold central to step merging.
- As τ increases, the HPS score improves, but excessively large values can merge useful entropy or gradient information and cause coarse updates.
- τ = 2.2 achieves the best overall performance by balancing entropy guidance against excessive merging.
7. Additional Visualizations
Additional qualitative results show that E-GRPO more faithfully follows difficult text prompts and improves aesthetic quality relative to baseline methods. The failure analysis also identifies reward hacking, including unwanted humanoid features, as a recurring limitation.
- 7.1. More Quality Results: E-GRPO produces more prompt-faithful results than baseline methods in additional qualitative comparisons.
- 7.1. More Quality Results: For a space-age lemon portrait and a curvy-road scene, E-GRPO better captures the requested composition or attribute while achieving higher aesthetic quality.
- 7.1. More Quality Results: The reported qualitative gains are attributed to focusing exploration on high-entropy steps for alignment with complex human preferences.
- 7.2. Failure Cases: Reward hacking remains a failure mode: HPS-only training can oversaturate images, while some challenging prompts produce unwanted human faces or humanoid shapes.
- 7.2. Failure Cases: The analysis identifies reward-model reliability as an important direction for improving reinforcement learning in visual generation.