Source-linked AI summary
GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping
Jing Wang, Jiajun Liang, Jie Liu, Henglin Liu, Gongye Liu, Jun Zheng, Wanyuan Pang, Ao Ma, Zhenyu Xie, Xintao Wang, Meng Wang, Pengfei Wan, Xiaodan Liang
TL;DR
GRPO-based reinforcement learning can over-optimize flow-matching models because left-shifted, timestep-inconsistent importance ratios leave positive updates insufficiently clipped. GRPO-Guard combines ratio normalization with gradient reweighting, and experiments show reduced over-optimization while preserving or improving generation quality across models and proxy tasks.
Problem
Left-shifted and timestep-inconsistent importance-ratio distributions can let overconfident positive updates escape clipping, degrading generation quality despite rising proxy rewards.
Method
GRPO-Guard combines ratio normalization with gradient reweighting to regulate clipping and stabilize policy updates across denoising steps.
Results
GRPO-Guard substantially reduces over-optimization across GRPO variants, diffusion backbones, and proxy tasks while maintaining or improving generation quality.
Takeaways & Limitations
The method offers a robust, generalizable approach for stable policy optimization in diffusion-based generation models without relying on heavy KL regularization.
Takeaways & Limitations
GRPO-Guard cannot fully eliminate reward hacking caused by limitations of the reward model and the gap between proxy and gold scores.
Abstract
from arXiv · showhide
Recently, GRPO-based reinforcement learning has shown remarkable progress in optimizing flow-matching models, effectively improving their alignment with task-specific rewards. Within these frameworks, the policy update relies on importance-ratio clipping to constrain overconfident positive and negative gradients. However, in practice, we observe a systematic shift in the importance-ratio distribution-its mean falls below 1 and its variance differs substantially across timesteps. This left-shifted and inconsistent distribution prevents positive-advantage samples from entering the clipped region, causing the mechanism to fail in constraining overconfident positive updates. As a result, the policy model inevitably enters an implicit over-optimization stage-while the proxy reward continues to increase, essential metrics such as image quality and text-prompt alignment deteriorate sharply, ultimately making the learned policy impractical for real-world use. To address this issue, we introduce GRPO-Guard, a simple yet effective enhancement to existing GRPO frameworks. Our method incorporates ratio normalization, which restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates across denoising timesteps. In addition, a gradient reweighting strategy equalizes policy gradients over noise conditions, preventing excessive updates from particular timestep regions. Together, these designs act as a regulated clipping mechanism, stabilizing optimization and substantially mitigating implicit over-optimization without relying on heavy KL regularization. Extensive experiments on multiple diffusion backbones (e.g., SD3.5M, Flux.1-dev) and diverse proxy tasks demonstrate that GRPO-Guard significantly reduces over-optimization while maintaining or even improving generation quality.
1. Introduction
GRPO-based reinforcement learning improves diffusion-model alignment, but importance-ratio clipping can fail under timestep-dependent distributional bias, causing implicit over-optimization. GRPO-Guard addresses this with ratio normalization and gradient balancing, reducing over-optimization across models and tasks.
- GRPO-style reinforcement learning has been applied to diffusion models, building on its success in large language model alignment.
- Importance-ratio clipping is intended to stabilize training by symmetrically constraining overconfident positive and negative updates around a ratio of 1.
- A left-shifted, timestep-inconsistent ratio distribution prevents positive-advantage samples from being clipped, allowing overconfident updates and degrading image fidelity and text–prompt alignment despite rising proxy rewards.
- GRPO-Guard normalizes ratios across denoising steps and balances gradients to restore consistent clipping and reduce timestep-specific over-optimization.
- GRPO-Guard converges quickly while substantially reducing over-optimization across Flow-GRPO and DanceGRPO, multiple diffusion backbones, and diverse proxy tasks.
2. Related Works
Related work transfers reinforcement-learning alignment methods from language models to diffusion models, but proxy-reward optimization remains vulnerable to reward over-optimization. In flow-matching models, importance-ratio bias can undermine clipping and motivate methods that restore stable policy updates.
- GRPO methods provide a lower-complexity alignment approach that has yielded stable improvements in instruction following and preference alignment for large language models.
- Reinforcement-learning methods such as PPO and DPO have been transferred to diffusion models for preference alignment and visual-generation improvement.
- Reward over-optimization arises because learned proxy rewards may improve while diverging from true objectives such as perceptual quality or human preference.
- Heavy KL regularization can reduce policy drift but may slow gains in both proxy and true-performance metrics, while clipping provides another update constraint.
- In flow-matching models, importance-ratio bias lets overconfident positive updates pass unchecked and drives the policy toward over-optimization.
3. Method
The method identifies timestep-dependent importance-ratio anomalies in diffusion-policy optimization and introduces RatioNorm plus gradient reweighting to restore effective clipping and balance updates across denoising steps.
- 3.2.1. Inherent Distributional Anomalies: Gaussian transition probabilities create a timestep-dependent negative bias, leaving positive-advantage updates insufficiently clipped when importance ratios remain below one.Scheduler-dependent variance further makes clipping inconsistent across timesteps, driving step-specific over-optimization.
- 3.2.2. Regulated Clipping: A dedicated clipping range for every timestep would require many hyperparameters and extensive model- and task-specific tuning.The method instead uses normalization to reduce timestep-dependent distribution differences more simply.
- 3.2.2. Regulated Clipping: Figure 2 shows that GRPO-Guard preserves a balanced ratio mean and consistent variance across timesteps, unlike FlowGRPO’s left shift and low-noise variance increase.The corrected distribution enables proper clipping and more stable policy updates.
- 3.2.2. Regulated Clipping: RatioNorm standardizes log importance ratios so their mean moves toward zero while preserving the sign and relative magnitude of the policy-mean difference.This removes denoising-scheduler effects from the ratio distribution and restores effective upper and lower clipping bounds.
- 3.2.3. Gradient Analysis: FlowGRPO gradients increase as noise decreases, and timestep-dependent gradient scales can cause particular noise conditions to dominate optimization.The analysis motivates reweighting to reduce gradient imbalance across denoising steps.
- 3. Method: GRPO-Guard combines RatioNorm with gradient reweighting to regulate clipping, reduce over-optimization, and maintain a similar upward proxy-score trend.The combined method is reported to substantially alleviate over-optimization while preserving proxy-score improvement.
4. Experiments
Experiments across diffusion backbones, proxy tasks, visual comparisons, ablations, and human evaluation show that GRPO-Guard mitigates reward hacking while preserving generation quality and alignment.
- Main Results: GRPO-Guard achieves superior gold scores under comparable proxy scores across GenEval, PickScore, and OCR settings on Flux and SD3.5M.The composite gold-score comparison evaluates FlowGRPO and DanceGRPO baselines across multiple tasks and backbones.
- Main Results: GRPO-Guard maintains high gold scores and image quality during training, whereas baseline proxy-score increases are followed by sharp gold-score declines.Validation curves compare proxy and gold scores across training tasks and baseline methods.
- Main Results: Visual comparisons show that GRPO-Guard preserves image quality and instruction following while FlowGRPO and DanceGRPO exhibit severe reward-hacking artifacts.FlowGRPO degrades in quality, text accuracy, and adherence; DanceGRPO produces horizontal and vertical stripe artifacts.
- Main Results: Across training steps, GRPO-Guard prevents the degeneration in which baselines increasingly favor text correctness while losing scene richness, diversity, and text-image consistency.The baseline models enter over-optimization around mid-training, while GRPO-Guard remains visually comparable to the base model and improves text accuracy.
- Ablation Study: Gradient reweighting provides a more moderate proxy-score increase while substantially reducing the gold-score decline in the ablation study.The ablation evaluates ratio mean correction, inter-step variance alignment, and gradient balancing on FlowGRPO with SD3.5M for OCR.
- Human Evaluation: Human preference evaluation on GenEval and OCR finds GRPO-Guard clearly superior in image quality and overall quality, indicating degraded visual fidelity for baselines.Evaluators compare 100 sample pairs using win/tie/lose ratios.
5. Conclusion and Limitation
GRPO-Guard regulates importance-ratio clipping and stabilizes policy updates, mitigating over-optimization while preserving or enhancing generation quality. Its remaining limitation is reward hacking caused by the reward model’s gap from gold evaluation, and improving that model would add computational cost.
- Conclusion: GRPO-Guard mitigates over-optimization, preserves or enhances generation quality, and stabilizes policy updates across denoising steps.The method combines ratio normalization and gradient reweighting to regulate clipping.
- Limitation: The approach cannot fully eliminate reward hacking because intrinsic reward-model limitations leave proxy scores misaligned with gold scores.This gap remains a direct limitation even after positive-sample clipping is reactivated.
- Limitation: Scaling the reward model could better approximate comprehensive gold scores, but sampling many outputs and rewards would substantially increase computation and prolong optimization.Designing a comprehensive, efficient reward model remains future work.