MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

Hongjun Wang; Wei Liu; Weibo Gu; Xing Sun; Kai Han

Source-linked AI summary

MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning

Hongjun Wang, Wei Liu, Weibo Gu, Xing Sun, Kai Han

arXiv:2603.16929v2Mar 14, 2026cs.LGcs.AIcs.CL

TL;DR

GRPO-based reinforcement learning remains vulnerable to unstable importance ratios, whose clipping-based controls can lose gradient fidelity and lack hazard-aware treatment of extreme deviations. MHPO addresses this with a bounded differentiable LFM and asymmetric DHP, and evaluations report consistent state-of-the-art performance and improved training stability across model families and reasoning tasks.

Problem
Extreme importance-ratio variance destabilizes GRPO training, while prevailing clipping methods introduce gradient discontinuities and vanishing-gradient regions and lack hazard-aware asymmetric regulation.
Method
MHPO combines an LFM that smoothly maps unbounded importance ratios into a bounded domain with a DHP that independently regulates positive and negative policy shifts.
Results
MHPO consistently achieves state-of-the-art results across instruction-tuned, base, and vision-language model configurations, with stable optimization reported across the evaluation suite.
Takeaways & Limitations
MHPO unifies gradient fidelity and damping at the gradient-multiplier level to regulate policy shifts while preserving stable reinforcement-learning updates.

Abstract

from arXiv · show

Regulating the importance ratio is critical for the training stability of Group Relative Policy Optimization (GRPO) based frameworks. However, prevailing ratio control methods, such as hard clipping, suffer from non-differentiable boundaries and vanishing gradient regions, failing to maintain gradient fidelity. Furthermore, these methods lack a hazard-aware mechanism to adaptively suppress extreme deviations, leaving the optimization process vulnerable to abrupt policy shifts. To address these challenges, we propose Modulated Hazard-aware Policy Optimization (MHPO), a novel framework designed for robust and stable reinforcement learning. The proposed MHPO introduces a Log-Fidelity Modulator (LFM) to map unbounded importance ratios into a bounded, differentiable domain. This mechanism effectively prevents high-variance outlier tokens from destabilizing the loss landscape while ensuring global gradient stability. Complementarily, a Decoupled Hazard Penalty (DHP) integrates cumulative hazard functions from survival analysis to independently regulate positive and negative policy shifts. By shaping the optimization landscape with hazard-aware penalties, the proposed MHPO achieves fine-grained regulation of asymmetric policy shifts simultaneously mitigating mode collapse from over-expansion and preventing policy erosion from catastrophic contraction within a stabilized trust region. Extensive evaluations on diverse reasoning benchmarks across both text-based and vision-language tasks demonstrate that MHPO consistently outperforms existing methods, achieving superior performance while significantly enhancing training stability.

1 Introduction

MHPO targets instability in GRPO-based reinforcement learning by combining a bounded, differentiable ratio modulator with hazard-aware regulation of asymmetric policy shifts. Evaluations across text and vision-language reasoning tasks report stronger performance and training stability than existing methods.

Motivation: Extreme token-level importance-ratio variance in long-form CoT generation can produce massive gradient spikes and severe training instability.Sequence-level multiplication can make ratios fluctuate across multiple orders of magnitude.
Motivation: Hard clipping constrains ratios but introduces gradient discontinuities and vanishing-gradient regions, while soft gating does not decouple directional policy-shift risks.These effects prevent out-of-region tokens from contributing to learning and limit gradient fidelity.
Approach: MHPO combines the Log-Fidelity Modulator (LFM) and Decoupled Hazard Penalty (DHP) to stabilize GRPO-based optimization.LFM maps unbounded importance ratios into a bounded, differentiable manifold; DHP regulates positive and negative policy shifts asymmetrically.
Approach: DHP uses cumulative hazard functions to encourage safe exploration while suppressing catastrophic policy erosion.The mechanism provides fine-grained control over the optimization landscape.
Results: MHPO consistently outperforms state-of-the-art baselines in performance and training stability across pure-text reasoning and multimodal vision-language benchmarks.The overview reports the largest performance gain on every benchmark shown.

2 Related Work

Prior GRPO-related methods stabilize policy optimization through clipping, soft gating, sequence-level control, or other trust-region mechanisms, but each leaves part of ratio-induced gradient control unresolved. MHPO instead operates directly at the token-level gradient multiplier, combining smoothness, bounded attenuation, and fidelity without changing the advantage or importance ratio.

Policy Optimization: GRPO uses group-based advantages from multiple rollouts, avoiding the learned critic required by PPO.The critic can impose substantial optimization burden in long chain-of-thought settings.
Ratio Control: DAPO uses asymmetric clipping boundaries, while SAPO replaces hard clipping with a sigmoid-based soft gate.GSPO moves ratio control to the sequence level to reduce token-level variance, at the cost of fine-grained credit assignment.
Ratio Control: Clipping bounds ratios but sacrifices gradient fidelity, soft gating restores smoothness without provable tail attenuation, and sequence-level control reduces token-level precision.These methods address different facets of ratio control rather than providing a unified gradient-level mechanism.
MHPO: MHPO operates at the gradient-multiplier level, unifying fidelity and damping at token granularity without altering the advantage or importance ratio.Its focus is distinct from rollout-, reward-, and advantage-centric methods that leave ratio-induced gradient scaling unchanged.
Trust Regions: Trust-region methods balance stable policy updates with computational efficiency through explicit KL constraints or implicit ratio-based mechanisms.Ratio-based mechanisms are practical at scale but can introduce gradient discontinuities and binary on/off control.
MHPO: MHPO’s gradient multiplier is smooth and differentiable everywhere while exhibiting principled tail decay that attenuates extreme deviations.This unifies operational goals associated with KL-based and ratio-based trust-region methods.

3 Methodology

MHPO combines GRPO preliminaries with a Log-Fidelity Modulator and Decoupled Hazard Penalty to stabilize importance-ratio optimization. LFM provides bounded, differentiable, directionally symmetric control, while DHP adds asymmetric hazard-aware regulation for positive and negative policy shifts.

GRPO Preliminaries: GRPO samples K responses from a reference policy, factorizes each response likelihood over tokens, and uses token-level importance ratios with group-relative advantages.The importance ratio measures deviation from the reference policy, while the group-relative advantage is shared within the response group.
Policy Shifts: Positive shifts increase token probability and negative shifts decrease it, with excessive expansion risking mode collapse and excessive contraction risking policy erosion.Shift direction is independent of advantage sign; their product determines the final update dynamics.
Log-Fidelity Modulator: The LFM projects importance ratios into symmetric log-space and applies a scaled tanh transformation that bounds contributions within [−c, c] while remaining differentiable.This smooth saturation contrasts with hard clipping, which introduces gradient discontinuities and vanishing regions.
Log-Fidelity Modulator: Near the on-policy anchor, the LFM preserves standard policy-gradient behavior, with dψ/dr = 1 and high fidelity for small policy updates.The local mapping is approximately the identity in log-space, supporting efficient learning without bias for on-policy samples.
Log-Fidelity Modulator: For extreme importance ratios, the LFM smoothly attenuates gradients so outlier tokens still contribute without dominating the global loss landscape.The resulting attenuation supports more predictable convergence while avoiding the complete truncation associated with hard clipping.
Decoupled Hazard Penalty: DHP complements symmetric LFM saturation by independently regulating positive and negative shifts through hazard-aware penalty shaping based on cumulative hazard functions.The Weibull cumulative hazard mechanism is introduced to dynamically modulate penalty severity, while bounded gradient multipliers support stable mini-batch second moments.

4 Experiments

MHPO is evaluated across model types, reasoning modalities, benchmarks, and stability analyses. It consistently improves performance while its bounded-gradient design supports stable optimization.

Experimental design: Evaluations span base and instruction-tuned models, general-purpose and domain-specialized backbones, and text-only and vision-language reasoning tasks.The experiments use multiple architectures, datasets, baselines, and Avg@32 evaluation.
Main results: MHPO achieves state-of-the-art results across instruction-tuned and base-model configurations.The reported evaluation covers Qwen2.5 instruction-tuned variants and Qwen3-4B-Base without instruction tuning.
Main results: 43.7% macro-average on Qwen2.5-7B-Instruct exceeds DAPO by 3.5 points, while 47.8% on Qwen2.5-Math-7B-Instruct exceeds DAPO by 4.2 points.On HMMT25 and AIME25, the domain-specialized model gains 6.4 and 6.6 points over DAPO, respectively.
Main results: 53.0% macro-average on Qwen2.5-VL-7B-Instruct surpasses DAPO by 0.7 points and the zero-shot baseline by 5.5 points.MathVision shows the largest reported multimodal gain: +5.3 points over the baseline and +1.4 points over DAPO.
Main results: 52.9% macro-average on Qwen3-4B-Base surpasses GSPO by 5.7 points, with AIME25 improving from 34.0% to 41.4%.AIME24 likewise increases from 39.7% to 44.7%, a 5.0-point gain.
Hyperparameter sensitivity: The best smooth-bound setting is c=1.5, while k=3.0 and λ=0.8 provide the strongest reported damping-related performance.The c=1.5 operator spans ω(r) ∈[0.22, 4.48], and performance deteriorates as λ increases toward 2.0.
Training stability: MHPO maintains bounded gradient norms, whereas baselines show high-variance spikes, premature plateaus, or later reward degradation.SAPO frequently exceeds twice the baseline gradient magnitude, while MHPO remains stable throughout the trajectory.
Training stability: MHPO’s Qwen3-4B-Base performance degrades by 0.8 points from best to latest checkpoint, compared with DAPO’s 12.2-point drop.GRPO, GSPO, and SAPO show degradations ranging from 7.7 to 8.2 points.

5 Conclusion

MHPO reformulates importance-ratio control as joint fidelity and damping design. Experiments across model families and modalities report state-of-the-art mathematical-reasoning performance with pronounced gains.

Conclusion: MHPO jointly uses a smooth bounded operator for gradient fidelity and controllable asymmetric damping for policy deviations.The design regulates the gradient multiplier that rescales the scorefunction gradient.
Conclusion: Experiments across general-purpose, domain-specialized, and vision-language models report consistent state-of-the-art performance on mathematical reasoning benchmarks.The conclusion highlights particularly pronounced gains across these settings.