Source-linked AI summary
DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
Hongbo Jin, Rongpeng Zhu, Zhongjing Du, Xu Jiang, Jingqi Tian, Qiaoman Zhang, Jiayu Ding
TL;DR
Existing critic-free reinforcement-learning methods struggle to assign credit within long reasoning sequences, while unbounded KL penalties destabilize exploration and encourage conservatism. DGPO redistributes sequence-level advantages using bounded Hellinger deviation and entropy gating, achieving strong AIME results and preserving critic-free efficiency. Its reported scope is primarily mathematical reasoning, with broader alignment domains still requiring validation.
Problem
Current critic-free RL methods broadcast coarse sequence-level advantages and use unbounded KL penalties, limiting fine-grained credit assignment and stable exploration.
Method
DGPO uses bounded Hellinger distance and policy entropy gating to redistribute sequence-level advantages toward meaningful exploratory tokens without an auxiliary value network.
Results
DGPO achieves 60.0% Avg@32 on AIME 2024 and 46.0% Avg@32 on AIME 2025, outperforming competitive baselines such as DAPO.
Takeaways & Limitations
DGPO provides fine-grained credit reallocation with process-level signals while retaining critic-free computational efficiency comparable to GRPO.
Takeaways & Limitations
Evaluations focus on AIME mathematical reasoning, so efficacy in creative generation, coding, and general instruction-following remains extensively unvalidated.
Abstract
from arXiv · showhide
Reinforcement learning is crucial for aligning large language models to perform complex reasoning tasks. However, current algorithms such as Group Relative Policy Optimization suffer from coarse grained, sequence level credit assignment, which severely struggles to isolate pivotal reasoning steps within long Chain of Thought generations. Furthermore, the standard unbounded Kullback Leibler divergence penalty induces severe gradient instability and mode seeking conservatism, ultimately stifling the discovery of novel reasoning trajectories. To overcome these limitations, we introduce Distribution Guided Policy Optimization, a novel critic free reinforcement learning framework that reinterprets distribution deviation as a guiding signal rather than a rigid penalty. DGPO replaces the volatile KL divergence with the bounded Hellinger distance to safely quantify token level exploration without the risk of gradient explosion. To effectively distinguish genuine reasoning breakthroughs from hallucinatory noise, we propose an entropy gating mechanism that scales this deviation by the policy`s epistemic uncertainty. By dynamically redistributing the coarse sequence-level advantage to individual tokens based on these gated scores, DGPO heavily incentivizes critical exploratory steps while suppressing unwarranted, low-entropy deviations. Consequently, DGPO completely eliminates the traditional token-level KL penalty and achieves fine-grained credit reallocation without the computational overhead of an additional value network. Extensive empirical evaluations demonstrate that DGPO sets a new state-of-the-art for critic free alignment. Notably, on the Qwen2.5-32B architecture, DGPO achieves 60.0% Avg@32 accuracy and 46.0% Avg@32 accuracy on the challenging AIME2024 and AIME2025 benchmarks respectively, substantially outperforming competitive baselines like DAPO.
1 Introduction
The paper identifies coarse sequence-level credit assignment and unbounded Reverse KL penalties as barriers to exploration, then proposes DGPO to redistribute advantages token-wise using bounded distribution deviation. DGPO combines Hellinger distance and entropy gating to emphasize critical reasoning steps, achieving strong benchmark performance while retaining critic-free efficiency.
- GRPO broadcasts one sequence-level advantage across thousands of tokens, diluting distinctions between pivotal reasoning breakthroughs and redundant syntax.
- Unbounded Reverse KL penalizes novel low-probability trajectories harshly, causing gradient instability and mode-seeking conservatism.
- DGPO reallocates coarse sequence-level advantages into fine-grained token-level signals and treats distribution deviation as an exploration guide.
- DGPO replaces Reverse KL with bounded Hellinger distance and uses entropy gating to filter confident hallucinations from useful exploratory deviations.
- 60.0% Avg@32 accuracy on AIME 2024 and 46.0% on AIME 2025 establish DGPO as a state-of-the-art critic-free alignment method, outperforming DAPO.
- DGPO preserves critic-free efficiency, adding only 3.6% time overhead while maintaining a memory footprint comparable to vanilla GRPO.
2 Related Works
Prior work addresses LLM reinforcement-learning credit assignment through critic-free optimization, process rewards, and heuristic reward redistribution. These approaches reduce some costs but leave open the need for theoretically grounded, self-contained token-level credit assignment.
- PPO relies on an auxiliary value network, creating prohibitive memory and computational overhead as model parameters scale.
- Process Reward Models assign scalar rewards to intermediate reasoning steps, but robust training requires exhaustive human-annotated step-level data.
- Attention-based and heuristic-decay methods redistribute sequence-level rewards but often lack rigorous theoretical justification and fail to isolate pivotal reasoning steps.
- DGPO bypasses external Process Reward Models and parameterized critics by extracting probabilistic deviations between current and reference policies for token-level credit assignment.
3 Distribution-Guided Policy Optimization
DGPO converts GRPO’s coarse sequence-level advantage into token-level guidance by treating distribution deviation as a bounded, entropy-gated exploration signal. It then reallocates credit with normalized weights to emphasize pivotal exploratory tokens while maintaining stable, critic-free optimization.
- Preliminaries and Motivation: GRPO broadcasts one sequence-level advantage across all tokens, preventing the model from distinguishing pivotal reasoning breakthroughs from redundant transitional phrases.
- Distribution-Guided Advantage Redistribution: DGPO interprets distribution deviation as token-level guidance rather than an extrinsic penalty, transforming the sequence-level advantage into fine-grained credit.
- Distribution-Guided Advantage Redistribution: The bounded Hellinger distance prevents penalty explosion when the reference policy assigns near-zero probability during exploration.
- Distribution-Guided Advantage Redistribution: Entropy gating filters confident hallucinations by reducing the influence of deviations that may represent fake innovations rather than meaningful reasoning steps.
- Fine-Grained Credit Reallocation and Objective: Because divergence enters as a bounded multiplicative reweighting factor, DGPO avoids unbounded KL-induced gradient spikes while retaining critic-free token-level credit assignment.
- Fine-Grained Credit Reallocation and Objective: DGPO converts gated scores into temperature-scaled, sequence-length-adjusted weights, preserving unit mean across tokens and reallocating the local advantage as Ai,t = Ai · wi,t.
4 Experiment
DGPO is evaluated on AIME benchmarks across model scales, component ablations, hyperparameter sensitivities, and computational profiling. The experiments show stronger reasoning performance, benefits from entropy gating and bounded Hellinger distance, and modest computational overhead.
- Main Results: 60.0% Avg@32 accuracy on AIME 2024 surpasses DAPO’s 50.0%, while AIME 2025 Avg@32 rises from 38.0% to 46.0%.Consensus accuracy on AIME 2024 increases from 60.0% to 73.0%.
- Main Results: 43.0% Pass@1 accuracy on Qwen2.5-7B-Math exceeds vanilla GRPO at 22.0% and DAPO at 36.0%.These results evaluate peak average Pass@1 across 32 samples.
- Ablation Studies: Removing entropy gating causes a 5% AIME 2024 degradation, supporting its role in filtering confident hallucinations from deviation-based updates.The ablation sets κ = 0 and relies only on token-level distribution deviation.
- Ablation Studies: Replacing Hellinger distance with Reverse KL yields only 34.0% Pass@1 and early convergence to suboptimal local minima.Token-level reallocation remains active in this comparison.
- Hyperparameter Sensitivity Analysis: Moderate reallocation temperatures, including τ ∈[0.5, 1.0], provide stable credit assignment, whereas extreme values dilute or sparsify updates.Very large τ approaches uniform token weights, while τ = 0.1 produces abrupt signals.
- Computational Profiling: DGPO reaches 188 tokens/s/GPU with only 3.6% time overhead relative to GRPO, while using roughly 46 GB per GPU.The profiling contrasts DGPO and GRPO with PPO’s approximately 72.4 GB per GPU requirement.
- Credit Assignment Analysis: DGPO amplifies gradients for critical exploratory steps instead of uniformly broadcasting one sequence-level advantage across all tokens.The analysis identifies pivotal mathematical substitutions and logical deductions as examples of amplified steps.
5 Conclusion
DGPO addresses coarse credit assignment and gradient instability by using bounded distribution deviation and entropy gating to guide token-level exploration. Evaluations on AIME 2024 and 2025 report superiority over competitive baselines.
- DGPO is a critic-free reinforcement learning framework designed to resolve coarse-grained credit assignment and gradient instability.
- Bounded Hellinger distance guides exploration, while policy entropy gating distinguishes genuine reasoning leaps from hallucinatory noise.
A Theoretical Analysis and Proofs
The theoretical analysis establishes that DGPO’s Hellinger-based mechanism avoids the gradient explosion associated with KL divergence under extreme exploration. It also analyzes the framework’s stochastic approximation properties.
- Appendix A.1 proves that Hellinger-distance gradients remain bounded when reference-policy probabilities approach zero, unlike KL gradients.
- Appendix A.2 provides a stochastic-approximation analysis of DGPO’s core mechanisms.
A.1 Gradient Stability and Boundedness: Hellinger vs. Kullback-Leibler
The analysis contrasts unbounded KL gradients with DGPO’s bounded Hellinger-based deviation and entropy-gated weighting. DGPO therefore keeps the resulting policy-gradient multiplier and overall gradient norm finite under extreme exploration.
- Gradient instability of KL: When reference probability approaches zero, the Reverse KL logarithmic term diverges and can produce an infinitely escalating gradient norm.
- Bounded Hellinger guidance: DGPO replaces the volatile KL penalty with bounded Hellinger distance used as an exploration-guiding signal.The Hellinger distance lies in the interval [0, 1].
- Entropy gating: Entropy gating multiplies distribution deviation by normalized policy entropy, producing a joint score bounded within [0, 1].
- Gradient boundedness: The DGPO weighting factor is strictly constrained, and the resulting gradient norm is capped by a finite upper bound M < ∞.
- Gradient boundedness: Even when reference probability approaches zero, deviation saturates smoothly at 1 without singularity, neutralizing gradient instability and alleviating mode-seeking conservatism.
A.2 Convergence Properties Preserved with Entropy Gating
DGPO reallocates sequence-level advantage using entropy-modulated token weights while preserving the total credit mass and the global policy-gradient direction. The analysis therefore retains the original update’s unbiased direction under trajectory expectation.
- Token-level reallocation: DGPO converts the sequence-level advantage into local token-level advantages using nonuniform, entropy-modulated weights.
- Credit conservation: Softmax reweighting enforces a unit-mean property across each sequence, preserving the total mass of reallocated credit.
- Convergence direction: Under trajectory expectation, DGPO maintains a positive inner product with the original policy gradient, keeping the global update direction unbiased.
A.3 Adaptive Step-Size Perspective of Entropy Gating.
Entropy gating modulates token-level update sizes according to epistemic uncertainty: it amplifies valid deviations during exploration and suppresses confident hallucinations. This preserves advantage signs while reducing gradient-estimator variance and supporting smoother optimization.
- Adaptive Scaling: The strictly positive scaling weight bifurcates according to epistemic uncertainty, determining whether deviation-driven updates are amplified or suppressed.The formulation defines the scaling weight as strictly positive before describing its entropy-dependent behavior.
- High-Entropy States (Deliberate Exploration): High-entropy states allow valid distribution deviations to amplify gradients at pivotal exploratory steps.When normalized entropy approaches one, the scaling weight is dominated by the distribution deviation.
- Low-Entropy States (Confident Hallucinations): Low-entropy, high-deviation states compress the joint score and diminish updates for likely confident hallucinations.The gate adaptively reduces the learning step size at spurious or overconfident tokens, preventing reinforcement of false positive signals.
- Optimization Consequences: Entropy gating changes relative update step sizes without changing advantage signs and satisfies the stated stochastic-approximation convergence prerequisites.The mechanism filters low-entropy, high-deviation noise, reducing gradient-estimator variance and producing a smoother optimization landscape.
B.1 Limitations
The study’s empirical validation is concentrated on mathematical reasoning, and its two new hyperparameters may require tuning when transferred to different tasks or architectures. Future work therefore targets broader task evaluation and adaptive parameter scheduling.
- Domain Specificity: DGPO’s empirical validation focuses on AIME 2024 and AIME 2025 mathematical reasoning benchmarks.Its effectiveness in creative generation, coding, and general instruction-following remains extensively unvalidated.
- Hyperparameter Sensitivity: The framework introduces reallocation temperature τ and entropy-gating factor κ, whose transfer to other tasks or architectures may require empirical tuning.The paper reports stability within moderate ranges, including τ ∈[0.5, 1.0], but warns about sparse updates or reinforcing confident hallucinations after transfer.
- Broader Task Generalization: Future work should evaluate DGPO across code generation, multi-turn conversational alignment, and complex agentic workflows.These settings are identified as broader reinforcement-learning tasks where fine-grained credit assignment may also matter.
- Dynamic Hyperparameter Scheduling: Future iterations could dynamically schedule τ and κ using training progress or moving-average advantages to reduce manual tuning.The proposed direction includes automatically decaying or increasing these parameters to further stabilize training.