PaperlayerView exact revision on arXiv

Source-linked AI summary

Holder Policy Optimisation

Yuxiang Chen, Dingli Liang, Yihang Chen, Ziqin Gong, Chenyang Le, Zhaokai Wang, Jiachen Zhu, Lingyu Yang, Jianghao Lin, Weinan Zhang, Jun Wang

arXiv:2605.12058v2cs.LGcs.AI

TL;DR

Fixed token-probability aggregation in GRPO limits adaptation and exposes a concentration–stability trade-off across tasks. HölderPO uses the Hölder mean with a dynamically annealed p to balance sparse-signal amplification and variance control, achieving state-of-the-art results across mathematical and agentic benchmarks.

  • Problem

    Fixed aggregation mechanisms limit adaptability because different task signal densities require different trade-offs between gradient concentration and variance.

  • Method

    HölderPO unifies token-level probability aggregation through the Hölder mean and dynamically anneals p from a higher positive value to a negative value during training.

  • Results

    54.9% average accuracy across five mathematical benchmarks and 93.8% success on ALFWorld are reported as state-of-the-art results.

  • Takeaways & Limitations

    Larger p amplifies sparse learning signals, while smaller p controls gradient variance, motivating dynamic scheduling rather than a universal static configuration.

  • Takeaways & Limitations

    The annealing schedule requires task-specific tuning of p_high, p_low, and decay shape, and its optimal shape lacks theoretical characterization.

Abstract

from arXiv · show

Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggregation mechanism for this step fundamentally limits the algorithm's adaptability. Empirically, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. To resolve this, we propose \textbf{HölderPO}, a generalised policy optimisation framework unifying token-level probability aggregation via the Hölder mean. By explicitly modulating the parameter $p$, our framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove that a larger $p$ concentrates the gradient to amplify sparse learning signals, whereas a smaller $p$ strictly bounds gradient variance. Because no static configuration can universally resolve this concentration-stability trade-off, we instantiate the framework with a dynamic annealing algorithm that progressively schedules $p$ across the training lifecycle. Extensive evaluations demonstrate superior stability and convergence over existing baselines. Specifically, our approach achieves a state-of-the-art average accuracy of $54.9\%$ across multiple mathematical benchmarks, yielding a substantial $7.2\%$ relative gain over standard GRPO and secures an exceptional $93.8\%$ success rate on ALFWorld.

1 Introduction

Fixed token-probability aggregations create a concentration–stability trade-off that varies with task signal density. HölderPO addresses this by tuning the Hölder parameter p and dynamically annealing it during training.

  • Motivation: Fixed aggregations can collapse on dense-signal tasks or underperform on sparse-signal tasks.Standard GRPO (p = 1) may over-weight minor token errors on dense-signal tasks, while other fixed choices fail to capture sparse correct reasoning signals.
  • Method: HölderPO unifies token-level probability aggregation through an adaptable Hölder mean parameterized by p.The framework makes p a controllable parameter for shaping policy-gradient behavior.
  • Theory: Larger p concentrates gradient weight on a small subset of tokens, amplifying sparse learning signals, whereas smaller p tightens gradient variance.These are the two theoretically characterized endpoints of the concentration–stability trade-off.
  • Method: Dynamic annealing schedules p from a higher positive value to a negative value, moving from early signal amplification toward later variance-controlled convergence.This schedule is designed because no static p simultaneously provides both endpoint advantages.
  • Results: 54.9% average accuracy across five mathematical benchmarks represents a 7.2% relative gain over standard GRPO.The reported benchmarks are AIME, AMC, MATH, Minerva, and OlympiadBench.
  • Results: 93.8% success rate on ALFWorld demonstrates the method’s reported performance on agentic tasks.This result is reported alongside the mathematical-benchmark results.

2 Related Work

Prior work uses fixed or restricted token-aggregation schemes and auxiliary token-reweighting signals. HölderPO extends aggregation across the full real-valued p range and schedules p across training steps.

  • Token-Level Aggregation: GRPO uses the arithmetic mean, while GMPO and GSPO use the geometric mean to mitigate outlier variance.These methods differ in how token-level importance ratios are converted into sequence-level signals.
  • Token-Level Aggregation: PMPO restricts the power-mean exponent to p ∈ [0, 1] and adapts it per trajectory using clip-aware ESS matching.HölderPO instead extends p to the full real range and adapts it along the training timeline.
  • Token-Level Aggregation: HölderPO introduces p < 0 as an inverse-concentration phase and schedules p across training steps rather than per trajectory.The paper describes these as two distinctions from prior token-aggregation methods.
  • Token Reweighting via Auxiliary Signals: Auxiliary reweighting methods use entropy, token probability, response-confidence contributions, or selective KL masking instead of importance-ratio aggregation.The paper characterizes these approaches as orthogonal and potentially combinable with HölderPO.

3 HölderPO: A Generalised Aggregation Framework

HölderPO replaces static token-level aggregation with a Hölder-mean operator controlled by p, which reweights policy-gradient contributions while preserving their directions. The resulting concentration–variance trade-off motivates dynamically decreasing p from positive to low or negative values during training.

  • 3.1 Aggregation via the Hölder Mean: HölderPO parameterises sequence-level aggregation with a single scalar p across the real numbers, encompassing arithmetic and geometric mean choices as special cases.The framework aggregates token-level importance ratios into a sequence-level surrogate and uses sequence-level clipping in a PPO-style objective.
  • 3.1 Aggregation via the Hölder Mean: Varying p reweights token-level gradient directions without changing the underlying per-token log-gradient directions.The weights W_i,t(p) form a probability distribution over tokens, determining how strongly each direction contributes.
  • 3.2 Distributional Deformation and Gradient Concentration: As |p| increases, gradient-weight entropy decreases from its maximum at p = 0, while weights concentrate on maximal-ratio tokens for p → +∞ and minimal-ratio tokens for p → −∞.Positive p emphasizes high-importance-ratio tokens, p = 0 distributes contribution uniformly, and negative p emphasizes ratios below 1.
  • 3.3 Policy Gradient Variance Bound: Larger p amplifies targeted learning signals but loosens the variance bound, whereas smaller p tightens variance control while diluting or redirecting concentration.The paper identifies this as a structural bottleneck for long-horizon reasoning and concludes that no fixed p is uniformly optimal.
  • 3.4 A Dynamic p-Scheduling Strategy: A monotonically decreasing schedule moves p from a positive initial value to a low, possibly negative, terminal value across training.The early phase uses positive concentration for sparse high-magnitude signals, while the late phase contracts the variance bound for stable convergence.
  • 3.4 A Dynamic p-Scheduling Strategy: Figure 2 shows that the linear schedule inherits the early concentration of p = +2 while converging toward the controlled regime of p = −2.The figure’s static envelopes indicate that decreasing p monotonically tightens the gap between the largest and smallest token-level ratios.

4 Experiment

Experiments show that optimal p depends on signal density, motivating linear annealing from 2 to −2, which combines early signal amplification with late variance contraction across mathematical and agentic tasks.

  • Task-Specific Sensitivity of p: The optimal p value diverges significantly between AIME24 and MATH500.These benchmarks represent sparse- and dense-signal regimes, respectively.
  • Task-Specific Sensitivity of p: Higher p performs best on sparse-signal AIME24, while lower p performs better on dense-signal MATH500.AIME24 rewards amplification of rare pivotal reasoning steps; MATH500 benefits from tighter gradient-variance bounds.
  • Dynamic Scheduling: The dynamic scheduler linearly decays p from 2 to −2 to combine early aggressive updates with later stable optimization.The schedule is evaluated alongside the best static configuration and existing state-of-the-art baselines.
  • Main Performance and Dynamic Scheduling: The dynamic schedule strictly outperforms every static setting on the overall task average.Static endpoint choices can remain best on individual tasks, including p = 3 on AIME24 and p = −1 on MATH500.
  • Generalisation to Agentic Reasoning: 93.8% average success rate is achieved on ALFWorld with a schedule decaying from 1 to −1.The conservative schedule accounts for the base model’s weaker embodied-environment intuitions and avoids overly aggressive early amplification.

5 Conclusion

HölderPO uses the Hölder mean to control the concentration–stability trade-off through p, and dynamic annealing delivers state-of-the-art results across mathematical and agentic benchmarks.

  • Conclusion: HölderPO unifies importance-ratio aggregation through the Hölder mean, using p to amplify sparse signals or tighten gradient variance.Larger p concentrates gradient weight, whereas smaller p provides variance contraction.
  • Conclusion: 54.9% accuracy across five mathematical reasoning benchmarks and 93.8% success on ALFWorld are reported for dynamic p-annealing.The reported results are state-of-the-art across mathematical and agentic benchmarks.
  • Limitations: Schedule endpoints and decay shape require empirical tuning per task, and the optimal shape lacks theoretical characterisation.Linear decay performed best in the reported setup.
  • Limitations: Positive-concentration settings are more susceptible to reward hacking when verifiers provide false-positive signals.This limitation arises because high-importance-ratio tokens are amplified.
  • Future Work: An adaptive scheduler based on batch-level gradient variance or token-ratio dispersion is proposed to remove manual tuning.This is identified as a primary future-work direction.

B Training Dynamics: Entropy and Gradient-Norm

Training dynamics differ by p: positive values reduce entropy through concentration, negative values increase it through dispersion, while linear annealing combines both regimes and narrows gradient fluctuations.

  • Entropy Dynamics: Positive p concentrates mass on high-likelihood tokens and pushes policy entropy down.Negative p disperses mass and pushes entropy up.
  • Entropy Dynamics: The linearly decaying schedule p:2→−2 inherits concentration early and dispersion later.It transitions from the positive-p regime to the negative-p regime during training.
  • Gradient-Norm Dynamics: The dynamic schedule keeps gradient norm in a tighter band than any constant-p choice.Figure 3 reports gradient norm on a log scale for Math and a linear scale for Alfworld.
  • Implementation: HölderPO computes sequence-level aggregation while allowing token-level or block-level aggregation without changing the algorithm or theory.Power operations are evaluated in log-space for numerical stability at large |p|.

D Token-Level Clipping of HölderPO

Token-level clipping weakens HölderPO’s controlled p-dependent trade-off by breaking the algebraic structure underlying monotonic variance bounds.

  • Token-Level Clipping: Token-level clipping produces a noticeably narrower performance spread across p than sequence-level clipping.The comparison is conducted on Qwen2.5-Math-7B across five mathematical benchmarks.
  • Token-Level Clipping: Token-level clipping breaks the algebraic structure underlying the variance bound’s monotonicity in p.This weakens the controlled concentration–stability trade-off motivating dynamic scheduling.

E Schedule-Shape Ablation

The schedule-shape ablation compares annealing families and tests HölderPO's transfer across Qwen3-Base model sizes. Descending linear annealing remains competitive while gains over aggregation baselines persist across scales.

  • E Schedule-Shape Ablation: 54.9% is achieved by the descending linear schedule, and no tested alternative annealing shape surpasses it on Qwen2.5-Math-7B.The sweep covers linear, square, cube, and sinusoidal interpolation in both ascending and descending directions.
  • F Generalisation to Qwen3 Base Models: HölderPO transfers beyond Qwen2.5-Math-7B through matched comparisons with GRPO, GSPO, and DAPO on the Qwen3-Base series.This experiment directly tests generalisation across base-model settings.
  • F Generalisation to Qwen3 Base Models: 50.9% average accuracy is achieved by HölderPO on Qwen3-4B-Base, exceeding DAPO by 4.5 points and GRPO by 7.0 points.The evaluation reports Pass@1 for MATH500, AMC23, Minerva, and Olympiad, and Pass@8 for AIME25.
  • F Generalisation to Qwen3 Base Models: 56.9% average accuracy is achieved by HölderPO on Qwen3-8B-Base, exceeding DAPO by 4.3 points and GRPO by 8.9 points.The reported advantage increases at the 8B model scale.

G Formulas and Derivation

This appendix derives HölderPO objectives and policy gradients for unclipped, token-level clipped, and sequence-level clipped settings, including the p = 0 geometric-mean case.

  • G Formulas and Derivation: The derivation defines prompt, response, token, policy, length, and advantage notation before omitting KL regularisation from PPO-style formulas.Rewards are assigned to generated responses and advantages are group-relative estimates.
  • G.1 No Clipping Formulas: HölderPO replaces arithmetic aggregation of token importance ratios with the Hölder-p mean ρi,p(θ).The resulting unclipped objective and policy gradient are derived from this sequence-level aggregation.
  • G.2 Token-Level Clipping Formulas: GRPO's token-level clipped objective is recovered as the p = 1 special case of the HölderPO formulation.The positive- and negative-advantage cases correspond to the respective clipped terms.
  • G.2 Token-Level Clipping Formulas: Token-level clipping reduces each token's gradient contribution to zero when its ratio crosses the relevant clipping boundary.The clipping function restricts ratios to [1 − ϵ, 1 + ϵ], with the active contribution depending on advantage sign.
  • G.3 Sequence-Level Clipping Formulas: Sequence-level clipping extends the same framework by clipping the aggregated ratio ρi,p(θ), with GSPO corresponding to the p = 0 case.The derivation identifies GSPO's sequence ratio as ρi,0(θ).
  • G.4 p = 0 Formulas: The p = 0 objective is defined by continuous extension when the relevant function has a removable singularity and is jointly C1.Under this condition, differentiation with respect to θ commutes with taking the limit as p approaches 0.
  • G.4 p = 0 Formulas: The p = 0 policy-gradient formulas follow because the extended objective is C1 and its θ-gradient is continuous at p = 0.The argument applies the commutativity result to the no-clipping objective.

H.1 Local Property

The local analysis characterises how p reallocates token-level gradient weights according to importance ratios. Increasing p favours maximal-ratio tokens, while non-maximal tokens can peak and then decay.

  • H.1 Local Property: The local theorem studies the token-weight allocation induced by the aggregation parameter p.The analysis introduces the weighted mean of token log-ratios as the central quantity.
  • H.1 Local Property: As p increases, maximal-ratio tokens gain weight monotonically and converge to uniform mass over the optimal-token subset.For finite p, the optimal-token weight rises because its log-ratio exceeds the weighted mean.
  • H.1 Local Property: The derivative of Wi,t(p) is governed by the token's log-ratio relative to the weighted sequence mean μyi(p).The proof rewrites the weights exponentially and applies the quotient rule.
  • H.1 Local Property: The weighted sequence mean μyi(p) is strictly increasing in p when a sequence contains at least two differing importance ratios.Its derivative reduces to a strictly positive weighted variance of log-ratios.
  • H.1 Local Property: Each non-maximal token can reach a peak at a critical p before its weight strictly decays as the weighted mean overtakes its log-ratio.As p approaches infinity, all sub-optimal token weights converge to zero.

H.2 Global Property

The global analysis connects p to entropy, gradient concentration, and variance. Positive p concentrates weights on high-ratio tokens, whereas lower p increases stability but does not alter rollout sampling.

  • H.2 Global Property: The weight distribution's Shannon entropy reaches its global maximum at p = 0 when importance ratios are non-uniform.At p = 0, weights are uniform and entropy equals ln |yi|.
  • H.2 Global Property: As p approaches positive or negative infinity, weights concentrate uniformly on maximal- or minimal-ratio token subsets, respectively.The limiting subsets are T+ = arg max_t ri,t(θ) and T− = arg min_t ri,t(θ).
  • H.3 Gradient Concentration vs. Exploration-Exploitation Trade-off: Positive p concentrates gradients on high-ratio tokens, p near 0 disperses them uniformly, and negative p upweights low-ratio hesitant tokens.The paper frames these as three gradient-concentration regimes.
  • H.3 Gradient Concentration vs. Exploration-Exploitation Trade-off: On AIME24, static p = 3 reaches 46.7% accuracy, while p = +2 rapidly lowers policy entropy during early training.These findings are presented as empirical support for aggressive concentration on sparse reasoning signals.
  • H.3 Gradient Concentration vs. Exploration-Exploitation Trade-off: The paper distinguishes negative-p diversity preservation from classical exploration because p reweights already-sampled trajectories without changing rollout generation.The mechanism operates retrospectively on gradient priorities rather than on the behavioural policy.
  • I Variance and Clipping: The unclipped gradient variance bound increases monotonically with p under a bounded token-gradient norm assumption.The theorem assumes ∥∇θ log πθ(yi,t | x, yi,<t)∥ ≤ M for all batch tokens.
  • I.2 Variance and Sequence-level Clipping: Without clipping, the surrogate scaling factor undergoes unbounded exponential growth, making explicit clipping necessary.The result follows by integrating the ratio's differential equation over time.
  • I.2 Variance and Sequence-level Clipping: Token-level clipping disrupts the structural relation supporting the monotonic variance bound, leaving p's variance control less predictable.The appendix reports that token-level clipping narrows the performance spread across p.

I.4 Monotonicity of Variance

Under idealised orthogonality and uniform expected token-gradient norms, HölderPO’s variance decreases from positive p toward zero, then has a global minimum at some p*≤0. The analysis attributes this pattern to the opposing behaviour of the Hölder mean and the weight-distribution concentration index.

  • Assumptions: The variance analysis assumes exact token-level gradient orthogonality and uniformly bounded expected gradient norms.These assumptions eliminate cross-terms and produce an exact second-moment expression for the aggregated estimator.
  • Positive p: As p decays from +∞ to 0, the variance strictly decreases.Both the squared Hölder mean and the HHI decrease over this positive-p range.
  • Conclusion: There exists a variance-minimising p*≤0.The variance is continuous and bounded below on the nonpositive domain, while it increases throughout the positive domain.
  • Positive p: For positive p, increasing p concentrates token weights, while decreasing p disperses them toward a uniform distribution.The HHI is minimised by the uniform distribution, linking weight dispersion to the variance trend.
  • Negative p: As p approaches −∞, weights collapse onto minimum-ratio tokens and the HHI rises exponentially toward 1.This concentration counteracts the continuing decay of the Hölder mean.

J Quantitative Advantage of Dynamic Scheduling

The quantitative analysis motivates dynamic scheduling by showing that high p is useful before sparse signals saturate, whereas lower p becomes appropriate afterward. This schedule shifts from signal amplification to variance contraction as training progresses.

  • Signal amplification: Before saturation, increasing p provides a positive lower-bounded amplification of a rare high-ratio token’s gradient weight.The condition R^p_high ≪ n^-1 ensures the high-ratio token has not yet monopolised the weight distribution.
  • Signal amplification: The pre-saturation regime represents a discovered high-reward reasoning token that remains insufficiently confident relative to the sequence background.Long trajectories and early-phase low confidence make this condition relevant to reasoning tasks.
  • Dynamic scheduling: Once saturation occurs, further increasing p becomes ineffective because the high-ratio token’s weight is already monopolised.At this point, the dynamic schedule decays p toward p_low≤0.
  • Dynamic scheduling: Dynamic scheduling decays p toward p_low≤0 after saturation, shifting the optimisation focus from signal amplification to variance contraction.This design connects the changing signal regime to the two-sided variance and concentration trade-off.
  • Broader impacts: Gradient amplification in the positive-p regime can intensify reward hacking when learning signals are misspecified.The paper identifies this as a framework-specific limitation alongside standard dual-use risks.
Loading 2605.12058v2…