PaperlayerView exact revision on arXiv

Source-linked AI summary

Holder Policy Optimisation

Yuxiang Chen, Dingli Liang, Yihang Chen, Ziqin Gong, Chenyang Le, Zhaokai Wang, Jiachen Zhu, Lingyu Yang, Jianghao Lin, Weinan Zhang, Jun Wang

arXiv:2605.12058v2cs.LGcs.AI

TL;DR

Fixed token-probability aggregation in GRPO limits adaptation and exposes a concentration–stability trade-off across tasks. HölderPO uses the Hölder mean with a dynamically annealed p to balance sparse-signal amplification and variance control, achieving state-of-the-art results across mathematical and agentic benchmarks.

  • Problem

    Fixed aggregation mechanisms limit adaptability because different task signal densities require different trade-offs between gradient concentration and variance.

    Source: §1 · p. 2, §4.2 · p. 6

  • Method

    HölderPO unifies token-level probability aggregation through the Hölder mean and dynamically anneals p from a higher positive value to a negative value during training.

    Source: §1 · p. 2

  • Results

    54.9% average accuracy across five mathematical benchmarks and 93.8% success on ALFWorld are reported as state-of-the-art results.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Larger p amplifies sparse learning signals, while smaller p controls gradient variance, motivating dynamic scheduling rather than a universal static configuration.

    Source: §1 · p. 2

    The annealing schedule requires task-specific tuning of p_high, p_low, and decay shape, and its optimal shape lacks theoretical characterization.

    Source: §5 · p. 9

Abstract

from arXiv · show

Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggregation mechanism for this step fundamentally limits the algorithm's adaptability. Empirically, we observe a critical trade-off: certain fixed aggregations frequently suffer from training collapse, while others fail to yield satisfactory performance. To resolve this, we propose \textbf{HölderPO}, a generalised policy optimisation framework unifying token-level probability aggregation via the Hölder mean. By explicitly modulating the parameter $p$, our framework provides continuous control over the trade-off between gradient concentration and variance bounds. Theoretically, we prove that a larger $p$ concentrates the gradient to amplify sparse learning signals, whereas a smaller $p$ strictly bounds gradient variance. Because no static configuration can universally resolve this concentration-stability trade-off, we instantiate the framework with a dynamic annealing algorithm that progressively schedules $p$ across the training lifecycle. Extensive evaluations demonstrate superior stability and convergence over existing baselines. Specifically, our approach achieves a state-of-the-art average accuracy of $54.9\%$ across multiple mathematical benchmarks, yielding a substantial $7.2\%$ relative gain over standard GRPO and secures an exceptional $93.8\%$ success rate on ALFWorld.

1 Introduction

Fixed token-probability aggregations create a concentration–stability trade-off that varies with task signal density. HölderPO addresses this by tuning the Hölder parameter p and dynamically annealing it during training.

Motivation

  • Fixed aggregations can collapse on dense-signal tasks or underperform on sparse-signal tasks.Standard GRPO (p = 1) may over-weight minor token errors on dense-signal tasks, while other fixed choices fail to capture sparse correct reasoning signals.Source: §1 · p. 2

Method

  • HölderPO unifies token-level probability aggregation through an adaptable Hölder mean parameterized by p.The framework makes p a controllable parameter for shaping policy-gradient behavior.Source: §1 · p. 2

Theory

  • Larger p concentrates gradient weight on a small subset of tokens, amplifying sparse learning signals, whereas smaller p tightens gradient variance.These are the two theoretically characterized endpoints of the concentration–stability trade-off.Source: §1 · p. 2

Method

  • Dynamic annealing schedules p from a higher positive value to a negative value, moving from early signal amplification toward later variance-controlled convergence.This schedule is designed because no static p simultaneously provides both endpoint advantages.Source: §1 · p. 2

Results

  • 54.9% average accuracy across five mathematical benchmarks represents a 7.2% relative gain over standard GRPO.The reported benchmarks are AIME, AMC, MATH, Minerva, and OlympiadBench.Source: §1 · pp. 2–3
  • 93.8% success rate on ALFWorld demonstrates the method’s reported performance on agentic tasks.This result is reported alongside the mathematical-benchmark results.Source: §1 · p. 3

2 Related Work

Prior work uses fixed or restricted token-aggregation schemes and auxiliary token-reweighting signals. HölderPO extends aggregation across the full real-valued p range and schedules p across training steps.

Token-Level Aggregation

  • GRPO uses the arithmetic mean, while GMPO and GSPO use the geometric mean to mitigate outlier variance.These methods differ in how token-level importance ratios are converted into sequence-level signals.Source: §2 · p. 3
  • PMPO restricts the power-mean exponent to p ∈ [0, 1] and adapts it per trajectory using clip-aware ESS matching.HölderPO instead extends p to the full real range and adapts it along the training timeline.Source: §2 · p. 3
  • HölderPO introduces p < 0 as an inverse-concentration phase and schedules p across training steps rather than per trajectory.The paper describes these as two distinctions from prior token-aggregation methods.Source: §2 · p. 3

Token Reweighting via Auxiliary Signals

  • Auxiliary reweighting methods use entropy, token probability, response-confidence contributions, or selective KL masking instead of importance-ratio aggregation.The paper characterizes these approaches as orthogonal and potentially combinable with HölderPO.Source: §2 · p. 3

3 HölderPO: A Generalised Aggregation Framework

HölderPO replaces static token-level aggregation with a Hölder-mean operator controlled by p, which reweights policy-gradient contributions while preserving their directions. The resulting concentration–variance trade-off motivates dynamically decreasing p from positive to low or negative values during training.

3.1 Aggregation via the Hölder Mean

  • HölderPO parameterises sequence-level aggregation with a single scalar p across the real numbers, encompassing arithmetic and geometric mean choices as special cases.The framework aggregates token-level importance ratios into a sequence-level surrogate and uses sequence-level clipping in a PPO-style objective.Source: §3 · p. 3, §3.1 · p. 4
  • Varying p reweights token-level gradient directions without changing the underlying per-token log-gradient directions.The weights W_i,t(p) form a probability distribution over tokens, determining how strongly each direction contributes.Source: §3.1 · p. 4

3.2 Distributional Deformation and Gradient Concentration

  • As |p| increases, gradient-weight entropy decreases from its maximum at p = 0, while weights concentrate on maximal-ratio tokens for p → +∞ and minimal-ratio tokens for p → −∞.Positive p emphasizes high-importance-ratio tokens, p = 0 distributes contribution uniformly, and negative p emphasizes ratios below 1.Source: §3.2 · p. 4

3.3 Policy Gradient Variance Bound

  • Larger p amplifies targeted learning signals but loosens the variance bound, whereas smaller p tightens variance control while diluting or redirecting concentration.The paper identifies this as a structural bottleneck for long-horizon reasoning and concludes that no fixed p is uniformly optimal.Source: §3.3 · p. 5

3.4 A Dynamic p-Scheduling Strategy

  • A monotonically decreasing schedule moves p from a positive initial value to a low, possibly negative, terminal value across training.The early phase uses positive concentration for sparse high-magnitude signals, while the late phase contracts the variance bound for stable convergence.Source: §3.4 · p. 5
  • Figure 2 shows that the linear schedule inherits the early concentration of p = +2 while converging toward the controlled regime of p = −2.The figure’s static envelopes indicate that decreasing p monotonically tightens the gap between the largest and smallest token-level ratios.Source: §3.4 · p. 5

4 Experiment

Experiments show that optimal p depends on signal density, motivating linear annealing from 2 to −2, which combines early signal amplification with late variance contraction across mathematical and agentic tasks.

Task-Specific Sensitivity of p

  • The optimal p value diverges significantly between AIME24 and MATH500.These benchmarks represent sparse- and dense-signal regimes, respectively.Source: §4.2 · p. 6
  • Higher p performs best on sparse-signal AIME24, while lower p performs better on dense-signal MATH500.AIME24 rewards amplification of rare pivotal reasoning steps; MATH500 benefits from tighter gradient-variance bounds.Source: §4.2 · p. 6

Dynamic Scheduling

  • The dynamic scheduler linearly decays p from 2 to −2 to combine early aggressive updates with later stable optimization.The schedule is evaluated alongside the best static configuration and existing state-of-the-art baselines.Source: §4.3 · p. 7, Figure 2 · p. 7

Main Performance and Dynamic Scheduling

  • The dynamic schedule strictly outperforms every static setting on the overall task average.Static endpoint choices can remain best on individual tasks, including p = 3 on AIME24 and p = −1 on MATH500.Source: §4.3 · p. 7

Generalisation to Agentic Reasoning

  • 93.8% average success rate is achieved on ALFWorld with a schedule decaying from 1 to −1.The conservative schedule accounts for the base model’s weaker embodied-environment intuitions and avoids overly aggressive early amplification.Source: §4.5 · p. 9

5 Conclusion

HölderPO uses the Hölder mean to control the concentration–stability trade-off through p, and dynamic annealing delivers state-of-the-art results across mathematical and agentic benchmarks.

Conclusion

  • HölderPO unifies importance-ratio aggregation through the Hölder mean, using p to amplify sparse signals or tighten gradient variance.Larger p concentrates gradient weight, whereas smaller p provides variance contraction.Source: §5 · p. 9
  • 54.9% accuracy across five mathematical reasoning benchmarks and 93.8% success on ALFWorld are reported for dynamic p-annealing.The reported results are state-of-the-art across mathematical and agentic benchmarks.Source: §5 · p. 9

Limitations

  • Schedule endpoints and decay shape require empirical tuning per task, and the optimal shape lacks theoretical characterisation.Linear decay performed best in the reported setup.Source: §5 · p. 9
  • Positive-concentration settings are more susceptible to reward hacking when verifiers provide false-positive signals.This limitation arises because high-importance-ratio tokens are amplified.Source: §5 · p. 9

Future Work

  • An adaptive scheduler based on batch-level gradient variance or token-ratio dispersion is proposed to remove manual tuning.This is identified as a primary future-work direction.Source: §5 · p. 9

B Training Dynamics: Entropy and Gradient-Norm

Training dynamics differ by p: positive values reduce entropy through concentration, negative values increase it through dispersion, while linear annealing combines both regimes and narrows gradient fluctuations.

Entropy Dynamics

  • Positive p concentrates mass on high-likelihood tokens and pushes policy entropy down.Negative p disperses mass and pushes entropy up.Source: Figure 3 · p. 14
  • The linearly decaying schedule p:2→−2 inherits concentration early and dispersion later.It transitions from the positive-p regime to the negative-p regime during training.Source: Figure 3 · p. 14

Gradient-Norm Dynamics

  • The dynamic schedule keeps gradient norm in a tighter band than any constant-p choice.Figure 3 reports gradient norm on a log scale for Math and a linear scale for Alfworld.Source: Figure 3 · p. 14

Implementation

  • HölderPO computes sequence-level aggregation while allowing token-level or block-level aggregation without changing the algorithm or theory.Power operations are evaluated in log-space for numerical stability at large |p|.Source: C Pseudocode · p. 15

D Token-Level Clipping of HölderPO

Token-level clipping weakens HölderPO’s controlled p-dependent trade-off by breaking the algebraic structure underlying monotonic variance bounds.

Token-Level Clipping

  • Token-level clipping produces a noticeably narrower performance spread across p than sequence-level clipping.The comparison is conducted on Qwen2.5-Math-7B across five mathematical benchmarks.Source: Table 4 · p. 15
  • Token-level clipping breaks the algebraic structure underlying the variance bound’s monotonicity in p.This weakens the controlled concentration–stability trade-off motivating dynamic scheduling.Source: Table 4 · p. 15

E Schedule-Shape Ablation

The schedule-shape ablation compares annealing families and tests HölderPO's transfer across Qwen3-Base model sizes. Descending linear annealing remains competitive while gains over aggregation baselines persist across scales.

E Schedule-Shape Ablation

  • 54.9% is achieved by the descending linear schedule, and no tested alternative annealing shape surpasses it on Qwen2.5-Math-7B.The sweep covers linear, square, cube, and sinusoidal interpolation in both ascending and descending directions.Source: Table 5 · p. 16

F Generalisation to Qwen3 Base Models

  • HölderPO transfers beyond Qwen2.5-Math-7B through matched comparisons with GRPO, GSPO, and DAPO on the Qwen3-Base series.This experiment directly tests generalisation across base-model settings.Source: F Generalisation to Qwen3 Base Models · p. 16
  • 50.9% average accuracy is achieved by HölderPO on Qwen3-4B-Base, exceeding DAPO by 4.5 points and GRPO by 7.0 points.The evaluation reports Pass@1 for MATH500, AMC23, Minerva, and Olympiad, and Pass@8 for AIME25.Source: Table 6 · p. 16
  • 56.9% average accuracy is achieved by HölderPO on Qwen3-8B-Base, exceeding DAPO by 4.3 points and GRPO by 8.9 points.The reported advantage increases at the 8B model scale.Source: Table 7 · p. 16

G Formulas and Derivation

This appendix derives HölderPO objectives and policy gradients for unclipped, token-level clipped, and sequence-level clipped settings, including the p = 0 geometric-mean case.

G Formulas and Derivation

  • The derivation defines prompt, response, token, policy, length, and advantage notation before omitting KL regularisation from PPO-style formulas.Rewards are assigned to generated responses and advantages are group-relative estimates.Source: G Formulas and Derivation · p. 17

G.1 No Clipping Formulas

  • HölderPO replaces arithmetic aggregation of token importance ratios with the Hölder-p mean ρi,p(θ).The resulting unclipped objective and policy gradient are derived from this sequence-level aggregation.Source: §G.1 · pp. 17–18

G.2 Token-Level Clipping Formulas

  • GRPO's token-level clipped objective is recovered as the p = 1 special case of the HölderPO formulation.The positive- and negative-advantage cases correspond to the respective clipped terms.Source: §G.2 · pp. 18–19
  • Token-level clipping reduces each token's gradient contribution to zero when its ratio crosses the relevant clipping boundary.The clipping function restricts ratios to [1 − ϵ, 1 + ϵ], with the active contribution depending on advantage sign.Source: §G.2 · pp. 18–19

G.3 Sequence-Level Clipping Formulas

  • Sequence-level clipping extends the same framework by clipping the aggregated ratio ρi,p(θ), with GSPO corresponding to the p = 0 case.The derivation identifies GSPO's sequence ratio as ρi,0(θ).Source: §G.3 · p. 20

G.4 p = 0 Formulas

  • The p = 0 objective is defined by continuous extension when the relevant function has a removable singularity and is jointly C1.Under this condition, differentiation with respect to θ commutes with taking the limit as p approaches 0.Source: §G.4 · p. 21
  • The p = 0 policy-gradient formulas follow because the extended objective is C1 and its θ-gradient is continuous at p = 0.The argument applies the commutativity result to the no-clipping objective.Source: §G.4 · pp. 20–21

H.1 Local Property

The local analysis characterises how p reallocates token-level gradient weights according to importance ratios. Increasing p favours maximal-ratio tokens, while non-maximal tokens can peak and then decay.

H.1 Local Property

  • The local theorem studies the token-weight allocation induced by the aggregation parameter p.The analysis introduces the weighted mean of token log-ratios as the central quantity.Source: §H.1 · p. 22
  • As p increases, maximal-ratio tokens gain weight monotonically and converge to uniform mass over the optimal-token subset.For finite p, the optimal-token weight rises because its log-ratio exceeds the weighted mean.Source: §H.1 · pp. 22, 24
  • The derivative of Wi,t(p) is governed by the token's log-ratio relative to the weighted sequence mean μyi(p).The proof rewrites the weights exponentially and applies the quotient rule.Source: §H.1 · pp. 22–23
  • The weighted sequence mean μyi(p) is strictly increasing in p when a sequence contains at least two differing importance ratios.Its derivative reduces to a strictly positive weighted variance of log-ratios.Source: §H.1 · pp. 23–24
  • Each non-maximal token can reach a peak at a critical p before its weight strictly decays as the weighted mean overtakes its log-ratio.As p approaches infinity, all sub-optimal token weights converge to zero.Source: §H.1 · p. 24

H.2 Global Property

The global analysis connects p to entropy, gradient concentration, and variance. Positive p concentrates weights on high-ratio tokens, whereas lower p increases stability but does not alter rollout sampling.

H.2 Global Property

  • The weight distribution's Shannon entropy reaches its global maximum at p = 0 when importance ratios are non-uniform.At p = 0, weights are uniform and entropy equals ln |yi|.Source: §H.2 · pp. 24–26
  • As p approaches positive or negative infinity, weights concentrate uniformly on maximal- or minimal-ratio token subsets, respectively.The limiting subsets are T+ = arg max_t ri,t(θ) and T− = arg min_t ri,t(θ).Source: §H.2 · pp. 24, 26

H.3 Gradient Concentration vs. Exploration-Exploitation Trade-off

  • Positive p concentrates gradients on high-ratio tokens, p near 0 disperses them uniformly, and negative p upweights low-ratio hesitant tokens.The paper frames these as three gradient-concentration regimes.Source: §H.3 · p. 26
  • On AIME24, static p = 3 reaches 46.7% accuracy, while p = +2 rapidly lowers policy entropy during early training.These findings are presented as empirical support for aggressive concentration on sparse reasoning signals.Source: §H.3 · p. 26
  • The paper distinguishes negative-p diversity preservation from classical exploration because p reweights already-sampled trajectories without changing rollout generation.The mechanism operates retrospectively on gradient priorities rather than on the behavioural policy.Source: §H.3 · pp. 26–27

I Variance and Clipping

  • The unclipped gradient variance bound increases monotonically with p under a bounded token-gradient norm assumption.The theorem assumes ∥∇θ log πθ(yi,t | x, yi,<t)∥ ≤ M for all batch tokens.Source: I Variance Behaviours · p. 27, §I.1 · p. 27

I.2 Variance and Sequence-level Clipping

  • Without clipping, the surrogate scaling factor undergoes unbounded exponential growth, making explicit clipping necessary.The result follows by integrating the ratio's differential equation over time.Source: §I.2 · p. 29
  • Token-level clipping disrupts the structural relation supporting the monotonic variance bound, leaving p's variance control less predictable.The appendix reports that token-level clipping narrows the performance spread across p.Source: §I.2 · p. 29

I.4 Monotonicity of Variance

Under idealised orthogonality and uniform expected token-gradient norms, HölderPO’s variance decreases from positive p toward zero, then has a global minimum at some p*≤0. The analysis attributes this pattern to the opposing behaviour of the Hölder mean and the weight-distribution concentration index.

Assumptions

  • The variance analysis assumes exact token-level gradient orthogonality and uniformly bounded expected gradient norms.These assumptions eliminate cross-terms and produce an exact second-moment expression for the aggregated estimator.Source: §I.4 · p. 30

Positive p

  • As p decays from +∞ to 0, the variance strictly decreases.Both the squared Hölder mean and the HHI decrease over this positive-p range.Source: §I.4 · p. 30

Conclusion

  • There exists a variance-minimising p*≤0.The variance is continuous and bounded below on the nonpositive domain, while it increases throughout the positive domain.Source: §I.4 · p. 30

Positive p

  • For positive p, increasing p concentrates token weights, while decreasing p disperses them toward a uniform distribution.The HHI is minimised by the uniform distribution, linking weight dispersion to the variance trend.Source: §I.4 · p. 30

Negative p

  • As p approaches −∞, weights collapse onto minimum-ratio tokens and the HHI rises exponentially toward 1.This concentration counteracts the continuing decay of the Hölder mean.Source: §I.4 · p. 30

J Quantitative Advantage of Dynamic Scheduling

The quantitative analysis motivates dynamic scheduling by showing that high p is useful before sparse signals saturate, whereas lower p becomes appropriate afterward. This schedule shifts from signal amplification to variance contraction as training progresses.

Signal amplification

  • Before saturation, increasing p provides a positive lower-bounded amplification of a rare high-ratio token’s gradient weight.The condition R^p_high ≪ n^-1 ensures the high-ratio token has not yet monopolised the weight distribution.Source: J Quantitative Advantage of Dynamic… · pp. 31–32
  • The pre-saturation regime represents a discovered high-reward reasoning token that remains insufficiently confident relative to the sequence background.Long trajectories and early-phase low confidence make this condition relevant to reasoning tasks.Source: J Quantitative Advantage of Dynamic… · p. 31

Dynamic scheduling

  • Once saturation occurs, further increasing p becomes ineffective because the high-ratio token’s weight is already monopolised.At this point, the dynamic schedule decays p toward p_low≤0.Source: J Quantitative Advantage of Dynamic… · p. 31
  • Dynamic scheduling decays p toward p_low≤0 after saturation, shifting the optimisation focus from signal amplification to variance contraction.This design connects the changing signal regime to the two-sided variance and concentration trade-off.Source: J Quantitative Advantage of Dynamic… · pp. 31–32

Broader impacts

  • Gradient amplification in the positive-p regime can intensify reward hacking when learning signals are misspecified.The paper identifies this as a framework-specific limitation alongside standard dual-use risks.Source: K Broader Impacts · p. 32

Related papers

Loading 2605.12058v2…