PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimization

Yun Kim, Nojun Kwak

arXiv:2609.39402v2cs.AIcs.LG

TL;DR

Value-model-free RLVR assigns identical advantages across rollout tokens, while global entropy proxies can confuse token importance with prompt difficulty and position. PEPO weights token advantages using proximal entropy, a local comparison to neighboring tokens. It reports consistent gains over batch-level methods across models and benchmarks, with the formulation also improving other algorithms and single-stream RL.

  • Problem

    Value-model-free RLVR methods such as GRPO give every generated token the same sequence-level advantage, although tokens contribute unequally, and global entropy proxies conflate importance with prompt difficulty and position.

    Source: §1 · p. 1

  • Method

    PEPO weights per-token advantages with proximal entropy, a softmax-based measure of each token’s entropy relative to a centered window of neighboring tokens.

    Source: §1 · pp. 1–2, §4 · p. 5

  • Results

    Across three model families and four mathematical benchmarks, PEPO consistently outperforms batch-level methods; on Qwen3-4B, it surpasses 80/20 by 2.27%p in mean accuracy across all benchmarks.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The local formulation also improves 80/20 when substituted without other changes and improves over baseline in single-stream RL, where batch-level entropy methods degrade below it.

    Source: §1 · p. 2

    The main benchmark suite focuses on mathematical reasoning; coding evidence is limited to one held-out LiveCodeBench split and Qwen3-4B, and the paper does not compare against value-based methods such as PPO.

    Source: §6 · p. 10

Abstract

from arXiv · show

Value-model-free RLVR methods such as GRPO assign uniform advantages to all tokens in a rollout, ignoring that tokens contribute unequally. Recent methods use token entropy as an importance proxy but compute it globally across the batch, conflating importance with prompt difficulty and positional trends. We argue that importance should instead be measured relative to the local context of each token. We introduce proximal entropy, a local measure of token importance relative to neighboring tokens, and prove it is invariant to both confounders. Proximal Entropy Policy Optimization (PEPO) uses it to weight per-token advantages and outperforms GRPO and entropy-based baselines on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct. We also show the formulation generalizes to other algorithms where substituting proximal entropy into existing methods improves, and applying it to single-stream RL succeeds where global entropy fails.

1 Introduction

Batch-level entropy can mistake prompt difficulty and early position for token importance, while value-model and extra-rollout approaches add overhead. PEPO instead measures token importance relative to neighboring tokens and reports consistent gains over batch-level methods.

1 Introduction

  • Batch-level entropy prioritizes some low-importance tokens in hard prompts or early positions over genuine decision points.These global entropy elevations can misguide the learning signal.Source: §1 · p. 1
  • PEPO weights per-token advantages using proximal entropy, a measure of uncertainty relative to neighboring tokens rather than the batch.The local formulation is reported invariant to prompt difficulty and robust to positional trends.Source: §1 · pp. 1–2
  • Proximal entropy adds negligible overhead because token entropy is precomputed and sliding-window operations are vectorized.The paper also reports that substituting the local reference frame into 80/20 improves performance, and that PEPO improves over baseline in single-stream RL where batch-level methods degrade below it.Source: §1 · p. 2
  • PEPO consistently outperforms batch-level methods across three model families and four mathematical benchmarks.Figure 1 shows higher validation accuracy than 80/20 for Qwen3-1.7B and Qwen3-4B.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Related value-free RLVR methods seek finer token credit with auxiliary signals or lightweight entropy proxies. Existing entropy approaches generally use global reference frames, while PEPO uses a centered local frame and is presented as transferable across algorithms and single-stream RL.

2 Related Work

  • Value-free methods give every token in a rollout identical credit, while learned values, extra rollouts, process rewards, attention maps, and attribution add overhead.Token entropy is a lightweight alternative computed from the policy forward pass.Source: §2 · pp. 2–3
  • Entropy-based methods use global batch or absolute-token statistics, which can conflate token importance with prompt difficulty and positional trends.The cited approaches include 80/20 [Wang et al. 2025a], Cheng et al. [2025a], GTPO [Tan et al. 2025], UCAS [Xie et al. 2025a], ERPO [Yu et al. 2026], and EDGE-GRPO [Zhang et al. 2025].Source: §2 · p. 3
  • Unlike ARES [Chen et al., 2026], PEPO uses a centered symmetric window with softmax-relative weighting rather than a forward-looking average and batch-global threshold.The symmetric offsets support cancellation of linear positional trends; the paper also reports transfer across algorithms and to single-stream RL.Source: §2 · p. 3

3 Background and Motivation

RLVR methods such as GRPO give every token in a rollout the same advantage, although tokens contribute unequally to the answer. Entropy can provide a lightweight token-importance signal, but batch-level entropy also reflects prompt difficulty and position, motivating comparison with each token’s local context.

3.1 Preliminaries

  • GRPO assigns one sequence-level advantage to every generated token, while fine-grained credit assignment seeks per-token weights reflecting each token’s contribution.The verifier’s binary reward does not directly indicate which tokens mattered, and learned value models add memory and training overhead.Source: §3.1 · pp. 3–4

3.2 Entropy as a Token Importance Signal

  • High-entropy tokens can mark reasoning branch points, but batch-wide entropy ranking uses a global reference frame.80/20 updates only the top 20% of tokens by batch-level entropy.Source: §3.2 · p. 4
  • Global entropy varies with prompt difficulty and generation position, so batch-level statistics can prioritize hard prompts and early tokens regardless of their importance.The section motivates measuring entropy against each token’s local context instead.Source: §3.2 · pp. 4–5

4 Proximal Entropy Policy Optimization

PEPO measures each token’s entropy relative to a centered window of neighboring tokens and uses that proximal entropy to weight GRPO advantages. The paper presents theoretical and empirical evidence that this local signal is robust to prompt difficulty and positional trends.

Proximal entropy

  • Proximal entropy uses a softmax over a centered sliding window to measure a token’s uncertainty relative to its neighbors rather than its absolute entropy.A token moderately uncertain among low-entropy neighbors receives a larger value than one surrounded by equally uncertain neighbors.Source: §4 · p. 5

Per-token advantage weighting

  • PEPO normalizes and scales proximal-entropy weights to preserve total advantage magnitude, applying them only to positive-reward rollouts.Failed rollouts remain unweighted because amplifying negative updates at high-entropy positions may discourage exploration.Source: §4 · p. 5

Theoretical justification

  • Proximal entropy is theoretically invariant to sequence-wide entropy shifts and approximately invariant to smooth positional trends for sufficiently small windows.Empirically, it remains flat across generation positions and approximately 0.00988 across prompt-difficulty groups, while global entropy varies.Source: §4 · p. 6

5 Experiments

Experiments compare PEPO with GRPO and entropy-based methods across mathematical benchmarks, model families, and RL settings. PEPO’s results and analyses test the local entropy formulation’s performance and transfer.

5.1 Experimental Setup

  • Main runs use ROLL on DeepMath-103K, train GRPO-based methods for 500 steps with W = 101, and evaluate three models on MATH500, AMC, and AIME 2024/2025.The models are Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct.Source: §5.1 · pp. 6–7

5.3 Proximal Entropy as a General Formulation

  • Replacing global entropy with proximal entropy improves 80/20 across benchmarks on both models; PEPO achieves the best overall performance.The 80/20 substitution changes its token ranking while leaving its other components unchanged.Source: Table 3 · p. 7, §5.3 · p. 7

5.2 Main Results

  • Across three models, PEPO consistently outperforms GRPO and batch-level entropy methods; on Qwen3-4B, it beats GRPO by 6.49%p on AIME 2024 and 9.62%p on AMC.On Qwen3-4B, PEPO also surpasses 80/20 by 2.27%p in mean accuracy across benchmarks.Source: §5.2 · p. 7

5.4 Transfer to Single-Stream RL

  • PESPO achieves 58.93% mean accuracy versus SPO’s 57.13%, and improves over SPO on AIME 2024 by 6.04%p.The single-stream experiments use the same hyperparameters as the main experiments without modification.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 Analysis

  • At a 5% replacement budget, proximal entropy reaches 31.62% accuracy versus 27.34% for global entropy, a 4.28-point gap.At 10%, proximal entropy reaches the GRPO checkpoint’s 33.33% accuracy; global entropy reaches that level only at 30%.Source: Table 5 · p. 8, §5.5 · p. 8
  • Performance is relatively robust across window sizes 51, 101, and 151; W = 101 achieves the best results and is used as the default.The authors relate this robustness to positional-trend cancellation when the window is small relative to sequence length.Source: §5.5 · p. 8, Table 6 · p. 9
  • Proximal entropy distinguishes high-leverage tokens within short phrases while retaining nontrivial weight on deterministic execution tokens that global entropy washes out.Examples include “calculate” and “periodicity” as decision tokens, alongside tokens carrying out the modular arithmetic derivation.Source: §5.5 · pp. 9–10

6 Conclusion

PEPO replaces global entropy statistics with a local token-importance measure and improves mathematical-reasoning performance across three model families. The conclusion also reports broader algorithmic applications, while the paper identifies limits in its coding evaluation and comparisons.

6 Conclusion

  • PEPO consistently improves over GRPO and batch-level entropy methods on mathematical reasoning across Qwen3-1.7B, Qwen3-4B, and Llama-3.2-3B-Instruct.The conclusion also reports improved pass@1 over GRPO and 80/20 on one held-out LiveCodeBench split with Qwen3-4B.Source: §6 · p. 10

B Proof of Proposition 1

  • The stated proposition concerns invariance to sequence-dependent constant entropy shifts and approximate invariance to smooth monotonic trends for sufficiently small windows.These are the two properties the conclusion cites in describing proximal entropy's treatment of prompt difficulty and positional trends.Source: §6 · p. 10, B Proof of Proposition 1 · p. 14

6 Conclusion

  • The paper limits its main benchmark suite to mathematical reasoning and its coding evaluation to one held-out LiveCodeBench split with Qwen3-4B.It says broader coding benchmarks and model families are needed to establish generalization to code generation, agentic tool use, and multi-turn dialogue; it also did not compare against value-based methods such as PPO.Source: §6 · p. 10

A Broader Impacts

  • The paper reports that PEPO matches or exceeds batch-level entropy methods at the same training budget and transfers to single-stream RL, removing the need for G parallel rollouts per prompt.It links the lower training cost to a lower barrier for academic and resource-constrained groups and a reduced energy footprint.Source: A Broader Impacts · p. 14
  • The work reports no new capability axis or qualitatively new behaviors, and releases no new pretrained models or scraped datasets.Its experiments use publicly available base models and mathematical-reasoning datasets released by their original authors.Source: A Broader Impacts · p. 14

B.1 Proof of (i): Invariance to Constant Shift

A sequence-dependent constant added to every token entropy in a rollout does not change proximal entropy. The proof relies on the shared offset canceling within each token's local window.

B.1 Proof of (i): Invariance to Constant Shift

  • Adding the same sequence-dependent constant δ_i to every token entropy leaves proximal entropy unchanged.The invariance applies to any sequence-dependent shift.Source: §B.1 · p. 14
  • All tokens in a window belong to the same rollout and therefore share the offset δ_i.The proof represents shifted entropy as ˜H_i,t = H_i,t + δ_i.Source: §B.1 · p. 14
  • Because the shared offset factors out and cancels, proximal entropy is independent of overall entropy-level differences across sequences.Source: §B.1 · p. 14

B.2 Proof of (ii): Approximate Invariance to Positional Trends

The proof models token entropy as a smooth, monotonic positional trend plus token-specific residuals. When the trend changes little across a window, proximal entropy is approximately independent of that trend and depends on residual variation.

B.2 Proof of (ii): Approximate Invariance to Positional Trends

  • The analysis decomposes token entropy into a smooth, monotonic positional trend shared across rollouts and a token-specific residual.The target is to show proximal entropy is approximately independent of the trend under mild regularity conditions.Source: §B.2 · p. 14
  • The proof uses a first-order Taylor expansion around position t, with the trend approximately linear within each window when W ≪ T_i.Monotonicity ensures the local trend does not change direction within the window.Source: §B.2 · p. 15
  • When |f′(t)| · W/2 ≪ 1, the linear trend term is negligible and the resulting proximal entropy depends only on residuals, not f.The numerator's linear correction vanishes at k = t.Source: §B.2 · p. 15
  • Under a smooth, monotonic trend that varies slowly within each window, proximal entropy is approximately invariant to the trend, with corrections of order O(f′(t) · W) and a Taylor remainder.The paper's example estimates |f′| · W/2 ≈ 0.004 for T_i = 2000 and W = 101, and reports stable performance for W ∈ {31, 51, 101}.Source: §B.2 · p. 15

C Implementation Details

The implementation uses the ROLL framework for main experiments and VeRL for single-stream experiments, with separate reported hyperparameter tables. Window edges use reflect padding, and experiments run on two NVIDIA H200 GPUs with vLLM for rollout generation.

C.1 Hyperparameters

  • Main experiments use the ROLL framework, while single-stream experiments use VeRL; Tables 7 and 8 report their hyperparameters.Table 7 covers GRPO, 80/20, Entropy Adv., and PEPO; Table 8 covers SPO, S-80/20, S-Entropy Adv., and PESPO.Source: §C.1 · p. 16, Table 7 · p. 16, Table 8 · p. 16

C.2 Window Edge Handling

  • Reflect padding fills window positions beyond sequence boundaries using mirrored entropy values rather than introducing artificially low values.The authors report boundary effects over roughly W/T_i ≈ 5% of tokens for default W = 101 and typical sequence lengths, and say the choice worked well empirically.Source: §C.2 · p. 16

C.3 Software and Compute

  • Experiments run on two NVIDIA H200 GPUs, using ROLL for main experiments, VeRL for single-stream experiments, and vLLM for rollout generation.Source: §C.3 · p. 17

D Runtime Analysis

Entropy-based methods add negligible runtime over GRPO under the tested training setup. The comparison used identical hardware, framework, and training configuration.

D Runtime Analysis

  • All entropy-based methods (80/20, Entropy Adv., PEPO) run within minutes of GRPO across both models.The comparison used 2×H200 GPUs, ROLL, and 500 training steps with G = 8 and maximum length 4096.Source: D Runtime Analysis · p. 17, Table 9 · p. 17

E Additional Bias Analyses of Proximal Entropy

PEPO’s positional invariance holds across the tested window sizes and does not depend on softmax aggregation. On held-out LiveCodeBench, PEPO also outperforms GRPO and 80/20 in pass@1.

E Additional Bias Analyses of Proximal Entropy

  • Across W = 31, 51, and 101, proximal entropy and simple windowed averages stay flat while global entropy decays, showing positional robustness comes from windowing rather than softmax.The softmax instead produces a normalized per-token weight emphasizing tokens with higher entropy than their local neighbors.Source: E Additional Bias Analyses of Proximal… · p. 17, Figure 4 · p. 18

F.1 Held-Out Code Generation: LiveCodeBench

  • 46.35% pass@1 for PEPO exceeds GRPO by 14.60 percentage points and 80/20 by 5.31 points on held-out LiveCodeBench.Its gains over GRPO are positive across all three difficulty groups and listed platforms; the Codeforces subset has only nine problems.Source: Table 10 · p. 18, §F.1 · p. 18

F.2 Weighting Design Ablation

The weighting ablation reports incremental accuracy gains from localization, continuous weighting, and positive-only application. The temperature sweep shows a narrow mean-accuracy range, with the default temperature highest among the reported point estimates.

F.2 Weighting Design Ablation

  • Localization adds 1.14 points, continuous weighting over all trajectories adds 0.66, and positive-only application adds a further 0.48.Even all-rollout weighting remains 7.38 points above GRPO (57.53 versus 50.15); the all-rollout comparison with positive-only weighting is descriptive because run-level uncertainty was not reported.Source: §F.2 · p. 19, Table 12 · p. 19

F.3 Temperature Sensitivity

  • Mean accuracy spans 20.68–21.00 across the temperature sweep, with default τ = 1 giving the highest reported mean.The sweep reports point estimates without seed-level uncertainty.Source: §F.3 · p. 19, Table 13 · p. 19

Main Results (Table 2); Proximal Entropy as a General Formulation (Table 3)

  • Per-run accuracy tables accompany the main results and the proximal-entropy formulation results.The main-results table reports MATH500 avg@1 and AMC, AIME 2024, and AIME 2025 avg@16; the formulation table’s mean rows match Table 3.Source: Table 14 · p. 20, Table 15 · p. 21

Single-Stream RL (Table 4); Window Size Ablation (Table 6)

  • Per-run accuracy tables also accompany the single-stream RL and window-size ablation results on Qwen3-4B.Their mean rows match the values in Tables 4 and 6, respectively.Source: Table 16 · p. 21, Table 17 · p. 21

Related papers

Loading 2609.39402v2…