PaperlayerView exact revision on arXiv

Source-linked AI summary

STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability

Haipeng Luo, Qingfeng Sun, Songli Wu, Can Xu, Wenfeng Deng, Han Hu, Yansong Tang

arXiv:2606.19236v1cs.LGcs.AIcs.CL

TL;DR

GRPO-style RLVR training commonly suffers policy entropy collapse, limiting stable long-horizon post-training. STARE analyzes the token-level credit-assignment mismatch, reweights entropy-critical tokens using surprisal quantiles with target-entropy feedback, and reports stable training across scales and tasks with 4%–8% average-accuracy gains over DAPO and other baselines.

  • Problem

    GRPO-style RLVR training commonly suffers policy entropy collapse, reducing output diversity and capping trainable steps for long-horizon post-training.

  • Method

    STARE reweights entropy-critical tokens selected by batch-internal surprisal quantiles and uses a target-entropy closed-loop gate to regulate their effective advantages.

  • Results

    Across 1.5B–32B models and three task families, STARE sustains thousands of stable RL steps and outperforms DAPO and other baselines by 4%–8% in average accuracy on AIME24/25.

  • Takeaways & Limitations

    STARE provides a surprisal-guided approach for stable entropy regulation and entropy-aware credit assignment in long-horizon RL post-training of LLMs.

  • Takeaways & Limitations

    The evaluation may not exhaustively cover all possible task distributions and optimization settings, and the trained model may sometimes generate unethical or misleading information.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.

1 Introduction

GRPO-style RLVR training can suffer policy entropy collapse, motivating STARE’s token-level credit rebalancing and target-entropy regulation. Across model scales and task regimes, STARE sustains stable training and improves accuracy over DAPO and other baselines.

  • Motivation: GRPO-style training can rapidly reduce entropy, eliminate output diversity, cause premature convergence, and homogenize within-group rollouts.These effects degrade relative advantage estimation and ultimately cap trainable steps.
  • Motivation: STARE analyzes token-level entropy dynamics and identifies an advantage–surprisal mismatch in GRPO’s shared trajectory-level credit assignment.The analysis establishes a near-criticality property in which mild token-weight perturbations can reverse entropy evolution.
  • Approach: STARE selects entropy-critical tokens using batch-internal surprisal quantiles and selectively reweights their effective advantages under a target-entropy closed-loop gate.The gate activates below the target entropy and reverts to GRPO after entropy recovers.
  • Results: STARE sustains stable RL training for over 5k steps at 7B and over 1.5k steps at 14B and 32B, keeping policy entropy within the target band.The evaluation spans Short CoT, Long CoT, and multi-turn tool-use agents.

2 Preliminaries

GRPO samples grouped responses and estimates advantages from group-normalized rewards, then optimizes a clipped per-token surrogate. The preliminaries define token surprisal, entropy, and the entropy response to logit updates.

  • GRPO: GRPO samples G responses from the old policy and computes each response’s group-normalized advantage from its reward relative to the group mean and standard deviation.The value network is replaced by group-normalized rewards as the baseline.
  • GRPO: The clipped surrogate uses per-token importance ratios between the current and old policies, with total token count N and no KL penalty when β = 0.The ratio compares the sampled token probability under the current policy with that under the old policy.
  • Entropy quantities: Token surprisal is sv = −ln πv, while position-level entropy is H = −∑v πv ln πv = Eπ[s].The batch mean entropy typically decreases during RL fine-tuning, a phenomenon described as entropy collapse.
  • Entropy quantities: In the unclipped regime, sampling token a produces the logit update Δzv = η Â(δva − πv), which determines the local policy change.Here η > 0 is an infinitesimal step size.
  • Entropy quantities: The entropy gradient is ∂H/∂zv = πv(sv − H), so increasing a rarer-than-average token’s logit raises entropy while increasing another token’s logit lowers it.The sign depends on whether surprisal exceeds the current entropy.

3 Theoretical Analysis

The analysis decomposes token-level entropy change into trajectory advantage and local entropy sensitivity, revealing four opposing advantage–surprisal regimes and near-critical reweighting. These dynamics explain collapse and motivate targeted intervention.

  • 3.1 First-Order Gradient Analysis of Token-Level Policy Entropy: GRPO’s token-level entropy variation factors into the trajectory advantage  and entropy sensitivity Φ(p), whose product determines the sign and magnitude of change.The sensitivity function is determined by the sampled token’s next-token distribution.
  • 3.1 First-Order Gradient Analysis of Token-Level Policy Entropy: A unique critical surprisal threshold s* partitions tokens into low- and high-surprisal regions according to the sign of Φ(p).The threshold exists for non-uniform, non-degenerate next-token distributions.
  • 3.2 Advantage–Surprisal Four-Quadrant Decomposition: Reinforcing low-surprisal positive-advantage tokens and suppressing high-surprisal negative-advantage tokens reduce entropy, while the opposite quadrants increase it.Each GRPO step therefore propagates four token-level entropy signals with opposing signs.
  • 3.2 Advantage–Surprisal Four-Quadrant Decomposition: Because low-surprisal tokens are sampled more frequently, positive-advantage trajectories reinforce an entropy-decreasing majority while underrepresenting high-surprisal entropy-increasing tokens.A corresponding asymmetry occurs among negative-advantage trajectories, producing a gradient-level mechanism for entropy collapse.
  • 3.3 Batch-Level Entropy Decomposition and Near-Criticality: A mild token-level weight change can reverse the batch entropy shift because the critical weight satisfies W* = 1 + Λ/Γ and approaches unity as sequence length and batch size grow.Beyond the critical threshold, W primarily controls shift magnitude rather than direction.
  • 3.4 Cross-Step Entropy Dynamics: Without reweighting, entropy reduction concentrates the policy, lowers future high-surprisal sampling, shrinks Γ, and reinforces collapse; entropy increases create the symmetric recovery loop.This cross-step feedback motivates selectively reweighting targeted tokens.

4 Method

STARE approximates entropy-critical tokens with separate high-surprisal quantiles for positive- and negative-advantage tokens, then reweights them within GRPO. Target-entropy gating limits intervention and enables automatic recovery to standard GRPO.

  • Core mechanism: STARE operates inside GRPO’s clipped surrogate as a token-level credit-rebalancing mechanism with differentiated weights for entropy-critical tokens.The method preserves token-level gradient directions because all weights remain positive.
  • 4.1 Entropy-Critical Token Partitioning via High-Surprisal Quantiles: Within positive- and negative-advantage sets separately, STARE ranks tokens by surprisal and selects the top P% as entropy-critical.This batch-internal proxy avoids computing the exact position-specific critical threshold.
  • Weighting variants: Positive high-surprisal tokens can be amplified with W > 1 to strengthen entropy-increasing updates, while negative high-surprisal tokens can be attenuated with 0 < M < 1.Two-sided regulation simultaneously amplifies entropy-increasing signals and attenuates entropy-decreasing signals.
  • Target-entropy regulation: A binary target-entropy gate activates reweighting when batch entropy falls below Htgt and restores unit weights once entropy recovers.The resulting bounded feedback drives entropy toward the target rather than relying on open-loop intervention.
  • Weight schedules: Near-criticality makes fixed weights generally sufficient because their value mainly controls entropy-shift magnitude beyond the critical point.Adaptive weights are also supported, increasing W and decreasing M below target entropy while relaxing toward GRPO otherwise.
  • Default configuration: The main experiments use Variant I with batch-level target-entropy gating and fixed weights, a minimal configuration reported to stabilize entropy and improve performance.Two-sided regulation and adaptive-weight variants are evaluated in ablations.

5 Experiments

Experiments evaluate STARE across model scales, task scenarios, benchmarks, and ablations. STARE maintains entropy stability over long training and improves mathematical performance relative to competitive baselines.

  • Experimental Setup: STARE is evaluated across Short CoT, Long CoT, and tool-use scenarios, using models from 1.5B to 32B and six mathematical benchmarks.The setup includes different decoding lengths and a cold-start SFT stage for the tool-use scenario.
  • Main Results: 54.4% average accuracy is achieved by STARE-O1 at 7B, exceeding STEER and GRPO-ds by 5.3 percentage points.At 14B and 32B, STARE-O1 exceeds GRPO-ds by 5.9 and 4.6 percentage points, respectively.
  • Main Results: 65.9% average accuracy is achieved by STARE-O1 at 1.5B in Long CoT, exceeding EAPO by 8.9 and DAPO by 10.8 percentage points.At 8B, STARE-O1 reaches 62.0% and STARE-C2 reaches 62.2%.
  • Main Results: 59.4% average accuracy is achieved by STARE-O1 in tool use, improving over GRPO-ds by 5.5 percentage points, while STARE-C2 reaches 60.4%.STARE-O1 reaches 53.2% on AIME24 and 37.5% on AIME25.
  • Training Dynamics: STARE stabilizes policy entropy near Htgt = 0.3 during 5000-step training, while accuracy continues rising beyond GRPO-ds’s early saturation.STARE also maintains higher Pass@32 and continued reward and response-length growth.
  • Mechanism Validation: The batch-internal surprisal proxy’s theoretical-region alignment rises from approximately 60% to around 95%, and its selected subset has positive cumulative net entropy contribution.The proxy identifies entropy-critical tokens without solving Φ(p*) = 0 at every position.
  • Cognitive Analysis: Tokens selected for advantage amplification concentrate on uncertainty and self-correction vocabulary, including “should be,” “but,” “instead,” and “verification.”This pattern is presented as evidence that the surprisal proxy identifies rare forking tokens with exploratory semantics.

6 Conclusion

The paper concludes that STARE addresses GRPO entropy collapse through entropy-aware token-level credit assignment and target-entropy regulation. Across scales and scenarios, it sustains stable training and improves benchmark performance.

  • Conclusion: STARE combines batch-internal surprisal-quantile reweighting with a target-entropy closed-loop gate to regulate policy entropy in GRPO-style RLVR.The method is designed as a minimally intrusive modification to the GRPO objective.
  • Conclusion: STARE sustains thousands of stable RL steps across 1.5B–32B models and three task scenarios, outperforming DAPO and other competitive baselines by 4%–8% on AIME24/25.The conclusion frames this as a foundation for entropy-aware credit assignment in long-horizon LLM post-training.
  • Related Work: The paper situates STARE among entropy-collapse mitigation methods involving clipping, trajectory-level weighting, and token-level perspectives.Trajectory-level methods cannot distinguish opposing entropy effects among tokens within one trajectory.

B.1 STARE vs. GRPO-ds: Detailed Comparison across Diverse Scenarios and Model Scales in RL Training

Detailed experiments show that STARE preserves entropy and continued accuracy gains across scales and scenarios, while ablations validate its surprisal proxy, gating strategy, and effective token reweighting variants.

  • Entropy stability and performance evolution: During 5000 Short CoT steps, GRPO-ds entropy approaches zero by step 1000, while STARE stabilizes near Htgt = 0.3 and continues improving accuracy.STARE’s Pass@32 remains higher throughout training, while GRPO-ds reward and Full-Solve Ratio plateau early.
  • Cross-scale and cross-scenario generalization: Across Short CoT, Long CoT, and tool-use settings, GRPO-ds shows entropy collapse and performance saturation, whereas STARE maintains entropy stability and continued accuracy improvement.On the 1.5B Long CoT model, STARE returns entropy to the target band around step 3500 while AIME accuracy improves in tandem.
  • Surprisal-quantile proxy validation: The selected high-surprisal fraction within the theoretical entropy-increasing region rises from approximately 60% to around 95% during training.The selected subset’s cumulative net entropy contribution remains positive and monotonically increasing.
  • Four-quadrant reweighting ablation: Among the ablated variants, O1 reaches 44.2%/23.8% and C2 reaches 42.5%/24.2% on AIME24/AIME25, motivating their use as default and dual-sided configurations.The experiments compare four single-polarity and four combined operations.

B.4 Detailed Ablation on Key Hyperparameters and Target-Entropy Gating

The ablations show that STARE’s target-entropy gate regulates the strength of token reweighting, preventing both entropy collapse and excessive exploration while supporting benchmark gains.

  • Key hyperparameters: W = 1.01 is sufficient to mitigate GRPO-ds entropy decay, whereas W ≥2.0 drives entropy upward too aggressively and triggers divergence.With Htgt = 0.3, W values from 1.05 to 1.5 steer entropy into the target band with bounded oscillation.
  • Target-entropy gating: Closed-loop STARE confines policy entropy to bounded oscillation around the target, unlike open-loop STARE, which induces over-exploration.The comparison fixes W = 1.1 and P = 10% over 1000 RL training steps.
  • Target-entropy gating: 38.0%/20.0% is achieved by closed-loop STARE on AIME24/AIME25, compared with 35.2%/17.3% for GRPO-ds.The result supports additional performance gains from stabilizing entropy within a reasonable band.
  • Target-entropy threshold: 44.2% on AIME24 and 23.8% on AIME25 are reached at Htgt = 0.3 after 4000 RL training steps, with gains of 7.1% and 6.1% over baseline.Both low and high target entropies are reported as suboptimal because they restrict exploration or promote over-exploration.

B.9 Ablation on Target-Entropy Gate Granularity

The batch-level target-entropy gate provides the strongest reported gating performance, while STARE’s adaptive surprisal quantile outperforms fixed probability-threshold token selection.

  • Gate granularity: 38.0%/20.0% is achieved by the batch-level gate on AIME24/AIME25, surpassing token-level 36.9%/19.1% and sample-level 37.6%/19.3%.The batch-level gate is adopted because local entropy estimates have higher variance and can switch the gating signal frequently.
  • Token selection: Fixed-threshold reweighting gains only 1.8% on AIME24 and 2.0% on AIME25, whereas STARE gains 7.1% and 6.1%, respectively.The batch-internal top-P% surprisal quantile adaptively tracks the current policy distribution.
  • Token selection: STARE’s quantile proxy agreement with the theoretical threshold rises from approximately 60% to over 95% throughout training.This supports adaptive alignment between the selected token subset and the theoretical entropy-critical boundary.
  • STARE procedure: The algorithm rolls out grouped responses, broadcasts each trajectory-level advantage to token positions, gates intervention using batch entropy, and selects top-surprisal positive-advantage tokens.The selected tokens are then reweighted inside the clipped-surrogate policy update.

D.4 Theorem 3.1 (Token-Level Entropy Variation)

Theorem 3.1 expresses first-order token-level entropy change as the product of trajectory-level advantage and local entropy sensitivity, yielding a surprisal-based four-quadrant sign structure.

  • Theorem statement: The first-order directional derivative of conditional policy entropy factors into trajectory-level advantage ˆA and entropy sensitivity Φ(p).The product determines the sign and magnitude of entropy variation at each decoding position.
  • Four-quadrant decomposition: For ˆA > 0, low-surprisal tokens decrease entropy while high-surprisal tokens increase entropy; for ˆA < 0, the directions reverse.At the boundary sa = s∗, the first-order entropy variation is zero.
  • Four-quadrant decomposition: The sign of first-order entropy variation is exactly the sign of −ˆAΦ(p).The unique threshold s∗ separates the low- and high-surprisal regions for non-uniform, non-degenerate distributions.
  • Asymmetric contributions: Under shared trajectory-level advantages, low-surprisal entropy-decreasing tokens form the sampling majority, while high-surprisal entropy-increasing tokens are a minority.This asymmetry makes the opposing token-level entropy effects difficult for GRPO to distinguish.
  • Reweighting implication: Reweighting positive-advantage high-surprisal tokens produces positive batch-level entropy variation when W > W∗ and negative variation when W < W∗.The first-order variation is zero at W = W∗.

E.11 Theorem 3.6 (Near-Criticality)

The near-criticality analysis shows that the critical reweighting factor remains close to one for sufficiently long sequences and large batches under stated assumptions.

  • Theorem 3.6: Under Assumptions E.1–E.4, the critical-weight offset satisfies W∗−1 = O(T−1) for sufficiently large sequence length T and batch size.The result follows from a weak residual term scaling as O(T−1) and a constant-order denominator.
  • Interpretation: In the entropy-collapse regime, baseline GRPO has Λ > 0, so W∗ exceeds one only by an O(T−1) offset.The proof is conservative because it lower-bounds the stabilizing contribution using a stronger subset; additional terms can move the offset closer to zero.

F Formalization of the Cross-Step Entropy Dynamics in Section 3.4

The appendix formalizes cross-step entropy dynamics under a mean-field approximation, then derives how fixed weights, proxy-set reweighting, and closed-loop gates affect entropy evolution.

  • F.1 Mean-Field Approximation: The formalization assumes batch-level quantities Λk and Γk depend primarily on current batch-averaged policy entropy through continuous functions, with Γ(h) > 0.
  • F.1 Mean-Field Approximation: The critical-weight curve W∗(h) is required to be strictly decreasing in entropy, formalizing the relation between entropy level and weight needed for entropy growth.
  • F.2 Sign of the One-Step Batch Entropy Change under a Fixed Weight: For fixed reweighting factor W, the one-step batch-entropy change has a sign determined entirely by W − W∗(H̄k).
  • F.3 Open-Loop Self-Reinforcing Entropy Collapse and Recovery: Under open-loop unit weighting, entropy below the critical curve decreases further, strengthening the entropy-decreasing pressure and reinforcing collapse.
  • F.3 Open-Loop Self-Reinforcing Entropy Collapse and Recovery: When W exceeds the critical curve, entropy rises and the increasing margin strengthens entropy recovery.
  • F.4 Local Stability of Batch-Level Target-Entropy Gating: The target-entropy gate is defined by gk = 1[H̄k < Htgt], activating the one-sided STARE update when batch entropy falls below its target.
  • G.1 Definition of Surprisal-Quantile Proxy Sets: Surprisal-quantile proxy sets split positive- and negative-advantage positions by separately selecting top or bottom P% surprisal positions to approximate the four theoretical quadrants.
  • G.2 Position-Level Logit Update under STARE Reweighting: STARE reweighting preserves every original token-level policy-gradient direction because all weights remain positive, while selectively changing update magnitudes; gates can operate at batch, sample, or token granularity.

I Limitations and Broader Impacts

The paper cautions that its empirical evaluation may not cover all task distributions and optimization settings, and that trained models may sometimes generate unethical or misleading information.

  • The empirical evaluation may not exhaustively cover all task distributions and optimization settings, and the trained model may sometimes generate unethical or misleading information.
Loading 2606.19236v1…