STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

Shiqi Liu; Zeyu He; Guojian Zhan; Letian Tao; Zhilong Zheng; Jiang Wu; Yinuo Wang; Yang Guan; Kehua Sheng; Bo Zhang; Keqiang Li; Jingliang Duan; Shengbo Eben Li

Source-linked AI summary

STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

Shiqi Liu, Zeyu He, Guojian Zhan, Letian Tao, Zhilong Zheng, Jiang Wu, Yinuo Wang, Yang Guan, Kehua Sheng, Bo Zhang, Keqiang Li, Jingliang Duan, Shengbo Eben Li

arXiv:2602.15620v5Feb 17, 2026cs.CLcs.AI

TL;DR

Existing RL fine-tuning methods use coarse stabilization techniques and can suffer entropy instability, while sequence-level rewards may disproportionately amplify updates to rare spurious tokens. The paper analyzes token-level optimization dynamics, introduces S2T, and incorporates it into STAPO. Across six mathematical reasoning benchmarks and three Qwen model scales, STAPO stabilizes policy entropy and consistently improves reasoning performance.

Problem
Existing RL stabilization methods regulate entropy globally, while sequence-level rewards can give around 0.01% of low-contribution spurious tokens disproportionately large gradient updates.
Method
STAPO combines a token-level analysis framework with S2T, which identifies spurious tokens and suppresses their gradient perturbations within a group-based objective.
Results
STAPO substantially stabilizes policy entropy and consistently improves reasoning performance across six mathematical reasoning benchmarks and Qwen 1.7B, 8B, and 14B models.
Takeaways & Limitations
The results support targeting spurious-token gradient influence as a way to refine large language models while maintaining stable reinforcement-learning optimization.
Takeaways & Limitations
The study focuses on incorrectly rewarded tokens in correctly answered examples and leaves incorrect responses and diverse task domains for future work.

Abstract

from arXiv · show

Reinforcement Learning (RL) has significantly improved large language model reasoning, but existing RL fine-tuning methods rely heavily on heuristic techniques such as entropy regularization and reweighting to maintain stability. In practice, they often suffer from late-stage performance collapse, leading to degraded reasoning quality and unstable training. We identify a key factor behind this instability: a small fraction of tokens, termed spurious tokens (around 0.01%), which contribute little to the reasoning outcome but receive disproportionately amplified gradient updates due to inheriting the full sequence-level reward. We present a unified framework for evaluating token-level optimization impacts across spurious risk, gradient norms, and entropy changes. Building on the analysis of token characteristics that severely disrupt optimization, we propose the Silencing Spurious Tokens (S2T) mechanism to efficiently suppress their gradient perturbations. Incorporating this mechanism into a group-based objective, we propose Spurious-Token-Aware Policy Optimization (STAPO), which promotes stable and effective large-scale model refinement. Across six mathematical reasoning benchmarks using Qwen 1.7B, 8B, and 14B base models, STAPO consistently demonstrates superior entropy stability and achieves an average performance improvement of 11.49% ($ρ_{\mathrm{T}}$=1.0, top-p=1.0) and 3.73% ($ρ_{\mathrm{T}}$=0.7, top-p=0.9) over GRPO, 20-Entropy, and JustRL.

1 Introduction

Existing RL fine-tuning methods can destabilize reasoning through entropy growth or coarse-grained stabilization, while STAPO targets rare spurious tokens to stabilize optimization and improve accuracy.

Entropy-focused exploration methods can drive entropy toward explosive growth, causing models to collapse into shallow, repetitive, or nonsensical outputs.
Global entropy regularization, sample augmentation, and advantage reweighting may over-suppress exploration or induce oscillatory entropy dynamics.
Spurious tokens comprise around 0.01% of tokens, contribute little to reasoning, and receive disproportionately large updates by inheriting the full sequence-level reward.
STAPO uses S2T to identify disruptive tokens, suppress their gradient perturbations, and integrate this mechanism into a group-based objective.
Across six mathematical reasoning benchmarks and Qwen 1.7B, 8B, and 14B models, STAPO stabilizes policy entropy and improves reasoning accuracy across evaluation settings.

2 Preliminaries

The paper formulates RL fine-tuning as stochastic autoregressive generation with sparse sequence-level rewards and reviews GRPO and clip-higher token normalization as optimization foundations.

2.1 Problem Formulation: RL fine-tunes an autoregressive stochastic policy that generates token sequences from prompts drawn from a distribution.
2.1 Problem Formulation: A sparse verifiable reward evaluates complete generated sequences for correctness using an external verifier.
2.2 Group Relative Policy Optimization (GRPO): GRPO estimates advantages from groups of sampled output sequences without relying on an explicit value function.
2.2 Group Relative Policy Optimization (GRPO): The GRPO objective uses an importance-sampling ratio and a sequence-level advantage standardized across group rewards.
2.3 Clip-Higher and Token Normalization: The baseline configuration removes the KL penalty and uses token-level normalization with asymmetric clip-higher bounds to constrain policy updates.

3 Methodology

The methodology identifies low-probability, low-entropy tokens as destabilizing contributors and develops S2T-based masking in STAPO to suppress harmful updates while preserving exploration.

3.1 The Hidden Threat: Spurious Tokens: All tokens in a trajectory inherit the same sequence-level advantage, potentially reinforcing tokens that contribute negligibly to correct reasoning.These tokens are defined as spurious tokens and can receive disproportionately large positive updates.
3.1 The Hidden Threat: Spurious Tokens: Spurious tokens introduce misleading update signals that steer the policy toward detrimental directions and destabilize training.The phenomenon was examined by recording generated-token statistics during JustRL training on DAPO-MATH-17K.
3.3 STAPO with the Silencing Spurious Tokens (S2T) Mechanism: STAPO uses a binary mask to zero spurious-token losses and dynamically averages the objective over remaining valid tokens.The mask is integrated into a group-based objective and approximated efficiently through S2T.
3.2 Token-Level Optimization Analysis: Low-probability tokens can sustain exploration, so stabilization must remove destructive spurious tokens without suppressing all exploratory updates.Full masking of low-probability tokens causes severe entropy collapse, whereas low-probability tokens in high-entropy states can represent legitimate exploration.
3.2 Token-Level Optimization Analysis: Low-probability, low-entropy tokens combine high spurious risk, disproportionately large gradient norms, and entropy explosion during optimization.Theorem 3.3 links larger gradient norms to tokens with both low probability and low entropy.
3.3 STAPO with the Silencing Spurious Tokens (S2T) Mechanism: S2T identifies low-probability tokens in correct responses, suppresses the bottom q entropy fraction, and preserves the top 1−q fraction for exploration.The probability threshold defines rarity, while the entropy quantile is computed dynamically over low-probability tokens in each mini-batch.

4 Related Work

Related work addresses RL optimization, entropy instability, and gradient domination in reasoning-oriented language models through increasingly targeted policy and entropy interventions.

Reinforcement Learning for LLMs: Recent reasoning-oriented RL work includes GRPO, DAPO, GSPO, SAPO, and related policy-optimization variants targeting stability, sample efficiency, and scalability.This work follows earlier RL and RLHF methods such as PPO and broader policy-gradient algorithms.
Entropy Instability in RL: Existing entropy interventions address collapse or excessive growth through regularization, entropy-enhancing samples, and modified clipping strategies.Prior approaches often treat entropy as a surface-level training signal rather than addressing underlying instability sources.
Gradient Domination by Low-Probability Tokens: Low-probability tokens are recognized as a microscopic source of instability because rare predictions can exert disproportionate gradient influence.This motivates token-level analysis beyond global entropy control.

5 Experiments

Across three model scales and six mathematical benchmarks, STAPO maintains more stable entropy and improves performance while masking fewer than 0.01% of tokens.

Training Behavior Analysis: STAPO maintains a stable, well-regulated entropy profile after warmup, unlike entropy explosion in JustRL and 20-Entropy or entropy collapse in GRPO.Its entropy concentrates at high quantiles, preserving exploration for critical tokens while keeping most tokens deterministic.
Training Behavior Analysis: STAPO achieves the highest reward and AIME24 accuracy while sustaining strong performance without noticeable degradation after 3000 training steps.The training curves show both superior performance and stable policy entropy on Qwen3-1.7B.
Main Results: 18.49%, 5.78%, and 10.20% are STAPO’s average relative accuracy improvements at the 1.7B, 8B, and 14B scales, respectively, in the training-aligned setting.These results surpass the strongest baselines across the three model scales.
Main Results: STAPO continues to achieve state-of-the-art performance under the JustRL configuration, where the performance gap narrows because decoding heuristics benefit high-entropy baselines more.The method remains less dependent on such decoding heuristics because of its inherently stable distribution.
Ablation and Sensitivity Analysis: Fewer than 0.01% of tokens are masked across the 1.7B, 8B, and 14B models while STAPO achieves significant performance gains.The results associate these sparse tokens with disproportionate effects on gradient updates.
Ablation and Sensitivity Analysis: Low-entropy masking consistently achieves the best performance, whereas high-entropy masking significantly degrades results.The ablation applies masking to positive-advantage, low-probability tokens on the Qwen3-1.7B base model with ρT=1.0 and top-p=1.0.

6 Conclusion

The paper concludes that sparse spurious tokens can disproportionately skew RL optimization and that STAPO mitigates this effect while stabilizing entropy and improving reasoning performance. Its current scope is limited to incorrectly rewarded tokens in correctly answered examples.

Conclusion: STAPO is presented as a general paradigm for mitigating the detrimental effects of spurious tokens in RL training.The S2T mechanism detects spurious tokens and curtails their gradient influence within STAPO.
Conclusion: Empirical results show that STAPO substantially stabilizes policy entropy and consistently improves reasoning performance.
Limitations and Future Work: The study primarily focuses on tokens in correctly answered examples that were incorrectly rewarded.The authors identify broader analysis of incorrect responses and diverse task domains as future work.

B Proof of Theorem 3.3

The proof establishes entropy-based gradient-norm bounds by reducing the optimization to a functional over the probability simplex. It shows the extremum is uniform on its support and that the global maximum uses the full vocabulary support.

Lower bound: The lower-bound argument relates collision probability to Rényi entropy and applies Jensen’s inequality to obtain an entropy-based gradient-norm bound.The relation e^-H2(πθ) ≥ e^-H(πθ) connects collision probability with Shannon entropy.
Upper bound: The proof rewrites the target inequality as bounding F(πθ) = H(πθ)^2/L over the probability simplex.At simplex vertices, both entropy and Gini impurity vanish, while non-vertex distributions permit the ratio formulation.
Upper bound: KKT conditions restrict any nontrivial maximizer to the relative interior of a sub-simplex with k non-zero components.Lagrange multipliers are applied subject to the probability-sum constraint.
Upper bound: The stationarity analysis rules out two-value extrema, forcing every non-zero component at a stationary point to be equal.Strict convexity permits at most two distinct component values, while monotonicity contradicts the existence of unequal values.
Upper bound: The stationary point is therefore uniform over a support of size k, with each non-zero component equal to 1/k.The proof then evaluates the objective as a function of support size.
Upper bound: The objective strictly increases with support size, so its unique global maximum occurs at full support k = |V|.This yields the entropy upper bound H(πθ)^2 ≤ |V|(ln |V|)^2 and completes the theorem after combining the bounds.

D Training Details

STAPO trains autoregressive policies with group-relative advantages and token-level masking based on probability and entropy thresholds. The implementation uses a fixed training configuration built on veRL and DAPO-Math-17K.

Implementation: STAPO is implemented with the open-source veRL alignment framework and uses DAPO-Math-17K as its training dataset.Prompts request step-by-step reasoning with the final answer inside boxed notation.
Hyperparameters: Training uses AdamW with learning rate 1 × 10^-6, 10 warm-up steps, global gradient clipping norm 1.0, batch size 256, and 64 NVIDIA H20 GPUs.The setup omits a separate value network and an additional KL penalty, relying on group-relative advantages and clipping.
Algorithm: The algorithm initializes policy parameters, samples grouped responses, computes advantages, and evaluates token probabilities and entropies during each update.Each prompt generates G = 8 rollouts, and token-level quantities are computed autoregressively.
Algorithm: Tokens are marked for masking when the response has positive advantage, token probability is below τp, and entropy is below the threshold τh.The masked-token condition is applied inside the per-response, per-token update loop.
Algorithm: The training update applies the S2T-masked objective to update θ after processing the token-level conditions.The procedure records the spurious-token mask before the policy update.

E.1 Supplementary Training Dynamics

Supplementary experiments show that STAPO’s training behavior extends from the 1.7B model to Qwen3-8B and Qwen3-14B. Across these scales, it maintains stable entropy and higher accuracy while suppressing premature entropy collapse.

Larger-model dynamics: Across Qwen3-8B and Qwen3-14B, STAPO maintains stable policy entropy and achieves higher accuracy across parameter scales.The larger-model training dynamics are presented in Figure 9.
Larger-model dynamics: STAPO prevents premature entropy collapse and maintains stable exploration throughout training on both larger model scales.The reported trends are described as consistent with those observed for the 1.7B model.
Plotting details: The plotted curves were downsampled by retaining every third point from evaluations originally performed every 20 gradient steps.The authors state that the true peak of each accuracy curve remains preserved.
Token characterization: For Qwen-1.7B, word clouds distinguish low-entropy spurious tokens from high-entropy exploratory tokens in low-probability correct answers.Frequent spurious tokens include digits such as “1”, “2”, and “3” and symbols such as “$” and “x”.

E.3 Supplementary Spurious Cases

The supplementary cases classify spurious tokens into uncommon syntax, hallucinations or mathematical errors, and formatting errors. Examples show that these tokens can be rare, locally incorrect, or noncanonical even when the final answer is correct.

Taxonomy: The supplementary analysis identifies three spurious-token categories: uncommon syntax, hallucinations and math errors, and formatting errors.The categories are organized in Tables 4–6.
Category I: Uncommon Syntax: Uncommon syntax includes valid but highly improbable tokens that replace standard mathematical terminology with awkward alternatives.In one graph example, “broken” has probability 0.05%, while “removed” dominates the Top-5 distribution at 85.53%.
Category II: Hallucinations and Math Errors: A mathematically false intermediate statement can receive reinforcement when the trajectory reaches the correct final answer.The example incorrectly writes 6901 = 67 × 103 - 1 even though 67 × 103 equals 6901 exactly.
Category III: Formatting Errors: Formatting errors include malformed LaTeX or noncanonical token sequences that answer extraction overlooks.One example omits canonical whitespace after a plus sign, while rendering makes the error invisible in the final output.
Case contexts: The examples span mathematical contexts including complete graphs, candy remainders, polygon triangulation, and boxed numerical answers.The cases illustrate how low-probability tokens occur within otherwise structured reasoning trajectories.