PaperlayerView exact revision on arXiv

Source-linked AI summary

When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL

Jiakang Wang, Runze Liu, Qingpeng Cai, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, Kun Gai, Ling Pan

arXiv:2510.06062v2cs.CL

TL;DR

In GRPO-style outcome-supervised RL, token-level importance-sampling ratios are used alongside advantages shared across a response, leaving their token-wise weighting role insufficiently understood. The paper proposes ASPO, which reverses ratio weighting for positive tokens and stabilizes extreme updates. Across math and coding benchmarks, it reports improved training stability and performance over GRPO-based baselines.

  • Problem

    In GRPO-style OSRL, response-level advantages are shared across tokens, leaving unclear what token-level importance-sampling ratios correct when they are no longer paired with token-specific credit.

    Source: §1 · p. 2

  • Method

    ASPO reverses ratio-induced weighting for positive-advantage tokens and uses soft dual clipping to constrain extreme ratios without discarding positive-token gradients.

    Source: §1 · p. 2

  • Results

    Across math reasoning and coding benchmarks, ASPO mitigates entropy collapse, improves training stability, and significantly outperforms GRPO-based baselines in final performance.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Under shared response-level advantages, token-level ratios primarily act as token-wise update weights, and correcting their positive-token mismatch is associated with more stable training and improved performance.

    Source: §6 · p. 8, §8 · p. 10

    The investigation focuses primarily on importance sampling in GRPO-based methods; extending it to paradigms such as PPO with token-level ratios and advantages remains a future direction.

    Source: §8 · p. 10

Abstract

from arXiv · show

Reinforcement learning (RL) has shown great promise in large language models (LLMs) post-training, which typically rely on token-level clipping to maintain stability during optimization. Despite the empirical success of GRPO-style methods, we identify a fundamental and previously overlooked challenge in this popular Outcome-Supervised RL (OSRL) paradigm. We reveal that in OSRL, where advantages are shared across tokens within a response, importance sampling (IS) ratios deviate from their traditional purpose of distribution correction as in classic RL, which become token-level weights that allocate the shared advantage signal across tokens. We show that this hidden role shift induces a critical mismatch for positive-advantage tokens, leading to unbalanced token weighting between positive and negative tokens. Specifically, it suppresses the update of underrepresented tokens that are lagging behind, while over-amplifying already high-probability tokens. This mismatch results in rich-get-richer dynamics that over-reinforce confident tokens, weaken catch-up learning that drive entropy collapse, excessive repetition, and premature convergence. To address this, we propose Asymmetric Importance Sampling Policy Optimization (ASPO), a simple yet effective strategy that reverses the ratio-induced weighting of positive-advantage tokens, while stabilizing extreme updates and maintaining gradient flow. This mismatch correction aligns their update direction with the learning dynamics of negative ones. Comprehensive experiments across math reasoning and coding benchmarks demonstrate that ASPO significantly mitigates entropy collapse, improves training stability, and enhances performance over strong GRPO-based baselines. Our analysis provides new insights into the role of token-level weighting in OSRL and highlights the critical importance of correcting ratio-induced weighting in LLM RL.

1 Introduction

In GRPO-style OSRL, a response-level advantage is shared across tokens, so importance-sampling ratios become token-level weights that allocate the shared signal. This can over-weight confident positive tokens and suppress lagging ones; ASPO reverses that weighting and stabilizes extreme updates.

1 Introduction

  • Shared response-level advantages turn token-level importance-sampling ratios into weights that allocate the learning signal across tokens, rather than token-specific credit estimates.This role shift differs from PPO's ratio paired with a per-step advantage.Source: §1 · p. 2
  • For positive-advantage tokens, standard ratio weighting overweights already high-probability tokens and down-weights lagging tokens, creating rich-get-richer updates linked to entropy collapse, repetition, and premature convergence.Existing DAPO clipping adjustments relax the upper bound but do not fix weak updates to low-probability positive tokens.Source: §1 · p. 2
  • ASPO reverses ratio-induced weighting for positive tokens, strengthening updates to low-probability tokens and down-weighting confident ones, while soft dual clipping constrains extreme ratios without discarding positive-token gradients.The method targets GRPO-style OSRL.Source: §1 · p. 2
  • The paper reports that ASPO mitigates entropy collapse and overfitting, stabilizes training, and outperforms GRPO-based baselines across math reasoning and coding benchmarks.The contribution summary characterizes the evidence as spanning multiple benchmarks.Source: §1 · pp. 2–3

2 Related Work

Prior LLM RL work extends GRPO and explores clipping choices for stabilizing policy updates. This paper focuses instead on ratio-induced relative weighting among updates that remain active.

RL for Large Language Models

  • The paper follows GRPO-based RL for LLMs, including work extending GRPO for reinforcement learning with verifiable rewards, but introduces a different clipping and importance-sampling mechanism.The cited extensions include [39] [40] [8] [35] [17] [33] [34].Source: §2 · p. 3

Clipping Mechanism in RL

  • Prior clipping methods alter when updates are clipped, masked, relaxed, or computed at sequence granularity; this work instead targets relative weighting of updates that remain active.DAPO [39] and DPPO [27] relax clipping ranges for low-probability positive tokens but do not address their weight mismatch.Source: §2 · p. 3

3 Preliminaries

The preliminaries introduce GRPO's group-based advantage estimation and its PPO-derived clipping mechanism. Clipping constrains policy updates, though clipped tokens can have their gradients masked.

3.1 Group Relative Policy Optimization

  • GRPO samples a group of G rollouts for advantage estimation and defines a loss using the resulting estimates.The supplied passage identifies the group-based estimation setup but does not provide the full loss expression.Source: §3.1 · p. 3

3.2 PPO Clipping and the Improvements

  • GRPO adopts PPO-Clip's clipping mechanism to constrain large policy updates, but clipped tokens can have their gradients masked.The gradient-masking observation is attributed to [3].Source: §3.2 · p. 3

4 The Role Shift of Importance Sampling Ratio in OSRL

With response-level advantages shared across tokens, GRPO's token-level ratios act as token-wise update weights rather than paired token-specific credit estimates. The paper tests this role shift and reports that removing ratio weights preserves comparable final accuracy while smoothing training.

4.1 Theoretical Understanding: PPO-style IS Ratios as Induced Token Weights in OSRL

  • GRPO's gradient is equivalent to a weighted log-likelihood update in which clipped token-level ratios modulate the shared response-level advantage.Tokens in one response can therefore receive different update magnitudes because their ratios differ, not because their reward contributions differ.Source: §4.1 · p. 4

4.2.1 Results

  • Standard GRPO and GRPO without IS weights achieve comparable final test accuracy, while removing the weights yields smoother convergence, slower entropy decay, and smaller increases in repetition and KL divergence.The no-IS variant fixes all ratios at 1.0; its optimization is less aggressive.Source: §4.2.1 · p. 5, Figure 2 · p. 5

4.2.2 Analysis

  • During GRPO training, average response-level ratios are around 1.0004, with positive-sample averages slightly above negative-sample averages; the gap accumulates across tokens and repeated off-policy updates.The paper links this imbalance to stronger positive-response learning and accelerating entropy decay.Source: §4.2.2 · p. 5
  • Removing ratio-induced weights preserves comparable final performance while making optimization smoother and less aggressive; ratios primarily act as effective update weights under shared advantages, with positive responses amplified more than negative responses are suppressed.The authors describe the ratios' dominant operational role as reallocating the outcome-level signal across samples and tokens.Source: §4.2.2 · p. 6

5 Ratio-induced Weight Mismatch in Positive-Advantage Responses

With a shared response-level advantage, PPO-Clip ratios act as token update weights, but misallocate positive-token learning toward already-high-probability tokens. A controlled reweighting improves training dynamics and performance compared with GRPO.

5.1 Rethinking the Role of IS-induced Token Weights

  • PPO-Clip’s low weights for low-current-probability tokens conflict with the proposed ideal of giving lagging tokens greater weight along the update direction.The rationale is to accelerate lagging-token learning while posing less risk to stability because those tokens are farther from the old policy.Source: §5.1 · p. 6

5.2 Weight Misallocation of Positive Tokens in PPO-Clip

  • For positive-advantage samples, PPO-Clip gives larger weights to already-reinforced tokens and very small weights to lagging tokens, unlike its expected behavior for negative samples.Figure 4 illustrates the contrasting weight patterns for positive- and negative-advantage samples.Source: §5.2 · p. 6, Figure 4 · p. 7
  • A token with old-policy probability 0.9 and current-policy probability 0.1 receives weight 1/9, weakening its update signal.Meanwhile, disproportionately high weights on already-strong positive tokens can reinforce them again in later updates.Source: §5.2 · pp. 6–7

5.3.1 Results

  • The authors link the positive-token weighting loop to entropy collapse, increased repetition, and movement toward a local optimum.Reweighting positive samples with response-level IS ratios mitigates these issues.Source: §5.2 · p. 7, §5.3.1 · p. 8
  • Replacing positive-token IS weights with response-level means slows entropy decline and moderates repetition, positive clipping, and KL increases.Training curves become substantially smoother, encouraging more diverse and exploratory behavior.Source: §5.3.1 · pp. 7–8
  • The modified method improves performance over GRPO while showing that stable training does not compromise final performance.The comparison is shown in Figure 2.Source: §5.3.1 · p. 7

6 Asymmetric Importance Sampling Policy Optimization

ASPO addresses the positive-token weighting mismatch by reversing the IS weights for positive-advantage tokens while keeping negative-token weighting aligned with GRPO. Its implementation retains hard masking and uses soft dual clipping to constrain extreme updates while preserving gradient flow.

Step 2: Weight Flipping

  • ASPO reverses positive-token IS weighting so lagging tokens receive higher learning weights and already-reinforced tokens receive lower weights.For tokens with negative advantages, ASPO uses the same ratio as GRPO.Source: §6 · p. 8

Steps 1 and 3: Token Masking and Dual Clipping

  • ASPO retains GRPO’s hard clipping for masked tokens and adds soft dual clipping for positive-sample tokens facing extreme ratios.Unlike hard-masked tokens, dual-clipped lagging tokens remain in training with their weight magnitude constrained.Source: §6 · p. 8

Step 3: Dual Clipping

  • Soft dual clipping constrains values while retaining gradient flow for tokens that lag significantly behind the old policy.The gradient analysis states that ASPO’s positive-token gradient grows as token probability decreases.Source: §6 · p. 8

7 Experiments

ASPO outperforms the base model and GRPO-based baselines across the reported math and coding evaluations. Its training curves show smoother entropy and repetition dynamics, although reward improvement is slightly slower early in training.

7.1 Setup

  • Experiments use DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-4B, Qwen3-8B, and Qwen3-30B-A3B1, evaluated on math and coding benchmarks.The math suite includes AIME24/25, AMC23, MATH-500, Minerva Math, and OlympiadBench; coding uses LiveCodeBench v5 and v6.Source: §7.1 · p. 9

7.2 Main Results

  • ASPO improves over the base model by 12.5% on math and 14.0% on coding, and outperforms GRPO and other OSRL methods across evaluated benchmarks.It also exceeds baselines on larger dense and MoE models and on OOD tasks.Source: §7.2 · pp. 9–10

7.3 Analysis

  • Training with ASPO yields more gradual entropy decline, slower growth in repetition and positive-token clipping, and eventual stabilization of these metrics.Entropy remains higher as performance steadily improves beyond the best GRPO-based results.Source: §7.3 · p. 10
  • ASPO models show slightly slower reward improvement in early training, but later catch up and ultimately surpass the other approaches.The passage attributes slower initial fitting to inversion of positive-sample weights reducing the overall average IS weight.Source: §7.3 · p. 10

8 Conclusion

The paper identifies token-level importance-sampling ratios as update weights under shared response-level advantages, rather than primarily as distribution-correction terms. It proposes ASPO and reports that it alleviates weight mismatch and entropy collapse while improving stability and performance across math and coding.

8 Conclusion

  • With shared response-level advantages in GRPO-style OSRL, token-level IS ratios primarily act as token-wise update weights rather than distribution-correction terms.The paper links the resulting positive-token mismatch to over-amplifying confident tokens and under-updating lagging ones.Source: §8 · p. 10
  • ASPO flips positive-token IS ratios and incorporates soft dual-clipping to stabilize OSRL for LLMs.Source: §8 · p. 10
  • Across mathematical and coding domains, ASPO alleviates token-level weight mismatch and entropy collapse, improves training stability, and outperforms baselines.Source: §8 · p. 10
  • The investigation primarily targets importance sampling in GRPO-based methods; extending it to other RL paradigms such as PPO with token-level ratios and advantages remains future work.Source: §8 · p. 10

A.1 Proof of Proposition 4.1

The proof establishes gradient equivalence between GRPO and a weighted log-likelihood objective. The gradient analysis characterizes ASPO's positive-token gradient as larger for lower-probability tokens.

A.1 Proof of Proposition 4.1

  • The proposition proves that GRPO's gradient equals the gradient of a weighted log-likelihood objective with detached token weights.The proof proceeds in three cases.Source: §A.1 · p. 14

A.2 Gradient Analysis

  • For positive tokens, ASPO's gradient is positively correlated with 1/πθ, so lower-probability tokens receive a larger gradient.Source: §A.2 · p. 15

B Experimental Details

The experiments compare ASPO with base and trained-model baselines across math and coding tasks. They report avg@K and pass@K, using specified training datasets and shared evaluation and optimization settings.

Baselines

  • The comparisons include the untuned base model, GRPO, DeepScaleR-1.5B, DeepCoder-1.5B, and Nemotron-1.5B, using four base-model families.Source: B Experimental Details · p. 15

Evaluation

  • Evaluation reports avg@K and pass@K on math and coding benchmarks, with K varying by benchmark sample count and task.Mathematical answers use official verification functions from DeepScaleR and Math-Verify, following the cited protocol.Source: B Experimental Details · pp. 15–16

Implementation Details

  • Training uses math and coding dataset mixtures, with 70.8k math and 8.9k coding samples after filtering, and sets GRPO and ASPO clipping ranges to ε = 0.2.The KL loss coefficient is β = 0.001, and each prompt receives 16 sampled responses.Source: B Experimental Details · p. 16

C Additional Experimental Results

Across Qwen3-4B and Qwen3-8B training curves, ASPO is reported to maintain steadier entropy and lower repetition despite slightly slower initial learning. The section also describes signs of healthy convergence and local-optimum behavior.

C Additional Experimental Results

  • In mathematical-task training, ASPO learns slightly more slowly than GRPO initially, yet maintains more stable model entropy and consistently lower repetition throughout training, demonstrating superior training stability.Training dynamics are shown for Qwen3-4B and Qwen3-8B, while test curves cover methods trained with Qwen3-4B on six mathematical benchmarks.Source: C Additional Experimental Results · p. 16, Figure 5 · p. 17, Figure 6 · p. 17
  • Without Dual Clipping, gradient norms show explosive spikes; with it, ASPO maintains stable gradient dynamics in extreme cases.Dual Clipping bounds extreme positive-token ratios shifted by Weight Flipping, complementing ASPO’s core mechanism.Source: C Additional Experimental Results · p. 16, Figure 8 · p. 18

D How to distinguish between “healthy convergence” and “local optima”?

  • Healthy convergence combines gradual entropy decline to a small positive level with steadily rising reward that eventually plateaus high.Stable clip ratios and KL divergence in later training are also described as characteristic of healthy convergence.Source: D How to distinguish between “healthy… · pp. 17–18
  • A local optimum is marked by entropy collapsing toward zero, stalled reward, and persistently high clip ratios without meaningful policy updates.Source: D How to distinguish between “healthy… · p. 18
  • Late-stage GRPO-based training can show entropy collapse, repetition spikes, rising clip ratios, and declining test performance; the paper links this degradation to positive-token weight mismatch.The mismatch is described as overfitting high-probability tokens, eventually leading to entropy collapse and training degradation.Source: D How to distinguish between “healthy… · p. 18

Related papers

Loading 2510.06062v2…