Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Haoming Meng; Kexin Huang; Shaohang Wei; Chiyu Ma; Shuo Yang; Xue Wang; Guoyin Wang; Bolin Ding; Jingren Zhou

Source-linked AI summary

Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, Jingren Zhou

arXiv:2603.22446v1Mar 23, 2026cs.CLcs.AIcs.LG

TL;DR

RLVR improves LLM reasoning, but the token-level mechanisms behind these gains remain unclear. This paper analyzes distributional shifts, links them to reasoning through cross-sampling, and finds that sparse, targeted refinements at critical token positions carry disproportionate functional importance. The study further shows that RLVR typically reallocates probability within existing candidate tokens and that divergence-weighted advantage variants can improve over baselines.

Problem
The token-level mechanisms by which RLVR changes LLM behavior remain unclear because prior evaluations mainly use aggregate response-level metrics.
Method
The paper conducts a systematic token-level study of RLVR distributional shifts, their sequence-level effects through forward and reverse cross-sampling, and their fine-grained distributional mechanics.
Results
RLVR changes are sparse and targeted, with high-divergence token decisions carrying disproportionate functional importance and probability mass usually reallocated within existing candidate sets.
Takeaways & Limitations
RLVR is best understood as selective refinement at a small set of high-impact decision points rather than a global policy shift.
Takeaways & Limitations
Divergences are computed on top-p truncated distributions under the generation sampling configuration, while entropy and probabilities use full estimated distributions.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR's distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on sequence-level reasoning performance through cross-sampling interventions, and (3) fine-grained mechanics of these shifts at the token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibiting meaningful divergence between the base and RL policies. We further characterize the structure and evolution of these shifts through analyses of token entropy, positional concentration, and reallocation of probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap token choices between the base and RL models with varying intervention budgets. We show that inserting only a small fraction of RL-sampled tokens into base generations progressively recovers RL performance gains, while injecting a similarly small number of base token choices into otherwise RL-generated sequences collapses performance to base levels, isolating a small set of token-level decisions directly responsible for RLVR's performance gains. Finally, we explore divergence-weighted variants of the advantage signal as a diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide a fine-grained, token-level lens for understanding RLVR fine-tuning as a targeted refinement process.

1 Introduction

RLVR improves reasoning through sparse, targeted token-level changes rather than broad behavioral rewriting. The paper characterizes these shifts, tests their functional importance through cross-sampling, and examines how probability mass is reallocated.

RLVR induces sparse token-level distributional shifts, with most token distributions nearly unchanged and a small subset showing substantial divergence.
The paper studies how these shifts vary across positions, entropy levels, and RLVR methods to reveal differences in exploration and refinement behavior.
Cross-sampling interventions show that modifying only a small fraction of token choices can recover base-model performance gains or erase gains in RL-model generations.
At high-divergence positions, RLVR primarily reallocates probability mass within an existing candidate set rather than introducing new tokens.
Divergence-weighted variants of the RLVR advantage signal can improve over baselines as a diagnostic objective modification.
Together, the findings portray RLVR as sparse, structured probability reallocation at critical token positions that steer downstream reasoning trajectories.

2 Token Distribution Analysis between Base and Fine-tuned Models

RLVR changes next-token distributions sparsely and selectively: most positions remain nearly unchanged, while divergence concentrates at particular positions, entropy regimes, and contexts. Compared with SFT, RLVR produces more targeted shifts whose spread depends on the training method.

2.1 Preliminaries: RLVR compares base and fine-tuned next-token distributions under identical RL-generated sequence contexts using JS divergence.The RL trajectory serves as a reference path for measuring how the base model would need to adapt.
2.2 Distribution Shifts Are Highly Targeted and Sparse: More than 83% of DAPO positions and over 98% of SimpleRL positions exhibit near-zero divergence, revealing highly sparse RLVR shifts.DAPO has a broader divergence distribution than SimpleRL.
2.3 Positional Concentration: Average divergence is highest near response beginnings, decreases through the middle, and rises modestly near the end, although individual sequences vary substantially.The early and late regions align with changes in branching decisions, answer formatting, and termination behavior.
2.4 Divergence–Entropy Relationship: DAPO’s high-divergence distributions span both high- and low-entropy regions, showing that RLVR can modify initially confident predictions.Low-divergence distributions are generally low-entropy, while more conservative methods focus more on higher-entropy regions.
2.5 Semantic Identity of Divergent Tokens: High-divergence tokens include function words, reasoning terms, and equation fragments, while low-divergence tokens more often include numerals, operators, and mathematical structure.Token identity alone is insufficient: the same token can have low or high divergence depending on context.
2.2 Distribution Shifts Are Highly Targeted and Sparse: Lower clip settings and more constrained methods concentrate updates on fewer token distributions, whereas clip-higher DAPO produces broader divergence.For Qwen2.5-Math-7B, the 0.2 upper clip yields sparser shifts but higher JS values within its high-divergence set.
2.6 Comparison with Supervised Fine-Tuning (SFT): SFT produces substantially denser, more globally distributed shifts than RLVR, indicating that RLVR sparsity is not generic fine-tuning behavior.SFT also shows elevated divergence across the response and stronger concentration in high base-model entropy regions than DAPO.

3 Cross-Sampling: Functional Importance of Divergent Distributions

Cross-sampling shows that RLVR’s sequence-level gains are functionally concentrated in a sparse set of high-divergence token decisions. Injecting a small number of RL choices into base trajectories recovers performance, whereas replacing RL choices with base choices rapidly removes it.

Intervention design: Cross-sampling directly tests whether divergent token decisions explain RLVR’s sequence-level performance gains.Forward interventions inject RL choices into base trajectories, while reverse interventions replace RL choices with base choices during RL generation.
Forward cross-sampling: On AIME 2024, fewer than 4% RL interventions raised SimpleRL accuracy from approximately 8% to approximately 25%.The interventions corresponded to fewer than 40 effective token substitutions per response.
Reverse cross-sampling: On AIME 2024, replacing approximately 5% of high-divergence SimpleRL distributions reduced accuracy from around 25% to around 8%.The reversions involved fewer than 30 effective base-sampled tokens per response.
Trajectory steering: Performance changes smoothly with intervention budget, indicating that gains are distributed across multiple sequential token decisions rather than requiring every RL-induced change.Local edits can redirect reasoning trajectories, which subsequent decoding continues under the primary policy.
Interpretation: RL and base policies behave similarly across most tokens, but sparse RL-refined choices steer generation toward more effective reasoning trajectories accessible to the base model.Even plausible base-token substitutions can alter downstream conditional distributions and derail reasoning.
Forward cross-sampling: A small fraction of RL-sampled tokens can recover or exceed RL-level accuracy in otherwise base-model generations.Across settings, modifying approximately 1–10% of tokens per response closes most or all of the performance gap and can sometimes outperform standalone RL decoding.

4 Fine-Grained Mechanics of Distribution Shifts

RLVR’s high-divergence updates usually reshape probabilities within an existing set of plausible candidates rather than introducing new token support. Training increasingly concentrates these changes, with method-specific differences in re-ranking and low-probability promotion.

Analysis question: The fine-grained analysis asks whether high-divergence updates introduce new candidate tokens or redistribute probability mass among existing candidates.It examines top-k overlap, rank shifts, tail probabilities, and training evolution.
Shared candidate sets: Top-k overlap remains high at high-divergence positions once k ≥2, showing that base and RL models usually share candidate sets.SimpleRL shows over 80% average overlap, often exceeding 85%, while DAPO remains substantially overlapping.
Selection rather than invention: Only about 5% of DAPO RL top-1 tokens at high-divergence positions have base probability below 0.01, compared with nearly zero for SimpleRL.Low-probability promotions remain rare, although the paper notes they may still matter for improved reasoning.
Training evolution: JS divergence becomes increasingly concentrated during training as upper percentiles grow faster than lower ones.This indicates that a small subset of token positions changes more strongly while most positions remain relatively stable.
Re-ranking over replacement: RLVR mainly re-ranks already-plausible tokens rather than replacing the candidate space.Around 30% of RL top-1 tokens are already base top-1, while over 80% of DAPO and 90% of SimpleRL RL top-1 tokens fall within the base top-3.
Method differences: Clip-higher DAPO produces more rank movement and low-probability promotion, while removing it concentrates probability on fewer dominant tokens.Both behaviors largely preserve the broader candidate space but differ in how probability is redistributed within it.

5 Exploratory Investigation: Divergence-Weighted Advantages

This section investigates whether divergence-weighted token advantages can focus RLVR learning on positions where the policy changes most. Using DAPO on Qwen2.5-Math-7B, it evaluates bounded weighting schemes across three mathematical benchmarks.

General formulation: Divergence-weighted advantages rescale standard group-normalized advantages using detached, per-token divergence weights.The formulation is ˜A_t = w_t · Â_t, with divergences detached so they affect weighting rather than gradient computation.
Choice of divergence: The study uses old-policy KL divergence as a proxy for recent local distributional change at each token position.For computational efficiency, the KL quantities are estimated over sampled tokens using the k3 estimator.
Weighting scheme: A sigmoid weighting scheme bounds the weights, with α > 0 emphasizing high-divergence tokens and α < 0 emphasizing low-divergence tokens.The sign of α determines whether the learning signal concentrates on regions that have changed or remained relatively unchanged.
Evaluation: The evaluation trains divergence-weighted DAPO on Qwen2.5-Math-7B and measures Mean@32 accuracy on AIME 2024, AIME 2025, and AMC.Results are averaged over three runs; Mean@32 is equivalent to pass@1 computed using 32 samples.
Results: Both low-KL and high-KL boost configurations can improve over the baseline, although the best weighting strategy and stability depend on further investigation.The findings support divergence weighting as a diagnostic intervention while leaving the optimal configuration unresolved.

6 Related Work

Related work studies RL fine-tuning through aggregate, parameter-level, entropy-based, and token-level perspectives. This paper extends those lines by quantifying RLVR-induced changes in token distributions and connecting them to reasoning outcomes.

Understanding RLVR and its differences with SFT: RL fine-tuning is increasingly viewed as a localized process that amplifies existing capabilities rather than uniformly changing model behavior.Prior evidence includes domain evaluations, catastrophic-forgetting analyses, parameter changes, aggregate KL divergence, and relatively small effective subnetworks.
Understanding RLVR and its differences with SFT: This paper complements parameter-level accounts by showing that RLVR changes are sparser at the token-distribution level than those induced by SFT.It also connects these sparse distributional changes to sequence-level reasoning performance.
Token-Level analyses of RLVR: The paper’s distinct focus is the fine-grained distributional structure of RLVR changes and their functional relationship to reasoning performance.This focus addresses token-position changes beyond aggregate divergence or broader reasoning-pattern analyses.
Token-Level analyses of RLVR: Token-level studies have linked RL gains to high-entropy minority tokens, exploratory reasoning steps, entropy-collapse risks, regularization, and specific sampling decisions.These works motivate examining how token-level uncertainty and choice structure relate to RLVR behavior.

7 Conclusion

The conclusion characterizes RLVR as a sparse, targeted refinement of token distributions rather than a global policy shift. It links a small set of high-impact decisions to reasoning gains and examines divergence-weighted learning as a related diagnostic.

Conclusion: Only a small subset of tokens undergo meaningful divergence, and cross-sampling shows that these positions carry disproportionate functional importance.Performance gains hinge on the high-divergence decisions identified by the token-level analyses.
Conclusion: At high-divergence positions, RLVR usually reallocates probability mass within an existing candidate set rather than introducing fundamentally new tokens.Rare substantial re-ranking and promotion of initially low-probability tokens may still matter for reasoning improvements.
Conclusion: Divergence-weighted advantages can influence learning dynamics, but stable performance may require model-specific choices and further investigation.The conclusion treats weighting as a promising but not yet uniformly settled modification.
Conclusion: Overall, RLVR appears to operate through selective refinements aligned with varying entropy levels at high-impact decision points.The authors suggest this perspective can inform future RL objectives and diagnostics.

8 Authors

The paper lists eight authors affiliated with four institutions, including the Qwen Pilot Team Lead.

Authors: The authors are Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou.
Authors: The affiliations are the University of Toronto, Peking University, Dartmouth College, and Alibaba Group.

A.1 Sequence-level Divergence Bounds for Cross-Sampling

The appendix formalizes cross-sampling as a mixed policy that switches between primary and intervention policies at selected token positions, inducing sequence-level distributions.

Policy construction: A switching rule selects whether each prefix uses the primary or intervention policy.The mixed next-token distribution is a convex combination controlled by the binary switch S_t.
Sequence distributions: The primary and intervention policies induce separate distributions over finite generated sequences.These sequence-level laws provide the objects compared by the divergence bounds.

A.1.1 KL Case

The KL analysis bounds sequence-level divergence between a cross-sampled policy and the intervention policy by the expected number of non-intervention steps and their effective token-level KL.

KL bound: The appendix first bounds cross-sampled versus intervention-policy sequence KL under a token-level KL switching rule.The result is stated for the simpler KL case before the corresponding JS analysis.
KL decomposition: A trajectory-level KL decomposition expresses sequence divergence as a sum of conditional token-level KL terms.The decomposition follows from policy factorizations, marginal prefixes, and iterated expectation.
Threshold control: The switching rule limits token-level KL on non-intervention steps, enabling a sequence-level bound proportional to their expected count.The analysis defines N_0 as the number of non-intervention steps and applies the threshold condition along the trajectory.
Effective KL: The effective non-intervention-step KL is typically much smaller than ε when the base and RL models are already close.The appendix denotes this average by κ̄ and notes the regime κ̄ ≪ ε.

A.1.2 JS Case

The JS analysis replaces the ordinary token-level decomposition with history-dependent skew JS divergence, yielding an analogous sequence-level control result for cross-sampling.

JS decomposition: Unlike KL, exact sequence-level JS decomposition requires a history-dependent skew Jensen–Shannon divergence.The ordinary JS divergence used in experiments is a special case when the mixture weight is 1/2.
JS decomposition: Lemma A.5 decomposes sequence-level JS by factoring the sequence laws and introducing prefix marginals of P, Q, and their mixture M.The proof uses the mixture factorization and Radon–Nikodym derivatives on prefix histories.
Token-level terms: The skew-JS decomposition sums history-weighted token-level terms across positions.The construction defines the relevant mixture conditional at each history and aggregates the resulting terms over t = 1, . . . , T_max.
JS threshold control: Proposition A.6 converts token-level skew-JS threshold control into a sequence-level JS bound for the mixed and intervention policies.The proposition uses the mixture M and a switching rule that constrains divergence at each eligible history.
Effective JS: The effective skew-JS average over non-intervention steps satisfies j̄ ≤ ε under Proposition A.6.This quantity is the JS analogue of effective KL on non-intervention steps.

A.2 Experimental Details

The appendix details divergence measurement, cross-sampling experiments, training configurations, and comparisons showing that RLVR changes fewer token distributions than SFT across several settings.

Experimental setup: JS divergence is computed between base and RL next-token distributions at identical prefixes, using top-p = 0.7 and temperature = 1 for AIME sampling.The truncated distribution matches the effective sampling distribution, while untruncated and alternative top-p analyses test robustness.
Cross-sampling: Cross-sampling swaps base and RL tokens where JS divergence exceeds thresholds to measure the functional importance of divergent positions.Forward and reverse experiments are run across SimpleRL, DAPO, and multiple model-dataset combinations.
Parameter-level comparison: RL fine-tuning produces substantially lower parameter relative-gap ratios than SFT across the evaluated foundation models.The paper interprets this as evidence that RL induces less extensive cumulative weight modification.
RLVR versus SFT: SFT has higher JS divergence, more high-divergence tokens, lower top-k overlap, larger rank shifts, and more frequent elevation of low-base-probability choices than RLVR.These axes jointly distinguish SFT’s broader edits from RLVR’s targeted editing behavior.
Robustness and generalization: The sparsity pattern persists on GPQA-Diamond and under alternative divergence-distribution truncation analyses.These evaluations extend the observed near-zero-divergence pattern beyond the primary AIME setting.
Training dynamics: A Qwen3-32B run plateaued around mean@32 = 57–60% between steps 80 and 180 before exceeding 70% after step 180.The delayed improvement occurred in a run reaching roughly step 500 and exceeding 140,000 GPU hours.