Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

Structure of Token-Level Distributional Shifts

• RLVR shifts are sparse: most token distributions remain nearly unchanged, while a small subset of high-divergence positions carries disproportionate functional importance.The paper characterizes shift structure through divergence, entropy, and positional analyses, including comparisons across RLVR methods.Source: §1 · p. 2, Figure 1 · p. 2

Fine-Grained Distribution Mechanics

• At high-divergence positions, RLVR primarily reallocates probability mass within an existing candidate set rather than introducing new tokens.The analyses include top-k overlap, rank, tail probability, and training evolution.Source: §1 · pp. 2–3

Divergence-Weighted Advantage

• Divergence-weighted variants of the RLVR advantage signal can improve over baselines as a diagnostic objective modification.Source: §1 · p. 3

2.1 Preliminaries

• The analysis compares base and RL next-token distributions along RL-generated trajectories, using JS divergence on matched top-p truncated distributions and full estimated distributions for entropy and probabilities.JS divergence is symmetric, bounded, and defined when distributions lack absolute continuity; comparisons across other top-p values and estimated full distributions are reported as robustness checks.Source: §2 · p. 3, §2.1 · p. 3

2.2 Distribution Shifts Are Highly Targeted and Sparse

• Near-zero JS divergence occurs at more than 83% of DAPO token positions and more than 98% of SimpleRL positions, with DAPO showing broader shifts than SimpleRL.For Qwen2.5-Math-7B, the lower 0.2 upper clip produces sparser shifts but higher divergence among its most divergent positions than the 0.28 clip; the sparsity pattern is stable across top-p settings and estimated full distributions. Extensions report consistent findings across models, datasets, and training paradigms.Source: Figure 2 · p. 4, §2.2 · pp. 4–5, §2.1 · p. 4

2.3 Positional Concentration

• Mean divergence is higher near response beginnings, falls through the middle, and rises modestly near the end, while individual sequences show divergence throughout.The early concentration aligns with changes to high-level branching decisions and the late rise with answer formatting and termination adjustments.Source: Figure 3 · p. 5, §2.3 · p. 5

2.4 Divergence–Entropy Relationship

• Low-divergence positions are largely low-entropy, but DAPO’s high-divergence positions span both low- and high-entropy predictions, unlike SimpleRL’s stronger concentration in high-base-entropy regions.At high-divergence positions, the 0.28 clip includes more low-base-entropy distributions and yields higher RL entropy, whereas the 0.2 clip concentrates divergence in higher-base-entropy regions and lower RL entropy.Source: Figure 4 · p. 6, §2.4 · p. 6

2.5 Semantic Identity of Divergent Tokens

• High-divergence samples include function words, reasoning terms, and equation fragments, whereas low-divergence samples are dominated by numerals, operators, and mathematical structure.Token identity alone does not determine divergence: “the,” for example, is frequent among high-divergence samples but overwhelmingly falls in the lower-divergence regime across its occurrences.Source: §2.5 · p. 7, §2.6 · p. 8

2.6 Comparison with Supervised Fine-Tuning (SFT)

• SFT produces a substantially larger high-divergence set and broader shifts than RLVR, with elevated divergence across responses and divergent tokens concentrated more in high-base-entropy regions.SFT’s fine-tuned entropy is substantially lower than the base model’s; the high-base-entropy concentration may partly reflect SFT outputs appearing more uncertain under base-model evaluation.Source: §2.6 · pp. 7–8, Figure 6 · p. 7

3.1 Cross-Sampling Framework

• The framework swaps tokens between primary and intervention policies only at high-divergence positions: forward sampling injects RL choices into base generations, while reverse sampling replaces RL choices with base tokens.Intervention budgets limit the number of swaps before generation continues under the primary policy.Source: §3.1 · p. 9

3.2 Results and Findings

• Sparse cross-sampling interventions recover RL-level accuracy in forward tests and erase it in reverse tests, sometimes pushing performance below base levels.The intervention effects emerge progressively as the budget increases.Source: Figure 7 · p. 10, Figure 8 · p. 11, §3.2 · pp. 12–13
• On AIME 2024, fewer than 4% RL-sampled tokens—fewer than 40 substitutions per response—close the SimpleRL gap from approximately 8% to approximately 25%.On AIME 2025, 1.53% effective cross-sampling, or roughly 13 substitutions per response, raises accuracy from about 5% to over 14%, exceeding standalone RL performance.Source: Figure 7 · p. 10, §3.2 · p. 11
• Stronger DAPO models require more substitutions to recover or erase their gains, but the required tokens remain a small fraction of each response.Forward interventions on AIME 2024 use approximately 7.8% effective tokens to raise accuracy from roughly 8% to over 44%; reverse interventions use roughly 10% to reduce accuracy from over 44% to near 8%. On AIME 2025, fewer than 6.5% forward interventions raise performance from around 5% to over 33%, while fewer than 10% reverse interventions reduce it from over 33% to below 4.5%.Source: §3.2 · pp. 11–13
• On AIME 2024, replacing approximately 5% of high-divergence SimpleRL choices—less than 30 base-sampled tokens per response—drops accuracy from around 25% to around 8%.On AIME 2025, roughly 4.7% effective base-sampled tokens, around 30 per response, reduces accuracy from approximately 12.7% to below 4%, below base-level performance.Source: Figure 8 · p. 11, §3.2 · pp. 11–12
• Accuracy changes smoothly as intervention budgets grow; even early local edits can redirect reasoning, and plausible or semantically interchangeable tokens can alter downstream outcomes.Forward interventions improve performance steadily, while reverse interventions degrade it near-monotonically; generation continues under the primary policy after the interventions.Source: §3.2 · pp. 12–13

4 Fine-Grained Mechanics of Distribution Shifts

• RLVR primarily reallocates probability within existing candidate sets, with rare substantial re-ranking or low-probability promotions potentially contributing to improved reasoning.The analysis concludes that RLVR sharpens and reorders plausible choices rather than globally altering effective token support.Source: §4 · p. 14, §4.3 · p. 17

4.1 Top-k Overlap and Rank Reordering

• At high-divergence positions, SimpleRL top-k overlap exceeds 80%, while over 80% of DAPO and 90% of SimpleRL RL top-1 tokens fall within the base top-3.Clip-higher produces more reshuffling among top-ranked tokens; without it, probability concentrates more on dominant tokens, though lower-ranked base tokens may occasionally rise in rank.Source: §4.1 · pp. 14–15, Figure 9 · p. 15

4.2 Low-Probability Behavior: Does RL Invent or Select?

• About 5% of DAPO RL top-1 tokens at high-divergence positions have base probability below 0.01, versus nearly zero for SimpleRL.Clip-higher increases the fraction of such promotions relative to its variant without clip-higher, although they remain rare and may still be consequential for reasoning.Source: §4.2 · pp. 15–16, Figure 10 · p. 16

4.3 Evolution Across Training

• During DAPO training, JS divergence rises monotonically, with higher percentiles growing faster and divergent-token overlap with the final set rising sharply near training’s end.This indicates increasingly concentrated distributional change, while most tokens remain relatively stable.Source: §4.3 · p. 16, Figure 11 · p. 16

5.1 GRPO-based Methods

• DAPO adapts GRPO with asymmetric clip-higher, dynamic correct/incorrect sampling, token-level averaging, and removal of the explicit KL penalty.Source: §5.1 · p. 18

5.2 Divergence-Weighted Advantage

• The method rescales each group-normalized advantage by a detached per-token divergence weight, using old-policy KL estimated on sampled tokens with the k3 estimator.Old-policy KL serves as a proxy for recent local policy change; sampled-token estimates may not capture the full distribution.Source: §5.2 · p. 18
• A sigmoid weighting scheme uses the sign of α to emphasize high-divergence tokens when α > 0 or low-divergence tokens when α < 0.The bounded, smooth weighting investigates whether focusing updates on already-changed or relatively stable token distributions is more effective.Source: §5.2 · p. 18
• Experiments use the DAPO recipe on Qwen2.5-Math-7B and evaluate Mean@32 accuracy on AIME 2024, AIME 2025, and AMC, averaging three runs.Source: §5.2 · p. 19, Table 2 · p. 19
• Both low-KL and high-KL boost configurations can improve over baseline, but benefits may depend on the model and training method.The authors frame divergence weighting as a complementary diagnostic tool, not a universally beneficial intervention.Source: Table 2 · p. 19, §5.2 · p. 19

RL fine-tuning in LLMs

• RLVR methods use policy-gradient variants such as GRPO (Shao et al., 2024), with extensions including Dr.GRPO, DAPO (Yu et al., 2025), and GSPO (Zheng et al., 2025).Related work also explores entropy-based updates, clipping/KL regularization, token probability or position reweighting, reward design, and reasoning boundaries.Source: §6 · p. 19

Understanding RLVR and its differences with SFT

• Prior analyses characterize RL as localized refinement using task evaluations, forgetting, parameter changes, and aggregate KL (Rajani et al., 2025; Chu et al., 2025; Shenfeld et al., 2025; Huan et al., 2025).Mukherjee et al. (2025) report concentrated effective updates in small subnetworks, while Zhu et al. (2025) analyze parameter-sparse update dynamics theoretically.Source: §6 · pp. 19–20
• This study finds RL changes smaller in aggregate than SFT and sparser at the token level, connecting those shifts to reasoning outcomes and probability-mass reallocation.This complements prior evaluations of locality across tasks, forgetting, parameter changes, and aggregate KL, as well as analyses of parameter-sparse updates.Source: §6 · pp. 19–20

Token-Level analyses of RLVR

• Prior work links high-entropy minority tokens to RL gains and exploratory steps (Wang et al., 2025; Cheng et al., 2025), and studies token choices, KL/rank shifts, and sparse rank shifts (Huan et al., 2025; Chen et al., 2026).This study adds empirical measures of divergence, entropy, and probability-mass redistribution, plus forward and reverse cross-sampling interventions tied to reasoning performance.Source: §6 · p. 20

• Cross-sampling interventions confirm that performance gains hinge on the small subset of token positions where RLVR induces meaningful divergence.The conclusion identifies these positions as having disproportionate functional importance.Source: §7 · p. 20
• At high-divergence positions, RLVR typically reallocates probability within existing candidate sets, although rare substantial re-ranking may also matter for reasoning improvements.Source: §7 · p. 20
• Divergence-weighted advantages can influence learning dynamics, but stabilizing performance may require model-specific choices and further investigation.Source: §7 · p. 20
• The findings frame RLVR as a sparse intervention on high-impact decision points and motivate future objectives and diagnostics that incorporate distributional structure.The conclusion links this perspective to more effective, interpretable, and controllable LLM post-training.Source: §7 · p. 20

A.1 Sequence-level Divergence Bounds for Cross-Sampling

• The setup fixes sequences at horizon Tmax by absorbing EOS, so all sequences have length Tmax.This is equivalent to the stopped process; terms after τ contribute zero because both policies become point masses on EOS.Source: §A.1 · p. 24
• After EOS, the process deterministically emits EOS for the remaining positions through Tmax.Source: §A.1 · p. 24
• A switching rule S maps token histories X<t to a binary indicator St that determines whether intervention is active.Source: §A.1 · p. 24
• At each history, the mixed policy is the primary policy when St=0 and the intervention policy when St=1.The policy is defined by πmix(· | X<t) = (1 − St) πprim(· | X<t) + St πint(· | X<t).Source: §A.1 · p. 24
• The mixed policy combines the primary and intervention next-token distributions using St as the selector.When St=0, πmix equals πprim; when St=1, πmix equals πint.Source: §A.1 · p. 24
• Pmix and Pint denote the sequence-level distributions induced by the mixed and intervention policies, respectively.Source: §A.1 · p. 24

A.1.1 KL Case

• Sequence-level KL decomposes into an expectation of conditional token-level KL contributions across positions.The lemma uses the factorizations of the sequence distributions and conditional expectation over each token given its history.Source: §A.1.1 · p. 24
• The token-level KL threshold switching rule yields a bound on sequence-level KL divergence between the cross-sampled and target intervention policies.The proof applies the KL decomposition and accounts for non-intervention steps before EOS.Source: §A.1.1 · pp. 24–25
• In regimes where the base and RL fine-tuned models are already close, effective KL on non-intervention steps is typically much smaller than ε.The appendix defines this effective KL over non-intervention steps.Source: §A.1.1 · p. 25

A.1.2 JS Case

• The exact sequence-level JS decomposition uses history-dependent skew-JS terms, rather than the ordinary JS divergence used in the experiments.At skew parameter 1/2, skew-JS reduces to the usual Jensen–Shannon divergence.Source: §A.1.2 · p. 26
• Token-level skew-JS control under the switching rule implies a sequence-level Jensen–Shannon bound for the induced mixture and intervention laws.The proposition applies the exact skew-JS decomposition to the sequence-level laws.Source: §A.1.2 · p. 27
• Under Proposition A.6’s assumption, the effective average skew-JS divergence on non-intervention steps is at most ε.This average captures the skew-JS terms in the exact decomposition and is the JS analogue of effective KL.Source: §A.1.2 · p. 28

A.2.1 Token Distribution Analyses; A.2.2 Cross-Sampling

• Cross-sampling swaps tokens between base and RL models at positions where JS divergence exceeds a threshold, using the token-analysis inference setup.AIME sampling uses top-p = 0.7 and temperature = 1; divergence on AIME is calculated from the top-p truncated distribution, while fine-training data uses top-p = 1.Source: §A.2.2 · p. 28, §A.2.1 · p. 28

A.2.3 Additional Training Details

• The appendix specifies DAPO and SFT training setups and divergence-weighted advantage schedules, including Qwen2.5-Math-7B DAPO clip ratios of 0.2 and 0.28 and Qwen2.5-32B SFT on 42k examples for five epochs.The Qwen2.5-Math-7B DAPO setup uses a 1 × 10^-6 learning rate and no explicit reference-KL penalty; divergence-weighted α rises from 0 to 50 in the high-KL setting or falls from 0 to −50 in the low-KL setting.Source: §A.2.3 · p. 29

A.3 Weight-Level Analysis of Changes

• Across Qwen2.5-32B and Qwen2.5-Math-7B, RL fine-tuning has lower relative gap ratios than SFT, indicating less extensive cumulative weight modification.The comparison standardizes dataset size and training steps across methods; the authors interpret the result as evidence for sparse, targeted RL parameter adjustments.Source: §A.3 · pp. 29–30, Table 3 · p. 30

A.4 Additional RLVR vs. Supervised Fine-Tuning Results

• On AIME 2024, SFT shows higher overall JS divergence and more high-divergence tokens than RLVR, lower top-k overlap with the base model, larger rank shifts, and more frequent elevation of low-base-probability choices.Source: §A.4 · p. 30, Figure 12 · p. 31, Figure 16 · p. 32

A.5.1 Supplementary Figures for Main Models

• Sparse divergence patterns extend to GPQA-Diamond and remain consistent across top-p sampling settings and when divergence is computed with full rather than truncated distributions.The sampling robustness check compares top-p values of 0.7, 0.8, and 0.9; the truncation comparison concerns Qwen2.5-32B DAPO and SimpleRL on AIME 2025.Source: §A.5.1 · pp. 34, 38, Figure 28 · p. 37, Figure 30 · p. 38

A.5.2 Comparison of DAPO Variants: Clip-Higher Settings

• With DAPO clip-higher set to 0.2 rather than 0.28, divergence shifts are sparser on AIME 2024 and 2025, but the divergent tokens have higher upper-percentile JS values.The more restrictive setting also promotes fewer tokens with low base probability.Source: Figure 33 · p. 39, §A.5.2 · p. 39, Figure 37 · p. 41

A.5.3 Qwen3-8B with DAPO

• Qwen3-8B with DAPO retains sparse distributional shifts despite nearly twice as many RLVR training steps as Qwen2.5-Math-7B, with divergence concentrated at early and late positions.The analysis covers AIME 2024 and AIME 2025.Source: §A.5.3 · pp. 44–45, Figure 43 · p. 45, Figure 44 · p. 45

A.5.4 Mistral-Small-24B with SimpleRL

• Mistral-Small-24B with SimpleRL shows sparse divergence patterns on both AIME 2024 and AIME 2025, extending the findings to another model architecture.The supplementary analysis also examines positional concentration, entropy across divergence bins, and tail behavior.Source: §A.5.4 · p. 47, Figure 48 · p. 47
• For Mistral-Small-24B with SimpleRL, divergence concentrated at response starts and ends; accompanying figures characterize entropy by divergence bin and RL top-1 choices’ base probabilities.The figures cover AIME 2024 and 2025 for positional divergence and base-probability thresholds, and AIME 2024 for entropy distributions.Source: Figure 49 · p. 47, Figure 50 · p. 47, Figure 51 · p. 48

A.6 Additional Cross-Sampling Results

• A single-prompt cross-sampling procedure generates mainly with a primary policy and intervenes with another policy when token-level divergence exceeds a fixed threshold.For each intervention count, the response is completed with the primary policy and evaluated before cross-sampling continues from the earlier prefix.Source: §A.6 · p. 48
• Injecting DAPO RL tokens into base generations progressively recovered accuracy, while replacing RL tokens with base tokens degraded it toward base performance on AIME 2024 and 2025.The same trend is shown for Mistral-Small-24B SimpleRL on AIME 2024.Source: Figure 53 · p. 50, Figure 54 · p. 50, Figure 56 · p. 51
• For DAPO on AIME 2024, random replacement improved performance more slowly than targeted cross-sampling, indicating the importance of selecting RL tokens at critical positions.The figure notes that random replacement may skip critical positions in reasoning trajectories.Source: Figure 55 · p. 51

A.7 Additional Results

• A single Qwen3-32B-Base DAPO training run exceeded 140,000 GPU hours to reach roughly step 500, so shorter runs may leave baselines undertrained.The authors identify this as a consideration for future work and do not investigate it further because of resource constraints.Source: §A.7 · p. 52
• With the standard DAPO recipe and dataset, Qwen3-32B-Base AIME24 performance plateaued around steps 80–180 at mean@32 57–60%, then improved steadily beyond step 180 to exceed 70%.Source: §A.7 · p. 52