Source-linked AI summary
Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation
Yuxuan Jiang, Runchao Li, Shubhashis Roy Dipta, Dawei Li, Zhao Yang
TL;DR
The paper asks whether persistent high-loss tokens in OPD are necessary reasoning signals or unproductive residuals. It characterizes Rock Tokens, analyzes their training dynamics and persistence, and intervenes by freezing their gradients. Rock Tokens remain stagnant despite substantial gradient magnitude, while deprioritizing them has negligible downstream-performance impact and increases training wall-clock speed by 1.4×.
Problem
The functional contribution of persistent high-loss tokens in OPD is unclear, particularly whether they are indispensable reasoning pillars or unproductive residuals.
Method
The paper identifies Rock Tokens, analyzes their gradient dynamics and persistence, and uses token knock-out and gradient-freeze interventions to test their functional necessity.
Results
1.4× training wall-clock speed increase accompanies freeze-weighting of Rock Tokens, with negligible impact on downstream performance.
Takeaways & Limitations
Rock Tokens occupy a dominant loss-landscape position without being decisive for performance gains, supporting their strategic deprioritization during OPD optimization.
Takeaways & Limitations
The evidence relies predominantly on competitive mathematical reasoning and a specific 30B-to-4B distillation pairing, so generalization across tasks and architectures remains open.
Abstract
from arXiv · showhide
While recent work in Reinforcement Learning with Verifiable Rewards (RLVR) has shown that a small subset of critical tokens disproportionately drives reasoning gains, an analogous token-level understanding of On-Policy Distillation (OPD) remains largely unexplored. In this work, we investigate high-loss tokens, a token type that--as the most direct signal of student-teacher mismatch under OPD's per-token KL objective--should progressively diminish as training converges according to existing studies; however, our empirical analysis shows otherwise. Even after OPD training reaches apparent saturation, a substantial subset of tokens continues to exhibit persistently high loss; these tokens, which we term Rock Tokens, can account for up to 18\% of the tokens in generated outputs. Our investigation reveals two startling paradoxes. First, despite their high occurrence frequency providing a disproportionately large share of total gradient norms, Rock Tokens themselves remain stagnant throughout training, resisting teacher-driven corrections. Second, through causal intervention, we find that these tokens provide negligible functional contribution to the model's actual reasoning performance. These findings suggest that a vast amount of optimization bandwidth is spent on structural and discourse residuals that the student model cannot or need not internalize. By deconstructing these dynamics, we demonstrate that strategically bypassing these ``stumbling blocks'' can significantly streamline the alignment process, challenging the necessity of uniform token weighting and offering a more efficient paradigm for large-scale model distillation.
1 Introduction
OPD research extends token-level analysis beyond RLVR by examining persistent high-loss tokens that remain after apparent convergence. The paper defines Rock Tokens, investigates their identity and persistence, and evaluates their functional impact on distillation.
- Motivation: OPD offers direct per-token KL measurements of student–teacher mismatch, but analogous token-level understanding remains largely unexplored.High-loss positions are therefore natural candidates for studying alignment corrections.
- Phenomenon: Rock Tokens remain consistently high-loss after apparent training saturation, comprising approximately 6% of the vocabulary but up to 18% of output tokens.This persistence challenges the expectation that high-loss tokens diminish as the student aligns with the teacher.
- Investigation framework: The paper investigates Rock Tokens across three questions: what they are, why they persist, and how they affect OPD training.The framework covers token properties, persistence mechanisms, and functional contribution during distillation.
- What are Rock Tokens?: Rock Tokens primarily consist of syntactic and structural scaffolding, including formatting delimiters, whitespace symbols, and frequent discourse markers.Gradient-dynamics analysis tests whether their persistent high loss yields constructive learning signals.
- Why do they persist?: Token knock-out experiments test whether student decoding path dependency explains why Rock Tokens resist teacher-driven correction.The study asks whether reliance on these tokens to maintain reasoning flow causes their persistence.
- How do they affect OPD?: Freezing Rock-Token gradients from the start of OPD isolates their functional necessity and tests whether they are reasoning pillars or optimization redundancies.The intervention directly evaluates the consequences of bypassing these tokens during alignment.
- Contribution: The paper identifies Rock Tokens as a persistent high-loss phenomenon and argues that substantial OPD training signals may contribute less to reasoning performance than expected.This frames Rock Tokens as a potential inefficiency in current alignment pipelines.
2 Rock Tokens
Rock Tokens are persistent high-loss token types that concentrate in structural and discourse scaffolding, occupy a substantial share of outputs, and remain resistant to OPD updates despite aligned gradients. Their small per-occurrence gradients accumulate through frequency, while their KL remains largely unchanged during training.
- Definition: Rock Tokens are defined by persistent high loss across training and context-consistent recurrence, distinguishing stable behavior from isolated trajectory spikes.The definition requires high-loss status before and after OPD plus stability across similar contexts.
- Prevalence: 18% median density: context-consistent Rock Tokens occupy nearly one-fifth of tokens in generated reasoning outputs.This remains true after context-window similarity filtering.
- Identity: Rock Tokens primarily mark structural or discourse boundaries, including math delimiters, whitespace, Markdown structure, discourse markers, and digits.Their clustering suggests disagreement centers on reasoning structure rather than content selection.
- Gradient geometry: Median gradient magnitude is approximately 0.016 for Rock Tokens versus approximately 0.54 for rare high-KL tokens, but frequency makes their aggregate update force dominant.The decomposition attributes this dominance to the occurrence-rate factor rather than per-occurrence gradient size.
- Gradient geometry: Median gradient alignment is approximately 0.040 for Rock Tokens versus 0.025 for high-KL and 0.006 for random tokens, with some rocks exceeding cos > 0.3.Their directions are positively aligned with the frequency-balanced global descent direction.
- Persistence: Rare high-KL tokens reduce median KL from 2.02 to 0.85, whereas Rock Tokens remain near 0.21 to 0.19 and cluster around zero change.The contrasting checkpoint behavior indicates learning for rare high-KL tokens but persistence for rocks.
- Open mechanism: The proposed mechanical explanation is optimizer suppression of frequent structural tokens through inflated Adam second-moment estimates, but formal verification remains future work.The paper presents this as a hypothesis rather than an established mechanism.
3 Pillar Tokens: The Scaffolding of Fragile Reasoning
Knockout experiments show that most screened Rock Tokens have negligible causal effects on benchmark accuracy, while a small minority are indispensable reasoning anchors. These Strong Pillars are rare, asymmetric with respect to harmful removals, and not predicted by standard distributional metrics.
- Knockout results: Most of the 200 screened candidates are Neutral on both benchmarks, while Pillars form only a short tail of the knockout distribution.Neutrality is defined by the ε-band and bootstrap significance procedure shown in Figure 4.
- Intervention: The knockout procedure masks a candidate token’s logit at every decoding step and measures the resulting benchmark accuracy change.The screened pool contains |eR| = 200 candidates, extending beyond the core K = 100 set.
- Pillar prevalence: Only 3.5% of Rock Tokens on MATH-500 and 1.5% on IFEval qualify as Strong Pillars.A Strong Pillar is a token whose removal causes significant degradation, with ΔB(v) ≤ −0.01.
- Directional asymmetry: No Strong Stumbling Blocks appear, suggesting persistent deviations are usually necessary anchors or harmless stylistic preferences rather than harmful reasoning errors.The result is described as directional asymmetry in the knockout effects.
- Metric orthogonality: Pillarhood is not significantly correlated with student or teacher entropy, log-frequency, or residual KL, with all reported correlations satisfying |r| < 0.07.Thus causal necessity is not captured by these standard distributional signals.
- Interpretation: Only 1.5%–3.5% of Rock Tokens are causally necessary, indicating a broad residual buffer surrounding a few critical semantic anchors.The paper frames this concentration as evidence against reliance on the entire high-loss set.
4 The genuine contribution of Rock Tokens
The paper tests whether persistent Rock Token residuals are essential reasoning signals or inefficient optimization burdens. Window-aware freezing shows that selectively reducing their optimization pressure can preserve reasoning while improving training efficiency.
- Rock Tokens are evaluated under competing hypotheses: late-stage stumbling blocks versus inherently useless stylistic residuals.The paper uses selective intervention to distinguish these possibilities.
- The intervention reweights local high-divergence windows rather than isolated Rock Tokens.The method uses a persistence set and associated windows to capture trajectory-level context.
- λ = 1 is baseline OPD, while Rock-Freeze sets λ = 0 for Rock Tokens and their associated windows.The three-regime design also compares selective freezing against random signal reduction.
- Rock Tokens provide essential anchoring gradients, whereas random removal of total gradient signals causes severe performance degradation.The comparison separates structured Rock-associated effects from stochastic and frequency-related artifacts.
- Selective Rock Token down-weighting maintains a higher accuracy ceiling and yields a 1.7× wall-clock speedup without substantial loss of reasoning integrity.In the reported setup, Rock Tokens comprise approximately 18% of output tokens.
- Freezing 30% high-cost tokens yields a 1.4× wall-clock speedup with minimal impact on reasoning integrity.The result supports non-uniform gradient allocation rather than treating Rock Tokens as entirely disposable.
5 Experiment Setups
The experiments use a two-stage teacher-to-student distillation pipeline, standardized zero-shot evaluation, and competitive mathematics benchmarks as the primary test setting.
- The pipeline transfers knowledge from Qwen3-30B-A3B-Instruct-2507 to Qwen3-4B-Instruct-2507 through off-policy and on-policy distillation stages.Both models use thinking mode disabled; Stage 1 trains on 20k teacher-generated solutions with forward KL and cross-entropy.
- Evaluation uses LM-EVALUATION-HARNESS in a zero-shot setting and reports Pass@1 averaged over five independent runs.The repeated runs account for decoding variance.
- The primary metric is aggregated average accuracy across AIME 24, AIME 25, and HMMT 25-Feb, covering 90 problems.MATH-500 and IFEval extend the analysis for token-level statistical evaluation.
6 Related Works
Related work positions OPD as dense token-level post-training while highlighting unresolved questions about teacher–student mismatch and which local signals actually support reasoning. This paper addresses that gap through Rock Token analysis.
- Reasoning improvement has traditionally used supervised fine-tuning and reinforcement learning, while OPD combines trajectory exploration with dense token-level guidance.Large-scale systems use OPD as a post-training component alongside other methods.
- Prior OPD studies examine teacher–student incompatibility through global factors such as vocabulary, reasoning style, teacher selection, and model family.The paper instead focuses on a fine-grained source of stagnation within OPD dynamics.
- Research on RLVR and reasoning anchors shows that optimization and inference contributions are non-uniform across tokens.High-entropy forking tokens and perturbation-sensitive anchor tokens motivate token-level analysis.
- Related work on structured reasoning, tool-use supervision, and evaluation emphasizes how intermediate steps and contextual structure affect local learning signals.These studies connect token-level supervision with broader organization of reasoning.
- Multimodal retrieval and structured representation studies examine fine-grained grounding, compositional matching, and robust alignment under complex modification signals.These works broaden the efficiency and alignment context beyond text-only reasoning.
- The paper asks whether persistent high-loss OPD positions are indispensable reasoning pillars or unproductive residues.It addresses this question through gradient-dynamics and training-effect analyses of Rock Tokens.
7 Conclusion
The conclusion argues that Rock Tokens dominate the OPD loss landscape without being decisive for reasoning gains. Freeze-weighting them preserves downstream performance while improving training efficiency, motivating token-selective alignment.
- Rock Tokens occupy a dominant position in the loss landscape but are not the decisive factor for performance gains.
- Freeze-weighting Rock Tokens produces negligible downstream-performance impact and a 1.4× increase in training wall-clock speed.
- The findings challenge the assumption that every high-loss token must be aligned and support bypassing computational redundancy through token-selective optimization.
A Limitations
The study’s evidence is concentrated in competitive mathematical reasoning and one 30B-to-4B distillation setup, while its binary mitigation leaves room for softer, adaptive approaches. The work also acknowledges limited direct societal risk but potential dual-use concerns.
- Scope boundaries: The evaluations rely predominantly on competitive mathematical reasoning, so Rock Token roles may differ in open-ended generation or coding.The stated boundary concerns tasks where structural boundaries are less rigid.
- Scope boundaries: The experiments use a specific 30B-to-4B distillation pairing, leaving behavior across architectures and much larger scales open.
- Methodological boundary: The binary gradient-freeze mitigation demonstrates causal necessity but is a blunt instrument compared with dynamic reweighting or soft penalties.
- Risk assessment: The work introduces no deployed system, human-subject data, or sensitive personal data, but improved language-model training may carry dual-use risks.
B Analysis on Rock Tokens
Rock Tokens are numerically identified high-loss tokens that cluster around structural and discourse boundaries rather than ordinary mathematical content. Their concentration suggests persistent OPD divergence is associated with how reasoning is structured, not simply what content is emitted.
- Token identity: The top-100 Rock Tokens form coherent structural or discourse categories despite being selected by a numerical criterion.
- Functional clusters: Four clusters dominate: LaTeX and math delimiters, Markdown and whitespace structure, discourse markers, and digits.
- Control comparison: Frequency-matched controls instead predominantly contain content-bearing nouns, verbs, mid-clause function words, and single-letter variables.
- Interpretation: Rock Tokens concentrate at structural boundaries where the model opens math mode, starts paragraphs or sections, and initiates reasoning steps.
C.1 off policy and on policy training
The analysis combines the distillation setup with causal knockout and multiple-testing analyses to distinguish persistently high-loss tokens from tokens that are functionally necessary. The results indicate that most screened candidates have no detectable causal effect, while a small, task-specific subset behaves as Pillars.
- C.1 off policy and on policy training: The pipeline distills Qwen3-30B-A3B-Instruct-2507 into Qwen3-4B-Instruct-2507 using two stages: off-policy KD followed by on-policy KD.
- C.1 off policy and on policy training: Stage 1 uses teacher responses with forward-KL distillation plus cross-entropy, whereas Stage 2 uses student rollouts with pure reverse-KL distillation.
- Causal analysis: Across 18 predictor–benchmark pairs, no correlation exceeds |r| > 0.075 and no p-value is below 0.30.
- Multiple-testing analysis: Bonferroni and Benjamini–Hochberg corrections yield zero per-token rejections, so individual Pillar identifications remain exploratory.
- Multiple-testing analysis: The combined sign-asymmetry test finds 10 negative and 0 positive effects, with two-sided exact binomial p = 1.95 × 10−3.
- Causal intervention: Pillarhood is defined through token- or window-level knockout effects that significantly degrade benchmark accuracy, while positive effects indicate Strong Stumbling Blocks.
- Causal intervention: The screened pool contains 200 candidates rather than the definitional 100-token Rock set, trading selection stability for broader search coverage.
D.4 Are Rock Tokens pillar? Experiments and findings
Pillarhood is largely independent of entropy, frequency, and loss-based predictors, while most screened candidates show no detectable causal effect when knocked out. The identified Strong Pillars are rare, task-specific, and content-bearing rather than structural Rock Tokens.
- Most screened candidates fall within the ε-band, indicating no detectable causal effect from their removal on either benchmark.The per-token knockout-effect distribution is concentrated near zero rather than in its tail.
- None of the six candidate predictors reaches |r| > 0.07 for MATH-500 knockout effects across 200 screened candidates.The predictors include student and teacher entropy, log-frequency, rock rate, and mean post-OPD KL.
- 10:0 is the sign split between Pillar-side and Stumbling-side rejections, with exact probability 2^-10 ≈0.002 under the multiplicity-noise null.Individual token identifications remain exploratory because neither Bonferroni nor Benjamini–Hochberg retains any token.
- Pillars comprise 3.5% of candidates on MATH-500 and 1.5% on IFEval, with no Strong Pillar shared across both benchmarks.The persistent high-KL criterion therefore over-selects causally relevant tokens by more than an order of magnitude, and Pillarhood is task-specific.
- Strong Pillars are content-bearing tokens at decision points, unlike the structural delimiters that dominate Rock Token surface clusters.Examples include “certain,” “strategic,” and “Initialize” on MATH-500, and “tools,” “trade,” and “-like” on IFEval.
- Pillarhood is an orthogonal, causal property rather than an inference-time projection of entropy-, frequency-, or KL-based importance.Pillars are scattered across the entropy range and are not the tokens with the largest residual KL.
E Selection of Ratio
The Rock-Token cutoff balances selection stability against corpus-level KL coverage. The procedure selects K = 100 for a stable definitional set, while using Kscreen = 200 for broader per-token causal searches.
- K = 100 yields average Jaccard above 0.70 while covering ≈60% of corpus-level KL, making it the selected Rock-Token cutoff.This corresponds to ≈2.5% of the observed vocabulary and remains within the stable regime.
- Average Jaccard exceeds 0.70 for K ≤100 and decays toward a ≈0.57 floor for K ≥150.The decay indicates that additional tokens beyond the stable regime are no longer reliably reproduced across sub-corpora.
- Kscreen = 200 is used for Appendix D.4 causal analysis as a strict superset intended to find rare-but-real Pillars.The broader pool increases detection probability while probing a regime where selection stability has fallen below J ≈0.6.