PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, Yu Cheng

arXiv:2608.14277v1cs.CLcs.AI

TL;DR

Distilling long-context reasoning into short-context students is hindered by tokenizer mismatch, distribution mismatch, and unstable response lengths. SimpleOPD addresses these issues with shared-text token alignment and stabilization losses, yielding consistent reasoning gains across model families, including a 21.2-point ProofBench improvement to 55.2 for Intern-S2-Preview.

  • Problem

    OPD for long-context reasoning teachers and short-context students lacks reliable heterogeneous-model token alignment and stable training under context-length and distribution mismatches.

  • Method

    SimpleOPD performs OPD in shared text space using identical-span token alignment, reference KL regularization, and termination-token advantage masking.

  • Results

    SimpleOPD consistently improves mathematical reasoning across model families, especially natural-language proving, while also improving science benchmarks; Intern-S2-Preview gains 21.2 points on ProofBench to 55.2.

  • Takeaways & Limitations

    OPD can transfer proof-reasoning capabilities from SU-01 to diverse short-context students without shared tokenization or supervised teacher trajectories.

  • Takeaways & Limitations

    Effectiveness can be limited by mismatched reasoning patterns between teacher and student and unreliable supervision over long sequences.

Abstract

from arXiv · show

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

1 Introduction

SimpleOPD addresses tokenizer mismatch and teacher–student distribution mismatch when distilling long-context reasoning teachers into short-context students. It uses shared-text token alignment and stabilization losses to transfer reasoning capabilities, yielding consistent mathematical-reasoning gains across model families.

  • OPD for long-context teachers and short-context students faces incompatible tokenizers and substantial teacher–student distribution mismatch.Differences in model capacity and context length contribute to the mismatch.
  • SimpleOPD aligns teacher and student tokens only when they occupy identical text spans in shared student-generated text.The teacher evaluates the student’s response under its native tokenizer, avoiding artificial correspondence between incompatible tokenizations.
  • 21.2 points on ProofBench is the gain reported for Intern-S2-Preview, while SimpleOPD consistently improves mathematical reasoning across same-family and cross-family students.The strongest gains occur on natural-language mathematical proof tasks.
  • Termination-token advantage masking and student reference KL loss stabilize training, reduce truncation, and constrain excessive length growth.Direct OPD can otherwise cause excessive length growth, frequent truncation, and training instability.

2 Method

SimpleOPD performs tokenizer-agnostic on-policy distillation in a shared response-text space, allowing the teacher to score student responses under its own tokenizer and chat template. It aligns only identical response spans, uses teacher supervision at aligned positions, and optimizes the resulting surrogate with PPO updates.

  • Shared text-space evaluation: The student response is decoded to text, appended to a teacher-formatted context, and re-encoded so the teacher evaluates identical response text with its native tokenizer and template.Student and teacher contexts may differ, but the response string being evaluated remains identical.
  • Cross-tokenizer alignment: A teacher token aligns with a student token only when both consume the same response prefix and contribute the same start-to-end text span.Partially overlapping tokens are excluded because their log-probabilities cannot be uniquely assigned across tokenizations.
  • Supervision target: Aligned student positions inherit teacher log-probabilities, while unmatched positions retain student log-probabilities in the student-length teacher target.The lexical overlap ratio ρ measures the fraction of student response tokens receiving teacher supervision.
  • Optimization objective: The distillation objective applies supervision only at tokenizer-agreeing positions and reduces to the standard token-level form when the student and teacher tokenizers are identical.For repeated updates on one rollout batch, unmatched positions use pre-update student log-probabilities within a PPO clipped policy loss.

3 Experiment

The experiments distill SU-01’s mathematical proof-generation capabilities into same-vocabulary and cross-tokenizer student models using proof-focused training data, standardized OPD training, and multiple reasoning benchmarks.

  • Teacher and students: SU-01 transfers mathematical reasoning and proof-generation capabilities to diverse student models through on-policy distillation.The teacher is an IMO gold-medal-level model developed by the authors.
  • Teacher and students: Experiments cover same-tokenizer Qwen3 students and cross-tokenizer Qwen3.5, Intern-S2-Preview, GLM-4.7-Flash, and Gemma-4-26B-A4B students.The same-tokenizer students are Qwen3-4B and Qwen3-30B-A3B; cross-tokenizer evaluation includes Qwen3.5-4B, Qwen3.5-35B-A3B, Intern-S2-Preview, GLM-4.7-Flash, and Gemma-4-26B-A4B.
  • Training data: All training examples are mathematical proof problems drawn from curated proof corpora, community contributions, and olympiad training materials.The listed sources are OPC, AoPS, Books, and Shuzhimi.
  • Training and evaluation setup: OPD samples responses from the student policy while using SU-01 as teacher, with Slime for training and SGLang for evaluation rollout.SU-01 is based on Qwen3-30B-A3B and can sustain over 100K tokens of natural-language reasoning for difficult Olympiad problems.
  • Training and evaluation setup: Training uses 100 rollout iterations, learning rate 1e-6, rollout batch size 64, four responses per prompt, and maximum rollout lengths of 32K or 6k tokens depending on model family.The PPO clipping coefficient is 0.2, and the policy is updated four times per rollout step.
  • Training and evaluation setup: Evaluation covers ProofBench, AnswerBench, AIME25, and AMOBench, using repeated proof evaluations and checkpoint selection by average AIME@4 and AnswerBench@1.ProofBench uses DeepSeek-V4-Flash as judge, with four evaluations averaged for the same proof rollout; generation uses temperature 1.0, top-p 0.95, and maximum response length 160,000 tokens.

4 Experimental Results

Section 4 shows that direct OPD causes response-length growth, truncation, repetition, and training instability, while special-token masking plus student-reference KL stabilizes training and yields consistent reasoning gains across models and tokenizer settings.

  • Training instability: Direct OPD increases response length, truncation, and repetition, with stronger degeneration observed for Qwen3.5-35B-A3B.Intern-S2-Preview shows modest task gains alongside these effects, whereas Qwen3.5-35B-A3B experiences more noticeable performance deterioration.
  • Mitigation strategies: Masking </think> and <|im_end|> advantages mitigates length-related instability but does not prevent later truncation-rate growth.These tokens primarily control output format and termination behavior rather than matching the teacher distribution.
  • Mitigation strategies: The student-reference KL loss reduces truncation to nearly zero and consistently improves AIME25 and AnswerBench performance.The loss coefficient is set to 0.5, preventing excessive deviation from the student’s initial distribution and mitigating response-length expansion.
  • Combined SimpleOPD results: Qwen3-30B-A3B gains 22.67 points on ProofBench, increasing from 13.80 to 36.47, alongside AnswerBench and AIME25 gains of 15.33 and 5.42 points.Among same-tokenizer models, Qwen3-4B-OPD gains 12.30 points on ProofBench, 17.00 points on AnswerBench, and 19.58 points on AIME25.
  • Cross-tokenizer distillation: Intern-S2-OPD improves from 34.0 to 55.2 on ProofBench, a gain of 21.2 points over the original Intern-S2-Preview model.Under the Gemini-2.5-Pro judging setting, Intern-S2-Preview nearly matches SU-01’s 45.00 score in another evaluation passage and surpasses SU-01 on AnswerBench and AIME25.
  • Cross-family distillation: Cross-family distillation raises GLM-4.7-OPD from 30.8 to 39.7 on ProofBench and from 69.6 to 72.0 on AnswerBench, while Gemma-4-26B-A4B-OPD raises ProofBench from 25.5 to 34.2 but lowers AnswerBench from 68.8 to 67.5.The results indicate consistent GLM gains but mixed Gemma outcomes under tokenizer mismatch.

5 Analysis

The analysis shows that SimpleOPD transfers reasoning effectively across tokenizer and model-family differences, with gains extending from proof-focused mathematics to scientific reasoning. Ablations further indicate that proof data, longer stable distillation, and moderate student-reference regularization improve outcomes.

  • Cross-method comparison: SimpleOPD improves Intern-S2 by 22.80, 4.07, 6.67, and 1.50 points on ProofBench, AnswerBench, AIME25, and AMOBench, respectively.The comparison includes the unmodified Intern-S2 model, EOPD, and G-OPD.
  • Tokenizer alignment: Aligned-token ratios are high at training onset and increase over time, preserving substantial shared-span supervision despite tokenizer differences.Partial alignment retains usable training signal without requiring full tokenizer compatibility.
  • Scientific generalization: Proof-focused OPD improves out-of-domain scientific reasoning, with HiPhO rising from 38.6 to 41.1 over Intern-S2-Preview.The analysis reports slight improvements on FrontierScience-Olympiad and HLE and a clearer gain on HiPhO.
  • Data composition: Using only proof data raises ProofBench@4 from 21.70 to 44.50, outperforming the alternative composition for natural-language proof-reasoning transfer.Both compositions substantially improve Intern-S2-Preview, while adding verifiable math data yields only a marginal AnswerBench improvement.
  • Distillation length: Increasing distillation length from 6k to 32k consistently improves evaluated benchmarks when training is stable, including Qwen3.5-OPD ProofBench@4 from 40.07 to 42.39.A 6k length is insufficient to fully capture teacher reasoning patterns on long-context proof-reasoning tasks.
  • Student-reference regularization: A student-reference KL coefficient of 1.0 gives GLM-4.7-Flash-OPD its best overall performance, raising ProofBench@4 from 30.75 to 39.71 and AnswerBench@8 from 69.59 to 71.97.All tested coefficients outperform the base model; 0.5 provides insufficient regularization, whereas 1.2 imposes a stronger constraint.

6 Related Work

Long chain-of-thought reasoning has become a central test-time scaling paradigm, while on-policy distillation provides dense supervision through student-generated rollouts. Existing OPD research mainly assumes shared vocabularies, leaving heterogeneous cross-model transfer underexplored amid teacher-student mismatch and long-sequence supervision challenges.

  • Long chain-of-thought reasoning enables LLMs to address challenging real-world tasks, including long-horizon agentic tasks and Olympiad-level reasoning.
  • On-policy distillation optimizes reverse KL divergence over student-generated rollouts, combining on-policy exploration with dense token-level supervision.
  • OPD effectiveness can be limited by mismatched reasoning patterns between teacher and student and unreliable supervision over long sequences.
  • Most existing OPD studies assume shared vocabularies, leaving heterogeneous models with different architectures and vocabularies underexplored.The paper investigates heterogeneous OPD for transferring long-context reasoning capabilities across diverse models.

7 Conclusion

The work studies on-policy distillation from the long-context reasoning model SU-01 to short-context students. It performs tokenizer-agnostic alignment in shared text space and identifies severe instability in naive distillation, including length growth, missing termination tokens, and truncation.

  • The study transfers reasoning from the long-context model SU-01 to short-context student models using on-policy distillation.
  • OPD handles tokenizer differences by operating in shared text space and aligning only tokens with identical spans under both tokenizers.
  • Naive distillation caused severe training instability, with output lengths continually increasing, termination tokens frequently missing, and many responses truncated.

A Cross-Tokenizer Scoring and Alignment · B Model Solutions for ProofBench

SimpleOPD scores teacher outputs across mismatched tokenizers and greedily aligns teacher and student token pieces in shared text space, enabling supervision for most semantically informative tokens. The section also illustrates the distilled model’s proof-generation behavior through two qualitative ProofBench examples from Intern-S2-OPD.

  • A Cross-Tokenizer Scoring and Alignment: Algorithm 1 defines the complete procedure for cross-tokenizer teacher scoring and greedy token-piece alignment.The procedure is presented as the implementation for cross-tokenizer distillation.
  • A Cross-Tokenizer Scoring and Alignment: Teacher scoring processes token-out, text-in representations, while initializing all positions with student log-probabilities.Teacher scores are computed over the student-generated sequence before alignment.
  • A Cross-Tokenizer Scoring and Alignment: The alignment loop compares tokenizer-native teacher and student token pieces while tracking accumulated text histories.It advances through both token sequences until either sequence is exhausted.
  • A Cross-Tokenizer Scoring and Alignment: When histories differ in length, the procedure greedily advances the shorter side or both sides to recover alignment before returning the scored sequence.The cases distinguish whether the teacher-side or student-side accumulated history is longer.
  • A Cross-Tokenizer Scoring and Alignment: The SU-01-to-Intern-S2-Preview visualization confirms that alignment gives most semantically informative tokens teacher supervision.Figure 10 displays per-token reverse KL values for aligned tokens and marks unaligned tokens separately.
  • B Model Solutions for ProofBench: The ProofBench section presents two qualitative outputs, ProofBench-Basic-001 and ProofBench-Advanced-028, generated by Intern-S2-OPD.Each example contains the original problem, a concise solution overview, and the complete model-generated proof.

B.1 ProofBench-Basic-001

The functional equation is solved by deriving identities that reduce it to an additive map on Z, hence to an affine function. Parameter comparison and verification yield exactly two solution families: the zero function and f(x)=2x+c.

  • Derivation: Evaluating the equation at specialized arguments produces two identities for f(f(t)), enabling a translated Cauchy relation.The identities are f(f(t)) = f(0) + 2f(t) and f(f(t)) = f(2t) + 2f(0).
  • Additivity: Subtracting f(0) defines an additive function g on Z, so cyclicity of Z implies f has the affine form f(x)=ax+b.The reduction gives g(x+y)=g(x)+g(y), and every additive map Z→Z is determined by its value at 1.
  • Parameter determination: Comparing coefficients after substitution forces a=0 or a=2; the first case gives b=0, while the second leaves b arbitrary in Z.Thus the candidate forms are f≡0 and f(x)=2x+b.
  • Verification: Both candidate families verify directly, so the complete solution set is f≡0 or f(x)=2x+c for c∈Z.The verification confirms that no additional affine candidates occur.

B.2 ProofBench-Advanced-028 · C Case Study

The case study proves the geometry claim by locating the circumcenter of △AFP above C, whose perpendicular to BC bisects chord XY. A separate appendix example shows direct OPD degenerating into endless repetition until the 160k-token limit.

  • B.2 ProofBench-Advanced-028: The target is to prove that C is the midpoint of XY in an acute triangle with H as orthocenter, F as an altitude foot, and P reflected across BC.
  • B.2 ProofBench-Advanced-028: Since O = (1, y0) and C = (1, 0), OC is perpendicular to BC; therefore, the circle’s perpendicular from its center bisects chord XY at C.
  • B.2 ProofBench-Advanced-028: A similarity transformation sets B = (0, 0), C = (1, 0), and A = (p, q), with q > 0 and 0 < p < 1 for an acute triangle.
  • B.2 ProofBench-Advanced-028: The coordinate construction derives F, H, and P, using the horizontal BC, vertical altitude x = p, and reflection across the x-axis.
  • B.2 ProofBench-Advanced-028: Because A and P share an x-coordinate, AP is vertical, so its perpendicular bisector is horizontal; the proof proposes a point O on that line as △AFP’s circumcenter.
  • B.2 ProofBench-Advanced-028: Verifying OA = OF establishes that O lies on the perpendicular bisectors of both AF and AP, making O the circumcenter of △AFP.
  • C Case Study: In two representative direct-OPD instances, distilling SU-01 to Qwen3.5-35B-A3B produced endless repetition through either a multi-sentence self-check block or a single-token loop.Both rollouts terminated only upon reaching the 160k-token generation limit.

C.1 Case A: correct answer, then 972 exact self-check repeats (AIME25)

In this AIME25 case, the model reaches the correct answer 279 but then enters a repetitive self-check loop, consuming the generation budget and ending in truncation.

  • C.1 Case A: correct answer, then 972 exact self-check repeats (AIME25): 972 exact repetitions: The model appends the same ten-sentence self-check unit after reaching the correct answer.The repeated unit is 542 characters long.
  • C.1 Case A: correct answer, then 972 exact self-check repeats (AIME25): 279: The model correctly solves the divisibility problem and computes N − 2025 = 279.It derives N = 2304 before reporting the final difference.
  • C.1 Case A: correct answer, then 972 exact self-check repeats (AIME25): 75,000–77,150 response characters: The repetitive failure begins after the model states the answer.At onset, the output contains the conclusion that N = 2304 and N − 2025 = 279.
  • C.1 Case A: correct answer, then 972 exact self-check repeats (AIME25): 160k tokens: The rollout reaches the generation limit inside the repeated unit and is recorded as truncated.The response ends mid-unit during the repeated self-check text.

C.2 Case B: reasoning collapses into a one-token loop (AnswerBench)

In AnswerBench Case B, the model begins reasoning about the 132nd exotic integer but collapses into an uninterrupted one-token “2?” loop, producing no progress or answer before truncation.

  • Case B: reasoning collapses into a one-token loop (AnswerBench): After initial reasoning about multiplicative orders, the model begins emitting the atomic token sequence “2?” indefinitely.The loop starts around response tokens 840–919, following a stalled calculation for m = 8.
  • Case B: reasoning collapses into a one-token loop (AnswerBench): Over 34,000 repetitions, the loop produces no new claim, calculation, answer, or termination.The repeated token occupies one uninterrupted run.
  • Case B: reasoning collapses into a one-token loop (AnswerBench): The model tackles a number-theoretic problem asking for the 132nd exotic integer, defined through counts of n-good functions.An n-good function satisfies g(1) = 1 and a divisibility condition involving a^n − b^n; n is exotic when the function count is twice an odd integer.
  • Case B: reasoning collapses into a one-token loop (AnswerBench): The rollout reaches the 160k-token generation limit inside the loop and is recorded as truncated.The response ends with repeated “2?” tokens.
Loading 2608.14277v1…