When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

1 Introduction

• OPD responses can become progressively longer and exhaust the generation budget; Qwen3 examples often answer correctly early, then continue with repetitive or redundant text.Length growth need not reflect more extensive reasoning; it can result from failure to stop.Source: §1 · p. 1, Figure 1 · p. 2
• Treating functionally equivalent EOS tokens as one semantic stopping action substantially mitigates mismatch-induced length inflation across Qwen3, Llama, and Gemma; aligning the decoding stop set alone is insufficient.The proposed corrections align termination in the distillation signal rather than relying only on the decoding interface.Source: §1 · pp. 2–3
• K2-Horizon checkpoints show that termination preferences shift across training stages, while a distinct late-run length inflation persists after termination alignment.The late behavior is not explained by EOS mismatch alone and points to additional mechanisms governing OPD length dynamics.Source: §1 · pp. 2–3
• Across Qwen3, Llama, and Gemma, students and teachers can prefer different tokens for the same stopping decision, even when their declared EOS sets are identical.Because OPD scores the student's sampled token, an EOS token preferred by the student can receive a negative signal when the teacher prefers an alternative, suppressing termination without reliably transferring that alternative.Source: §1 · pp. 1–2
• Prompt and evaluation protocol choices also modulate measured OPD behavior, adding a source of variation across reported results.The authors report sensitivity to both prompt and evaluation protocols.Source: §1 · p. 3
• The authors release an implementation incorporating termination-handling corrections and evaluation protocols.Source: §1 · p. 3

2 Experimental Setup

• In sampled-token OPD, the fixed teacher scores the token sampled from the student's rollout; the update uses its log-probability difference from the student and averages valid generated-token contributions.The teacher scores only the sampled token, and the score-function update treats the advantage as detached without reward-to-go.Source: §2 · p. 3
• The main full ablation uses Qwen3-1.7B-Base as student and Qwen3-4B as teacher; Llama and Gemma pairs test generality, while K2-Horizon checkpoints support stage-wise comparisons.Training uses DAPO-Math-17K; the K2-Horizon final checkpoint is the teacher and earlier stages initialize students.Source: §2 · p. 3
• Training and evaluation usually use the TTRL template, with DAPO and raw-question templates added for prompt comparisons; training permits 7,168 response tokens and checkpoint evaluation 8,192.The main setup allows at most 1,024 prompt tokens during training.Source: §2 · p. 3
• The setup tracks mean response length and clipping ratio, reports Avg@16 across AMC23, AIME24, and AIME25, and compares token-level and total stopping probabilities.Total stopping mass is evaluated at the final position of student rollouts and differs from the fraction of rollouts that terminate naturally.Source: §2 · p. 3
• Table 1 reports declared EOS token IDs and groups models sharing the same stopping set; K2-Horizon's IDs and declared set remain unchanged across stages.For K2-Horizon, token strings were renamed after pretraining, but the IDs and declared stopping set stayed unchanged.Source: Table 1 · p. 4

3.1 Decoding EOS Alignment Alone Is Insufficient

• Adding both EOS tokens to the Qwen3 rollout stopping set leaves length and clipping trajectories near vanilla OPD because it changes neither token probabilities nor distillation supervision.The student assigns approximately 10^-11 probability to the teacher-preferred token around terminal states, while sampled-token OPD suppresses its frequently sampled native EOS.Source: §3.1 · p. 5

3.2 Probability-Level Termination Alignment

• Probability-level corrections outperform stopping-set alignment; semantic EOS aggregation supervises the total probability of termination as one action without choosing a canonical surface token.Teacher-side EOS mapping and canonical single-EOS decoding also behave similarly in the Qwen3 ablation, but require a canonical token.Source: §3.2 · pp. 6, 8

3.3 Termination Mismatch Across Model Families

• Gemma 3’s student and teacher favor different EOS tokens despite sharing a declared stopping set, while semantic correction recovers termination without forcing their surface preferences to coincide.Source: Figure 3 · p. 7, §3.3 · p. 8
• Semantic EOS correction substantially reduces response length and clipping in Qwen3, Llama 3.2, and Gemma 3; Qwen3 and Gemma 3 approach teacher lengths more closely than Llama 3.2.Recovery dynamics differ across families, and residual length gaps remain.Source: Figure 4 · p. 7, §3.3 · p. 8

3.2 Probability-Level Termination Alignment

• Cross-template evaluation can shorten observed responses, especially for DAPO-trained models evaluated with TTRL, while OPD performance gains can persist despite length inflation.The observed accuracy effect also depends on the grader; length inflation does not necessarily imply a comparable loss of reasoning capability.Source: §3.2 · p. 8

4.1 OPD Across Post-Training Stages

• Vanilla OPD from midtraining and SFT checkpoints avoids the severe termination failure seen in more mismatched base-to-post-trained settings; semantic correction produces similar dynamics in these already-aligned cases.Source: §4.1 · p. 9
• Most of the pretrained-to-final termination-preference shift occurs by midtraining, and the SFT student’s preference is already close to the final teacher’s.The pretrained student favors <|ifm|endoftext|>, while the final teacher favors <|ifm|im_end|>; the pretrained student has enough probability on the latter to sample it and receive direct supervision.Source: §4.1 · p. 9

4.2 Residual Length Dynamics

• With semantic correction, response length initially rises across all three model families; corrected K2-Horizon then reaches an elevated, fluctuating plateau rather than recovering to teacher length.The early drift is more pronounced in Llama 3.2 and Gemma 3, and students remain longer than teachers at later checkpoints, especially Llama 3.2. A trajectory-level reverse-KL analysis identifies an omitted future mismatch as consistent with early length growth.Source: §4.2 · p. 10
• In Phase 3 of K2-Horizon Pretrain-to-Final OPD, response length rises again, termination probabilities approach zero, and almost all rollouts reach the generation budget.The renewed inflation is more gradual with semantic EOS correction than with vanilla OPD, and also occurs when both EOS tokens are treated as one stopping action.Source: §4.2 · pp. 10–11, Figure 7 · p. 11

• Prior studies attribute OPD length inflation to long-rollout preference, degraded teacher supervision with trajectory depth, and reverse-KL-induced entropy collapse.These mechanisms are not mutually exclusive and may apply where identified.Source: §5 · p. 11
• Rollout-quality work addresses teacher drift by limiting supervision to early prefixes or improving student rollouts through teacher intervention.The cited approaches include Prefix OPD, Prune-OPD, ADWIN, SWITCH, teacher-blended behavior, and Relay-OPD.Source: §5 · p. 12
• Fu et al. (2026) mask problematic special tokens, while Plyusov et al. (2026) canonicalize Qwen3 EOS tokens before sampling and KL evaluation.Source: §5 · p. 12
• Termination mismatch can penalize student stopping from the outset, and probability-level alignment consistently mitigates the failure whereas decoding-set alignment alone does not.The comparison varies decoding rules, teacher distributions, and semantic and student action spaces.Source: §5 · p. 12
• Qwen3, Llama, and Gemma can prefer different surface tokens for stopping; Gemma 3 exhibits this mismatch despite identical declared EOS sets.Source: §5 · p. 12

• Termination mismatch is not a complete explanation: length rises again after the teacher-preferred token is acquired, including under semantic EOS correction, and the residual mechanism remains unknown.Source: §6 · p. 12
• The experiments cover single-turn mathematical reasoning; multi-turn or agentic use needs context-dependent termination classes to preserve distinct interaction events.Most base models lack tool-use and multi-turn capability, so comparison would require a different pairing and experimental design.Source: §6 · p. 13
• Prompt-template findings are limited to Qwen3, three templates, and two graders, so their generalization to Llama or Gemma is unestablished.The authors call for broader analysis across model families, task domains, and evaluation protocols.Source: §6 · p. 13

• Base students and post-trained teachers can use different response-ending tokens, causing token-level distillation to suppress student termination and responses to grow.Source: §7 · p. 13
• Aligning termination tokens as one probability-level stop action restores response length and clipping ratio to teacher reference levels; enlarging the decoding stop set does not.Source: §7 · p. 13
• Training templates persistently affect response length, while evaluation format substantially changes measured accuracy, and the two effects need not move together.Source: §7 · p. 13
• The authors identify termination semantics as an overlooked alignment axis and leave multi-turn and agentic extensions and other length-inflation mechanisms for future work.Source: §7 · p. 13

• YY, TY, SL, and KZ contributed equally; SL observed length inflation and proposed mechanisms and mitigations, while YY identified EOS mismatch in adjacent projects.YY verified the mismatch with SL on Qwen; SL, TY, and KZ extended the work to Gemma and Llama and examined K2-Horizon stages.Source: Author Contributions and Acknowledgments · p. 13
• The authors thank Dr. Mikhail Yurochkin and Dr. Hector Liu for discussions checking the K2-Horizon implementation.Source: Author Contributions and Acknowledgments · p. 13

A Experimental Details

• Sampled-token OPD trains Qwen3-1.7B-Base against a fixed Qwen3-4B teacher on DAPO-Math-17K, with 16 prompts and four responses per prompt per step.The main setup uses TTRL formatting with thinking disabled, trains for 200 steps, and compares five termination-action treatments.Source: A Experimental Details · p. 16
• Checkpoint evaluation samples each problem 16 times and reports benchmark averages; main training allows 7,168 response tokens, while checkpoint evaluation allows 8,192 generated tokens.The reported unweighted macro average aggregates AIME24, AIME25, and AMC23.Source: A Experimental Details · pp. 16–17

B.1 Prompt Templates

• Three rollout prompt templates are considered, while evaluation uses TTRL-style boxed-answer extraction or DAPO-style extraction following the final Answer: marker.The raw-question template is used only for training because it does not prescribe a canonical final-answer format for reliable automatic extraction.Source: §B.1 · p. 17, §B.2 · p. 18

B.2 Evaluation Graders

• 6.28% versus 24.34% Avg@16: the Qwen3-4B teacher scores differ under the original DAPO grader and the format-compatible extension.The original grader misses valid equivalent Markdown and display-math answer layouts, so some student checkpoints can appear to outperform the teacher.Source: §B.2 · p. 18, Figure 8 · p. 19

B.3 DAPO-Compatible Evaluation Grader

• The DAPO-compatible fallback supports equivalent inline and display LaTeX layouts while retaining the official final-300-character window, answer marker, normalization, and exact-match criterion.It is invoked only when the official scorer returns an incorrect result; it adds neither semantic matching nor answer searches elsewhere in the response.Source: §B.3 · p. 19

C Future Mismatch and Transient Length Drift

• After termination-token correction, persistent budget saturation disappears, but response lengths can rise early and fall later, especially for Llama and Gemma.The paper identifies this transient drift as distinct from the persistent saturation observed in vanilla Qwen3 OPD.Source: C Future Mismatch and Transient Length… · pp. 19, 21
• The update comparison assumes a fixed teacher, aligned termination semantics, common support, finite KL divergences, and unclipped on-policy updates; it need not hold exactly with stale rollouts, clipping, or length-dependent reweighting.The analysis uses a common finite generation budget and trajectory-summed losses before optimizer preconditioning.Source: C Future Mismatch and Transient Length… · pp. 19–20
• Local sampled-token OPD omits the successor value that accounts for future student–teacher mismatch along continuation trajectories.The trajectory-level reverse-KL objective includes this future mismatch; the analysis compares its expected updates with local OPD.Source: C Future Mismatch and Transient Length… · p. 20
• At a fixed prefix, local OPD applies weakly less upward pressure to the stopping logit than the trajectory-consistent update.This is a relative bias, not an absolute suppression: local OPD can still raise the stopping logit when the immediate teacher signal favors termination.Source: C Future Mismatch and Transient Length… · pp. 20–21
• The missing stopping pressure comes from continuation actions: penalizing their future mismatch increases the relative preference for stopping through softmax coupling.The successor value is zero after stopping, but continuation actions carry future mismatch penalties that local OPD omits.Source: C Future Mismatch and Transient Length… · pp. 20–21
• The authors consider omitted future mismatch a plausible contributor to transient length drift, not a guarantee that local OPD increases response length.The proposed account is especially relevant when termination probabilities are low and the teacher itself assigns little mass to stopping at the current prefix.Source: C Future Mismatch and Transient Length… · p. 21, §D.1 · p. 23

D.1 Termination Mismatch and Residual Length Dynamics Across Model Families

• In Gemma 3 and Llama 3.2, vanilla OPD mismatches student and teacher surface EOS preferences, while semantic correction matches total termination probability.The student favors <eos> in Gemma and <|end_of_text|> in Llama; their teachers prefer <end_of_turn> and <|eot_id|>, respectively.Source: §D.1 · p. 22
• K2-Horizon initially transfers the teacher-preferred termination token because students assign it non-negligible probability, but both termination probabilities later decline again.That later decline occurs after the teacher-preferred form has been learned, so surface EOS mismatch alone does not explain it.Source: §D.1 · p. 22

D.2 Semantic EOS Correction at Later Training Stages

• Later-stage checkpoints do not show the severe termination collapse seen in base-to-post-trained settings, so semantic correction is not needed to recover the teacher-preferred termination form.When termination is already sufficiently aligned, aggregation is largely inactive rather than forcing a different surface convention.Source: §D.2 · pp. 23–24
• For SFT and midtraining K2-Horizon initializations, semantic correction leaves response-length, clipping, termination-probability, and downstream-performance trajectories broadly similar to vanilla OPD.The checkpoints already place substantial probability on the teacher-preferred <|ifm|im_end|> token, and no systematic accuracy penalty is observed.Source: §D.2 · pp. 23–24, Figure 11 · p. 24

E Influence of Training and Evaluation Templates

• The study uses three training templates and evaluates checkpoints with both TTRL and DAPO formats, holding other experimental settings fixed.Semantic EOS correction is used throughout these experiments.Source: E Influence of Training and Evaluation… · p. 24, Figure 13 · p. 26
• Evaluation under TTRL generally yields higher measured accuracy than evaluation under DAPO across all three training settings, without a consistent advantage from matching training and evaluation formats.Because the evaluation formats differ in both prompting and grading, the authors interpret this as an overall protocol effect rather than a prompt-only effect.Source: E Influence of Training and Evaluation… · pp. 24–25
• DAPO-trained models generally produce shorter responses than TTRL- or raw-question-trained models under both evaluation formats.The authors interpret this persistence across evaluation prompts as evidence that the training prompt induces a lasting generation-behavior change.Source: E Influence of Training and Evaluation… · p. 25
• Under TTRL evaluation, DAPO-trained models achieve comparable performance while producing substantially shorter responses.Response length and measured task performance therefore need not move together.Source: E Influence of Training and Evaluation… · p. 25
• The authors identify systematic characterization of the training-prompt and evaluation-format interaction as future work.Their results suggest the training prompt shapes generation behavior while evaluation format affects measured performance.Source: E Influence of Training and Evaluation… · p. 25

F Rollout Examples from Figure 1b

• The rollout examples reach the full generation budget; the section presents reasoning through the first correct answer and selected repetitive continuation segments.Repeated middle segments are omitted for readability.Source: F Rollout Examples from Figure 1b · p. 26

F.1 Example 1: Answer Repetition

• 1,094 tokens yielded the correct AMC23 answer before 7,098 redundant tokens (86.6% of the response); generation then hit the budget without EOS.The final-answer block repeats 704 times; the reasoning identifies a four-power cycle in units digits and gives 5 as the answer.Source: §F.1 · pp. 26–27

F.2 Example 2: Self Correction Loop

• 759 tokens yielded the correct AIME24 answer before 7,433 redundant tokens (90.7% of the response); generation hit the budget without EOS.A “Let's try” block repeats 371 times, after an initial incorrect choice of x = 5 and y = 5; the derivation gives xy = 25.Source: §F.2 · pp. 28–29

F.3 Example 3: Token Repetition

• 462 tokens yielded the correct AMC23 answer before 7,730 redundant tokens (94.4% of the response); generation hit the budget without EOS.The check-mark token repeats 3,836 times after the answer that they meet 27 miles from City A.Source: §F.3 · pp. 29–30