PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Vocabulary Language Model for Cross-Tokenizer Distillation

Kedi Chen, Chen Lin, Yutao Sun, Wei Zhang

arXiv:2609.33816v1cs.AI

TL;DR

Cross-tokenizer OPD faces mismatches in token boundaries and logit dimensions, while existing alignment approaches can lose distribution information. DVLM combines student-vocabulary logits with PTS and HPA to align teacher supervision with student inputs; experiments report student improvements across six reasoning benchmarks.

  • Problem

    Cross-tokenizer OPD must align teacher and student distributions despite differences in input token boundaries and output-logit spaces.

    Source: §1 · pp. 1–2

  • Method

    DVLM uses a student-vocabulary projection head with PTS and HPA to produce teacher logits aligned with student tokens and logit dimensions during OPD.

    Source: §1 · pp. 2–3

  • Results

    DVLM-based OPD consistently improves student models across all six reasoning benchmarks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The approach provides distribution-aligned teacher supervision matching student input tokenization and output logits during cross-tokenizer OPD.

    Source: §1 · p. 2

    Limited GPU resources and the need to account for teacher and student baseline performance prevented experiments with larger models.

    Source: §A.7 · p. 22

Abstract

from arXiv · show

On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grained token information or disrupting the native-token paradigm, while for the latter, strategies such as ranking, padding, or key-token selection retain only shared logit dimensions, resulting in much distribution loss. In this paper, we propose Dual-Vocabulary Language Model (DVLM), which replaces the teacher's LM head with a new student-vocabulary projection head and obtains full-dimensional student logits (for #2). To support student tokens (for #1), it takes a Parallel-Tokenized Sequence (PTS) as input, which concatenates the original teacher-tokenized sequence and a re-tokenized sequence formed by independently converting each student token into a teacher-token group. To avoid inference inconsistency with the original teacher tokens, the Hybrid-Prefix Attention (HPA) further restricts re-tokenized groups to their corresponding teacher prefix and uses its last state as the aggregation of the original student-token representation for projection into the student vocabulary space. Similarly, via the combined use of PTS and HPA, the DVLM teacher can provide distribution-aligned supervision with the student's input-tokenization and output-logit during OPD. Experimental results demonstrate that our DVLM teacher has a similar converged loss as the original teacher model and enables student models to improve performance across six reasoning tasks.

1 INTRODUCTION

Cross-tokenizer OPD must align teacher and student distributions despite differences in input token boundaries and output vocabulary dimensions. DVLM addresses both mismatches, and experiments report teacher-level performance and student gains across reasoning benchmarks.

1 INTRODUCTION

  • On-policy distillation uses student-generated responses to mitigate mismatch between teacher supervision and student behavior, making cross-tokenizer alignment relevant to OPD.Teacher and student models often use different tokenizers.Source: §1 · p. 1
  • Cross-tokenizer distillation must align distributions across different teacher–student token boundaries and output-logit spaces.The differing vocabularies give the models logits with different dimensionalities.Source: §1 · p. 1
  • Prior input-alignment methods match shared text spans or convert tokens to bytes, while output-alignment methods select shared logit dimensions and lose distribution information.Span aggregation loses fine-grained token information, and byte splitting departs from the native-token paradigm.Source: §1 · pp. 1–2
  • DVLM replaces the teacher’s LM head with a student-vocabulary projection head, while PTS and HPA support student-token inputs and one-to-one teacher–student token mapping.The new head produces full-dimensional student logits; HPA restricts re-tokenized groups to their corresponding teacher prefix.Source: §1 · pp. 2–3
  • DVLM teachers reach the original teachers’ performance level on four language-understanding benchmarks, while distilled students outperform conventional and cross-tokenizer methods on six math and code reasoning benchmarks.The authors also report similar converged loss to the original teacher and student improvements across diverse reasoning tasks.Source: §1 · pp. 2–3

2 RELATED WORK

Related work frames OPD as learning from teacher feedback on student-generated trajectories. In cross-tokenizer settings, prior methods align input distributions through spans or bytes and output distributions through selected shared logit dimensions.

On-Policy Distillation

  • OPD has students generate response trajectories and learn from teacher feedback on logit states, unlike off-policy distillation from fixed teacher-generated or ground-truth text.The paper focuses on OPD in cross-tokenizer settings.Source: §2 · p. 3

Cross-tokenizer Distillation

  • Input-alignment approaches measure token distances, aggregate distributions over shared text spans (Sun et al., 2026), or map tokens into a shared byte space (Phan et al., 2026; Wang et al., 2026a).The paper characterizes these approaches as incurring distribution loss and disrupting the native-token paradigm.Source: §2 · p. 3
  • Output-alignment methods use ranking (Cui et al., 2025; Sreenivas et al., 2026), padding, or key-token selection (Nguyen et al., 2026) to retain shared logit dimensions.The paper also describes dividing vocabularies into matched and unmatched tokens (Patiño et al., 2025).Source: §2 · p. 3

3 METHOD

DVLM adapts a teacher for cross-tokenizer OPD with student-compatible inputs and a student-vocabulary output head. PTS and HPA preserve teacher-prefix context while producing student-aligned supervision for student-generated tokens.

3 METHOD

  • DVLM replaces the teacher’s original LM head with a student-vocabulary projection head, then uses the trained DVLM teacher to guide the student through OPD.The method has two steps: training the new head on the teacher backbone, then using DVLM as the OPD teacher.Source: §3 · p. 3, §3.1 · p. 3

3.1 DUAL-VOCABULARY LANGUAGE MODEL

  • PTS concatenates the original teacher-tokenized sequence with groups formed by independently decoding each student token and re-tokenizing it into teacher tokens.The original teacher sequence is prepended to preserve native-tokenization prefix context.Source: §3.1 · pp. 3–4
  • HPA keeps original teacher tokens under causal attention and restricts each re-tokenized group to prior aligned teacher context and its group-local preceding tokens.The mask also permits attention to earlier student-token groups within the same alignment group.Source: §3.1 · pp. 4–5
  • For each student token, DVLM uses the hidden state at its group’s last re-tokenized token and projects it into the student vocabulary to predict the next student token.The new head is initialized from the teacher head; shared-token weights are inherited, while unmatched student-token weights average their re-tokenized teacher-token weights.Source: §3.1 · p. 5
  • The teacher backbone is frozen during DVLM training, and only lightweight LM heads are trained using the standard cross-entropy objective.The head initialization transfers shared teacher-token weights directly and initializes unmatched student tokens by averaging their re-tokenized teacher-token weights.Source: §3.1 · p. 5

3.2 DVLM-BASED OPD

  • In DVLM-based OPD, the student samples a response, the prompt and response are converted to PTS, and the frozen DVLM teacher supplies logits from each response token’s group-ending state.The resulting teacher distributions have shape LS × |VS|, aligned with the student’s input tokens and output-logit dimensions, and the student is optimized with forward KL divergence.Source: §3.2 · p. 5

4 EXPERIMENTS

Across the experiments, DVLM improves student performance on reasoning benchmarks while its teacher reaches approximately the original teacher’s loss and overall understanding-task performance. Ablations and difficulty tests assess the contribution of its components and the effect of sample difficulty.

4 EXPERIMENTS

  • The experiments distinguish “DVLM teacher” for the architecture from “method DVLM” for the overall method.The evaluation covers training configurations and protocols across both stages.Source: §4 · p. 6

4.1.1 TRAINING

  • The experiments use Qwen3-4B as teacher and Llama-3.2-1B-Instruct and OLMo-2-1B-Instruct as students, with vocabulary overlaps of 83.8% and 59.5%, respectively.The new LM heads are trained on 0.22B tokens, and the DVLM-based OPD training set contains around 8K samples.Source: §4.1.1 · p. 6

4.1.2 EVALUATION

  • Stage I evaluates the DVLM teacher on four language-understanding benchmarks; Stage II evaluates students on three math and three code benchmarks against SFT, RL, and cross-tokenizer distillation baselines.The cross-tokenizer baselines include DSKD, ALM, GOLD, and BPM.Source: §4.1.2 · p. 6

4.2 MAIN RESULTS

  • DVLM improves the two students by 3.10 and 2.86 points across six reasoning benchmarks, and beats the second-best cross-tokenizer OPD method by 2.16 and 1.89 points, respectively.The results attribute the gains to supervision dimensionally aligned with the student’s original distribution, with particular effectiveness for large vocabulary gaps.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 ABLATION STUDY

  • Removing any DVLM component reduces performance for both students; PTS construction is particularly important, while HPA limits information leakage.The new LM head and extracting the last-position logit of each re-tokenized group also matter for student-output logits.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 IN-DOMAIN EVALUATION

  • DVLM achieves the best distillation performance for both student models on Countdown and LeetCodeDataset, with margins of 2.31 and 2.68 points on Countdown and 1.31 and 2.20 on LeetCodeDataset.The authors attribute OLMo’s lower scores primarily to its larger vocabulary discrepancy with the teacher.Source: §4.4 · p. 8, Figure 3 · p. 9

4.5 EVALUATION OF THE DVLM ARCHITECTURE

  • The original teacher’s loss stays approximately 1.9, while the Llama and OLMo heads reach 1.81 and 1.79; both DVLM teachers overall match teacher text-understanding performance.The OLMo-head model scores 1 point lower on MMLU(Algebra), which the authors attribute to the subset’s small sample size.Source: §4.5 · pp. 8–9, Figure 4 · p. 9

4.6 DIFFICULTY SELECTION FOR CROSS-TOKENIZER DISTILLATION

  • Increasing sample difficulty makes effective distillation less likely and may degrade the original backbone, so difficulty should match student capability.The study increases difficulty across three datasets while keeping sample count and the math-code ratio fixed.Source: §4.6 · p. 9

5 CONCLUSION

DVLM provides supervision aligned with the student’s input tokenization and output logits during on-policy distillation. The paper reports consistent student improvements across six reasoning benchmarks.

5 CONCLUSION

  • DVLM-based OPD provides distribution-aligned supervision using the student’s input tokenization and output-logit dimensions.The conclusion identifies the Dual-Vocabulary Language Model as the approach.Source: §5 · p. 9
  • The method consistently improves student models across all six reasoning benchmarks.Source: §5 · p. 9
  • Figure 5 examines sample-difficulty scaling on four benchmarks with Llama-3.2-1B.Source: Figure 5 · p. 10

AI USE STATEMENT

The authors report using generative AI for coding assistance, manuscript editing, and example images. They state that AI-assisted work was reviewed and that they take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools assisted with coding, manuscript polishing, and generating example images for figures.Source: AI USE STATEMENT · p. 10
  • The authors say they reviewed AI-assisted code, manuscript revisions, and generated content before inclusion.Source: AI USE STATEMENT · p. 10
  • The authors take responsibility for the final work, including AI-assisted text, claims, code, and artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The work uses open-source academic models and datasets, with publicly available training and evaluation frameworks. The authors report identifying no concerns in the listed ethics areas.

ETHICS STATEMENT

  • All models and datasets used in the work come from open-source resources released by the academic community.Source: ETHICS STATEMENT · p. 10
  • Training and LLM evaluation frameworks are publicly available on GitHub, and the authors adapted their code from those frameworks.Source: ETHICS STATEMENT · p. 10
  • The authors report identifying no concerns related to misuse, conflicts, fairness, privacy, security, legal compliance, or research integrity.Source: ETHICS STATEMENT · p. 10

A.1 MORE ABOUT CROSS-TOKENIZER DISTILLATION

Cross-tokenizer distillation must address mismatches in input tokenization and output logits, while some approaches instead align representations. This work focuses on distribution-aligned supervision using the student’s tokenization and logits.

A.1 MORE ABOUT CROSS-TOKENIZER DISTILLATION

A.2 TOKEN ALIGNMENT GROUP

Token alignment groups pair student and teacher token sequences by matching their decoded text spans. The construction procedure greedily extends decoded text until the two tokenizers agree.

A.2 TOKEN ALIGNMENT GROUP

  • The algorithm incrementally decodes token prefixes, greedily extends the shorter text buffer until both buffers match, and records the corresponding token indices as an alignment group.Any remaining token indices are retained as a residual group for complete sequence coverage.Source: §A.2 · p. 17
  • The procedure retains unmatched residual indices as a final group rather than dropping them.This ensures complete sequence coverage.Source: §A.2 · p. 17
  • A student token can align with multiple teacher tokens when their decoded spans match, as with “distillation” represented by “distill” and “ation.”The example also shows one-to-one matches for “is” and a one-to-many match for “challenging.”Source: §A.2 · p. 17

A.3 MORE ABOUT OUR METHODS

The method preserves aligned logical positions in the concatenated sequence, constructs token alignment groups through incremental decoding, and filters tokens that cannot be independently decoded and re-tokenized.

A.3 MORE ABOUT OUR METHODS

  • Position IDs encode logical text positions, placing re-tokenized tokens according to the teacher prefix, earlier expanded tokens in the group, and their offset.This lets the tokens retain aligned logical positions under the HPA attention mask despite physical concatenation.Source: §A.3 · p. 17
  • Samples containing student tokens that cannot be independently decoded and correctly re-tokenized are discarded during preprocessing.This ensures retained student tokens can be converted into valid teacher-token sequences for PTS construction.Source: §A.3 · pp. 17–18
  • Incremental decoding and greedy buffer extension construct alignment groups by matching decoded student and teacher text.The algorithm records token indices when the buffers match and retains any residual indices.Source: §A.3 · p. 18

A.4 SUPPLEMENTARY TRAINING SETTINGS

The supplementary sections describe training and evaluation settings, additional comparisons and ablations, and a scope limitation: experiments with larger models could not be expanded.

A.4 SUPPLEMENTARY TRAINING SETTINGS

  • Training uses Qwen3-4B with Llama-3.2-1B-Instruct and OLMo-2-1B-Instruct; the new heads use mixed synthetic, math, and code data, while evaluations specify benchmarks, baselines, and metrics.The cross-tokenizer baselines are reproduced under the paper’s own training settings, and understanding tasks use single-token prediction.Source: §A.4 · pp. 18–19, §A.5 · p. 20

A.6.1 TRAINABLE PARAMETERS, TRAINING RUNS, AND TRAINING VARIANCE

  • About 8%: Stage I adds this same-GPU-hour cost for both teacher–student pairs, while the two LM heads increase trainable parameters by 26% and 18%, respectively.Stage II results are averaged across three random seeds; reported variance is limited to trainable baselines.Source: §A.6.1 · p. 20, Table 7 · p. 21

A.6.3 THE EXTRA EVALUATION OF DVLM ARCHITECTURE

  • Both DVLM heads achieve language-modeling performance comparable to the original teacher by BPB, and show only a minor performance drop on MATH-500 and HumanEval+.The language-modeling evaluation uses 500 held-out passages; reasoning evaluation uses a custom autoregressive generation pipeline.Source: Table 8 · p. 21, §A.6.3 · p. 21, Table 9 · p. 22

A.6.2 MORE ABOUT ABLATION STUDY

  • Randomly initializing the new LM head prevents it from converging to the original teacher’s loss level.The ablations also compare removing HPA, the teacher prefix, re-tokenization, the LM head, or logit extraction.Source: §A.6.2 · pp. 20–21

A.6.4 TRAINING TIME COMPARISON WITH GOLD

  • 0.35 and 0.45 days: GOLD takes substantially longer than DVLM for the Llama-3.2-1B and OLMo-2-1B backbones, respectively.The authors attribute the difference mainly to GOLD processing matched and unmatched logits separately.Source: §A.6.4 · p. 21

A.6.5 COMPARISON WITH SAME-TOKENIZER DISTILLATION

  • Five of six benchmarks: cross-tokenizer distillation outperforms same-tokenizer distillation for OLMo-2-1B-Instruct, with LiveCodeBench v5 the exception.The authors suggest the result may reflect Qwen3-4B being substantially stronger than OLMo-2-7B-Instruct.Source: Table 10 · p. 22, §A.6.5 · p. 22

A.7 LIMITATIONS

  • GPU resources and the need to account for teacher and student baseline performance prevented more extensive experiments with larger models.This bounds the scale of the reported experimental coverage.Source: §A.7 · p. 22

A.6.6 COMPARISON WITH TOKENIZER TRANSFER METHOD

  • All four benchmarks: DVLM consistently outperforms TokAlign in the Llama-3.2-1B-Instruct comparison.The authors attribute TokAlign’s weaker performance partly to replacing the teacher’s input tokenization scheme, which creates pretraining–adaptation mismatch.Source: §A.6.6 · p. 22, Table 11 · p. 23
Loading 2609.33816v1…