PaperlayerView exact revision on arXiv

Source-linked AI summary

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, Bing Qin

arXiv:2608.29662v1cs.CL

TL;DR

Cross-tokenizer distillation must handle vocabulary and sequence misalignment, while approximate mapping can add supervision noise. ACTD combines vocabulary and sequence alignment with anchor loss and residual regularization, and extends the framework to multiple teachers. Across five reasoning benchmarks, ACTD and Multi-ACTD outperform cross-tokenizer baselines, with gains of up to 4.5 and 4.9 AVG@8 points, respectively.

  • Problem

    Cross-tokenizer distillation faces vocabulary and sequence misalignment, and approximate vocabulary alignment introduces additional noise into supervision.

  • Method

    ACTD aligns vocabularies and sequences, focuses distillation on teacher high-probability tokens with an anchor loss, and applies residual regularization; Multi-ACTD extends it to multiple teachers.

  • Results

    Across five reasoning benchmarks, ACTD improves AVG@8 by up to 4.5 absolute points and Multi-ACTD improves AVG@8 by 4.9 points over baselines.

  • Takeaways & Limitations

    ACTD transfers logits-level knowledge across heterogeneous tokenizer and teacher settings, while Multi-ACTD leverages complementary supervision from multiple teachers.

  • Takeaways & Limitations

    ACTD requires teacher predictive distributions during data construction, limiting applicability to models with accessible outputs such as open-source models.

Abstract

from arXiv · show

Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.

1 Introduction

Cross-tokenizer distillation broadens teacher–student compatibility but faces vocabulary and sequence misalignment plus alignment-induced noise. ACTD addresses these issues with alignment, anchor-focused supervision, residual regularization, and a multi-teacher extension.

  • Cross-tokenizer distillation enables knowledge transfer between models with different tokenizers and broader teacher-model flexibility.
  • Vocabulary mismatch, sequence misalignment, and lossy mapping noise make cross-tokenizer supervision difficult.
  • 4.5 absolute points: ACTD improves AVG@8 by up to 4.5 points over cross-tokenizer distillation baselines across five reasoning benchmarks.
  • ACTD aligns teacher and student vocabularies using first-subtoken mapping and sequences using nearest monotonic character-boundary matching.
  • ACTD focuses supervision on teacher high-probability tokens through an anchor loss and uses residual regularization to suppress alignment noise.
  • Multi-ACTD extends ACTD to multiple teachers and consistently outperforms multi-teacher variants of existing cross-tokenizer methods.

2 Preliminaries

Knowledge distillation matches teacher and student next-token distributions, while cross-tokenizer distillation requires transformations because their vocabularies and sequence segmentations differ.

  • At each decoding position, teacher and student models produce next-token distributions that knowledge distillation trains the student to match.
  • Homogeneous distillation: When teacher and student share vocabulary V, their distributions occupy a common token space and can be matched directly.
  • Homogeneous distillation: Forward KL divergence is a common homogeneous-distillation objective, while reverse KL and Jensen–Shannon divergence offer alternative matching behaviors.
  • Cross-tokenizer distillation: Different teacher and student vocabularies require a cross-vocabulary transformation Γ from teacher distributions into the student vocabulary space.
  • Cross-tokenizer distillation: Sequence segmentation differences mean cross-tokenizer distillation typically requires both sequence alignment and vocabulary alignment.

3 Method

ACTD addresses vocabulary and sequence mismatches between heterogeneous tokenizers, then reduces alignment noise by concentrating supervision on teacher high-probability tokens and regularizing residual student regions.

  • Vocabulary and sequence alignment: ACTD uses first-subtoken vocabulary alignment and nearest monotonic boundary alignment to connect teacher and student token spaces and decoding positions.Unshared teacher tokens are decoded and re-tokenized, with the first student subtoken selected as the match; sequence positions are aligned using character-level boundaries.
  • Cross-tokenizer noise analysis: Vocabulary mapping introduces noise because reverse-mapped teacher probabilities differ from the original distribution.The analysis maps teacher probabilities into the student vocabulary and back, using discrepancies as evidence of imperfect alignment.
  • Cross-tokenizer noise analysis: Teacher long-tail probability accumulates more heavily on residual than anchor tokens, leaving residual regions with more low-confidence mapped signal.Anchor tokens come from mapped teacher top-k tokens, while residual tokens are the remaining student vocabulary.
  • Anchor-based loss with residual regularization: The anchor loss matches the teacher’s renormalized top-k distribution with student probabilities on mapped anchor tokens.This transfers fine-grained teacher probability signals to the student-side anchor subset.
  • Anchor-based loss with residual regularization: Residual regularization sets residual target probabilities to zero and penalizes the student’s aggregate residual probability mass using an L1 distance.The design targets low-probability regions rather than matching the complete distribution, helping mitigate mode averaging.
  • Multi-teacher distillation: Multi-ACTD independently computes ACTD losses for multiple teachers and averages their teacher-specific losses during training.Each teacher supplies cached top-k logits for the same sample and target position.

4 Experiments

The experiments evaluate ACTD and its multi-teacher extension on mathematical reasoning benchmarks using heterogeneous teacher models, standardized inference, and comparisons with single- and multi-teacher baselines.

  • Experimental setup: The study uses three primary teachers, Qwen3-1.7B-Base as student, and GLM-4-9B-Chat for additional cross-family generalization evaluation.The primary teachers are DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-0528-Qwen3-8B.
  • Experimental setup: Training uses 30,000 OpenThoughts-114k math examples, ground-truth solution targets, and cached top-k logits with k = 128 for each teacher.The setup follows an off-policy distillation procedure using the problem field as input.
  • Baselines and comparisons: ACTD consistently outperforms cross-tokenizer distillation baselines under the same teacher settings.The comparisons include ULD, DSKD, ALM, and other specified student baselines.
  • Experimental setup: Evaluation covers AIME24, AIME25, AMC23, MATH500, and Minerva, reporting accuracy as the main metric.The benchmarks span competition mathematics, a MATH subset, and college-level quantitative reasoning.
  • Evaluation procedure: Inference uses max_gen_tokens = 8192, temperature = 1.0, and top_p = 0.8, with boxed answers checked using math_verify2.Accuracy is computed as the proportion of generated responses judged mathematically correct.

5 Results

ACTD consistently outperforms cross-tokenizer baselines across teacher, student, and model-family settings, while ablations show benefits from prefix-truncated targets, residual regularization, and focused top-k supervision.

  • ACTD consistently outperforms cross-tokenizer distillation baselines under the same teacher settings.
  • 18.4 AVG@8 is achieved by ACTD with Llama-3.2-3B-Instruct, improving 2.6 points over DSKD’s 15.8.ACTD is best on four benchmarks and ties for best on AIME25.
  • AVG@8 rises from 22.7 to 26.0 and PASS@8 from 44.9 to 47.2 with ACTD under the GLM-4-9B-Chat setting.
  • Removing prefix truncation decreases AVG@8 by 0.7–1.6 points across teacher settings.The drop is largest for DeepSeek-R1-Distill-Qwen3-8B.
  • λres = 1.0 achieves the best balance, while lower or higher values reduce AVG@8 or PASS@8.At λres = 0, AVG@8 is 26.1 and PASS@8 is 49.5; at λres = 2.0, they are 26.6 and 47.6.
  • k = 128 yields the best average AVG@8 of 27.8, whereas increasing supervision to larger top-k sets lowers it to 26.6.The results attribute the decline to noisy low-probability long-tail tokens.
  • Residual regularization produces more concentrated probability distributions and lower entropy across inference steps.It assigns more probability mass to high-probability regions and reduces over-smoothed predictions.

6 Related Work

Prior work transfers teacher supervision across tokenizers through vocabulary or distribution alignment, while multi-teacher distillation exploits complementary supervision from multiple teachers.

  • Cross-tokenizer distillation relaxes the shared-vocabulary assumption by transferring distribution-level supervision across heterogeneous token spaces.Methods include optimal transport, cross-model attention, approximate likelihood matching, and span-level preference projection.
  • Multi-teacher distillation leverages complementary supervision from multiple teachers to improve student learning.Existing approaches use generated responses, Chain-of-Thought rationales, consistency, distribution fusion, data selection, or reasoning-behavior distillation.

7 Conclusion

ACTD aligns heterogeneous teacher–student vocabularies and sequences, then focuses distillation on high-probability predictions while regularizing residual probability mass. Multi-ACTD extends this approach to multiple heterogeneous teachers and improves performance across reasoning benchmarks.

  • ACTD uses first-subtoken mapping and nearest-position matching based on character-level boundaries to align heterogeneous teacher–student vocabularies and sequences.
  • An anchor loss distills aligned high-probability predictions while residual regularization suppresses probability mass outside the anchor subset.The combination is intended to mitigate alignment noise and mode averaging.
  • Multi-ACTD integrates logits-level supervision from multiple heterogeneous teachers and further improves performance by leveraging complementary teacher supervision.The reported results show robustness across teacher settings and tokenizer heterogeneity levels.

Limitations

The reported scope is constrained by teacher-output access and computational resources, particularly for multi-teacher combinations.

  • ACTD requires teacher predictive distributions during data construction, limiting applicability to models with accessible outputs such as open-source models.
  • The experiments are constrained by computational resources, especially for multi-teacher combinations.The paper identifies broader heterogeneous settings and more efficient distributional supervision as future directions.

B Cross-tokenizer Noise Analysis

The analysis measures cross-tokenizer projection noise in student vocabulary space by separating mapped teacher probability mass into anchor and residual regions. It treats excess mass outside retained high-confidence teacher support as potentially unintended supervision.

  • Noise decomposition: Non-top-k teacher probability mass accumulated on anchor tokens is anchor noise, whereas mass accumulated on residual tokens is residual noise.The analysis reports these quantities across valid target positions.
  • Student-side mapping noise: Teacher distributions are projected from the teacher vocabulary onto student tokens using a teacher-to-student tokenizer alignment map.All subsequent noise quantities are computed over student tokens rather than teacher tokens.
  • Anchor and residual regions: Teacher top-k tokens are mapped to form the student-side anchor set, while the remaining student vocabulary forms the residual set.The two regions partition the student-side vocabulary for noise analysis.
  • Noise interpretation: Projection noise is defined as excess probability mass outside retained high-confidence teacher support after projection into the student vocabulary.This mass still contributes to the student-side distillation target and may introduce unintended supervision.

C Data Construction

The paper constructs prefix-truncated math instruction data for ACTD by formatting prompts with the student tokenizer, concatenating reference solutions, and preserving final answers while truncating reasoning prefixes. Baselines and ACTD use closely matched full-parameter training configurations.

  • Data formatting: Each math example contains an id, instruction, and reference output, which are formatted into a student-tokenizer chat prompt before concatenating the reference solution.A fixed instruction requests step-by-step reasoning and places the final answer within \boxed{}; the assistant-completion start position is recorded.
  • Prefix truncation: Reasoning prefixes are truncated while answer suffixes are preserved, using </think>, the final \boxed{} expression, or the last sentence to locate the answer boundary.Figure 7 illustrates truncation before a coordinate calculation, followed by a forced </think> marker and preserved final-answer suffix.
  • Baseline configuration: Baseline comparisons keep the main optimization, memory, and distillation settings identical where applicable.The shared setup uses full-parameter fine-tuning with FSDP2, two GPUs, bfloat16 precision, gradient checkpointing, global batch size 36, and 300 training steps.
  • ACTD configuration: ACTD is implemented by modifying DistillKit3 and fine-tuning all student-model parameters rather than using parameter-efficient adaptation.Its general setup uses two GPUs, bfloat16 precision, effective batch size 36, 300 steps, learning rate 1.0 × 10−5, cosine scheduling, and 30 warmup steps.

E Experimental Results on Other Tasks

The paper tests ACTD beyond mathematical reasoning on cross-tokenizer code generation. Distilling DeepSeek-R1-Distill-Llama-8B into Qwen3-1.7B-Base yields higher HumanEval+ Pass@1 than both the strongest baseline and the original student.

  • Evaluation setup: Code-generation experiments train on OpenCodeInstruct and evaluate distilled models on HumanEval+ using Pass@1.The setting distills DeepSeek-R1-Distill-Llama-8B into Qwen3-1.7B-Base, representing different model families and tokenizers.
  • Results: 23.2 Pass@1 is achieved by ACTD, outperforming ULD by 1.9 absolute points.ULD is identified as the strongest cross-tokenizer baseline in this comparison.
  • Results: ACTD improves on the original Qwen3-1.7B-Base student model by 2.5 points.The reported result is presented as evidence that ACTD transfers code-generation capabilities across different tokenizers.
Loading 2608.29662v1…