PaperlayerView exact revision on arXiv

Source-linked AI summary

BitNet Distillation

Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, Li Dong, Yan Xia, Furu Wei

arXiv:2510.13998v1cs.LGcs.CL

TL;DR

Deploying large LLMs on resource-constrained devices is challenging because memory and computational demands are high. BitDistill fine-tunes existing full-precision models into 1.58-bit models for downstream tasks, combining SubLN, continued pre-training, and attention distillation. The paper reports performance comparable to full-precision counterparts alongside substantial memory and CPU inference gains.

  • Problem

    Deploying LLMs for downstream applications is challenging on resource-constrained devices, while direct 1.58-bit quantization-aware training can be unstable, degrade performance, and scale poorly.

    Source: §1 · p. 2

  • Method

    BitDistill fine-tunes existing full-precision LLMs to 1.58-bit weights using SubLN refinement, continued pre-training, and MiniLM-based multi-head attention distillation.

    Source: §1 · p. 2

  • Results

    Up to 10× memory savings and 2.65× faster inference on CPUs accompany downstream performance comparable to full-precision counterparts across model scales.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    BitDistill offers a route to resource-constrained deployment and insights into scalable low-bit quantization strategies.

    Source: §6 · p. 10

Abstract

from arXiv · show

In this paper, we present BitNet Distillation (BitDistill), a lightweight pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into 1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream tasks, achieving strong task-specific performance with minimal computational cost. Specifically, BitDistill incorporates three key techniques: the SubLN module, as introduced in BitNet; multi-head attention distillation, based on MiniLM; and continual pre-training, which serves as a crucial warm-up step to mitigate the scalability issue of the performance gap between finetuned full-precision and 1.58-bit LLMs on specific tasks. Experimental results show that BitDistill achieves performance comparable to the full-precision counterpart models across model size, while enabling up to 10x memory savings and 2.65x faster inference on CPUs. Code is available at https://github.com/microsoft/BitNet.

1 Introduction

Deploying large language models on resource-constrained devices is difficult because their memory and computational demands grow with model size. BitDistill fine-tunes existing LLMs to 1.58-bit precision and reports performance comparable to full-precision models alongside efficiency gains.

1 Introduction

  • Large model size makes memory consumption and computational overhead prohibitive on resource-constrained devices such as smartphones.The introduction frames downstream applications including recommendation, classification, and retrieval as contexts for LLM deployment.Source: §1 · p. 2
  • Directly applying 1.58-bit quantization-aware training is often unstable, degrades performance, and scales poorly, with the gap to full precision growing from 13.9 to 15.3 as model size rises from 0.6B to 4B.Training BitNet from scratch can also impose substantial computational and energy overhead.Source: §1 · p. 2
  • BitDistill combines SubLN refinement, continued pre-training, and MiniLM-based multi-head attention distillation to stabilize optimization, mitigate scale-related gaps, and recover full-precision accuracy.The paper presents the framework as a tailored approach to fine-tuning existing LLMs for specific downstream tasks at 1.58-bit precision.Source: §1 · p. 2
  • Up to 10× memory savings and 2.65× faster inference on CPUs accompany downstream performance comparable to full-precision counterparts across benchmarks and model scales.The reported results are intended to support deployment on resource-constrained hardware.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe ternary weight quantization, 8-bit activation quantization, and the gradient approximation used to train quantized LLMs.

2 Preliminaries

  • Weights are mapped per tensor to ternary values {-1, 0, 1} using absmean quantization, with Δ defined as mean(|W|).The rounding-and-clipping operation is specified in Eq. 2.Source: §2 · p. 2
  • LLM inputs use 8-bit activation quantization with per-token absmax and absmean functions.The quantization is formulated in Eq. 3.Source: §2 · p. 3
  • Because RoundClip is non-differentiable, the method uses a Straight-Through Estimator to approximate gradients during backpropagation.The cited approach follows prior 1.58-bit BitNet work.Source: §2 · p. 3

3 BitDistill: Finetuning LLMs into 1.58-bit BitNet For Downstream Tasks

BitDistill adapts pretrained LLMs to 1.58-bit for specific downstream tasks through modeling refinement, continued pre-training, and distillation-based fine-tuning.

3.1 Stage-1: Modeling Refinement

  • SubLN layers before the attention and feed-forward output projections stabilize hidden-state variance entering quantized projections.Low-bit models can have excessive activation variance, which is linked to optimization instability and degraded convergence.Source: §3.1 · p. 3

3.2 Stage-2: Continue Pre-Training

  • Directly fine-tuning 1.58-bit models can leave a widening performance gap versus full-precision models as model size increases.The paper attributes this to limited downstream training tokens being insufficient to adapt full-precision weights to the constrained representation.Source: §3.2 · p. 3
  • Continued pre-training on a small corpus precedes downstream fine-tuning to adapt the model before task training.The passage describes this as a two-stage pipeline after modeling refinement.Source: §3.2 · p. 4

3.3 Stage-3: Distillation-based Fine-tuning

  • The final fine-tuning phase uses a full-precision model as teacher and its 1.58-bit counterpart as student for two kinds of knowledge distillation.The loss also includes downstream cross-entropy, with coefficients controlling the trade-off between distillation and model fitting.Source: §3.3 · pp. 4–5
  • Logits distillation uses downstream examples to align the student and teacher output distributions, with temperature controlling their softening.The cited passage identifies the divergence as Kullback–Leibler divergence.Source: §3.3 · p. 4
  • Attention distillation matches fine-grained structural dependencies in the teacher’s attention patterns through query, key, and value relations.The method recommends distilling from a single layer rather than all layers, reporting that this can give the student greater optimization flexibility and superior downstream performance.Source: §3.3 · p. 4

4 Experiments

Across tasks and model sizes, BitDistill delivers performance comparable to full-precision fine-tuning with better memory and CPU inference efficiency. The experiments examine its setup, component contributions, and choices affecting distillation.

4.1 Experimental Setup

  • BitDistill is evaluated on three GLUE classification datasets and CNNDM summarization, using Qwen3 models at 0.6B, 1.7B, and 4B scales.The comparisons include directly fine-tuned FP16 models and directly converted, fine-tuned BitNet models.Source: §4.1 · p. 5

4.2 Main Results

  • Across tasks and model sizes, 1.58-bit BitDistill performs comparably to FP16 while providing 2× faster inference and 10× memory reduction.The classification and CNNDM summarization results are reported in Tables 1 and 2.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • Removing any BitDistill stage causes a non-trivial downstream performance drop, indicating that the full pipeline’s stages contribute complementarily.Table 5 ablates modeling refinement, continued pre-training, and distillation-based fine-tuning on Qwen3-0.6B.Source: §4.3 · p. 7, Table 5 · p. 9
  • Combining logits and multi-head attention distillation yields the most consistent performance across benchmarks, beyond the partial improvements from either alone.The comparison of the two Stage-3 techniques is reported in Table 6.Source: §4.3 · p. 7, Table 6 · p. 9
  • Across alternative quantization methods, BitDistill generally matches full-precision baselines, while GPTQ and AWQ provide additional gains.The comparison covers Block-Quant, GPTQ, AWQ, and min–max quantization.Source: §4.3 · p. 7, Table 4 · p. 7

4.4 Analysis

  • With SubLN inserted at specific positions, fine-tuning existing LLMs stabilizes optimization and improves performance.The training-loss comparison is shown in Figure 3(a).Source: §4.4 · p. 7, Figure 3 · p. 8
  • Continued training alleviates the growing performance gap with model size; its weights become more scratch-like, with boundary-near weights able to shift under small gradient steps.The authors link this distributional adaptation to better downstream fitting and reduced risk of suboptimal local minima.Source: §4.4 · pp. 7–8, Figure 2 · p. 8
  • On Qwen3-0.6B MNLI, a well-chosen single later layer outperforms all-layer distillation, and larger FP16 teachers yield greater gains, sometimes surpassing same-size FP16 models.Results vary substantially by selected layer; Figure 3(b,c) compares layer choices and teacher sizes.Source: §4.4 · pp. 8–9, Figure 3 · p. 8

5 Related Work

Prior work uses quantization and distillation to improve LLM efficiency while preserving accuracy, but existing quantization and distillation methods still face performance limitations, especially on downstream tasks.

5 Related Work

  • Post-training quantization applies low-bit quantization using calibration data without end-to-end training loss, while quantization-aware training continues optimizing quantized LLMs to raise their performance ceiling.The passage notes that PTQ often degrades performance, especially below 4 bits.Source: §5 · p. 9
  • Knowledge distillation transfers knowledge from a high-capacity teacher to a compact student, including through layer-to-layer or self-distillation approaches for quantized LLMs.The passage gives TSLD and BitDistiller as examples.Source: §5 · p. 9
  • Most existing distillation methods primarily target general language modeling and retain noticeable downstream-application performance gaps relative to full-precision models.Source: §5 · p. 9

6 Conclusion

BitDistill adapts pretrained LLMs to 1.58-bit weights through a three-stage framework. Experiments report performance competitive with FP16 models while reducing computational and memory footprint.

6 Conclusion

  • The framework refines the model with SubLN, continues pretraining to recover representation capacity, then distills hidden-state and attention-relation knowledge to narrow the accuracy gap.Source: §6 · p. 9
  • BitDistill achieves downstream-task performance competitive with FP16 models while significantly reducing computational and memory footprint.Source: §6 · p. 10
  • The authors present BitDistill as a strategy for resource-constrained deployment and as a source of insights into low-bit quantization, pretraining, and distillation dynamics.Source: §6 · p. 10

Related papers

Loading 2510.13998v1…