Source-linked AI summary
BitNet: Scaling 1-bit Transformers for Large Language Models
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
TL;DR
Large language models are expensive to deploy, while lower-precision quantization can lose accuracy or become difficult to optimize. BitNet introduces a scalable, stable 1-bit Transformer with quantization-aware training, achieving competitive performance with lower memory and energy consumption and scaling behavior similar to full-precision Transformers.
Problem
Large language models have high inference, memory, and communication costs, while lower-precision quantization can reduce accuracy and make optimization more difficult.
Method
BitNet trains 1-bit weights from scratch using binary weights, quantized activations, and BitLinear as a replacement for Transformer linear projections.
Results
BitNet achieves competitive perplexity and downstream task performance while significantly reducing memory footprint and energy consumption compared with the baselines.
Takeaways & Limitations
BitNet follows a scaling law similar to full-precision Transformers, indicating potential for effective scaling to even larger language models.
Takeaways & Limitations
Quantization-aware training becomes more difficult to optimize as precision decreases, and its scaling-law behavior had been unknown before this work.
Abstract
from arXiv · showhide
The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.
1 Introduction
BitNet addresses the deployment costs and quantization challenges of large language models with a scalable 1-bit Transformer trained through quantization-aware methods. It achieves competitive performance while reducing memory and energy use, and follows a scaling law similar to full-precision Transformers.
- Motivation: Large language models are costly to host because inference, memory bandwidth, and inter-device communication consume substantial resources.Quantization is presented as a way to reduce memory footprint and computational cost while maintaining competitive performance.
- Quantization challenges: Post-training quantization is simple to apply but causes greater accuracy loss at lower precision because the model is not optimized for quantized representations during training.
- Quantization challenges: Quantization-aware training can improve accuracy and support continued training, but lower precision makes optimization harder and its scaling behavior was unknown.
- Approach: BitNet is a 1-bit Transformer that uses binary weights, quantized activations, and high-precision optimizer states and gradients during training.It is implemented by replacing Transformer linear projections with BitLinear and is designed for scalable, stable training.
- Results: BitNet achieves competitive perplexity and downstream accuracy while significantly reducing memory footprint and energy consumption versus quantization and FP16 baselines.The evaluation covers a range of language-modeling benchmarks.
- Results: BitNet follows a scaling law similar to full-precision Transformers, supporting effective scaling to larger language models with potential performance and efficiency benefits.
2 BitNet
BitNet replaces conventional Transformer matrix multiplication with BitLinear, using 1-bit weights and quantized activations while retaining high precision where needed for training stability. Group-wise quantization and normalization support efficient model parallelism, while the design targets lower energy and memory costs.
- BitLinear: BitLinear replaces conventional matrix multiplication with binarized 1-bit weights while retaining the Transformer layout and high-precision components.BitNet stacks self-attention and feed-forward blocks like a standard Transformer; residual connections, layer normalization, and other components remain higher precision.
- BitLinear: Weights are centralized, binarized to +1 or −1, and scaled by β to reduce the l2 error from binarization.The signum function produces the binary weights after zero-mean centralization, and β rescales them toward the real-valued weights.
- BitLinear: Activations are quantized with absmax scaling into [−Qb, Qb], with 8-bit activation quantization used in this work.Per-tensor quantization is used during training and per-token quantization during inference for stability and efficiency; lower precision is left for future work.
- BitLinear: Layer normalization before activation quantization preserves output variance at approximately 1, matching the scale of full-precision computation.This SubLN-based design addresses training stability after quantization.
- Model Parallelism: Group Quantization and Normalization estimates parameters independently for weight, activation, and normalization groups, avoiding additional communication during model parallelism.Dividing tensors into groups makes the parameters local to each device partition and supports scaling to large language models.
- Model Training: BitNet uses straight-through estimation, high-precision gradients and optimizer states, and a large learning rate to train the 1-bit model stably.The latent high-precision weights are binarized during the forward pass, while experiments report that BitNet benefits from a large learning rate even when FP16 diverges at the same rate.
- Computational Efficiency: BitNet’s matrix multiplication is dominated by additions rather than multiplications, producing substantially lower estimated energy than full- and half-precision Transformers.Multiplications are mainly used to rescale outputs with β and γ/Qb; the reported savings focus on matrix multiplication energy across model sizes.
3 Comparison with FP16 Transformers
BitNet is evaluated against FP16 Transformers across model scales, scaling behavior, inference efficiency, downstream capabilities, and training stability. The results indicate similar loss scaling, improved inference efficiency, and greater stability for BitNet.
- Experimental setup: 125M to 30B models are trained on the same datasets and settings for BitNet and Transformer baselines.The corpus includes the Pile, Common Crawl snapshots, RealNews, and CC-Stories.
- Scaling behavior: BitNet’s loss scaling is similar to the FP16 Transformer’s power-law scaling as parameter count increases.The comparison fixes training tokens while varying model size.
- Scaling behavior: The fitted scaling law predicts BitNet’s loss with high accuracy from 125M–6.7B models to 13B and 30B models, while the FP16 gap narrows at larger sizes.The extrapolation uses smaller models to fit the power-law parameters and larger models for prediction.
- Inference efficiency: At 7nm, BitNet has higher inference scaling efficiency, achieving better loss under a fixed budget and lower cost for the same FP16 performance.The inference-optimal law models loss against energy consumption rather than training FLOPs.
- Downstream capabilities: Zero-shot and few-shot downstream performance are evaluated across multiple BitNet and FP16 scales against inference cost.The tasks include Hellaswag, Winogrande, Winograd, and Storycloze.
- Training stability: BitNet converges with larger learning rates than FP16 Transformers and benefits from them through better convergence in PPL.The stability tests vary peak learning rates across model series.
4 Comparison with Post-training Quantization
BitNet is compared with post-training quantization methods across bit levels, perplexity, and downstream accuracy. It achieves competitive or superior performance at low precision while reducing inference cost relative to higher-bit baselines.
- Comparison setup: BitNet is compared with Absmax, SmoothQuant, GPTQ, and QuIP, which post-train quantize FP16 Transformer models.Absmax and SmoothQuant quantize weights and activations, whereas GPTQ and QuIP reduce weight precision only.
- Evaluation: The evaluation spans 16- to 1-bit levels using validation perplexity and zero-shot downstream accuracy.The downstream evaluation covers multiple benchmark tasks.
- Results: BitNet’s zero-shot scores are comparable with 8-bit models while its inference cost is much lower.The comparison uses 6.7B models for fairness.
- Results: For 4-bit models, weight-only quantization outperforms weight-and-activation quantization because activations are more difficult to quantify.This comparison distinguishes the two quantization configurations.
- Results: At 1 bit, BitNet significantly outperforms both weight-and-activation and weight-only post-training quantization methods.The passage reports consistently superior scores for BitNet at lower bit levels.
- Results: BitNet’s advantage remains consistent in zero-shot and few-shot results as model size scales from 1.3B to 6.7B.Figure 6 summarizes downstream results against post-training quantization baselines.
5 Ablation Studies
The ablation study examines activation quantization and training-stability choices in BitNet. It selects absmax and SubLN because they outperform the tested alternatives and support stable training.
- Ablation design: BitNet uses absmax activation quantization and SubLN for training stability.The study compares these choices with elastic activation scaling, Pre-LN, and BMT.
- Activation quantization: Absmax outperforms the elastic activation quantization function and provides more stable training.Its stability enables BitNet to use a larger learning rate.
- Training stability: SubLN outperforms both Pre-LN and BMT in the ablation experiments.Pre-LN is the default GPT pretraining architecture, while BMT is intended to stabilize binarized models.
- Configuration: The final BitNet implementation chooses absmax and SubLN based on the ablation results.These choices are used as the reported implementation configuration.
6 Conclusion and Future Work
BitNet is presented as a scalable and stable 1-bit Transformer that maintains competitive performance while reducing memory footprint and energy consumption. Its scaling behavior resembles that of full-precision Transformers, supporting investigation at larger model sizes.
- BitNet is a scalable and stable 1-bit Transformer architecture for large language models.
- BitNet achieves competitive perplexity and downstream task performance while significantly reducing memory footprint and energy consumption versus baselines.
- BitNet follows a scaling law similar to full-precision Transformers, indicating effective scaling to even larger language models.
- Future work includes increasing BitNet’s model size and training steps and applying it to other architectures such as RetNet.
A Hyperparameters
The appendix provides configuration and hyperparameter information for BitNet’s scaling experiments, stability tests, and ablations.
- Table 5 gives the model configuration used for BitNet’s scaling experiments.
- Table 6 lists hyperparameters for BitNet and FP16 Transformers in the scaling experiments, including a 0.05 weight decay setting for 13B and 30B models.
- Table 7 reports hyperparameters for stability tests comparing BitNet with FP16 Transformers.
- Table 8 reports hyperparameters used in BitNet ablation experiments.