Source-linked AI summary
OneBit: Towards Extremely Low-bit Large Language Models
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che
TL;DR
LLM deployment faces substantial computational and memory demands, while existing approaches struggle to preserve effectiveness at 1-bit weight precision. OneBit combines a 1-bit weight representation with SVID-based initialization, and the paper reports advantages over representative baselines with a tradeoff between model size and performance.
Problem
LLMs require substantial memory and computation, and existing quantization approaches struggle to maintain effectiveness at 1-bit weight precision.
Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 4Method
OneBit represents weights with a sign matrix and two value vectors, using SVID to initialize the 1-bit model from high-bit weights.
Source: §1 · p. 2, §3.3 · p. 5Results
OneBit has clear advantages over representative strong baselines and achieves a good tradeoff between model size and performance.
Source: §6 · p. 10Takeaways & Limitations
The paper analyzes the capabilities of extremely low-bit quantized models and provides guidance for future research.
Source: §6 · p. 10OneBit incurs performance loss relative to the original model, requires relatively costly KD for capability transfer, and does not quantize activations.
Source: B Limitations · p. 19
Abstract
from arXiv · showhide
Model quantification uses low bit-width values to represent the weight matrices of existing models to be quantized, which is a promising approach to reduce both storage and computational overheads of deploying highly anticipated LLMs. However, current quantization methods suffer severe performance degradation when the bit-width is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to quantize models. This paper boldly quantizes the weight matrices of LLMs to 1-bit, paving the way for the extremely low bit-width deployment of LLMs. For this target, we introduce a 1-bit model compressing framework named OneBit, including a novel 1-bit parameter representation method to better quantize LLMs as well as an effective parameter initialization method based on matrix decomposition to improve the convergence speed of the quantization framework. Sufficient experimental results indicate that OneBit achieves good performance (at least 81% of the non-quantized performance on LLaMA models) with robust training processes when only using 1-bit weight matrices.
1 Introduction
OneBit addresses the difficulty of quantizing LLM weights to 1 bit by introducing a model architecture and an initialization method; experiments cover multiple model sizes and series.
1 Introduction
- LLM deployment is constrained by computational and memory demands; loading LLaMA-13B in FP16 requires around 26GB.These overheads can restrict deployment beyond mid-to-high-end GPUs and mobile devices.Source: §1 · p. 1
- Quantization methods such as GPTQ, SpQR, and AWQ compress LLM weights to 4 bits while maintaining their main abilities.The passage describes quantization as balancing time and space efficiency with model performance.Source: §1 · p. 1
- At extremely low bit-widths, PTQ efficacy declines, and existing methods struggle to preserve effectiveness when compressing weights to 1 bit.The paper attributes the difficulty to precision loss in weight matrices, which increases loss in the linear projection WX.Source: §1 · pp. 1–2
- OneBit’s 1-bit quantization method can outperform the examined 2-bit baselines on LLaMA-7B perplexity.Figure 1 reports results on Wikitext2; lower perplexity is better.Source: Figure 1 · p. 2
- OneBit represents each high-bit weight matrix with a sign matrix and two value vectors, then uses quantization-aware knowledge distillation to transfer model capabilities.The vectors retain floating-point precision in the linear projection at little cost, while the sign matrix preserves high rank with a small space cost.Source: §1 · p. 2
- SVID-based initialization decomposes high-bit matrices into low-bit ones and improves model performance and convergence speed.The paper presents SVID as essential for initializing its 1-bit architecture.Source: §1 · p. 2
- Experiments cover OneBit model sizes from 1.3B to 13B across OPT, LLaMA, and LLaMA2.The paper presents this range as evidence of generalizability.Source: §1 · p. 2
2 Related Work
The related work situates OneBit among model-compression approaches and contrasts PTQ and QAT, emphasizing the distinct representation and initialization challenges of 1-bit quantization.
2.1 Large Language Model Compression
- Model compression includes quantization, pruning, knowledge distillation, and low-rank factorization; OneBit uses quantization and KD for knowledge transfer.Quantization reduces weight precision, pruning removes weights or modules, KD trains a smaller student with a larger teacher, and low-rank factorization approximates a matrix with lower-rank factors.Source: §2.1 · pp. 2–3
2.2 Large Language Model Quantization
- Quantization is classified as PTQ or QAT according to when it is applied.The paper presents quantization as a compression method that can achieve a significant compression ratio with relatively small loss.Source: §2.2 · p. 3
- PTQ converts trained models using solvers and limited calibration data without additional training; GPTQ and other methods address precision loss through weight adjustment or outlier handling.The passage describes GPTQ as nearly lossless at 4-bit weight quantization.Source: §2.2 · p. 3
- QAT applies quantization during training or fine-tuning, and OneBit distinguishes itself with 1-bit weight representation and initialization methods.Prior approaches described include learnable parameters, equivalent transformations, randomized Hadamard transforms, vector quantization, and fine-tuning.Source: §2.2 · p. 3
3 Methodology
The methodology develops a 1-bit Linear layer and a decomposition-based initialization, then describes knowledge transfer from the trained model to its quantized counterpart.
3.1 Background
- At 1 bit, round-to-nearest quantization leaves only two available values, makes its scale and zero point practically insignificant, and severely reduces weight-matrix precision.The resulting loss of precision undermines model performance.Source: §3.1 · pp. 3–4
3.2 1-bit Linear Layer Architecture
- OneBit pairs an INT1 ±1 sign matrix with two FP16 value vectors to mitigate precision loss; a 4096 × 4096 matrix averages 1.0073 bits.The extra vectors distinguish OneBit from the sign-only design discussed in the paper.Source: §3.2 · p. 4
3.3 Sign-Value-Independent Decomposition
- SVID separates W into sign and magnitude components, then approximates the magnitude with the rank-1 outer product of two vectors.The paper states that including the sign matrix makes the approximation closer to W in Frobenius norm than decomposing |W| alone.Source: §3.3 · p. 5
- SVID initializes the quantized layer by mapping the sign matrix to W±1 and the rank-1 factors to h and g, without restoring W in FP16.This is an approximate initialization, not an attempt to precisely replicate the original parameters; it is intended as a starting point for further training.Source: §3.3 · p. 5
3.4 Knowledge Transfer
- The student is trained with teacher-guided cross-entropy on logits and mean-square error on hidden states; language-modeling loss is not used.The combined objective is LKD = LCE + αLMSE, with α balancing the two terms.Source: §3.4 · pp. 5–6
4 Experiments
The experiments evaluate 1-bit weight-only quantization with 16-bit activations across OPT, LLaMA, and LLaMA2. OneBit is compared with stronger-bit baselines and smaller models on language modeling, zero-shot tasks, and practical capability benchmarks.
4.1 Settings
- The evaluation uses W1A16 quantization on OPT-1.3B/2.7B, LLaMA-7B/13B, and LLaMA2-7B/13B, testing perplexity and zero-shot accuracy.Perplexity is measured on WikiText2 and C4; zero-shot tasks include Winogrande, HellaSwag, PIQA, BoolQ, and ARC.Source: §4 · p. 6, §4.1 · p. 6
4.2 Main Results
- On LLaMA-13B Wiki2, OneBit reaches 9.18 perplexity versus 5.09 for FP16, and the authors report it as the only method with perplexity comparable to the strongest FP16 baseline.The authors also report that OneBit outperforms the W2A16 baselines across model sizes and moves closer to FP16 as model size increases.Source: Table 2 · p. 7, §4.2 · p. 7
- On LLaMA-7B zero-shot tasks, OneBit averages about a 4% improvement over OmniQuant, while achieving the closest performance to FP16 among most evaluated models.The reported exceptions to its advantage over OmniQuant are BoolQ and ARC-e.Source: §4.2 · p. 7
4.3 Problem Solving Ability
- OneBit-7B retains commonsense advantages over models with more parameters or more training, and outperforms fully trained Pythia-1B in world-knowledge domains except social sciences.Figure 3 compares model capabilities and compression degree.Source: §4.3 · p. 8, Figure 3 · p. 8
5 Analysis and Discussion
The analysis reports efficiency and training robustness benefits for OneBit, then examines architectural and initialization choices that contribute to training and performance.
5.1 Efficiency
- OneBit offers a good trade-off between model space and performance, with comparable performance to FP16 at 0.2x the model space.The authors also report that compression ratios increase with model size and that 1-bit weights can accelerate CPU matrix multiplication through bit operations.Source: §5.1 · p. 9
5.2 Robustness
- OneBit training is more stable and less sensitive to learning rates than the instability reported for extremely low-bit quantization.The paper attributes sensitivity to large gradients as weights fluctuate between +1 and -1, producing substantial linear-layer output fluctuations.Source: §5.2 · p. 9
5.3 Effect of Different Components
- Post-LayerNorm is used to address possible floating-point overflow as activations grow with depth; Pre-LayerNorm may occasionally be ineffective.The components analyzed include Post-LayerNorm, value vectors, and parameter initialization.Source: §5.3 · p. 9
- The two value vectors distinguish OneBit structurally from BitNet and are reported to facilitate stable training and knowledge transfer.The paper also identifies them as effective in the main results.Source: §5.3 · p. 10
- NMF initialization may accelerate convergence, and Figure 5 shows better performance with NMF initialization.Both NMF and SVD can decompose |W| in SVID; the authors recommend NMF.Source: §5.3 · p. 10, Figure 5 · p. 10
6 Conclusion
The conclusion presents OneBit as a 1-bit weight-quantization structure with a corresponding initialization method, supported by experiments across LLM sizes and series.
6 Conclusion
- OneBit pairs a novel 1-bit weight-quantization structure with a parameter initialization method to address difficulties in 1-bit quantization.Source: §6 · p. 10
- Experiments across LLM sizes and series show advantages over representative strong baselines and a trade-off between model size and performance.Source: §6 · p. 10
- The paper analyzes the capabilities of extremely low-bit quantized models and offers guidance for future research.Source: §6 · p. 10
A.1 Proofs of Propositions
The appendix proves propositions about SVID’s linear-layer formulation and approximation, then gives implementation details for the comparison baselines.
A.1 Proofs of Propositions
- Proposition 1 reformulates the Linear layer according to SVID, with the proof using the approximation wij ≈ sij · aibj.Source: §A.1 · p. 13
- Proposition 2 states that SVID is closer to the original matrix W in Frobenius norm, with its proof considering SVD rank-1 approximation error.The proof describes that error as the sum of squared singular values other than the largest.Source: §A.1 · p. 14
A.2 Details on Baselines
- The GPTQ setup calibrates OPT and LLaMA models on 128 2048-token C4 samples; LLM-QAT is reimplemented for W2A16, while OmniQuant uses WikiText2 calibration samples.The appendix also specifies baseline-specific training and quantization settings.Source: §A.2 · p. 15
A.3 Results of LLaMA2
On LLaMA2-7B/13B, OneBit has advantages in perplexity and zero-shot accuracy, with larger-model scaling associated with more pronounced perplexity gains. The reported perplexity reduction is around 1.0 for OneBit versus around 0.5 for FP16 when scaling from 7B to 13B.
A.3 Results of LLaMA2
- OneBit has advantages in both perplexity and zero-shot accuracy on LLaMA2-7B/13B.Table 4 presents the LLaMA2 results.Source: §A.3 · p. 15, Table 4 · p. 15
- Around 1.0, OneBit's perplexity reduction when scaling from LLaMA2-7B to 13B on both Wiki2 and C4, versus around 0.5 for FP16.The passage reports the comparison as approximate reductions.Source: §A.3 · p. 15
- OneBit's advantages are more pronounced in larger LLaMA2 models.The passage contrasts the 7B-to-13B scaling pattern with FP16.Source: §A.3 · p. 15
A.4 Instrution Following Ability
After supervised fine-tuning, the 7B model produces higher-quality content and exhibits instruction following in zero-shot and few-shot settings. In the BitNet comparison, distillation training can remain unstable, while OneBit's value vectors are described as supporting forward and backward stability.
A.4 Instrution Following Ability
- After SFT, the 7B model smoothly produces high-quality content and exhibits instruction following in zero-shot and few-shot settings.The experiment fine-tunes for 3 epochs on alpaca_en_52k using Alpaca templates; Table 5 presents cases before and after SFT.Source: §A.4 · pp. 15, 17, Table 5 · p. 16
A.5 Comparison with BitNet
- BitNet training may remain unstable when distilling LLaMA-7B, even with larger learning rates.The paper conjectures that large gradients arise as weight elements fluctuate between +1 and -1, aggravating the Linear-layer output.Source: §A.5 · p. 17
- OneBit's value vectors supplement floating-point precision and limit fluctuations in quantized matrix-multiplication results.The paper describes forward stability through restoring activation variation to the FP16 level, and backward stability through smaller value vectors and a tanh-based derivative for Sign(·).Source: §A.5 · p. 17
A.6 Discussion on Knowledge Distillation
The discussion compares distillation losses and reports that hidden-state alignment improves perplexity, while adding attention-score alignment prevents convergence. It also describes the quantization cost and gives an example average Linear-layer bit-width of approximately 1.0073.
A.6 Discussion on Knowledge Distillation
- Hidden-state alignment improves perplexity, but adding attention-score alignment on top leads the model to fail to converge.Distillation loss alone achieves satisfactory transfer compared with other baselines; the paper suggests hidden-state regularization may reduce learning variance and support generalization.Source: §A.6 · pp. 17–18
- Figure 6 compares BitNet learning rates during LLaMA-7B distillation in the W1A16 setting, with weights copied from the original model.The passage specifies the comparison setup but does not state a winning learning rate.Source: Figure 6 · p. 18
- Approximately 7 days on 8 A100-80GB GPUs are required to quantize LLaMA-7B, less than 2% of its pretraining time.The comparison is with 82,432 GPU hours to train LLaMA-7B from scratch; the paper calls quantization time an acceptable cost.Source: §A.6 · p. 18
A.7 Average Bit-width of Linear Layer
- Approximately 1.0073 bits per parameter is the average bit-width for the described 4096 × 4096 Linear-layer example.The bit count includes a 1-bit quantized weight matrix and two FP16 value vectors.Source: §A.7 · pp. 18–19
B Limitations
The method has several stated boundaries: it loses performance relative to the original model, requires costly knowledge distillation, is not naturally extendable to higher bit-width, and omits activation quantization. The ethics statement says the study uses public open-source models for academic research in accordance with their licenses.
B Limitations
- Extremely low-bit quantization inevitably loses performance relative to the original model.The limitation passage also says the mathematical principles behind optimal 1-bit parameters remain unclear.Source: B Limitations · p. 19
- Capability transfer currently requires relatively costly knowledge distillation, though the cost is a one-time expense.The paper says it does not yet understand the mathematical principles behind optimal 1-bit model parameters.Source: B Limitations · p. 19
- The method is not naturally extendable to higher bit-width, and activation quantization remains future work.Source: B Limitations · p. 19
C Ethics Statement
- The study uses public open-source models for academic research, following their intended purposes and open-source licenses.Source: C Ethics Statement · p. 19