Source-linked AI summary
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei
TL;DR
Large language models deliver strong task performance, but their growing size creates deployment, energy, memory, and computational challenges. This paper introduces BitNet b1.58, a Transformer trained from scratch with ternary weights and 8-bit activations, and reports comparable perplexity and end-task performance alongside lower latency, memory, throughput, and energy costs.
Problem
Growing LLM size creates deployment challenges and raises energy and economic concerns because of high computational and memory requirements.
Method
BitNet b1.58 replaces Transformer linear layers with BitLinear and trains from scratch using 1.58-bit ternary weights and 8-bit activations.
Results
BitNet b1.58 matches full-precision LLaMA performance from 3B model size while reducing latency and memory, with larger models also showing substantial energy and throughput gains.
Takeaways & Limitations
The results support 1.58-bit LLMs as a cost-effective basis for scaling language models and improving deployment prospects for MoE, long-context, edge, and mobile applications.
Abstract
from arXiv · showhide
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.
1 The Era of 1-bit LLMs
LLMs deliver strong capabilities but their growing size increases deployment, energy, and economic costs. BitNet-style 1-bit architectures address these pressures by replacing costly floating-point computation and reducing parameter-memory demands while preserving performance.
- Growing LLM size creates deployment challenges and raises environmental and economic concerns through high energy consumption.
- Post-training quantization reduces weight and activation precision, lowering LLM memory and computational requirements.
- BitNet replaces floating-point matrix multiplication with integer addition, saving orders of energy cost for LLMs.
- 1-bit LLMs reduce memory capacity and bandwidth requirements, lowering the cost and time of loading weights from DRAM.
- BitNet b1.58 assigns every parameter one of {-1, 0, 1}, adding zero to the original 1-bit BitNet representation.
2 BitNet b1.58
BitNet b1.58 is a Transformer variant trained from scratch with ternary weights and 8-bit activations. Its design combines absmean quantization, per-token activation scaling, and LLaMA-alike components for efficient implementation.
- BitNet b1.58 replaces nn.Linear with BitLinear in a Transformer architecture and trains from scratch with 1.58-bit weights and 8-bit activations.
- Quantization Function: Absmean quantization scales the weight matrix by its average absolute value before rounding values to {-1, 0, +1}.
- Activation Quantization: Activations are scaled per token to [−Qb, Qb] without zero-point quantization, simplifying implementation and system-level optimization.The experiments report negligible effects on performance from this choice.
- LLaMA-alike Components: The architecture adopts LLaMA-alike RMSNorm, SwiGLU, rotary embedding, and no biases for integration with open-source software.
3 Results
BitNet b1.58 matches full-precision LLaMA performance from 3B while reducing inference cost, with efficiency gains widening at larger model sizes. It also improves energy, throughput, and scaling efficiency, and maintains strong generalization after 2T-token training.
- BitNet b1.58 matches full-precision LLaMA performance from 3B model size in perplexity and zero-shot end-task accuracy.
- 4.1 times faster: BitNet b1.58 70B outperforms the LLaMA LLM baseline in decoding latency, with speed-up increasing as model size scales.
- 71.4 times lower arithmetic-operations energy: BitNet b1.58 uses mainly INT8 addition, whereas LLaMA combines FP16 addition and multiplication.
- 8.9 times higher throughput: BitNet b1.58 70B supports up to 11 times LLaMA 70B's batch size on two 80GB A100 cards.
- 13B BitNet b1.58 is more efficient in latency, memory usage, and energy consumption than a 3B FP16 LLM, supporting a new cost-performance scaling equivalence.
- BitNet b1.58 achieves superior performance across all evaluated end tasks against StableLM-3B after training with 2T tokens.
4 Discussion and Future Work
BitNet b1.58 is positioned as a way to reduce memory, communication, and energy constraints across MoE, long-sequence, edge, and mobile deployments. The discussion also calls for hardware designed specifically for 1-bit LLM computation.
- 1-bit Mixture-of-Experts (MoE) LLMs: 1.58-bit LLMs reduce MoE deployment costs by lowering memory requirements and activation-transfer overhead across networks.These reductions address limitations from high memory consumption and inter-chip communication in MoE systems.
- Native Support of Long Sequence in LLMs: 8-bit activations allow BitNet b1.58 to double context length under the same resources, while further compression to 4 bits or lower remains future work.
- LLMs on Edge and Mobile: Reduced memory and energy consumption could enable 1.58-bit LLMs to run on resource-constrained edge and mobile devices.The models are also described as more CPU-friendly, supporting efficient execution on these devices.
- New Hardware for 1-bit LLMs: The new computation paradigm enabled by 1-bit LLMs motivates designing hardware and systems specifically optimized for their workloads.The discussion points to specialized hardware as a direction beyond general-purpose LLM accelerators.