PaperlayerView exact revision on arXiv

Source-linked AI summary

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi

arXiv:2402.04291v2cs.LGcs.AIcs.CL

TL;DR

Ultra-low-bit quantization must reduce LLM resource demands without the performance collapse seen in existing methods. BiLLM uses distribution-aware salient-weight residual approximation and non-salient-weight splitting, achieving strong results near 1 bit across LLM families and evaluations.

  • Problem

    Existing PTQ methods lose substantial performance at ultra-low bit-widths, limiting effective 1-bit compression for large, resource-intensive LLMs.

  • Method

    BiLLM structurally selects salient weights for binary residual approximation and optimally splits non-salient weights for grouped binarization.

  • Results

    8.41 perplexity is achieved on LLaMA2-70B with 1.08-bit weights, while experiments report SOTA performance across multiple LLM families and evaluation metrics.

  • Takeaways & Limitations

    BiLLM demonstrates that pretrained LLMs can undergo one-time post-training binarization at an average bit rate close to 1 bit without substantial precision loss.

Abstract

from arXiv · show

Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.

1. Introduction

LLM deployment is constrained by large memory and computation demands, while existing PTQ methods collapse at ultra-low bit-widths. BiLLM addresses this with distribution-aware salient-weight selection, residual approximation, and split binarization, achieving compact weights with strong perplexity results.

  • LLaMA2-70B requires 150 GB of FP16 storage, motivating compression for memory-constrained inference.
  • Existing PTQ methods perform well at 8-bit and 4-bit precision but suffer performance collapse at ultra-low bit-widths of 3 bits or fewer.PB-LLM maintains perplexity around 800 with average 1.7-bit weights.
  • LLM weights show a bell-shaped magnitude distribution, while a small fraction has high Hessian values and strongly influences layer outputs.
  • BiLLM structurally selects salient weights using a Hessian-based metric and applies residual approximation to preserve them with low storage overhead.
  • BiLLM splits and separately binarizes non-salient weights, searching for an optimal breakpoint to reduce errors under their bell-shaped distribution.
  • 8.49 and 8.41 perplexity are achieved on LLaMA-65B and LLaMA2-70B, respectively, using only 1.08-bit weights on WikiText2.The LLaMA2-70B result surpasses the reported 9.34 perplexity of FP16 OPT-66B.

2. Related Works

Related work spans QAT and PTQ methods for compressing neural networks, with PTQ favored for efficient LLM quantization. Prior approaches preserve selected features or use binarization, but existing LLM binarization retains substantial higher-precision weights.

  • Quantization maps high-precision parameters to discrete values without changing model structure, reducing storage and computational overhead.
  • PTQ methods improve accuracy through custom grouping or by retaining outlier features and high-error features at FP16 or INT8.
  • Binarization represents parameters using only 1 bit, typically as ±1, with the sign function producing the binary weight tensor.
  • BWN binarizes weights with full-precision activations, while XNOR-Net extends the approach by binarizing both weights and activations.
  • PB-LLM requires more than 30% of weights to remain at 8 bits for LLMs to produce reasonable answers, limiting its compression.

3. Method

BiLLM binarizes salient and non-salient weights with distinct strategies: structural selection and residual approximation protect salient weights, while distribution-based splitting reduces errors for the remainder. The workflow adds about 0.1 weight bits while retaining approximately 1.1-bit compression.

  • Salient Weight Binarization: BiLLM structurally selects salient columns using Hessian-based sensitivity and searches their number to balance quantization error and storage overhead.Sensitive Hessian values are concentrated in specific columns or rows, so column-wise selection avoids the bitmap overhead of unstructured selection.
  • Salient Weight Binarization: Residual approximation binarizes salient weights twice, using the residual after the first binary approximation to reduce quantization error.The first binary tensor is subtracted from the original salient weights, and the residual is binarized again; the resulting error is lower than direct binarization.
  • Bell-shaped Distribution Splitting: After salient-weight removal, BiLLM splits bell-shaped non-salient weights at an optimized break-point into concentrated and sparse regions, then binarizes each region separately.The concentrated region is [−p, p], while the sparse region is [−m, −p] ∪ [p, m]; scales and binary tensors are computed separately.
  • Bell-shaped Distribution Splitting: BiLLM uses percentile search to find the splitting point for actual non-salient distributions, which can deviate from the ideal Gaussian model after block-wise error compensation.For an ideal Gaussian distribution, the error objective is convex with a global minimum, but the method searches empirically for the actual distribution.
  • Binarization Workflow: Structural searching and residuals add only about 0.1 weight bits, while the overall search strategy maintains approximately 1.1-bit average weights across OPT, LLaMA, and LLaMA2.BiLLM primarily binarizes Linear weights within Transformer blocks and applies block-wise compensation by default.

4. Experiments

Experiments evaluate BiLLM across OPT, LLaMA, and Vicuna families using perplexity, zero-shot tasks, model size, and memory comparisons. Across these settings, BiLLM preserves performance near 1 bit while substantially reducing storage and memory requirements.

  • Results: BiLLM maintains reasonable OPT outputs at 1.1 average weight bits, while RTN and GPTQ collapse at 1 bit.Compared with PB-LLM at 1.7 bits, BiLLM reduces weight bit-width by 35% and improves OPT performance by 49.4% to 77.0%.
  • Results: BiLLM achieves better Vicuna-7B and Vicuna-13B perplexity performance at an average weight bit-width of 1.08.The instruction-tuned evaluation uses Wikitext2, PTB, and C4 datasets.
  • Ablation results: Ablations show that both salient-weight residual approximation and non-salient-weight splitting significantly improve binary performance.OPT-6.7B is more sensitive to splitting, whereas LLaMA-7B is more responsive to salient-weight residual approximation.
  • Model size: BiLLM achieves close to tenfold weight compression across LLaMA and LLaMA2 models ranging from 7B to 70B parameters.The model-size comparison covers FP16 and BiLLM-compressed models.
  • GPU memory: For OPT-30B, BiLLM at 1.1 bits improves memory compression by 41.57% versus PB-LLM and 27.07% versus GPTQ.It also improves accuracy by 49.44% and 19.10%, respectively; its memory occupancy is about 69.9% of 2-bit quantization.

5. Conclusions

BiLLM is a post-training binary quantization method for pretrained LLMs that combines residual approximation for salient weights with optimal segmentation for non-salient weights. Experiments across open-source LLM families and an instruction-tuned model show performance near a 1-bit average rate.

  • 5. Conclusions: BiLLM uses binary residual approximation for structurally salient weights and optimal segmentation for grouped binarization of non-salient weights.The design is inspired by weight-value and Hessian distributions.
  • 5. Conclusions: BiLLM enables one-time ultra-low-bit quantization without substantial precision loss across multiple LLM families and a fine-tuned instruction model.The reported average bit rate is close to 1 bit.
  • 5. Conclusions: BiLLM advances the LLM bit-width quantization frontier and is intended to facilitate deployment on edge and resource-constrained devices.The conclusion also encourages further exploration of LLM compression.

A. BiLLM Implementation

BiLLM structurally searches for salient columns, binarizes them with residual approximation, and separately splits the bell-shaped non-salient distribution at an optimized breakpoint. These steps produce the final binary weights while limiting quantization error and bit overhead.

  • A. BiLLM Implementation: Non-salient weights are divided into sparse and concentrated regions, then binarized separately at a searched breakpoint p* that minimizes quantization loss.The regions are defined around the central interval and the outer magnitude ranges of the bell-shaped distribution.
  • A. BiLLM Implementation: Structural salient selection avoids the bitmap overhead associated with unstructured selection because sensitive Hessian values concentrate in particular rows or columns.The method uses this concentration to balance accuracy and storage.

B. Quantization Error

The quantization-error analysis relates uniform-quantizer MSQE to the weight probability density and target bit-width. After removing salient weights, BiLLM models the remaining bell-shaped distribution and optimizes its binary split accordingly.

  • B. Quantization Error: A uniform quantizer maps [Xmin, Xmax] into M intervals, where M typically equals 2^b for target bit-width b.The quantization step size and interval boundaries define the discretization.
  • B. Quantization Error: Under binarization, quantization levels are restricted to −α, 0, and α, and interval means determine the quantized representation.The resulting mean squared quantization error is then derived from the quantizer intervals.
  • B. Quantization Error: The MSQE of a uniform quantizer depends on the probability density function and quantization bit-width.For BiLLM, the non-salient distribution is treated as non-uniform and approximately Gaussian after salient weights are removed.
  • B. Quantization Error: BiLLM substitutes the binary scale α into the error formulation to analyze binarization under the non-salient weight distribution.This connects the general quantization-error expression to the method’s bell-shaped-weight setting.

C. Searching Curve of Salient Column and Non-salient Distribution

BiLLM searches for a compact set of salient columns and an optimal split for non-salient weights. These searches preserve compression near 1.1 bits while following the distributions observed in LLM weights.

  • Salient-column search: The search begins with the most salient columns and restricts candidate groups to 3–30 columns to control residual-approximation overhead.The method uses column-level segmentation and formulates a minimal-error column-number search.
  • Salient-column search: Structured selection captures most salient weights despite block-wise fluctuations in the search curve.The Feedforward layer has more dispersed salient weights and therefore favors residual approximation across more columns.
  • Salient-column search: Approximately 1.1 average weight bits are maintained after the salient-column search strategy.Table 1 confirms the compression level across various LLMs.
  • Non-salient distribution: The non-salient search curve is convex, indicating an optimal split point p* for separately binarizing distribution segments.The curve uses p divided by the maximum weight value as its horizontal-axis ratio and resembles Gaussian or Laplacian behavior.

D. Multi-evaluation Comparisons

BiLLM is evaluated against GPTQ and PB-LLM on perplexity and zero-shot accuracy across multiple datasets and model families. The reported comparisons generally favor BiLLM, particularly at lower bit-widths.

  • Perplexity comparisons: The evaluated perplexity experiments cover PTB and C4, with additional model-size comparisons reported in the appendix.Main-text results include LLaMA-7B, LLaMA2-7B, and OPT-6.7B.
  • Perplexity comparisons: BiLLM generally achieves lower perplexity than GPTQ and PB-LLM under lower-bit configurations across PTB and C4.The comparisons include LLaMA-13B, LLaMA2-13B, OPT-13B, and other models.
  • Zero-shot comparisons: BiLLM is superior overall in side-by-side zero-shot accuracy comparisons on PIQA, BoolQ, and other datasets.Random perturbations are reported but do not reduce BiLLM’s performance across the board.

E. Ablation of BiLLM with different block size

The ablation varies BiLLM’s block size to examine the trade-off between quantization granularity, perplexity, and bit-width. Finer chunks improve the overall perplexity, while block size 128 balances compression and quantization effect.

  • Block-size effects: Finer chunk granularity produces lower overall perplexity and relatively smaller bit-widths.The tested block sizes range from 32 to 512 columns.
  • Block-size effects: Block size 128 better balances bit-width and quantization effect.Smaller chunks provide finer representation and more scale, but increase quantization-result diversity and weighting overhead.
  • Evaluation setup: Table 9 reports BiLLM perplexity on Wikitext2, PTB, and C4 for different block-size settings.

G. Magnitude and Hessian Distribution of LLMs

The paper examines weight-magnitude and Hessian distributions across Transformer layers and blocks. The examples show bell-shaped weight distributions and concentrated salient weights in several attention-related layers.

  • Distribution analysis: Additional examples illustrate bell-shaped weight values and long-tailed Hessian distributions across OPT-1.3B and LLaMA-7B Transformer blocks.Figures 13 and 14 cover multiple linear layers in selected blocks.
  • Salient-weight distribution: Figure 15 marks the top 10% most significant weights within Hessian distributions across five OPT-1.3B Transformer blocks.The analysis covers both attention and feedforward blocks.
  • Salient-weight distribution: Salient Q, K, and V weights in OPT models tend to concentrate in particular columns or rows.
  • Dialogue examples: The dialogue examples compare binarized LLaMA-13B and Vicuna-13B responses with PB-LLM using colored text for reasonable or inappropriate outputs.
Loading 2402.04291v2…