PaperlayerView exact revision on arXiv

Source-linked AI summary

The case for 4-bit precision: k-bit Inference Scaling Laws

Tim Dettmers, Luke Zettlemoyer

arXiv:2212.09720v2cs.LGcs.NE

TL;DR

Quantization reduces model memory requirements, but choosing precision and parameter count to maximize zero-shot accuracy at a fixed bit budget remains an open trade-off. This paper studies that trade-off through large-scale inference scaling experiments and finds that 4-bit parameters are almost universally optimal, with small blocks and float data types improving scaling.

  • Problem

    LLMs have large memory footprints, while the precision and parameter count that maximize zero-shot accuracy for a fixed model-bit budget remain uncertain.

  • Method

    The paper analyzes bit-level inference scaling through 35,000 zero-shot experiments across LLM families and parameter scales, testing quantization methods, data types, and block sizes.

  • Results

    4-bit quantization is almost universally optimal for reducing model bits and maximizing zero-shot accuracy, while small block sizes and data types improve 4-bit scaling.

  • Takeaways & Limitations

    Use 4-bit models with small blocks and float data types; vary parameter count rather than precision when trading off model size and predictive performance.

  • Takeaways & Limitations

    The scaling laws do not model low-bit inputs and therefore apply only when the mini-batch fits into the L1 cache, limiting their relevance to high-throughput inference latency.

Abstract

from arXiv · show

Quantization methods reduce the number of bits required to represent each parameter in a model, trading accuracy for smaller memory footprints and inference latencies. However, the final model size depends on both the number of parameters of the original model and the rate of compression. For example, a 30B 8-bit model and a 60B 4-bit model have the same number of bits but may have very different zero-shot accuracies. In this work, we study this trade-off by developing inference scaling laws of zero-shot performance in Large Language Models (LLMs) to determine the bit-precision and model size that maximizes zero-shot performance. We run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2. We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size -- splitting the parameters into small independently quantized blocks -- and the quantization data type being used (e.g., Int vs Float). Overall, our findings show that {4-bit} precision is almost universally optimal for total model bits and zero-shot accuracy.

1. Introduction

The paper studies how model size and quantization precision jointly determine zero-shot accuracy at a fixed total bit budget. Across tested models, 4-bit parameters provide the strongest trade-off, while small block sizes and floating-point or quantile data types improve 4-bit scaling.

  • LLM memory footprints can reach 352 GB for 175B models, while quantization reduces bits and may reduce latency at the expense of accuracy.
  • A 60B model at 4 bits and a 30B model at 8 bits use the same total number of bits but may differ in zero-shot accuracy.
  • The study evaluates bit-level inference scaling laws to identify the precision maximizing zero-shot accuracy for a fixed total number of model bits.
  • 35,000+ experiments cover 19M–176B-parameter models from OPT, Pythia/NeoX, GPT-2, BLOOM, and BLOOMZ across 3–16-bit precision.
  • Reducing precision from 16 to 4 bits improves fixed-bit zero-shot performance, but 3-bit precision reverses this trend, making 4-bit precision optimal across tested scales.
  • No tested method improves scaling at 6–8 bits; at 4 bits, quantile or floating-point types and block sizes of 64–128 are most effective.
  • For inference trade-offs, the paper recommends retaining 4-bit precision and varying model parameter count rather than changing precision.

2. Background

The background connects total parameter bits to inference latency and introduces the quantization data types and block-wise procedures evaluated in the study. Smaller independently normalized blocks can use quantization bins more effectively, at an explicit bit overhead.

  • Inference latency depends mainly on memory loading and computation, with modern hardware often spending over 100 times longer loading a number than performing an arithmetic operation.
  • When the mini-batch fits in L1 cache, smaller models or lower-bit parameters can reduce inference latency.
  • The study evaluates Integer, Float, quantile, and dynamic exponent data types with distinct representations and quantization behavior.
  • A nominal 4-bit type can behave like 3 bits when only half of its 16 bins are used on average.
  • Blocking divides a tensor into one-dimensional chunks, independently quantizing each block with its own normalization constant.
  • Block size 64 with 16-bit normalization constants adds 0.25 bit per parameter, independently of model hidden dimension.

3. Outlier-dependent quantization through proxy quantization

The paper develops proxy quantization to test whether outlier-dependent higher precision can stabilize sub-8-bit weights without task-specific inputs. It uses weight statistics to identify dimensions for higher-precision treatment.

  • Outlier features can cause large quantization errors and severe performance degradation below the previously studied 8-bit weight setting.
  • Proxy quantization assigns higher precision to weights associated with outlier feature dimensions while seeking a constant memory footprint across models and tasks.
  • Hidden-state standard-deviation thresholding can detect too many outliers in later OPT layers because hidden-state variation increases with depth.
  • Proxy quantization is input-independent and task-independent because it uses each layer’s hidden-unit weight standard deviation as an outlier proxy.
  • For each linear layer, the method identifies output dimensions for higher precision and quantizes corresponding next-layer input dimensions at 16 bits instead of k bits.

4. Experimental setup

The experiments use 16-bit inputs with quantized parameters across several evaluation metrics and model configurations. Perplexity is less noisy than zero-shot accuracy, while linear interpolation better represents the observed precision scaling curves.

  • Experiments use 16-bit inputs, 3–8-bit quantized parameters, and an unquantized 16-bit floating-point baseline.
  • Inference performance is measured with Pile CommonCrawl perplexity and mean zero-shot performance on LAMBADA, Winogrande, HellaSwag, and PiQA.
  • Across 35,000+ experiments, Pile CommonCrawl perplexity and zero-shot performance have a Pearson correlation coefficient of -0.94.
  • Perplexity is preferred for reliable evaluation, but zero-shot accuracy is used in the main paper because perplexity is difficult to interpret.
  • Bivariate power functions fit the parameter-count and precision data poorly, whereas linear interpolations yield nearly parallel precision-specific scaling curves.

5. Results & Analysis

Across model families and scales, 4-bit precision generally provides the best zero-shot scaling for a fixed bit budget, while 3-bit inference can become unstable or degrade performance. Small blocks and suitable data types improve lower-precision scaling, but tested methods do not surpass 4-bit scaling.

  • 5.1. Bit-level Inference Scaling Laws: 4-bit precision yields optimal scaling for almost all model families and scales, with BLOOM-176B as the only reported exception.
  • 5.1. Bit-level Inference Scaling Laws: 3-bit inference is unstable for Pythia and OPT, with performance near random at the largest models.The reported performance is close to 35%.
  • 5.2. Improving Scaling Laws: None of the tested quantization methods improve bit-level scaling for 6 to 8-bit precision.The authors attribute this to sufficient precision relative to 16-bit weights.
  • 5.2. Improving Scaling Laws: A block size of 64 adds 0.24 bits per parameter and improves zero-shot accuracy almost as much as increasing precision from 4 to 5 bits.Small block sizes also improve 3-bit scaling, but it remains inferior to 4-bit scaling.
  • 5.2. Improving Scaling Laws: Quantile and float data types generally produce better scaling than integer and dynamic exponent quantization.Quantile quantization is generally the strongest data type across models, scales, and precisions.
  • 5.2. Improving Scaling Laws: Outlier-dependent quantization improves 3-bit stability for OPT and Pythia but still scales worse than 4-bit precision.It has no scaling benefit for 4-bit precision and is unnecessary for stable 3-bit BLOOM and GPT-2 models.

6. Related work

Related work covers LLM quantization methods, including one-shot and zero-shot approaches, blocking, centering, and learned data types. It also situates this study within inference scaling-law research by extending precision analysis beyond prior 4-bit versus 16-bit comparisons.

  • Large language model quantization: One-shot quantization methods such as GPTQ can be more accurate but may require hours of optimization before deployment.
  • Large language model quantization: Zero-shot quantization methods can be used immediately but often fail at lower precisions.
  • Quantization methods: Prior quantization methods include blocking, grouping, centering, and learned data types found through clustering.
  • Scaling Laws for Inference: This study extends inference scaling-law analysis from prior 4-bit versus 16-bit comparisons to precisions from 3 to 16 bits and separates factors affecting scaling.

7. Recommendations & Future Work

The paper recommends 4-bit quantization with small blocks and floating-point or quantile data types for zero-shot inference, while identifying low-bit methods and hardware efficiency as future directions.

  • Use 4-bit quantization by default because it offers the best trade-off between total model bits and zero-shot accuracy.
  • Use block sizes of 128 or lower to stabilize 4-bit quantization and improve zero-shot performance.
  • Use floating-point or quantile data types; integer types may be preferable when hardware support improves inference latency.
  • 2-bit GPTQ with blocking outperforms 3-bit Float with blocking on WikiText-2 perplexity, but GPTQ requires blocking for good scaling.
  • 3-bit GPTQ without blocking scales poorly, whereas blocking is needed to make one-shot quantization bit-level efficient.
  • A higher precision can be preferable when available memory fits a larger-bit smaller model but not a larger 4-bit model.
  • Future work should improve sub-4-bit scaling, preserve outlier information without adding many bits, and develop hardware-efficient data types.

8. Discussion & Limitations

The study’s conclusions are bounded by omitted quantization classes, missing optimized GPU implementations, and the indirect treatment of deployment latency.

  • The study did not consider quantization methods optimized with additional input data or with model weights alone.
  • Unstudied optimization-based methods could improve inference scaling, representing a missed opportunity in the experiments.
  • The lack of optimized GPU implementations leaves the speedups of lookup-table data types unclear.
  • Low-bit models with 16-bit inputs may become less latency-efficient for busy APIs because large mini-batches no longer fit into cache.
  • Weight-matrix loading is only one part of inference latency; unoptimized attention operations can remain a large latency component.

9. Conclusion

The paper studies how parameter count, quantization precision, and zero-shot accuracy trade off during inference across many LLMs and scales. It finds that 4-bit quantization is almost universally optimal for reducing model bits while maximizing zero-shot accuracy, with data types and block size most important for improving scaling.

  • 35,000 zero-shot experiments across diverse LLMs and parameter scales analyze trade-offs among model size, bit precision, and zero-shot accuracy.
  • 4-bit quantization is almost universally optimal for reducing model bits and maximizing zero-shot accuracy.
  • Data types and block size are the most critical measures for improving bit-level scaling behavior.

A. Data type details

The paper unifies k-bit quantization as a mapping from integer codes to normalized floating-point values, then defines several data types through their codebooks. Quantized tensors are normalized, mapped to nearby codebook values, stored as indices, and later dequantized for computation.

  • General quantization: Quantization maps k-bit integers to floating-point values in [−1, 1], allowing data types to be compared through their codebooks.The codebook and bit count fully specify the quantization data type.
  • Quantization pipeline: Normalized inputs are assigned the nearest codebook value by binary search, stored as mapping indices, and reconstructed through a lookup followed by rescaling.The normalization constant restores the original value range during dequantization.
  • Data types: Integer quantization uses uniformly spaced signed values, with an offset representation and symmetric truncation around zero.For Int8, the normalized range is [−127/c, 127/c] with c = 127.
  • Data types: Floating-point quantization represents values using sign, exponent, and mantissa bits; the experiments vary exponent–mantissa allocations across 3–8-bit formats.The study reports that 2–3 exponent bits perform best, using 3 exponent bits for 4–8-bit formats and 2 for 3-bit formats.
  • Data types: Dynamic exponent quantization combines a sign bit, a base-10 exponent region, an indicator bit, and an unsigned linear fraction.Its codebook can also be constructed by evenly bisecting [0.1, 0.9] across fractional values and adding zero.
  • Data types: Quantile quantization allocates equal numbers of values to each quantization bin by using estimated inverse cumulative-distribution-function quantiles.The codebook contains quantile values for all 2^k indices plus an additional zero.

B. Further negative results: distribution centering

The paper evaluates distribution centering and related scaling results, finding that centering weight distributions does not improve scaling. Additional observations identify precision- and model-specific exceptions.

  • Distribution centering: Distribution centering does not improve scaling for weight quantization in any tested scenario.The method subtracts the mean before quantization and adds it back during dequantization.
  • Additional observations: Pythia 5-bit models perform as well as Pythia 4-bit models in an exception to the main scaling trends.
  • Additional observations: BLOOM and BLOOMZ exhibit almost identical quantization behavior, suggesting that fine-tuning an existing model does not change their quantization properties.
  • Precision-specific results: The main quantization-method improvements occur at 3- and 4-bit precision, with small block sizes improving 4-bit bit-level scaling.The 4-bit experiments did not include block-size grid searches for BLOOM-176B and OPT-175B because of limited compute.

C.3. No scaling improvements for 6 to 8-bit models through quantization methods

The paper finds that quantization methods do not improve bit-level scaling at 6–8 bits, while data types and block sizes matter more at lower precision. Cross-entropy evaluations likewise identify 4-bit quantization as bit-level optimal.

  • 6–8-bit scaling: 6–8-bit quantization methods do not improve bit-level scaling, likely because these precisions already represent weights sufficiently accurately.The paper specifically reports no improvement from additional quantization techniques in this range.
  • 6–8-bit scaling: Data types and block size do not affect 6-bit scaling behavior, with similar results reported for 7- and 8-bit precision.
  • Exponent configurations: For float data types with block-wise quantized weights, exponent bits should generally make up at least half the precision, rounded up.The corresponding exponent-bit choices for 3–8 bits are 2, 2, 3, 3, 4, and 4; 5-bit precision is the exception.
  • Cross-entropy evaluation: 4-bit quantization is bit-level optimal across all evaluated models for cross-entropy loss on CC-Pile.The evaluation uses cross-entropy because perplexity provides a continuous token-level measure.
  • Cross-entropy evaluation: Quantile quantization is the best data type on average for cross-entropy scaling, while smaller block sizes are consistently better except for BLOOMZ.The difference between block sizes is small for many models.
Loading 2212.09720v2…