PaperlayerView exact revision on arXiv

Source-linked AI summary

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh

arXiv:2210.17323v2cs.LG

TL;DR

Large GPT and OPT models are expensive to store and run, while existing compression methods are limited by retraining costs or poor scalability and accuracy at high compression. GPTQ introduces an efficient approximate second-order, one-shot quantizer that compresses massive models to low bitwidths with little accuracy loss. It also enables single-GPU generative inference and reported end-to-end speedups, while leaving multiplication reduction and activation quantization unresolved.

  • Problem

    GPT and OPT models have high inference and storage costs, while existing accurate compression methods are difficult to retrain or scale to billion-parameter models.

  • Method

    GPTQ is a one-shot weight-quantization method based on approximate second-order information and layer-wise reconstruction.

  • Results

    GPTQ quantizes 175-billion-parameter models to 3 or 4 bits per weight in approximately four GPU hours with minimal accuracy loss, while reporting 3.25× A100 and 4.5× A6000 inference speedups.

  • Takeaways & Limitations

    The method enables compressed OPT-175B generative inference on a single NVIDIA A100 GPU or two NVIDIA A6000 GPUs.

  • Takeaways & Limitations

    GPTQ reduces memory movement rather than multiplication cost, and the study does not include activation quantization or secondary measures such as bias effects.

Abstract

from arXiv · show

Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

1 INTRODUCTION

GPT and OPT models achieve strong language-modeling performance but are difficult to use because their scale creates substantial inference and storage costs. GPTQ addresses the open challenge of accurate, efficient one-shot quantization at higher compression rates.

  • Motivation: GPT3-175B has approximately 175 billion parameters and requires 326GB in float16, exceeding the capacity of a single high-end GPU for inference.This necessitates more complex and expensive multi-GPU deployments.
  • Motivation: Existing accurate low-bitwidth compression methods often require retraining or are difficult to scale to billion-parameter models.Basic round-to-nearest methods scale to GPT-175B but fail to preserve accuracy at higher compression rates.
  • Contribution: GPTQ quantizes models with hundreds of billions of parameters to 3 or 4 bits per parameter without significant accuracy loss.OPT-175B and BLOOM-176B can be quantized in approximately four GPU hours with minimal perplexity increase.
  • Contribution: GPTQ supports robust 2-bit and ternary quantization and enables compressed OPT-175B inference on a single NVIDIA A100 GPU or two NVIDIA A6000 GPUs.The implementation also reports approximately 3.25× and 4.5× speedups on A100 and A6000 GPUs, respectively.
  • Novelty: GPTQ is presented as the first method to quantize highly accurate language models with hundreds of billions of parameters to 3–4 bits per component.The paper notes tradeoffs among perplexity, bit-width, and original model size.
  • Limitations: The current method does not speed up the multiplication itself and does not include activation quantization.The stated limitation arises from lacking mainstream hardware support for mixed-precision operands; activation quantization can be added using orthogonal techniques.

2 RELATED WORK

Related work distinguishes retraining-based quantization from one-shot post-training methods and highlights the difficulty of applying accurate compression to massive language models. GPTQ targets this gap with a more accurate quantizer that remains efficient at large scale.

  • Quantization methods: Training-time quantization typically relies on extensive retraining or finetuning, whereas post-training methods quantize pretrained models using modest data and computation.Post-training approaches are especially relevant when full training or finetuning of massive models is expensive.
  • Post-training quantization: Prior post-training methods mainly developed for vision models use layer-wise or small-block reconstruction objectives to determine quantized weights.Examples include AdaRound, BitSplit, and AdaQuant.
  • Large-model quantization: Large-language-model methods such as ZeroQuant, LLM.int8(), and nuQmm select quantization granularity but ultimately use round-to-nearest weight quantization.This design choice helps maintain acceptable runtimes for very large models.
  • GPTQ: GPTQ implements a substantially more complex and accurate quantizer efficiently at large model scale.The paper reports more than doubling compression relative to prior techniques at similar accuracy.

3 BACKGROUND

The background formulates post-training quantization as layer-wise reconstruction and describes OBQ as a second-order, greedy procedure for compensating quantization errors. GPTQ builds on OBQ while modifying it to make large-language-model quantization tractable.

  • Layer-Wise Quantization: GPTQ quantizes each linear layer by finding quantized weights that minimize reconstruction error on layer inputs from a small calibration set.The method assumes a fixed quantization grid while allowing individual weights to move freely.
  • Optimal Brain Quantization: GPTQ builds on Optimal Brain Quantization and introduces major modifications that provide more than three orders of magnitude computational speedup.OBQ is presented as the starting point for solving the layer-wise quantization problem.
  • Optimal Brain Quantization: OBQ independently processes each row, quantizing one weight at a time while updating unquantized weights to compensate for the incurred error.The greedy choice selects the next weight according to its additional quantization error.
  • Optimal Brain Quantization: After quantizing a weight, OBQ updates the inverse Hessian efficiently by removing the corresponding row and column through one-step Gaussian elimination.This avoids expensive full recomputation of the inverse.
  • Scalability: OBQ can quantize medium-sized models but has cubic input dependency O(d_row · d_col^3), making billion-parameter models extremely expensive.Its vectorized implementation handles multiple rows in parallel, yet the runtime scaling remains prohibitive.

4 THE GPTQ ALGORITHM

GPTQ accelerates layer-wise post-training quantization by processing columns in shared blocks, using inverse-Hessian information and Cholesky reformulation to maintain accuracy and numerical stability. Its optimizations reduce runtime and address GPU memory-throughput bottlenecks, enabling quantization of very large models.

  • Step 1: Arbitrary Order Insight: Quantizing all rows in the same column order reduces inverse-Hessian update complexity from O(drow · dcol^3) to O(max {drow · dcol^2, dcol^3}), a factor of min {drow, dcol}.For larger models, this reduction amounts to several orders of magnitude.
  • Step 2: Lazy Batch-Updates: GPTQ lazily batches 128 columns, restricting intermediate updates to each block before globally updating the weights and inverse-Hessian matrix.The procedure exploits the fact that later-column updates do not affect the current column’s final rounding decision.
  • Step 2: Lazy Batch-Updates: Lazy batching addresses the memory-throughput bottleneck and provides an order of magnitude speedup for very large models in practice.It improves GPU utilization without reducing the theoretical amount of computation.
  • Step 3: Cholesky Reformulation: Cholesky reformulation precomputes the required inverse-Hessian rows using a numerically stable decomposition, with mild dampening improving robustness on huge models.The method leverages optimized Cholesky kernels for additional speedup.
  • The Full Algorithm: The full algorithm quantizes each column within a block, records normalized quantization errors, updates weights inside the block, then updates all remaining weights.Algorithm 1 uses inverse Hessian H−1 = (2XX⊤+ λI)−1 and a specified blocksize B.

5 EXPERIMENTAL VALIDATION

GPTQ is evaluated against post-training baselines across model sizes, precisions, language tasks, runtime, and practical inference. It maintains strong accuracy at 3–4 bits, scales to very large models, supports extreme quantization, and enables faster generation.

  • Small-model validation: GPTQ remains competitive with state-of-the-art post-training methods on small vision and language models while taking under 1 minute instead of approximately 1 hour.On ResNet and smaller language models, GPTQ is comparable at 4 bits and competitive at 3 bits, while substantially reducing quantization time.
  • Runtime: GPTQ quantizes 1–3 billion-parameter models in minutes and 175-billion-parameter models in a few hours on one NVIDIA A100 GPU.The runtime is substantially more favorable for scaling than methods whose smaller-model runtimes extrapolate to hundreds of hours at 175B.
  • Language generation: At 4 bits, GPTQ loses only 0.03 perplexity on OPT-175B, while RTN loses 2.2 points; at 3 bits, RTN collapses but GPTQ retains reasonable perplexity.Across OPT and BLOOM language-generation evaluations, larger models generally appear easier to quantize, with OPT-66B as an exception.
  • 175-billion-parameter models: At 4 bits, GPTQ is within 0.25 perplexity of full precision on BLOOM-176B and OPT-175B; at 3 bits, it loses 0.3–0.6 points on most tasks.Grouping improves accuracy further: group-size 1024 adds approximately 0.02 bits, while group-size 128 adds approximately 0.15 bits.
  • Practical speedups: A 3-bit OPT-175B model fits on one 80GB A100, compared with five such GPUs for FP16 execution and three for LLM.int8().The quantized model occupies approximately 63GB, while the maximum 2048-token key-value history adds approximately 9GB.
  • Practical speedups: 3-bit OPT-175B generation is approximately 3.25× faster on one A100 and 4.5× faster on two A6000 GPUs than the corresponding FP16 baselines.The reported A6000 latency falls from 589 milliseconds to 130 milliseconds per token.
  • Grouping: Grouping reduces accuracy losses for medium models at 4 bits and improves the largest models at 3 bits by determining group parameters during quantization.The group parameters use the most current updated weights, making grouping compatible with GPTQ’s layerwise procedure.
  • Extreme quantization: At approximately 2.2 bits, GPTQ increases WikiText2 perplexity by less than 1.5 points, while ternary OPT-175B quantization reaches 9.20 perplexity.Smaller group sizes improve 2-bit results and enable ternary values, which may suit custom hardware implementations.

6 SUMMARY AND LIMITATIONS

GPTQ accurately compresses very large language models to 3 and 4 bits, improving usability and enabling end-to-end speedups with low accuracy loss. The authors also identify technical and evaluation-scope limitations.

  • GPTQ is an approximate second-order method that compresses some of the largest publicly available models to 3 and 4 bits.
  • The compressed models provide significant usability improvements and end-to-end speedups at low accuracy loss.
  • Speedups come from reduced memory movement rather than computational reductions.
  • The study focuses on generative tasks and does not consider activation quantization.

7 ETHICS STATEMENT

The authors identify no significant ethical implications arising directly from GPTQ's technical details, while noting unresolved questions about secondary measures and broader accessibility. They specifically call for studying compression's effects on bias.

  • GPTQ uses a tiny amount of randomly chosen calibration data and is task-agnostic.
  • The study evaluates leading accuracy metrics such as perplexity, leaving secondary measures, particularly bias effects, for further study.
  • Making inference on extremely large language models more accessible increases the need to understand their power and limitations.

8 REPRODUCIBILITY STATEMENT

The supplementary materials provide code to reproduce the paper's experiments, including model compression, evaluation, inference benchmarking, and ZeroShot experiments.

  • The release includes code for compressing all OPT and BLOOM models to 2, 3, and 4 bits.
  • It includes code for evaluating perplexity of the quantized models.
  • The materials provide a 3-bit CUDA kernel and compressed-inference benchmarking features.
  • The release also contains code for ZeroShot experiments.
  • A README supplies sample commands and instructions for running the scripts.

A.1 ADDITIONAL COMPARISON WITH OBQ

The appendix adds a comparison of GPTQ with OBQ on BERT-base/SQuAD and OPT-125M/WikiText2, with the results presented in Table 8.

  • The comparison evaluates GPTQ relative to OBQ on BERT-base/SQuAD and OPT-125M/WikiText2.

A.2 EXPERIMENT DETAILS

The experiments evaluate language-model perplexity and zero-shot task performance, while timing tests measure inference using distributed consecutive-layer execution across GPUs. The kernels target low-batch generative inference, with additional results reported across perplexity and accuracy benchmarks.

  • Evaluation: Perplexity is computed by concatenating validation data, splitting it into 2048-token segments, and exponentiating the average next-token log-probability.Each model uses its default HuggingFace tokenizer.
  • Evaluation: Zero-shot evaluation follows the EleutherAI harness, scoring each sample separately without padding.
  • Timing experiments: Timing experiments distribute consecutive layer chunks across GPUs using the HuggingFace/accelerate setup, with communication below 5% of total runtime even on eight GPUs.Reported speedups therefore primarily reflect quantized-matrix full-precision vector-product kernels.
  • Timing experiments: Non-quantized overheads, including framework, attention, residual, and LayerNorm operations, remain identical between quantized models and the baseline.
  • Timing experiments: The kernels target batchsize 1 generative inference, whereas larger-batch non-generative workloads may require decompressing matrices before matrix-matrix computation.For batchsize 16×1024 tokens, decompression takes under 1.5ms on an A100 and under 3ms on an A6000, compared with 76ms and 365ms for the subsequent OPT-175B FC2 layer.
  • Additional results: Additional experiments report OPT and BLOOM perplexity on PTB and C4, plus accuracy on LAMBADA, PIQA, ARC-easy, ARC-challenge, and StoryCloze.C4 evaluations use calibration data sampled from C4 training data and are therefore not fully zero-shot.
Loading 2210.17323v2…