PaperlayerView exact revision on arXiv

Source-linked AI summary

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song Han

arXiv:2211.10438v7cs.CLcs.AIcs.LG

TL;DR

LLM quantization must reduce serving cost without sacrificing accuracy or hardware efficiency, but activation outliers make INT8 activation quantization difficult. SmoothQuant uses an offline, mathematically equivalent scaling transformation to shift quantization difficulty to weights, enabling efficient W8A8 inference with substantial speed and memory gains.

  • Problem

    Activation outliers make LLMs difficult to quantize accurately, while hardware-efficient methods cannot preserve performance on large models.

  • Method

    SmoothQuant applies an offline per-channel scaling transformation that migrates quantization difficulty from activations to weights for W8A8 quantization.

  • Results

    SmoothQuant maintains FP16 accuracy across evaluation datasets and achieves up to 1.56× speedup with halved memory usage in FasterTransformer.

  • Takeaways & Limitations

    SmoothQuant enables lossless 8-bit weight and activation quantization for LLMs up to 530B parameters and serving a 530B model within one 8-GPU node.

  • Takeaways & Limitations

    Efficient GEMM kernels restrict vector-wise scaling to outer dimensions, so per-channel activation quantization is not hardware-compatible.

Abstract

from arXiv · show

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.

1 Introduction

LLMs impose substantial memory, computation, and latency costs, motivating quantization; however, activation outliers make efficient INT8 quantization difficult. SmoothQuant addresses this by moving quantization difficulty from activations to weights through an equivalent transformation and achieves substantial serving gains.

  • Motivation: GPT-3’s 175B parameters require at least 350GB of FP16 memory, making LLM serving costly and hardware-intensive.The paper gives examples requiring 8×48GB A6000 GPUs or 5×80GB A100 GPUs for inference.
  • Motivation: INT8 quantization of weights and activations can halve GPU memory usage and nearly double matrix-multiplication throughput versus FP16.Quantization targets GEMM operations in linear layers and BMM operations in attention.
  • Challenge: Activation outliers make large LLMs difficult to quantize, while mixed-precision handling preserves outliers but is hard to implement efficiently on accelerators.Prior approaches either lose accuracy on very large models or introduce hardware-unfriendly decomposition.
  • SmoothQuant: SmoothQuant uses a mathematically equivalent offline scaling transformation to migrate quantization difficulty from activations to weights and smooth channel magnitudes.This makes both adjusted weights and activations more quantization-friendly.
  • Results: 1.56× speedup and halved memory usage were achieved in FasterTransformer, while serving a 530B model within one 8-GPU node.The PyTorch implementation reached up to 1.51× speedup and 1.96× memory saving.

2 Preliminaries

The paper formulates symmetric INT8 uniform quantization using a step size derived from the tensor’s maximum absolute value. It distinguishes quantization granularities and explains why W8A8 is needed to exploit integer hardware kernels.

  • Uniform quantization: INT8 uniform quantization maps floating-point tensors to discrete integer levels using rounding and a quantization step size.The formulation assumes symmetry around zero for simplicity.
  • Uniform quantization: The step size is computed from the maximum absolute tensor value, preserving activation outliers that matter for accuracy.The paper describes both offline static and runtime dynamic estimation of this step size.
  • Quantization granularity: Per-tensor quantization uses one step size, whereas per-token, per-channel, and group-wise schemes use progressively finer-grained scaling.Per-token scaling applies to activation tokens, per-channel scaling to weight output channels, and group-wise scaling to channel groups.
  • Hardware execution: W8A8 quantization is required to use widely supported integer kernels for faster Transformer inference, beyond the memory savings from INT8 weights alone.The relevant hardware includes NVIDIA GPUs, Intel CPUs, and Qualcomm DSPs.
  • Quantization granularity: Per-tensor quantization is the most efficient to implement, while vector-wise INT8 GEMM scaling is restricted to outer dimensions T and Co.Scaling along the inner input-channel dimension Ci is not used for efficient GEMM kernels.

3 Review of Quantization Difficulty

LLM activations are harder to quantize than weights because large, persistent outliers waste effective quantization levels. Although per-channel activation quantization can preserve accuracy, it is incompatible with efficient INT8 GEMM kernels.

  • Activation outliers: Weights are relatively uniform and can be quantized to INT8 or even INT4 without reported accuracy degradation, unlike activations.This asymmetry motivates transferring quantization difficulty away from activations.
  • Activation outliers: LLM activation outliers can reach approximately 100× the magnitude of most values, causing large errors under per-tensor quantization.Non-outlier channels may receive only 2–3 effective quantization levels.
  • Hardware constraint: Per-channel activation quantization preserves accuracy but does not map well to hardware-accelerated INT8 GEMM kernels.Such kernels cannot tolerate lower-throughput conversions or CUDA Core operations inserted into their high-throughput sequence.
  • Outlier structure: Outliers persist in a small fraction of fixed channels across tokens, making per-channel activation quantization more effective than per-token quantization.The cited evidence reports that simulated per-channel quantization bridges accuracy to the FP16 baseline.
  • Hardware constraint: Previous methods therefore use per-token activation quantization for linear layers, which remains only slightly better than per-tensor quantization.The hardware constraint prevents directly exploiting the more accurate per-channel activation scheme.

4 SmoothQuant

SmoothQuant smooths input activations by scaling channels offline while applying the inverse scaling to weights, preserving the linear layer mathematically. It applies this transformation and INT8 quantization selectively across Transformer computations to balance accuracy and inference efficiency.

  • SmoothQuant divides input activations by per-channel factors and scales weights in the reversed direction to preserve mathematical equivalence.The smoothing factors can be fused into previous layers offline, avoiding extra runtime kernel-call overhead.
  • Activation outliers are migrated to weights, leaving activations greatly smoothed while the weight distribution remains relatively flat.The transformation targets channels with large activation magnitudes while preserving a smooth weight distribution.
  • SmoothQuant obtains smoothing factors from calibration samples and performs the transformation offline, so runtime activations require no scaling.
  • The migration strength α controls how much quantization difficulty moves from activations to weights, with α = 0.5 often balancing the two.For models with more significant activation outliers, such as GLM-130B, a larger α such as 0.75 can be used.
  • The Transformer flow quantizes linear layers and attention BMM operators with INT8 while retaining FP16 for lightweight element-wise operations.This precision mapping covers compute-heavy operators while keeping ReLU, Softmax, and Layer-Norm in FP16 to balance accuracy and inference efficiency.
  • SmoothQuant’s efficiency levels progress from O1 to O3, with higher levels providing lower latency.

5 Experiments

Experiments show that SmoothQuant preserves accuracy across diverse LLM families, sizes, architectures, and instruction-tuned models while delivering substantial inference speedups and memory savings. Its benefits extend to models beyond 500B parameters and depend on choosing an appropriate quantization configuration.

  • Accuracy: SmoothQuant matches FP16 accuracy across all evaluation datasets and quantization schemes, whereas W8A8, ZeroQuant, and Outlier Suppression produce nearly random results.LLM.int8() also matches floating-point accuracy but uses floating-point outlier representations, creating latency overhead.
  • Accuracy: SmoothQuant maintains accuracy across OPT, BLOOM, GLM-130B, instruction-tuned OPT-IML-30B, LLaMA, Llama-2, Falcon, Mistral, and Mixtral models with minimal degradation.For GLM-130B, SmoothQuant-O1 matches FP16 accuracy and O3 degrades accuracy by 1%; for BLOOM-176B, O3 degrades average accuracy by 0.8%.
  • Accuracy: SmoothQuant works consistently across OPT model sizes, matching FP16 accuracy with INT8 quantization.Figure 7 reports accuracy preservation across different OPT scales using the most efficient SmoothQuant-O3 setting.
  • Efficiency: 1.56× lower execution latency and almost 2× lower memory usage are achieved with SmoothQuant-O3 in FasterTransformer compared with FP16.The method also achieves similar or better latency for larger distributed models while using half the number of GPUs.
  • Efficiency: 1.42× lower per-token decoding latency and halved memory footprints are reported for autoregressive LLM decoding.SmoothQuant consistently reduces decoding latency compared with FP16.
  • Scaling: SmoothQuant enables W8A8 quantization of MT-NLG 530B with negligible accuracy loss and serving on eight rather than sixteen GPUs.This places a model above 500B parameters within a single node of eight A100 80GB GPUs at similar latency.
  • Efficiency: Coarser quantization granularity lowers latency, while static quantization accelerates inference by avoiding runtime step-size calculation.SmoothQuant is faster than the FP16 baseline under all tested settings, whereas LLM.int8() is usually slower.
  • Configuration: Migration strength α must balance activation and weight difficulty: values below 0.4 burden activations, values above 0.6 burden weights, and 0.4–0.6 is the sweet spot.The ablation is conducted on OPT-175B with LAMBADA.

6 Related Work

Related work spans large-scale LLM development, quantization methods, and hardware-efficient inference. SmoothQuant-related results emphasize reduced latency and memory, while prior approaches face model-scale or hardware-efficiency constraints.

  • SmoothQuant reports up to 1.56× lower inference latency and nearly half the memory usage than FP16 in FasterTransformer.For smaller models, latency improves by up to 1.56×; larger OPT models can use half as many GPUs.
  • SmoothQuant can serve MT-NLG 530B with half the memory and a similar latency using half as many GPUs, within one node.
  • Large language models have expanded beyond 500B parameters, making inference increasingly expensive and challenging.
  • With coarser quantization settings, latency decreases; SmoothQuant is faster than FP16 across settings, whereas LLM.int8() is mostly slower.The comparison uses GPU latency with batch size 4.
  • Prior LLM quantization methods include weight-only, per-token or group-wise, and mixed INT8/FP16 schemes, with limitations in scale, accuracy, or hardware efficiency.ZeroQuant and nuQmm require customized CUDA kernels, while LLM.int8() uses mixed precision.
  • The migration strength α has a sweet spot: excessive migration makes weights difficult to quantize, while insufficient migration leaves activations difficult to quantize.

7 Conclusion

The conclusion presents SmoothQuant as a post-training method for lossless 8-bit weight-and-activation quantization in LLMs up to 530B parameters. Its integrations provide faster inference and substantially lower memory use, supporting lower-cost LLM serving.

  • SmoothQuant enables lossless 8-bit weight and activation quantization for LLMs up to 530B parameters.
  • SmoothQuant quantizes weights and activations for all LLM GEMMs, reducing inference latency and memory usage compared with mixed-precision activation quantization.
  • Up to 1.56× inference acceleration and half the memory footprint are achieved through PyTorch and FasterTransformer integrations.
  • SmoothQuant offers a turnkey approach for reducing LLM serving costs and broadening access to LLM applications.

A Discussion on Weight-Only Quantization

The paper distinguishes SmoothQuant’s W8A8 focus from weight-only quantization methods such as GPTQ. It discusses implementation differences, workload-dependent trade-offs, and the additional memory relevance of activation quantization for long-context batching.

  • The authors leave integration with GPTQ for potentially better weight quantization and W4A4 as future work.They connect this possibility to INT4 support on NVIDIA Hopper GPUs.
  • SmoothQuant studies W8A8 quantization to use INT8 GEMM kernels for higher throughput and faster inference.
  • Weight-only methods such as GPTQ convert quantized weights to FP16 during matmul and can accelerate generation through reduced data loading.
  • GPTQ’s kernel supports generation with batch size 1 but not context-stage or batch-based settings, complicating direct comparison with SmoothQuant.
  • GPTQ may suit small-token, memory-bounded inputs, whereas SmoothQuant may perform better for batching or context processing.
  • Long-context chatbot workloads can make KV-cache memory substantial, so activation quantization can reduce memory used for storing the cache.The cited discussion reports a 3TB KV cache under batch size 512 and context length 2048.
Loading 2211.10438v7…