PaperlayerView exact revision on arXiv

Source-linked AI summary

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, Baris Kasikci

arXiv:2310.19102v3cs.LG

TL;DR

LLM serving faces rising compute, memory, and energy costs, while prevalent quantization schemes underuse modern low-bit hardware. Atom combines mixed-precision, fine-grained, and dynamic quantization with serving-system and kernel co-design, achieving up to 7.73× throughput over FP16 with less than 1.4% zero-shot accuracy loss. Its results support accurate, high-throughput low-bit serving, while the paper notes that effective-bit compression is not the primary metric for weight-activation serving.

  • Problem

    Rising LLM inference demand and model complexity increase serving costs, while existing quantization schemes do not fully exploit modern GPUs’ lower-bit arithmetic.

  • Method

    Atom uses mixed-precision quantization with reordering, fine-grained group quantization, dynamic activation quantization, KV-cache quantization, and integrated low-bit serving kernels.

  • Results

    7.73× throughput enhancement over FP16 and less than 1.4% zero-shot accuracy loss are achieved in an end-to-end serving framework.

  • Takeaways & Limitations

    Atom achieves high LLM serving throughput while maintaining high accuracy through low-bit weight-activation quantization and hardware-efficient serving integration.

  • Takeaways & Limitations

    Atom does not use effective bit as a primary metric because serving benefits mainly come from computation efficiency rather than memory reduction.

Abstract

from arXiv · show

The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization in the serving context. Atom improves end-to-end throughput (token/s) by up to $7.7\times$ compared to the FP16 and by $2.5\times$ compared to INT8 quantization, while maintaining the same latency target.

1 INTRODUCTION

Atom addresses the throughput and efficiency limits of LLM serving by combining 4-bit weight-activation quantization with mixed-precision and fine-grained designs. Integrated kernels and serving optimizations preserve accuracy while substantially improving end-to-end throughput.

  • Motivation: LLM serving costs are rising as model capabilities and inference demand grow, increasing providers’ compute, memory, and energy burdens.Reported ChatGPT usage exceeds 100 million users and 1 billion monthly website accesses, while operational costs approach $1 million daily.
  • Motivation: Existing weight-only and INT8 weight-activation schemes do not fully exploit modern GPUs’ lower-bit arithmetic, limiting serving throughput.A100 GPUs provide 1248 TOPS for INT4 and 624 TOPS for INT8, compared with 312 TFLOPS for FP16 Tensor Cores.
  • Atom’s Design: Atom co-designs serving execution with customized low-bit CUDA kernels, fused operators, and quantized KV-cache handling to reduce quantization and memory-movement overheads.The design reorders data for regular memory access and fuses quantization with existing operators.
  • Atom’s Design: Atom combines mixed-precision quantization, channel reordering, fine-grained group quantization, and dynamic activation quantization to retain accuracy at low precision.High-precision values preserve salient outliers, while group quantization and dynamic activation quantization reduce quantization error.
  • Evaluation: Atom’s 4-bit weight-activation quantization causes negligible accuracy loss, with a 1.4% average zero-shot accuracy drop and 0.3 WikiText2 perplexity increase for Llama-65B.These results are reported relative to the corresponding unquantized evaluations and outperform prior methods at the same precision.
  • Evaluation: 7.7× higher end-to-end serving throughput than FP16 and 2.5× higher throughput than W8A8 are achieved at similar latency.Atom also improves throughput by up to 5.5× over W4A16.

2 BACKGROUND

Quantization maps high-precision tensors to discrete low-bit values using scale and zero-point parameters. Finer-grained quantization improves precision but increases overhead, while symmetric quantization avoids extra matrix-multiplication cross-terms.

  • Quantization determines scale and zero-point parameters before calculating a discrete low-bit tensor.Uniform asymmetric quantization approximates floating-point values with integers for hardware efficiency.
  • Dynamic quantization computes scale and zero-point parameters during inference using runtime statistics, whereas static quantization uses calibration data.
  • Weight-activation quantization applies quantization to both activation and weight matrices, unlike weight-only quantization, which quantizes only weights.
  • Symmetric quantization avoids three additional cross-terms required by asymmetric weight-activation matrix multiplication.
  • Per-tensor, per-channel, and per-group quantization provide progressively finer granularity with increasing precision and overhead.Atom adopts group quantization and dedicated kernels to manage its overhead.

3 PERFORMANCE ANALYSIS OF LOW-BIT LLM SERVING

LLM serving throughput is constrained by memory-bound decode operations and by dense and self-attention layers that dominate execution time. Low-bit weight-activation quantization accelerates both layers, while Atom addresses outlier accuracy and hardware-efficiency challenges through reordering and specialized designs.

  • Decode-stage GEMV is heavily memory-bound because it loads a large weight matrix for relatively few multiplications, reducing GPU utilization and throughput.
  • Batching converts dense-layer operations into larger GEMMs that become compute-bound, but decode-stage self-attention cannot benefit because requests do not share KV-cache.
  • Over 90% of processing time is consumed by dense and self-attention layers, motivating quantization of both components.
  • Weight-activation quantization improves dense-layer compute throughput and self-attention throughput by using low-bit arithmetic and reducing KV-cache movement.
  • Atom reorders outlier channels and uses higher precision for them, reducing quantization error while preserving regular memory access.
  • Weight-only quantization does not improve dense-layer throughput or self-attention because weights require dequantization and the KV-cache remains unquantized.

4 DESIGN

Atom combines mixed precision, fine-grained group quantization, dynamic activation quantization, and kernel fusion to preserve accuracy while using low-bit hardware efficiently. It also quantizes the KV-cache because self-attention is memory-bound and the cache is relatively amenable to quantization.

  • 4.1 Mixed-precision quantization: Atom reorders activation outlier channels and correspondingly reorders weights, preserving computation while enabling regular memory access.Reorder indices are determined during offline calibration, while activation reordering is performed dynamically.
  • 4.1 Mixed-precision quantization: Atom uses mixed precision, retaining salient activations and weights at higher precision while quantizing other values at low bit width.Outlier channels are handled separately because their large dynamic range increases quantization error.
  • 4.2 Fine-grained group quantization: Fine-grained group quantization reduces local quantization error but introduces dequantization overhead that Atom incorporates into a fused GEMM pipeline.Group products are computed on low-bit Tensor Cores, then dequantized and accumulated with FP16 units.
  • 4.2 Fine-grained group quantization: With group size 128 and 128 high-precision channels, Atom has an effective bit width of 4.251 on Llama-7b.The effective-bit metric includes quantization parameters and represents actual compression, but the paper emphasizes computation efficiency for weight-activation quantization.
  • 4.3 Dynamic quantization: Atom dynamically quantizes each activation matrix to match its input distribution and fuses the added operations into preceding operators.Dynamic quantization avoids distribution mismatch from static calibration while keeping runtime overhead negligible relative to dense and self-attention layers.
  • 4.4 KV-cache quantization: Atom quantizes the KV-cache in low bit precision because self-attention is memory-bound and the cache exhibits fewer outliers than activation matrices.The cache is dequantized before FP16 calculation, reducing memory movement while preserving accuracy through asymmetric quantization at attention-head granularity.

5 EVALUATION

Atom’s evaluation measures both accuracy and efficiency, using standard accuracy metrics, bottom-up performance profiling, and ablations of its design choices. The implementation also includes calibration and preprocessing procedures with nontrivial cost for large models.

  • Evaluation methodology: Atom is evaluated with generation perplexity and zero-shot accuracy for model quality, and with per-kernel, end-to-end throughput, and latency for efficiency.Ablation studies examine how individual techniques affect the efficiency–accuracy trade-off.
  • Evaluation configuration: Atom uses 128 sampled WikiText2 sentences for calibration, selects 128 outlier channels by square-sum magnitude, and evaluates with group size 128.Weights and activations use symmetric quantization, while the KV-cache uses asymmetric quantization.
  • Evaluation configuration: 4 hours are required to quantize the Llama-65B model layer-by-layer on a single RTX Ada 6000.This preprocessing includes weight quantization and outlier identification.

5.2 Accuracy evaluation

Atom maintains substantially better accuracy than prior low-bit weight-activation methods across zero-shot tasks and perplexity benchmarks. The evaluation covers Llama models using INT4 and INT3 quantization across multiple datasets and tasks.

  • Evaluation setup: The accuracy evaluation uses INT4 and INT3 weight-activation quantization on Llama models, measuring perplexity and zero-shot accuracy.Perplexity is measured on WikiText2, PTB, and C4, while zero-shot evaluation uses multiple common-sense tasks.
  • Baselines: The baselines are SmoothQuant, OmniQuant, and QLLM, evaluated using their supported or implemented post-training quantization configurations.The comparison includes W8A8 and W4A4 settings where applicable.
  • Zero-shot accuracy: Atom’s W4A4 average accuracy loss versus FP16 is 2.3%, 1.7%, 0.4%, and 1.4% for Llama models at 7B, 13B, 30B, and 65B.Prior methods lose 9.6% to 23.8% under the same settings across the six zero-shot tasks.
  • Perplexity: Atom increases Llama-65B perplexity by less than 0.4 on WikiText2, PTB, and C4 under W4A4 quantization.Under W3A3, Atom has an average perplexity increase of 2.3 for Llama-65B.

5.3 Efficiency evaluation

Atom improves efficiency at both kernel and serving levels by using fused low-bit computation and reducing memory pressure in self-attention. Its end-to-end throughput advantage is accompanied by lower decode latency across batch sizes.

  • Evaluation setup: Atom’s efficiency evaluation uses W4A4 quantization on an RTX 4090 with CUDA 11.3 because NVIDIA GPUs support efficient INT4 arithmetic.Performance is profiled from individual kernels through end-to-end serving.
  • 5.3.1 Kernel evaluation: At batch size 512, Atom’s matrix multiplication achieves 3.4× and 1.9× speedup over FP16 and INT8 kernels.The fused GEMM operator is compared with W8A8 and W4A16 baselines.
  • 5.3.1 Kernel evaluation: At batch size 128, Atom’s self-attention achieves 1.8× speedup over INT8 quantization and 3.5× over FP16.Lower-bit KV-cache representation reduces memory usage proportionally in the memory-bound setting.
  • 5.3.2 End-to-end serving: Atom outperforms the other quantization methods in end-to-end throughput across all evaluated batch sizes.The evaluation integrates Atom, W8A8, and W4A16 into the Punica serving framework using a ShareGPT-derived workload.
  • 5.3.2 End-to-end serving: Using the same amount of memory, Atom achieves 7.73× throughput versus FP16 and 2.53× versus INT8 quantization.Atom supports larger batch sizes under fixed memory, while weight-only quantization remains limited by FP16 dense-layer computation and KV-cache movement.
  • 5.3.2 End-to-end serving: At batch size 256, Atom’s average decode latency remains below 100 ms, and at batch size 64 it is lower than INT8 or FP16 at batch size 8.Latency excludes queuing time and is measured as average decoding time per token.

5.4 Ablation study of quantization techniques

Atom’s ablations show that mixed precision and fine-grained group quantization improve accuracy, while fused implementations retain high throughput despite quantization overhead.

  • Accuracy ablation: Keeping outlier channels in FP16 significantly reduces perplexity, while quantizing them to INT8 adds only 0.05 perplexity.This supports mixed precision as an effective treatment for activation outliers.
  • Accuracy ablation: Fine-grained group quantization produces a major perplexity reduction, while clipping and GPTQ each lower perplexity by 0.09.KV-cache quantization causes a slight 0.12 perplexity increase.
  • Efficiency ablation: 770 TOPS is achieved after fusing group dequantization, while mixed-precision fusion reaches 900 TOPS and adds 8% overhead.A pure INT4 GEMM reaches nearly 980 TOPS before quantization operations are fused.
  • Efficiency ablation: 18% higher than the theoretical INT8 throughput is the fused GEMM kernel’s measured performance despite group-quantization overhead.The result indicates that fused low-bit execution remains faster than the INT8 throughput limit.
  • Efficiency ablation: 25% to 35% higher performance is achieved by Atom than the baseline across batch sizes from 16 to 256.The comparison measures inference latency for layer normalization and GEMM operations.

6 DISCUSSION

Atom generalizes across newer transformer-based models, MoE models, and FP4 representations while maintaining high accuracy and outperforming baselines.

  • Generality on models: Atom’s mixed precision for outliers and fine-grained quantization for normal values generalize to Llama-2 and Mixtral.For MoE models, Atom adapts by using different reorder indices for different experts’ FFNs.
  • Generality on models: Atom maintains high accuracy and outperforms baselines on Llama-2 and Mixtral after adapting reorder indices for different experts’ FFNs.The paper also reports that shared reorder indices across experts can provide similar accuracy with greater efficiency.
  • Generality on data formats: Atom maintains accuracy similar to INT4 when quantizing both weights and activations into FP4.The authors conclude that INT4 and FP4 have similar representation capability.

7 RELATED WORK

Related work improves LLM serving through batching, memory management, offloading, and quantization, while Atom focuses on low-bit weight-activation execution for serving.

  • LLM serving: Continuous batching, page tables, and offloading improve GPU utilization, KV-cache memory utilization, and support for larger serving batches.These approaches target serving throughput and resource management rather than the quantization operator design itself.
  • Weight-only quantization: Weight-only methods reduce weight precision and preserve salient or sensitive weights, but their serving benefit is limited when batching amortizes weight-loading costs.Under batching, dense layers become compute-bound, motivating weight-activation quantization.
  • Weight-activation quantization: Weight-activation quantization is more challenging because activation outliers require mixed precision or mathematical transformations.Prior methods include LLM.INT8, SmoothQuant, and related transformations for handling outliers.

8 CONCLUSION

Atom combines low-bit quantization and hardware-efficient execution to improve LLM serving throughput while preserving accuracy.

  • Conclusion: Up to 7.73× throughput enhancement over the FP16 baseline is achieved with less than 1.4% zero-shot accuracy loss.Atom combines mixed precision with reordering, fine-grained group quantization, dynamic quantization, and KV-cache quantization.
  • Conclusion: Atom leverages underlying hardware efficiently to achieve both high accuracy and high throughput for LLM serving.The conclusion presents this combination as the method’s central outcome.
Loading 2310.19102v3…