PaperlayerView exact revision on arXiv

Source-linked AI summary

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han

arXiv:2405.04532v3cs.CLcs.AIcs.LGcs.PF

TL;DR

Existing INT4 approaches can incur substantial GPU dequantization overhead, limiting their benefits for efficient LLM serving. QServe addresses this with QoQ W4A8KV4 quantization and system co-design, achieving reported throughput improvements over TensorRT-LLM.

  • Problem

    Existing INT4 quantization methods incur substantial GPU runtime overhead during dequantization, limiting performance gains for large-batch LLM serving.

  • Method

    QServe combines QoQ W4A8KV4 quantization—4-bit weights, 8-bit activations, and 4-bit KV caches—with progressive quantization, SmoothAttention, and GPU system optimizations.

  • Results

    1.88× speedup over TRT-LLM is reported for Llama-2-7B, with comparable dequantization overhead and 2× higher throughput from INT8 tensor-core computation.

  • Takeaways & Limitations

    W4A8KV4 is presented as a precision choice that reduces memory footprint while enabling efficient GPU computation for LLM serving.

  • Takeaways & Limitations

    Enabling saturation during dequantization can severely reduce computation throughput, by as much as 67%.

Abstract

from arXiv · show

Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented by the QServe inference library that achieves measured speedup. The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores. Building upon this insight, in QoQ algorithm, we introduce progressive quantization that can allow low dequantization overhead in W4A8 GEMM. Additionally, we develop SmoothAttention to effectively mitigate the accuracy degradation incurred by 4-bit KV quantization. In the QServe system, we perform compute-aware weight reordering and take advantage of register-level parallelism to reduce dequantization latency. We also make fused attention memory-bound, harnessing the performance gain brought by KV4 quantization. As a result, QServe improves the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of LLM serving by 3x. Code is available at https://github.com/mit-han-lab/omniserve.

1 INTRODUCTION

Existing INT4 methods do not reliably improve cloud LLM serving because GPU dequantization overhead can offset lower precision. QServe addresses this with QoQ W4A8KV4 quantization and system-level optimizations, improving throughput across GPUs and models.

  • Problem: 20–90% dequantization overhead in existing INT4 methods can prevent lower bit precision from accelerating LLM inference.W4A16 dequantizes weights, while W4A4 dequantizes weights and partial sums with shared FP16 scales.
  • Approach: QoQ quantizes weights, activations, and KV caches to W4A8KV4 precision.The configuration uses 4-bit weights, 8-bit activations, and 4-bit KV caches.
  • Approach: Progressive quantization keeps W4A8 GEMMs on INT8 tensor cores, while SmoothAttention mitigates accuracy loss from KV4 quantization.The method first quantizes weights to 8 bits, then quantizes those intermediates to 4 bits; SmoothAttention shifts activation-quantization difficulty from keys to unquantized queries.
  • System: QServe reduces dequantization and CUDA-core overhead through register-level parallelism, compute-aware weight reordering, and memory-bound fused attention.These optimizations support efficient execution of W4A8 GEMM and KV4 attention.
  • Results: QServe achieves 1.2–2.4× higher throughput than TensorRT-LLM on A100 and 1.5–3.5× higher throughput on L40S.The evaluation covers seven widely used LLMs and also reports 2.5–2.9× gains over Atom and QuaRot on A100.

2 BACKGROUND

LLM serving uses transformer layers with attention, feed-forward, and normalization components across prefilling and decoding stages. Quantization represents tensors with discrete levels using scaling and zero-point parameters at different granularities.

  • Large Language Models: LLMs are causal transformer models whose layers combine attention, feed-forward, and normalization components.Each layer receives an N × HD tensor, where N is the number of input tokens.
  • Large Language Models: Prefilling processes all prompt tokens together, whereas decoding processes one token per prompt at a time.Thus, N > 1 for each request during prefilling and N = 1 during decoding.
  • Attention: Attention projects inputs into queries, keys, and values, combines them with cached features, and computes attention outputs.The KV cache stores features from S previous tokens before attention computation.
  • Integer Quantization: Integer quantization maps floating-point tensors to discrete n-bit levels using a scaling factor and zero point.The resulting representation can be dequantized using the same parameters.
  • Integer Quantization: Quantization granularity ranges from shared tensor parameters to per-channel, per-token, and per-group parameters.Per-group quantization uses different scale and zero-point values for every g columns within each row.

3 MOTIVATION

Roofline analysis motivates W4A8KV4 as a balance between W4A16’s memory advantages and W8A8’s compute advantages, while KV4 improves attention bandwidth. Existing W4A4 systems struggle because dequantization and long GEMM main loops shift work onto low-throughput CUDA cores.

  • W4A4 Limitations: W4A4 systems can lag behind W8A8 despite higher theoretical peak performance because their practical GPU efficiency is limited.The gap reflects both inefficient runtime implementations and the difficulty of mapping per-group W4A4 GEMMs onto GPUs.
  • Roofline Analysis: W4A8 can combine W4A16’s memory efficiency with W8A8’s INT8 tensor-core throughput across batch sizes.W4A16 is favored when m < 78, while W8A8 performs better when m > 78 on the A100 roofline.
  • GEMM Structure: LLM-serving GEMMs have small m and large n and k, making the reduction-dimension main loop long and sequential.This structure applies particularly to decoding, where m is the number of sequences.
  • Roofline Analysis: KV4 effectively increases memory bandwidth and offers 2× peak attention performance over KV8.Attention accounts for more than 50% of runtime at batch=64, creating end-to-end speedup opportunity.
  • Dequantization Overhead: W4A16 and Atom-W4A4 incur different conversion costs: INT4-to-FP16 weights versus INT32-to-FP32 partial sums.These conversions contribute to main-loop overhead on CUDA cores.
  • QServe Design: QServe uses progressive quantization, weight dequantization, and four-way register-level parallelism to keep computation on INT8 tensor cores and reduce main-loop overhead.The design favors weight dequantization over partial-sum dequantization because it creates lower register pressure.

4 QOQ QUANTIZATION

QoQ combines progressive group quantization for efficient W4A8 GEMM with SmoothAttention and additional transformations to preserve accuracy under W4A8KV4 quantization.

  • Progressive Group Quantization: QoQ reduces W4A8 GEMM dequantization overhead by keeping computation on INT8 tensor cores and using register-level parallelism.QServe chooses weight rather than partial-sum dequantization and decodes four INT4 weights simultaneously.
  • Progressive Group Quantization: Progressive group quantization first applies per-channel INT8 quantization, then per-group INT4 quantization to the intermediate weights.The method uses channel-wise FP16 scales, group-wise INT4 scales, and zero points before W4A8 computation.
  • Progressive Group Quantization: A protective INT8 range prevents intermediate dequantized weights from overflowing the 8-bit representation range.The range is shrunk to [-119, 119]; saturation would otherwise reduce throughput by as much as 67%.
  • SmoothAttention: SmoothAttention mitigates KV4 accuracy loss by scaling down outlier Key channels while leaving Queries unquantized.The method concentrates smoothing on Keys, and α = 0.5 is reported as sufficient in practice.
  • Activation optimizations: QServe further suppresses activation outliers by rotating block inputs and smoothing block intermediate activations with channel-wise transformations.The rotation is unitary and can be absorbed into preceding weights, while output modules use per-channel smoothing factors.

5 QSERVE SERVING SYSTEM

QServe combines system-level optimizations to reduce W4A8 GEMM overhead and accelerate KV4 attention. Its design targets pointer arithmetic, dequantization, memory access, and CUDA-core bottlenecks.

  • Design goals: QServe’s system design focuses on reducing GEMM main-loop overhead and accelerating KV4 attention to realize W4A8KV4 throughput benefits.These are the two explicit directions of the serving-system design.
  • Runtime and precision mapping: QServe’s W4A8 GEMMs use INT8 tensor cores and produce FP16 outputs, while attention computes in FP16 on CUDA cores.Activation quantization is fused into preceding normalization or activation layers, with a separate node before attention output projection.
  • KV4 attention: QServe uses SmoothAttention-related transformations to address activation and KV-cache quantization challenges, while requiring per-head dynamic KV4 scaling factors and zero points.The supplied system passages describe per-head dynamic KV quantization and CUDA-core attention bottlenecks from fused KV-cache processing.
  • W4A8 GEMM optimization: Compute-aware weight reordering stores weights in computation order, reducing pointer arithmetic and enabling high-bandwidth 128-bit-per-thread loads without runtime reordering.The method partitions GEMMs into 32×32 tiles and concatenates channels into words matching thread usage.
  • W4A8 GEMM optimization: Progressive quantization and subtraction-after-multiplication move zero-point correction into the epilogue, reducing main-loop dequantization overhead while enabling register-level parallelism.The progressive method prevents intermediate overflow, and per-channel scaling is fused into the GEMM epilogue.

6 EVALUATION

The evaluation measures QoQ accuracy and QServe throughput across quantized LLMs, GPUs, and serving baselines. Results show competitive accuracy and substantial throughput improvements under matched memory constraints.

  • Experimental setup: QoQ is evaluated with per-token INT8 activations, per-head asymmetric INT4 KV caches, and per-channel or grouped progressive weight quantization.The W4A8KV4 g128 variant uses group size 128, while W4A8KV4 denotes the per-channel counterpart.
  • Accuracy evaluation: QoQ increased Llama-2-7B WikiText2 perplexity by at most 0.16 versus W8A8 SmoothQuant and W4A16 AWQ, while outperforming Atom and QuaRot in the reported comparisons.QoQ improved perplexity by up to 0.49 versus W4A4 QuaRot.
  • Accuracy evaluation: QoQ achieved 4.82% higher Winogrande accuracy than QuaRot and introduced only 1.03%, 0.89%, and 0.40% accuracy loss versus FP16 for Llama-2 at 7B, 13B, and 70B.The evaluation also reports minimal degradation on long-context performance relative to BF16.
  • Throughput evaluation: QServe achieved 1.2–2.4× higher throughput than TensorRT-LLM on A100 and 1.5–3.5× higher throughput on L40S across the evaluated models.The comparison uses maximum achievable throughput under fixed memory constraints, with 1024 input and 512 output tokens.
  • Throughput analysis: For Llama-2-7B, QServe’s 1.88× speedup over TensorRT-LLM combines 1.45× same-batch speedup with 1.3× from enlarged batch size.For Llama-2-13B, batch-size scaling and single-batch speedup each contribute to a reported 1.7× improvement.
  • Ablation and breakdown: KV4 attention optimization reduced A100 latency from 0.48ms to 0.28ms for a 64×1024 input and produced a 1.7× end-to-end improvement.The reductions combine bit tricks, simpler control flow, FP16 products, and asynchronous prefetching.

7 RELATED WORK

Prior work spans quantization methods and systems for efficient LLM deployment, including approaches targeting memory traffic, scheduling, KV-cache management, programming, and compilation.

  • Quantization: Weight-only quantization benefits memory-bound edge workloads through faster weight loading but does not accelerate computation-bound cloud-serving scenarios.The related-work discussion contrasts edge-device benefits with high-traffic batched cloud inference.
  • Serving systems: LLM serving systems improve deployment through scheduling, batching, virtualized or blocked KV-cache management, programming primitives, and compiler acceleration.Examples include Orca, vLLM, SGLang, LMDeploy, MLC-LLM, and TensorRT-LLM.
  • LLM accelerators: Domain-specific LLM accelerators have explored pruning and quantization for transformer and attention workloads.Representative systems include A3, ELSA, SpAtten, GOBO, EdgeBERT, and DOTA.

8 CONCLUSION

QServe co-designs the QoQ algorithm and inference system for W4A8KV4 quantization, achieving higher throughput than TensorRT-LLM when serving seven representative LLMs on A100 and L40S GPUs.

  • 8 CONCLUSION: Progressive quantization enables W4A8 GEMM operations on INT8 tensor cores, while SmoothAttention reduces accuracy loss from KV4 quantization.The system also uses register-level parallelism for INT4-to-INT8 dequantization and compute-aware weight reordering to reduce pointer-arithmetic overhead.
  • 8 CONCLUSION: Up to 2.4-3.5× higher throughput over TensorRT-LLM is achieved when serving seven representative LLMs on A100 and L40S GPUs.The reported result covers both the A100 and L40S evaluation platforms.

A.1 Abstract

The artifact provides scripts and dependencies for reproducing the paper’s crucial experiments, with an x86 64 CPU host and at least one A100 or L40S NVIDIA GPU.

  • A.1 Abstract: The artifact contains scripts and dependencies needed to reproduce the paper’s crucial experiments.A pre-built Docker image is provided to simplify environment setup.
  • A.1 Abstract: Experiments require an x86 64 CPU host and at least one A100 or L40S NVIDIA GPU.The stated hardware requirement covers either an A100 or L40S GPU.

A.2 Artifact check-list (meta-information)

The artifact checklist specifies the benchmark programs, models, runtime environment, hardware, metrics, licenses, and approximate resource requirements for QServe evaluation.

  • A.2 Artifact check-list (meta-information): The artifact includes efficiency benchmarking code for QServe and baseline systems such as TensorRT-LLM.The reported output and metric are LLM generation throughput in tokens per second.
  • A.2 Artifact check-list (meta-information): The listed models are Llama-3-8B, Llama-2-7B, Mistral-7B, and Llama-2-13B, with no dataset required.Benchmarks run on NVIDIA GPUs, while some preprocessing runs on the host CPU.
  • A.2 Artifact check-list (meta-information): The runtime uses NVIDIA Container Toolkit, with Docker as the workflow framework and an x86 64 host plus an A100 or L40S GPU.The checklist identifies the run-time state as N/A.
  • A.2 Artifact check-list (meta-information): Approximately 512G of disk space is required, with around 1 hour to prepare the workflow and around 1 hour for QServe benchmarks.TensorRT-LLM benchmarks require approximately 2-4 GPU hours, depending on GPU performance and the number of tasks.
  • A.2 Artifact check-list (meta-information): The artifact is publicly available under the Apache License 2.0, with data listed under the MIT license and an archived DOI provided.The archived DOI is 10.5281/zenodo.14991385.

A.3 Description

The artifact uses Docker and GPU-compatible infrastructure to run QServe and TensorRT-LLM throughput benchmarks, while reporting reference results with reproducibility caveats and omitting accuracy evaluation from the setup.

  • A.3 Description: A pre-built Docker image packages QServe, TensorRT-LLM, and the necessary dependencies for artifact evaluation.The recommended setup runs experiments inside a GPU-supported Docker container.
  • A.3 Description: The environment requires an x86 64 host with an A100 or L40S GPU and a GPU-compatible Docker runtime.The artifact provides a docker run command using all available GPUs.
  • A.3 Description: Generation throughput for QServe and TensorRT-LLM is measured using the provided benchmark commands, with results written to CSV files.The QServe and TensorRT-LLM evaluations use separate benchmark scripts and output paths.
  • A.3 Description: Table 6 provides reference QServe throughput numbers, but absolute measurements may vary across identical GPU platforms while relative acceleration ratios should remain consistent.This makes relative acceleration the more stable comparison across machine conditions.
  • A.3 Description: Users can modify scripts to test different models and batch sizes, while accuracy evaluation is omitted from the simplified environment setup.Accuracy results can be reproduced with the open-source deepcompressor library.
Loading 2405.04532v3…