PaperlayerView exact revision on arXiv

Source-linked AI summary

SqueezeLLM: Dense-and-Sparse Quantization

Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer

arXiv:2306.07629v4cs.CLcs.LG

TL;DR

Generative LLM deployment is constrained by resource demands, while prior low-precision quantization can degrade performance. SqueezeLLM combines sensitivity-based non-uniform quantization with Dense-and-Sparse decomposition, achieving ultra-low-bit compression with strong accuracy and inference results.

  • Problem

    Deploying large LLMs is difficult because their resource requirements are high, and prior low-precision quantization often degrades performance.

  • Method

    SqueezeLLM is a post-training framework combining sensitivity-based non-uniform quantization with Dense-and-Sparse decomposition for weight compression.

  • Results

    Across evaluated models and datasets, SqueezeLLM consistently outperforms previous state-of-the-art methodologies, including at 3-bit precision.

  • Takeaways & Limitations

    SqueezeLLM demonstrates that memory-focused weight quantization can support ultra-low-precision LLM inference without compromising model performance.

  • Takeaways & Limitations

    The comparison with SpQR relies on reported speedup numbers because SpQR does not release its kernel implementation.

Abstract

from arXiv · show

Generative Large Language Models (LLMs) have demonstrated remarkable results for a wide range of tasks. However, deploying these models for inference has been a significant challenge due to their unprecedented resource requirements. This has forced existing deployment frameworks to use multi-GPU inference pipelines, which are often complex and costly, or to use smaller and less performant models. In this work, we demonstrate that the main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, specifically for single batch inference. While quantization has emerged as a promising solution by representing weights with reduced precision, previous efforts have often resulted in notable performance degradation. To address this, we introduce SqueezeLLM, a post-training quantization framework that not only enables lossless compression to ultra-low precisions of up to 3-bit, but also achieves higher quantization performance under the same memory constraint. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format. When applied to the LLaMA models, our 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x as compared to the state-of-the-art methods with the same memory requirement. Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup compared to the baseline. Our code is available at https://github.com/SqueezeAILab/SqueezeLLM.

1. Introduction

SqueezeLLM addresses the resource and memory-bandwidth challenges of generative LLM inference with post-training quantization designed for ultra-low precision without compromising model performance.

  • LLaMA-65B requires at least 130GB of RAM in FP16, exceeding current GPU capacity and making deployment costly and complex.
  • Generative LLM inference is primarily bottlenecked by memory bandwidth rather than compute.The bottleneck motivates reducing parameter memory traffic through quantization.
  • SqueezeLLM combines sensitivity-based non-uniform quantization with Dense-and-Sparse decomposition in a post-training framework.The decomposition retains outlier and sensitive values in sparse full precision, while the remaining weights are quantized.
  • 7.75 perplexity replaces 28.26 for 3-bit LLaMA-7B on C4 with sensitivity-based non-uniform quantization.This compares the proposed method with round-to-nearest uniform quantization.
  • 0.45% sparse extraction further improves LLaMA-7B perplexity from 7.75 to 7.58 on C4.The sparse component stores selected outlier values in full precision.
  • SqueezeLLM is evaluated across language modeling, problem-solving, and instruction-following benchmarks, with deployed models achieving up to 2.4× latency gains over FP16 on A6000 GPUs.

2. Related Work

Prior work largely uses weight-only uniform quantization or targets activation outliers, while SqueezeLLM introduces dynamic sensitivity-aware quantization and applies Dense-and-Sparse decomposition to weight matrices.

  • GPTQ, AWQ, and SpQR use weight-only quantization schemes, whereas SqueezeLLM employs non-uniform quantization and targets preserving the model’s final output.
  • SqueezeLLM’s dynamic representation accounts for both weight distributions and value sensitivity, unlike a static NF datatype assuming normally distributed weights.
  • Earlier outlier-aware methods primarily address activation outliers, while SqueezeLLM’s Dense-and-Sparse quantization addresses weight outliers with floating-point activations.
  • SqueezeLLM uses sensitivity-aware non-uniform quantization to achieve precise quantization with 0.05% or zero sparsity, supporting smaller models and faster inference than higher-sparsity approaches.
  • SqueezeLLM is presented as the first application of Dense-and-Sparse decomposition to weight matrices for improving quantization performance.Its sparse matrix incorporates both outlier and sensitive values.
  • Reducing weight precision lowers modeled LLaMA-7B runtime at sequence lengths 128 and 2048 even when activations remain unreduced.

3. Memory Wall

Single-batch generative LLM inference is memory-bound because matrix-vector operations provide little data reuse, so reducing weight traffic is more valuable than reducing computation.

  • Generative LLM inference has extremely low arithmetic intensity because matrix-vector operations limit data reuse across tokens.
  • The single-batch scope is important: compute can become important for large-batch inference.
  • On an A5000, peak computational throughput is 290× higher than peak memory bandwidth, illustrating the Memory Wall disparity.
  • Roofline modeling shows latency decreasing linearly as weight bit precision is reduced while computations remain in FP16.
  • Quantizing only weights can reduce model size and improve speed because weight loading dominates while dequantization and FP16 computation are relatively small costs.

4. Methodology

SqueezeLLM combines sensitivity-based non-uniform quantization with Dense-and-Sparse decomposition to preserve important weights and make low-bit quantization more accurate. Its method uses second-order sensitivity information, isolates outliers, and supports efficient compressed inference.

  • Sensitivity-Based Non-Uniform Quantization: Uniform quantization is sub-optimal because LLM weights are non-uniform and inference arithmetic remains in FP16.The method instead places quantized values according to weight distributions and sensitivity.
  • Sensitivity-Based Non-Uniform Quantization: 1-dimensional k-means clustering selects k centroids to represent weights with low-bit precision, with k=8 for 3-bit quantization.The centroids represent the quantized values assigned to weight clusters.
  • Sensitivity-Based Non-Uniform Quantization: Second-order sensitivity weights quantization errors by their impact on the final loss, pulling centroids toward more sensitive values.The Hessian provides the importance weighting, while Fisher information approximates it efficiently; diagonalization assumes negligible cross-weight interactions.
  • Sensitivity-Based Non-Uniform Quantization: 7.75 perplexity versus 28.26 for round-to-nearest uniform quantization on C4 demonstrates the benefit of sensitivity-based non-uniform quantization for 3-bit LLaMA-7B.The approach places centroids near sensitive values while minimizing quantization error.
  • Dense-and-Sparse Quantization: Dense-and-Sparse decomposition separates outliers into a sparse matrix S and quantizes the narrower-range dense matrix D.The decomposition is W = D + S; the sparse component can retain full precision with low overhead.
  • Efficient Inference: 3/4-bit CUDA LUT-based kernels dequantize compressed weights piece-by-piece during matrix-vector multiplication to reduce memory-bandwidth use.Compressed indices reference FP16 lookup-table values for the non-uniform quantization bins.

5. Evaluations

SqueezeLLM is evaluated across language modeling, instruction-following, accuracy, perplexity, latency, and memory settings. It consistently preserves or improves model quality at low precision while reducing inference cost.

  • Evaluation Setup: Evaluations cover LLaMA models on C4 and WikiText2, Vicuna models on MMLU and instruction-following tasks, and latency on an A6000 GPU.
  • Perplexity Results: 4-bit dense-only quantization reduces model size by 3.95× with only ∼0.1 perplexity degradation on C4 and WikiText2.
  • Perplexity Results: 1.80/1.22 perplexity points separate SqueezeLLM from GPTQ on C4/WikiText2 at 3 bits and a 5.29× compression rate.
  • Latency and Memory: GPTQ activation ordering creates a latency penalty from distributed memory accesses, whereas removing it trades latency gains for substantial perplexity degradation.
  • Dense-and-Sparse Results: 0.45% sparsity yields around 0.2 perplexity improvement at 3 bits and keeps deviation from FP16 below 0.5 perplexity points.
  • Comparison with Baselines: At comparable model sizes, 0.45% sparse SqueezeLLM outperforms grouped GPTQ and AWQ by up to ∼0.3 perplexity at 3 bits.
  • Larger and Instruction-Tuned Models: The performance trend extends to larger models, while 3-bit SqueezeLLM also improves zero-shot MMLU and instruction-following results over competing methods.
  • Latency and Memory: LUT-based SqueezeLLM reaches up to 2.4× speedup over FP16, while adding 0.45% FP16 parameters retains up to 2.2× speedup with roughly 10% added latency.

6. Conclusion

The paper presents SqueezeLLM to address memory-bound generative LLM inference through ultra-low-precision quantization. Across evaluated models and tasks, it reports negligible generation-performance degradation and consistent improvements over prior methods.

  • SqueezeLLM targets the Memory Wall in memory-bound generative LLM inference.
  • The framework combines sensitivity-based non-uniform quantization with Dense-and-Sparse decomposition to support ultra-low precision.
  • Evaluations span language modeling, problem-solving, and instruction-following capabilities across a wide range of models and datasets.
  • The method consistently outperforms previous state-of-the-art methodologies in the reported evaluations.

B. Experiment Setup (Details)

The experiments evaluate SqueezeLLM against established post-training quantization baselines across multiple models, datasets, tasks, and deployment measurements. Implementation details include channelwise lookup-table quantization and specialized latency profiling on an A6000 GPU.

  • Models and Datasets: Language-modeling experiments use LLaMA, LLaMA2, and OPT on C4 and WikiText2, while additional evaluations measure MMLU and instruction-following ability.
  • Baselines: SqueezeLLM is compared with RTN, GPTQ, AWQ, and SpQR, using GPTQ with activation ordering unless otherwise specified.
  • Latency Profiling: Latency and peak memory are measured for 128- and 1024-token generation on an A6000 using the Torch CUDA profiler.
  • Latency Profiling: Because an official grouped GPTQ implementation is unavailable, the study uses an optimized kernel based on GPTQ-For-LLaMA.
  • Latency Profiling: Latency comparisons with SpQR rely on reported speedups because its kernel implementation is not publicly available, while AWQ uses the GPTQ kernel without activation ordering.

C. Data Skew in Per-channel Sparsity Pattern

The appendices examine skewed sparsity patterns and the design choices behind Dense-and-Sparse quantization. They find that sensitivity extraction and outlier handling provide better trade-offs than grouping alone or hybrid grouping.

  • Data Skew: Nonzero entries are heavily skewed across output channels, with a few channels containing substantially more nonzeros than others.
  • Sensitivity Ablation: The FP16 baseline for the 3-bit LLaMA-7B sensitivity ablation has perplexity 7.08.
  • Sparse Kernels: Balanced hybrid sparse kernels assign equal nonzero counts per thread to mitigate workload imbalance from skewed sparse distributions.
  • Sensitivity Ablation: Sensitivity-based weighted clustering is compared with sensitivity-agnostic non-weighted k-means at 0%, 0.05%, and 0.45% sparsity.
  • Sensitivity Ablation: Perplexity gains diminish beyond 0.05% sensitive-value sparsity, motivating a fixed 0.05% setting in the experiments.
  • Grouping Comparison: The pure Dense-and-Sparse decomposition achieves better size-perplexity trade-offs than grouping or a hybrid grouping-and-sparsity approach.
  • Grouping Comparison: Grouping can isolate outliers within groups but adds lookup-table storage overhead under non-uniform quantization.

D.4. Comparison of Optimization Objectives for Non-uniform Quantization: Minimizing Layer-wise Perturbation versus Final Output Perturbation

SqueezeLLM’s final-output perturbation objective outperforms layer-wise perturbation minimization for 3-bit LLaMA-7B quantization. The ablations also show that sensitivity-based non-uniform quantization and carefully limited sparse decomposition improve perplexity, while additional sparsity has diminishing returns.

  • Optimization objectives: Final-output perturbation minimization outperforms layer-wise perturbation minimization by up to around 0.3 perplexity points across sparsity levels.The comparison uses 3-bit LLaMA-7B quantization evaluated on C4.
  • Non-uniform quantization: Sensitivity-based non-uniform quantization improves perplexity over uniform quantization across all tested bitwidths and sparsity levels.The comparison uses LLaMA-2 7B perplexity results on WikiText2.
  • Sparsity levels: Higher sparsity consistently improves performance, but returns diminish because few weights are outliers or sensitive and added sparse storage increases average bitwidth.The main experiments identify 0.45% sparsity as sufficient for performance gains.
  • Bitwidth versus sparsity: Increasing dense-component bitwidth improves perplexity more than increasing sparsity in the compared LLaMA-2 7B configurations.A 4-bit model without sparsity outperforms 3-bit counterparts with 1.5% and 2.5% sparsity at similar or larger model sizes.
  • Quantization cost: Quantization requires more memory than GPTQ because SqueezeLLM computes the Fisher information matrix, although this is a one-time cost taking 2–3 minutes for a 65B model.The Fisher-information computation supports loss-perturbation minimization, while GPTQ minimizes individual-layer activation perturbation.

E.3. Data Efficiency

The supplied passages describe comparisons with QuIP and OmniQuant, but do not provide a substantive data-efficiency discussion despite the section title. They report that SqueezeLLM generally performs better under matched model-size or memory constraints, with specific exceptions at 2 bits.

  • Data efficiency: SqueezeLLM uses a calibration set of 100 samples in the paper, while as few as 10 examples are typically sufficient for desired quantization performance.The passage contrasts this with the 100–200 calibration points reported for GPTQ and AWQ.
  • Comparison with QuIP: Dense-only SqueezeLLM consistently outperforms QuIP across model sizes and quantization bitwidths.At 2 bits, adding 0.1% sparsity, equivalent to an additional 0.05 bit, significantly improves performance over QuIP.
  • Comparison with OmniQuant: SqueezeLLM generally outperforms OmniQuant at the same model size and memory constraints for 4-bit and 3-bit quantization.The comparison covers LLaMA and LLaMA2 models on WikiText2.
  • 2-bit comparison: At 2 bits, OmniQuant without grouping outperforms dense-only SqueezeLLM on the 13B and 70B models.The passage attributes this to OmniQuant’s learnable clipping ranges, which account for outliers more effectively in that setting.

G. Additional Hardware Profiling Results

Additional profiling shows that SqueezeLLM’s custom kernels retain substantial speedups across hardware and that its performance trends extend across supplementary model and evaluation settings.

  • Kernel profiling: 1.5–2.5× speedups over the FP16 matrix-vector kernel are achieved on an A100 across model sizes without additional optimization or tuning.The kernels include configurations with and without outliers.
  • Downstream evaluation: SqueezeLLM consistently outperforms baseline quantization methods with the same model size on five-shot MMLU evaluation.The supplementary results use Vicuna v1.1 and v1.3 models.

I. Limitations

The paper’s empirical evaluation primarily targets generation tasks and decoder architectures, while broader architectural coverage remains untested. Its hardware performance modeling also relies on simplified simulation assumptions.

  • Scope of evaluation: Thorough assessments are not yet available for encoder-only, encoder-decoder, or other neural network architectures.The proposed ideas are described as not inherently limited to decoder architectures, but the empirical evidence remains narrower.
  • Hardware modeling: Hardware performance modeling uses a simulation-based roofline approach that makes simplified assumptions about the inference pipeline.This limits how directly the modeled hardware results can represent complete inference behavior.
Loading 2306.07629v4…