PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, Ping Luo

arXiv:2308.13137v3cs.LGcs.CL

TL;DR

LLM deployment is constrained by substantial computational and memory requirements, while existing PTQ methods degrade at extremely low bit widths. OmniQuant addresses this gap by optimizing learnable clipping and equivalent-transformation parameters within block-wise differentiable PTQ. It reports strong results across weight-only and weight-activation settings while retaining PTQ efficiency and hardware compatibility.

  • Problem

    Existing PTQ methods reduce LLM memory and computation costs but hand-craft quantization parameters, causing performance degradation especially in extremely low-bit settings.

  • Method

    OmniQuant freezes full-precision weights and uses Learnable Weight Clipping and Learnable Equivalent Transformation within block-wise differentiable error minimization.

  • Results

    OmniQuant outperforms prior methods across diverse quantization configurations, model families, and sizes, including average accuracy gains of +4.99% ∼+11.80% for LLaMA W4A4 quantization.

  • Takeaways & Limitations

    OmniQuant supports efficient low-bit quantization for weight-only and weight-activation models, including instruction-tuned LLMs, while its added parameters can be absorbed for hardware compatibility.

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized natural language processing tasks. However, their practical deployment is hindered by their immense memory and computation requirements. Although recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM, they hand-craft quantization parameters, leading to low performance, especially in extremely low-bit quantization. To tackle this issue, we introduce an Omnidirectionally calibrated Quantization (\textbf{OmniQuant}) technique for LLMs, which achieves good performance in diverse quantization settings while maintaining the computational efficiency of PTQ by efficiently optimizing various quantization parameters. OmniQuant comprises two innovative components including Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET). LWC modulates the extreme values of weights by optimizing the clipping threshold. Meanwhile, LET tackles activation outliers by shifting the challenge of quantization from activations to weights. Operating within a differentiable framework using block-wise error minimization, OmniQuant can optimize the quantization process efficiently for both weight-only and weight-activation quantization. For instance, the LLaMA-2 model family size 7-70B can be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using 128 samples. Extensive experiments validate OmniQuant's superior performance across diverse quantization configurations such as W4A4 (4-bit weight, 4-bit activation), W6A6, W4A16, W3A16, and W2A16. Additionally, OmniQuant demonstrates effectiveness in instruction-tuned models and delivers notable improvements in inference speed and memory reduction on real devices. Codes are available at \url{https://github.com/OpenGVLab/OmniQuant}.

1 INTRODUCTION

OmniQuant addresses the degradation of low-bit LLM quantization by optimizing a small set of learnable parameters within an efficient PTQ framework. Its LWC and LET components support weight-only and weight-activation quantization while preserving PTQ efficiency.

  • Motivation: Low-bit settings such as W2A16 and W4A4 cause significant performance degradation in existing quantization methods.These methods perform better in settings including W4A16 and W8A8 but struggle at lower bit widths.
  • OmniQuant: OmniQuant freezes full-precision weights and optimizes a restrained set of learnable quantization parameters instead of training the whole model.This differentiable approach is designed to preserve PTQ time and data efficiency while incorporating gradient updates.
  • OmniQuant: Learnable Weight Clipping optimizes clipping thresholds to modulate extreme weight values, while Learnable Equivalent Transformation addresses activation outliers through mathematically equivalent transformations.Together, the components make weights and activations more amenable to quantization.
  • Optimization: Block-wise quantization-error minimization sequentially quantizes one layer before moving to the next and enables efficient optimization with SGD.The framework jointly optimizes LWC and LET for both weight-only and weight-activation quantization.
  • Efficiency: LLaMA-2 models from 7B to 70B can be quantized on one A100-40G GPU using 128 samples in 1–16 hours.The reported time depends on model size.
  • Results: OmniQuant outperforms prior methods across multiple quantization settings, model families, and model sizes, with speedup and memory reduction demonstrated on real devices.The reported settings include W4A16, W3A16, W2A16, W6A6, and W4A4.

3 OMNIQUANT

OmniQuant addresses low-bit LLM quantization by learning a small set of parameters within a block-wise differentiable error-minimization framework. Its LWC and LET components jointly improve quantization of weights, activations, and attention operations without optimizing the full-precision model weights.

  • LLM quantization is challenged by activation outlier channels and weight quantization error, especially for low-bit settings.
  • OmniQuant freezes full-precision weights and learns only additional quantization parameters through block-wise quantization error minimization.Blocks are optimized sequentially with a simple gradient-based procedure rather than jointly optimizing the entire model.
  • Learnable Weight Clipping: LWC learns upper and lower clipping strengths to determine an effective weight range, with γ and β constrained to [0, 1].The resulting quantizer uses a learnable normalization factor and clamps values to the target N-bit integer range.
  • Learnable Weight Clipping: LWC reduces weight quantization difficulty, and experiments report that it significantly outperforms previous weight-only quantization techniques.When γ = 1 and β = 1, LWC reduces to vanilla MinMax quantization.
  • Learnable Equivalent Transformation: LET learns channel-wise scaling and shifting to make activations more quantization-friendly while transferring quantization difficulty to weights.The transformation is applied to linear layers and extended to attention operations, including low-bit quantization of Q/K/V matrices.
  • Learnable Equivalent Transformation: Equivalent transformations can be absorbed into preceding normalization or linear layers and fused into original weights, avoiding additional inference parameters or costs.The method excludes the second FFN linear layer and omits explicit V transformation because of feature sparsity and an inverse output-projection transformation.

4 EXPERIMENTS

OmniQuant is evaluated across weight-only, weight-activation, instruction-tuned, and deployment settings using multiple models, tasks, and baselines. It consistently improves low-bit quantization performance while reducing memory use and increasing inference speed in supported deployments.

  • 4.1 SETTINGS: Experiments cover weight-only and weight-activation quantization across LLaMA, OPT, Falcon, and instruction-tuned LLaMA-2-chat models.Evaluations use perplexity on WikiText2, PTB, and C4, plus zero-shot accuracy on PIQA, ARC, BoolQ, and HellaSwag.
  • 4.2 WEIGHT-ONLY QUANTIZATION RESULTS: OmniQuant consistently outperforms prior weight-only methods across OPT, LLaMA-1, and LLaMA-2 under W2A16 through W4A16 configurations.The gains extend to both channel-wise and group-wise quantization and become more pronounced at lower bit sizes.
  • 4.3 WEIGHT-ACTIVATION QUANTIZATION RESULTS: At W4A4, OmniQuant improves average zero-shot accuracy by +4.99% ∼+11.80% across models.For LLaMA-7B, it surpasses LLM-QAT by +6.22%.

5 CONCLUSION

OmniQuant advances low-bit weight-only and weight-activation quantization by retaining full-precision weights and adding learnable parameters. It maintains PTQ-like training efficiency, improves evaluated language-generation and zero-shot results, supports instruction-tuned LLMs, and allows added parameters to be absorbed for hardware compatibility.

  • 5 CONCLUSION: OmniQuant retains original full-precision weights while adding learnable parameters for low-bit weight-only and weight-activation quantization.Its learnable weight clipping and equivalent transformation optimize weights and activations for quantization.
  • 5 CONCLUSION: OmniQuant maintains training efficiency comparable to existing PTQ methods while incorporating gradient updates.
  • 5 CONCLUSION: OmniQuant outperforms current methods in language generation and zero-shot tasks and is suited for instruction-tuned LLMs.
  • 5 CONCLUSION: OmniQuant’s added parameters can be absorbed, supporting hardware compatibility for quantized models.

A1 OVERALL ALGORITHM

OmniQuant calibrates each model block sequentially by initializing and optimizing learnable clipping and equivalent-transformation parameters against full-precision outputs. It then folds the learned transformation and clipping into the quantized block before propagating quantized activations.

  • A1 OVERALL ALGORITHM: The block-wise calibration strategy initializes learnable parameters, trains them, transforms the model, and then quantizes each block.
  • A1 OVERALL ALGORITHM: For each block, OmniQuant updates full-precision inputs, initializes clipping and equivalent-transformation parameters, and optimizes them with gradient descent.The algorithm uses paired quantized and full-precision inputs during training.
  • A1 OVERALL ALGORITHM: The output of the procedure is a quantized model produced from calibration data and a pretrained LLM.
  • A1 OVERALL ALGORITHM: After optimization, the algorithm applies LET, quantizes with LWC, updates the quantized input, and proceeds to the next block.

A2 DISTINCTION OF EXISTING EQUIVALENT TRANSFORMATION METHODS

OmniQuant broadens equivalent transformations and learns their parameters through end-to-end gradient descent. Compared with prior methods, it supports scaling and shifting, attention matrix multiplication, and both weight-only and weight-activation quantization.

  • A2 DISTINCTION OF EXISTING EQUIVALENT TRANSFORMATION METHODS: SmoothQuant and AWQ use channel-wise scaling, whereas OP+ and OmniQuant use both channel-wise scaling and shifting.
  • A2 DISTINCTION OF EXISTING EQUIVALENT TRANSFORMATION METHODS: Prior methods apply equivalent transformations only to linear layers, while OmniQuant also transforms matrix multiplication within attention.This enlarges the transformation solution space and facilitates quantization of Q and K.
  • A2 DISTINCTION OF EXISTING EQUIVALENT TRANSFORMATION METHODS: SmoothQuant uses predefined migration strength and AWQ and OP+ use heuristic grid searches, whereas OmniQuant optimizes transformation parameters through end-to-end gradient descent.
  • A2 DISTINCTION OF EXISTING EQUIVALENT TRANSFORMATION METHODS: OmniQuant’s LWC and LET design supports both weight-only and weight-activation quantization, unlike prior methods designed for one of these scenarios.

A3 ABLATION STUDIES

The ablations examine OmniQuant’s component combination, LET design and placement, initialization, Softmax quantization, training strategy, calibration data, and runtime settings. Across these studies, LWC and LET generally contribute positively, with their joint differentiable training providing the strongest performance.

  • Component combination: The LWC+LET combination is evaluated against variants replacing LET with SmoothQuant or removing components.The reported study emphasizes that LET and LWC are integrated through a differentiable framework rather than a simple additive combination.
  • Component efficacy: Both LWC and LET improve performance, while LET is essential for weight-activation quantization.Removing LET for W4A4 markedly increases perplexity because activation outliers remain difficult to quantize; LET is disabled for LLaMA weight-only quantization because its advantage is limited.
  • LET design: Channel-wise shifting and attention transformation both enhance weight-activation quantization, while attention transformation provides comparatively smaller incremental benefit.These modifications are evaluated against the equivalent transformation used in SmoothQuant.
  • LET placement: All four tested LET placements improve performance, especially the [ln1, (q proj, k proj, v proj)] pair.The second linear layer’s LET is excluded because feature sparsity after the nonlinear layer causes unstable gradients.
  • Initialization: Careful initialization of LET scaling and shifting improves final performance, with scaling initialization more important than shifting initialization.The experiments compare SmoothQuant- and Outlier Suppression+-based initialization with scaling initialized to 1 and shifting initialized to 0.
  • Training and calibration: Simultaneous LWC and LET training outperforms iterative alternatives, while most configurations converge within 20 epochs and W2A16 requires longer training.The training-time ablation identifies W2A16 as an exception to the typical convergence schedule; calibration experiments use 128 segments of 2048 tokens from WikiText2 and compare Pile and c4.

A4 TRAINING TIME

OmniQuant’s runtime remains practical for PTQ, although it requires more training time than GPTQ. Weight-only quantization is faster than weight-activation quantization because fewer learnable parameters are activated.

  • Runtime comparison: OmniQuant requires approximately 5× the training time of GPTQ but remains markedly faster than QAT methods.For LLaMA, only learnable weight clipping is activated for weight-only quantization, reducing its training time relative to weight-activation quantization.

A5 PERFORMANCE ANALYSIS

The performance analysis attributes OmniQuant’s behavior to learnable clipping, activation homogenization, and differentiable block-wise optimization. These mechanisms reduce quantization error and support favorable low-bit trade-offs.

  • Learnable weight clipping: LWC substantially decreases the l1 distance between quantized and full-precision weights and activations.Its lower activation distance indicates improved balance between outlier and regular values.
  • Learned clipping scales: Lower-bit settings learn more pronounced clipping, with 50% of W2A16g128 clipping scales larger than 0.95 versus 70% for W3A16g128.Per-channel W3A16 shows an approximately normal clipping-scale distribution, whereas group-wise quantization is long-tailed.
  • Learnable equivalent transformation: LET reduces activation outlier magnitudes and homogenizes activation distributions compared with the original activation and SmoothQuant.The analysis reports an outlier magnitude reduction from 70 to 2 for SmoothQuant before describing further homogenization from LET.
  • Quantization error: OmniQuant significantly reduces block-wise quantization loss compared with grid-search methods such as AWQ and Outlier Suppression+.The comparison is presented as evidence for gradient-based optimization.
  • Scaling laws: OmniQuant is reported to make 3-bit quantization achieve a comparable bits-perplexity trade-off to 4-bit quantization.The scaling-law analysis concerns model-bit constraints and the choice of model size and quantization bits.

A6 COMPARISONS WITH CLIPPING-BASED METHODS

The paper compares LWC with clipping-based alternatives including PACT and LSQ, using the same broader quantization pipeline. It also reports comparisons with SpQR and SqueezeLLM.

  • Clipping-based methods: LWC adaptively determines the weight clipping threshold by scaling the original minimum and maximum values.PACT directly determines the clipping threshold, whereas LSQ directly derives the scaling factor and zero-point.
  • Clipping-based methods: WikiText2 perplexity is compared for clipping-based methods by replacing LWC with LSQ or PACT in OmniQuant’s pipeline.The comparison is designed to evaluate these methods under a shared pipeline.
  • Other quantization methods: OmniQuant is additionally compared with SpQR and SqueezeLLM.The supplied comparison reference identifies the methods but does not state their numerical outcomes.

A7 COMPARISONS WITH OTHER WEIGHT-ONLY QUANTIZATION METHODS

OmniQuant uses asymmetrically uniform quantization and achieves comparable performance to mixed-precision alternatives while supporting both weight-only and weight-activation quantization.

  • OmniQuant is an asymmetrically uniform quantization method compared with AWQ and GPTQ.
  • OmniQuant achieves comparable performance to SpQR and SqueezeLLM, although it performs slightly worse than SqueezeLLM.
  • Uniform INT quantization gives OmniQuant simplicity and flexibility across weight-only and weight-activation quantization, unlike SpQR and SqueezeLLM.

A8 FULL RESULTS

The full results cover multiple datasets, model families, quantization settings, and evaluation formats. The performance overview reports improved perplexity–model-size trade-offs for OmniQuant variants.

  • The full evaluation includes perplexity results for weight-only and weight-activation quantization across LLaMA and OPT model families.Reported datasets include WikiText2, PTB, and C4, with additional results for Falcon-180B and LLaMA-7B MMLU accuracy.
  • The evaluation covers asymmetric-bit configurations including W4A8, W4A6, and W8A4.
  • 0.81 lower perplexity is achieved at equivalent model size, while matching perplexity requires only 0.33x the model size in the performance overview.Figure A6 compares W4A16g128 and W3A16g128 variants across three model families.
Loading 2308.13137v3…