PaperlayerView exact revision on arXiv

Source-linked AI summary

DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs

Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, Ying Wei

arXiv:2406.01721v3cs.CL

TL;DR

LLM quantization must handle activation outliers, especially sparse Massive Outliers that existing methods struggle to smooth. DuQuant combines rotation and permutation transformations to redistribute and smooth both outlier types, and it outperforms 4-bit baselines across models and tasks. Its main scope boundary is calibration-data selection, which remains based on randomly selected WikiText2 samples.

  • Problem

    Activation outliers enlarge quantization step sizes and hurt low-bit accuracy, while existing methods struggle with sparse, extremely large Massive Outliers.

  • Method

    DuQuant uses orthogonal rotation and zigzag permutation transformations to redistribute outliers across channels and blocks and smooth activations and weights.

  • Results

    DuQuant establishes state-of-the-art 4-bit weight-activation quantization results, surpassing QLLM by +9% and Atom by +5% in Commonsense QA across model sizes.

  • Takeaways & Limitations

    DuQuant simplifies low-bit quantization while improving performance and supporting faster, lower-memory LLM deployment.

  • Takeaways & Limitations

    Calibration uses 128 randomly selected WikiText2 samples, and more tailored calibration-data selection remains for future study.

Abstract

from arXiv · show

Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.

1 Introduction

LLM quantization is challenged by activation outliers that increase quantization step sizes and reduce accuracy. DuQuant addresses both Normal and Massive Outliers through rotation and permutation transformations, improving low-bit quantization results.

  • Activation outliers enlarge quantization step sizes and can cause significant accuracy loss in LLM quantization.
  • Normal Outliers persist across several channels and token sequences, while Massive Outliers occur at few tokens with extremely high magnitudes.
  • SmoothQuant struggles with Massive Outliers, while OmniQuant and AffineQuant exhibit training instability in their presence.
  • DuQuant redistributes activation outliers through orthogonal rotation and permutation transformations to facilitate quantization.
  • DuQuant outperforms existing 4-bit weight-activation quantization baselines across benchmarks, including 5% higher Commonsense QA performance across LLaMA sizes.

2 Motivation

The paper distinguishes Normal Outliers from Massive Outliers and examines how both complicate quantization. Massive Outliers are especially problematic because they are exceptionally large, sparse, and difficult for existing smoothing methods to handle.

  • Normal Outliers appear in specific feature dimensions across all token sequences, whereas Massive Outliers are confined to limited tokens and have much larger magnitudes.Massive Outliers can exceed 100 and reach approximately 1000 times the median of other activations.
  • The study investigates how these two distinct outlier types affect quantization.
  • Massive Outliers occur at the input of the FFN down-projection layer, including an approximately 1400-magnitude activation isolated to one token in LLaMA2-7B Layer 1.
  • SmoothQuant can transfer Massive Outliers into the down-projection weights because their large scaling factors introduce noticeable weight outliers.

3 Method

DuQuant combines block-wise rotation, zigzag permutation, and a subsequent rotation to redistribute and smooth activation outliers. Its design targets both Normal and Massive Outliers while also smoothing weights for quantization.

  • Preliminaries: Integer uniform quantization uses per-token activation step sizes and per-channel weight step sizes for hardware-oriented quantization.The linear layer is represented as Y = X · W, with activations and weights quantized separately.
  • Integrated transformation: After smoothing, the transformation sequence performs an initial rotation, permutation, and second rotation, while DuQuant also mitigates weight outliers induced by smoothing.This avoids reliance on complex weight quantization techniques such as GPTQ used by some prior methods.
  • Method overview: DuQuant applies rotation and channel permutation to redistribute activation features and mitigate both Normal and Massive Outliers.The method builds on the smooth technique but adds transformations in activation space.
  • Rotation transformation: The rotation transformation uses an orthogonal matrix and greedy search guided by outlier dimensions to reduce large activation magnitudes.The searched dimensions identify concentrated outliers; iterative rotations select the step minimizing the maximum transformed activation.
  • Rotation transformation: A block-diagonal rotation approximates the full rotation for faster multiplication and lower storage, but its local blocks cannot share information across blocks.Each block covers 2^n dimensions, with the number of blocks determined by the input channel count.
  • Permutation transformation: The zigzag permutation assigns high-activation channels across blocks in alternating directions to reduce variance in block-wise outlier magnitudes.The procedure balances block means by reordering channels from highest activations through descending and ascending passes.

4 Experiment

DuQuant is evaluated across multiple LLM families, quantization settings, language and QA benchmarks, ablations, and efficiency measurements. Results show stronger accuracy and perplexity than baselines, with rotation and permutation contributing to outlier management and acceptable inference overhead.

  • Main Results: DuQuant is evaluated on LLaMA, LLaMA2, LLaMA3, Mistral, Phi2, and Vicuna models using language generation, commonsense QA, MMLU, MT-Bench, and LongBench tasks.The main settings focus on 4-bit and 6-bit weight-activation quantization, with per-token activation and per-channel weight quantization.
  • Main Results: DuQuant outperforms baselines on WikiText2, C4, and commonsense QA, surpassing QLLM by +9% and Atom by +5% in average accuracy across model sizes.DuQuant+LWC achieves performance comparable with FP16 models on the reported perplexity evaluations.
  • Main Results: DuQuant surpasses baselines across Vicuna MMLU task categories and achieves stronger MT-Bench win rates against Atom and OmniQuant.For Vicuna-7B, it records 68 wins versus Atom and 155 versus OmniQuant, with 16 and 1 losses, respectively.
  • Main Results: Under W6A6, DuQuant achieves LLaMA3-8B performance comparable to FP16 while maintaining competitive performance under W4A4.The paper attributes this robustness to improved handling of outliers.
  • Ablation Study: Ablations show that smoothing provides a baseline, the initial rotation improves performance, and permutation plus a second rotation further improves quantized models.The full method combines these operations to address both normal and massive outliers; removing rotation and permutation causes greater degradation for massive outliers.
  • Efficiency Analysis: “Perm 1” improves performance over “Perm 0” at an additional 8.9% computational cost on LLaMA2-7B and 9.3% on LLaMA2-13B, while further permutations reduce efficiency.DuQuant achieves 2.08× and 2.34× pre-filling speedups over FP16 for LLaMA2-7B and LLaMA2-13B, respectively; pre-filling memory reduction reaches up to 3.2×.

5 Conclusion

DuQuant addresses activation outliers in low-bit LLM quantization by combining transformations that mitigate massive and normal outliers. The paper reports improved performance and state-of-the-art results for 4-bit weight-activation quantization.

  • DuQuant combines rotation and permutation transformations to mitigate both massive and normal activation outliers.The approach redistributes outliers to simplify quantization.
  • DuQuant establishes state-of-the-art results in 4-bit weight-activation quantization scenarios.
  • The paper organizes supplementary material around related work, proofs, implementation details, empirical results, ablations, QuaRot comparisons, rotation algorithms, limitations, and visualizations.
  • The work focuses on enhancing low-bit post-training quantization performance, especially for 4-bit weight-activation quantization.

B Proofs

The proofs analyze block-wise rotation and zigzag permutation for controlling activation outliers. They establish that permutation redistributes outliers so block means share a common upper bound.

  • Rotation: The rotation theorem analyzes a block-diagonal orthogonal rotation applied to activation blocks and their maximum dimension-wise outliers.
  • Rotation: The rotation construction uses switch operations to place the largest outlier in a designated column within a block.
  • Zigzag Permutation: The zigzag permutation reorders outliers across K blocks, where each block contains 2^n channels.
  • Zigzag Permutation: After zigzag permutation, all block mean values M_bi share the same upper bound.

C Additional Implementation Details

The implementation evaluates DuQuant on NVIDIA GPUs with fixed sequence lengths and WikiText2 calibration data. It uses block transformations, Hadamard rotations in selected attention multiplications, and specified clipping and smoothing settings.

  • Experiments use RTX 3090 GPUs for small models, A100 GPUs for large models, and sequence length 2048 across evaluation tasks.
  • Calibration uses 128 WikiText2 sequences of length 2048, with rotation block size 128 and 256 maximum greedy-search steps.
  • Hadamard rotations are applied to selected activation multiplications in the attention module for rapid processing.
  • DuQuant uses smooth parameter α=0.6, activation clipping ratio 0.9, and weight clipping ratio 0.8.

D More Empirical Results

Additional evaluations report strong DuQuant performance across models, tasks, quantization settings, and deployment measurements. Ablations examine rotation size, search steps, and permutation algorithms.

  • Zero-shot QA: DuQuant outperforms Atom by 5.43% on LLaMA2-7B and by 5.18% on LLaMA2-13B in zero-shot commonsense QA.
  • MMLU: DuQuant shows the smallest performance decline among compared methods on Vicuna-v1.5-7B MMLU and generalizes to instruction-tuned models.
  • MT-Bench: DuQuant performs comparably to FP16 models on MT-Bench for both 7B and 13B models.
  • Mistral and Phi2: DuQuant largely surpasses other baselines on 4-bit Mistral-7B and Phi2-2.8B, while Phi2 can still experience matrix-multiplication overflow issues.
  • LLaMA3-70B: DuQuant outperforms SmoothQuant by 12.9% on LLaMA3-70B Commonsense QA and reduces perplexity across WikiText2, C4, and PTB.
  • W6A6: Under W6A6 quantization, DuQuant surpasses baselines and achieves nearly lossless performance relative to FP16 models.
  • Ablations: Increasing rotation block size generally improves performance, while 256 greedy-search steps balance performance and time usage.
  • Ablations: The permutation ablation evaluates algorithms by their ability to minimize variance among activation blocks.

E.4 Effects of Calibration Datasets

DuQuant remains robust across calibration datasets and sample counts because calibration data is used to identify outlier channels and compute mean activations, rather than learn parameters.

  • Calibration-dataset selection has a relatively minor impact on DuQuant quantization performance.The method uses calibration data to identify outlier channels, not for gradient-based parameter learning.
  • Random calibration data still produces results competitively close to those from actual calibration data.The calibration-free experiment uses 256 random samples generated within the model vocabulary range.
  • Quantization performance shows low correlation with the number of WikiText2 calibration samples.DuQuant constructs rotation matrices from mean activation values across the samples.

F Detailed Comparison with QuaRot

Compared with QuaRot, DuQuant combines prior-guided rotations, zigzag permutation, and direct weight smoothing to handle outliers efficiently. It consistently outperforms QuaRot variants across reported models, tasks, and datasets while avoiding QuaRot’s time-consuming GPTQ step.

  • Method comparison: DuQuant uses prior knowledge to target outliers through multiple rotations, producing smoother activations than QuaRot’s Hadamard transformation.Its optimal rotation matrix is derived through a greedy search guided by prior knowledge.
  • Method comparison: Zigzag channel permutation reduces activation variance across blocks and rapidly smooths outlier features.The permutation provides a distinct advantage for handling massive outliers.
  • Method comparison: Direct weight-matrix smoothing avoids QuaRot’s time-consuming GPTQ algorithm and accelerates quantization.The paper reports high quantization efficiency in Table F24.
  • Experimental comparison: DuQuant consistently outperforms QuaRot-RTN across perplexity and zero-shot benchmarks for LLaMA1-7B, LLaMA2-7B, and LLaMA3-8B.DuQuant and DuQuant+LWC also achieve better average accuracy across six QA tasks and stronger WikiText and C4 performance, particularly for LLaMA3-8B.
  • Experimental comparison: DuQuant without permutation outperforms QuaRot under W4A4 quantization, supporting the effectiveness of its prior-guided rotation.The comparison omits DuQuant’s permutation step and contrasts its rotation with QuaRot’s Hadamard matrices.

H Limitations and Broader Impacts

DuQuant’s main limitation is calibration-data selection, while its broader impact is faster, lower-memory LLM deployment through effective INT4 weight-activation quantization.

  • Limitations: DuQuant lacks a specialized calibration-data selection strategy and currently uses 128 randomly selected WikiText2 samples.The authors report promising calibration-free results but leave tailored calibration choices for future study.
  • Broader Impacts: DuQuant smooths massive and uniform outliers with rotation and permutation matrices for fast INT4 weight-activation post-training quantization.The authors position this approach as a new state-of-the-art and as a way to reduce deployment memory usage and accelerate LLMs.

I More Visualizations

The supplementary visualizations show DuQuant’s activation changes across LLaMA, Vicuna, and Mistral models. For models other than Mistral, the figures separately display normal and massive outlier changes before and after transformation.

  • Visualization layout: Except for Mistral, each figure’s left side shows normal-outlier changes and its right side shows massive-outlier changes before and after transformation.The visualizations indicate that massive outliers consistently occur in the FFN down-projection layer.
  • LLaMA1: Figures I1–I4 visualize DuQuant activation changes for LLaMA1-7B, LLaMA1-13B, and LLaMA1-65B.Figure I3 provides additional LLaMA1-13B examples.
  • LLaMA2: Figures I5–I6 visualize DuQuant activation changes for LLaMA2-13B and LLaMA2-70B.
  • Vicuna: Figures I7–I8 visualize DuQuant activation changes for Vicuna-v1.5-7B and Vicuna-v1.5-13B.
  • Mistral: Figure I9 visualizes massive activation changes after applying DuQuant to Mistral7B.
Loading 2406.01721v3…