PaperlayerView exact revision on arXiv

Source-linked AI summary

OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, Eunhyeok Park

arXiv:2306.02272v4cs.CL

TL;DR

Large language models require substantial memory and computation, motivating quantization that reduces footprint without unacceptable quality loss. OWQ protects activation-outlier-sensitive weight columns in high precision while quantizing the rest at very low precision, and WCT adapts the resulting models efficiently. The paper reports that 3.1-bit OWQ is comparable to 4-bit OPTQ, while WCT achieves strong task-specific adaptation with low memory overhead.

  • Problem

    LLM memory and computation demands hinder practical deployment, while quantized fine-tuning still depends on high-quality low-precision base weights.

  • Method

    OWQ identifies activation-outlier-sensitive weight columns, retains them in high precision, and quantizes remaining weights with tuned low-precision parameters; WCT fine-tunes the retained weak columns.

  • Results

    3.1-bit OWQ has comparable quality to 4-bit OPTQ, and OWQ consistently improves quality over OPTQ across model sizes and few-shot tasks.

  • Takeaways & Limitations

    OWQ reduces quality loss in extremely low-bit weight quantization, while WCT enables task-specific adaptation with minimal memory overhead.

Abstract

from arXiv · show

Large language models (LLMs) with hundreds of billions of parameters require powerful server-grade GPUs for inference, limiting their practical deployment. To address this challenge, we introduce the outlier-aware weight quantization (OWQ) method, which aims to minimize LLM's footprint through low-precision representation. OWQ prioritizes a small subset of structured weights sensitive to quantization, storing them in high-precision, while applying highly tuned quantization to the remaining dense weights. This sensitivity-aware mixed-precision scheme reduces the quantization error notably, and extensive experiments demonstrate that 3.1-bit models using OWQ perform comparably to 4-bit models optimized by OPTQ. Furthermore, OWQ incorporates a parameter-efficient fine-tuning for task-specific adaptation, called weak column tuning (WCT), enabling accurate task-specific LLM adaptation with minimal memory overhead in the optimized format. OWQ represents a notable advancement in the flexibility, efficiency, and practicality of LLM optimization literature. The source code is available at https://github.com/xvyaward/owq

Introduction

LLM deployment is constrained by substantial memory and computation costs, motivating low-precision weight quantization. OWQ addresses quantization degradation by protecting sensitive weight columns while quantizing the remaining weights, and supports efficient task-specific fine-tuning through WCT.

  • Motivation: GPT3-175B requires approximately 330 GB in fp16 merely to store parameters, making server-grade multi-GPU deployment costly.The paper identifies minimizing serving costs as crucial for widespread LLM adoption.
  • Motivation: 3-bit weight and fp16 activation matrix multiplication can outperform fp16 weight and activation across multiple GPUs on a single GPU.Low-precision weight quantization can reduce memory bottlenecks and communication costs.
  • Limitations of Existing Quantization: The compressed 3-bit OPT-175B model occupies around 63 GB and outperforms the fp16 OPT-30B model, but 3-bit quantization still causes degradation.The degradation is more pronounced in smaller models, motivating improved accuracy preservation.
  • Task-Specific Adaptation: QLoRA enables fine-tuning quantized LLMs with added high-precision low-rank tensors, but suboptimal dense-matrix quality can diminish fine-tuning benefits.The paper therefore treats high-quality weight quantization as essential for adaptation.
  • Proposed Approach: OWQ uses mixed-precision quantization based on weight-column sensitivity to address degradation associated with activation outliers.The method is designed around LLM-specific activation-outlier characteristics.
  • Contributions: 3.1-bit OWQ has comparable quality to the 4-bit OPTQ model.The paper also introduces WCT for fine-tuning with minimal memory overhead and fewer trainable parameters than existing methods.

Background and Related Works

LLM quantization balances low-precision efficiency against quality degradation, with PTQ favored over costly training-based approaches. OWQ extends sub-4-bit weight quantization by accounting for activation-outlier-driven sensitivity and selectively protecting vulnerable weights.

  • Quantization Approaches: Quantization exploits low-precision representations for size and performance benefits, but can degrade network quality.QAT and PTQ represent training-based and post-training strategies for managing this trade-off.
  • Quantization Approaches: PTQ is favored for LLM quantization because QAT is less practical at extremely low precision due to the high cost of the training environment.The paper positions PTQ as an important approach for resource-intensive LLMs.
  • LLM Quantization: Sub-4-bit weight-only quantization is distinguished from approaches that quantize both activations and weights to int8.OWQ follows the weight-only direction.
  • Activation Outliers: LLM intermediate activations contain a few values significantly larger than others, creating a challenge for activation quantization.These activation outliers motivate examining their impact on weight quantization as well.
  • Activation Outliers: Activation outliers increase the sensitivity of certain weight columns even when only weights are quantized.The paper identifies accounting for activation outliers as essential for accurate weight quantization.
  • OPTQ: OPTQ uses Hessian-based layer-wise quantization and compensation, but accuracy decreases relative to fp16 baselines as model size or problem complexity increases.OPTQ parallelizes quantization across output-channel elements for rapid processing.
  • OWQ: OWQ selectively applies high precision to weights vulnerable to activation-outlier-induced quantization errors and applies modified OPTQ to the remaining weights.This design targets the accuracy limitations of uniform low-bit quantization.
  • Parameter-Efficient Fine-Tuning: LoRA freezes pretrained weights and adds a small fraction of learnable parameters through low-rank decomposition.It exemplifies parameter-efficient fine-tuning for task adaptation.

Problem Definition and Motivation

The paper formulates quantization as minimizing output-activation error under a compression constraint. Its motivation is that activation outliers create highly sensitive weight columns, concentrating much of the error in a small number of channels.

  • Problem Definition: OWQ maps a full-precision weight matrix to low precision by minimizing the difference between pre- and post-quantization output activations.The formulation considers an input feature matrix with Cin input channels and N sequence length.
  • Problem Definition: The quantization objective is constrained by a target compression ratio, with layer-wise quantization applied sequentially through the model.Embedding and head weights remain full precision.
  • Motivation: Weights linked to activation outliers are particularly susceptible to quantization.This relationship motivates the sensitivity-aware design of OWQ.
  • Error Decomposition: The layer-wise squared error decomposes into individual output-channel errors, with no Hessian interaction between output channels.The individual Hessians share the same input-channel dimensionality.
  • Error Analysis: Taylor expansion relates each output-channel error to the Hessian and the magnitude of weight perturbation.This connects quantization error to both curvature and the deviation between original and quantized weights.
  • Sensitivity to Outliers: Activation outliers create exceptionally large Hessian values, increasing the quantization sensitivity of corresponding weight channels.Equal bit-width quantization can therefore produce disproportionately large errors in channels associated with outliers.
  • Sensitivity to Outliers: A large portion of output error originates from a limited number of channels aligned with weak columns.Figure 2 sorts channel indices by their influence on output error and supports focusing special treatment on these channels.

OWQ: Outlier-aware Weight Quantization

OWQ identifies weight columns that are especially sensitive to quantization, retains them in full precision, and quantizes the remaining weights with tuned low-precision methods. This design reduces reconstruction error while supporting memory-efficient fine-tuning through WCT.

  • Weak-column selection: OWQ selects the top-k weight columns by sensitivity, using Hessian-based column metrics, and excludes them from quantization.The sensitivity metric combines each column’s perturbation norm with the corresponding Hessian diagonal element.
  • Low-precision quantization: The remaining weights are integrated into OPTQ sequential column-wise quantization, with full-precision weak columns helping compensate quantization errors.Reordering the weak columns to the end allows their retained values to preserve compensation during OPTQ.
  • Weak-column selection: Unlike magnitude-based outlier methods, OWQ selects columns by their effect on output activation error rather than weight magnitude.Figure 3 contrasts weight ranges with the selected weak columns, while the paper explicitly distinguishes sensitivity-based selection from magnitude-based selection.
  • Low-precision quantization: OWQ modifies OPTQ with truncation-aware parameter search, using a 2D grid search over step size and zero point before applying OPTQ.The searched configuration uses nearest rounding with truncation to narrow the quantization range and reduce reconstruction error.
  • Quantization quality: Truncation-aware OWQ lowers WikiText-2 perplexity from 12.14 to 11.21 for OPT-6.7B, whereas applying truncation directly to conventional OPTQ raises it from 12.88 to 48.26.The paper attributes this contrast to exceptionally large values in weak key and query columns, which OWQ retains in full precision.
  • Weak Column Tuning: WCT fine-tunes only OWQ’s full-precision weak columns, preserving the OWQ format and adding negligible memory overhead.The paper reports that WCT can support per-task weak columns while sharing the dense low-precision matrix across task-specific models.
  • Weak Column Tuning: WCT surpasses leading fine-tuning methods in memory usage and output quality, which the paper attributes to OWQ’s denser representation and preselected sensitive update targets.These claims are presented as experiment-backed outcomes and are linked to the quality of OWQ’s low-precision representation.

Experiments

Experiments evaluate OWQ on OPT and LLaMA models using perplexity, few-shot accuracy, latency, quantization time, and fine-tuning comparisons. Across these evaluations, OWQ improves low-bit quality, preserves acceleration, and supports parameter-efficient adaptation.

  • Evaluation setup: OWQ experiments evaluate OPT and LLaMA families using WikiText-2, PTB, C4, and few-shot ARC-challenge, Hellaswag, and MMLU tasks.Perplexity is used as an indicator of generative performance, while few-shot scores assess reasoning and general knowledge.
  • Perplexity and accuracy: 3.01-bit OWQ mitigates the quality degradation of 3-bit OPTQ, while 3.1-bit OWQ achieves performance comparable to 4-bit OPTQ across LLM families and model sizes.OWQ 4.01-bit also produces noteworthy improvements, highlighting the value of treating weak columns.
  • Perplexity and accuracy: OWQ consistently outperforms OPTQ with true-sequential and act-order options, while those options provide no additional benefit to OWQ.The results suggest sensitivity-aware quantization accounts for the benefit of act-order, whereas act-order alone cannot sufficiently address weak-column degradation.
  • Real-device acceleration: 3.21% latency overhead is measured for OWQ 3.01-bit versus 3-bit OPTQ acceleration on LLaMA 7B, with overhead generally amortized for larger models.A customized CUDA kernel selects weak-column activation channels on the fly while avoiding irregular memory access.
  • Quantization efficiency: OWQ can quantize a 66B model in under 3 hours on an A100 GPU despite additional weak-column selection and hyperparameter-tuning operations.Sharing the Hessian with OPTQ reduces OWQ’s quantization overhead.
  • WCT-based fine-tuning: WCT with 8 weak columns uses 6.8% of QLoRA’s learnable parameters, while OWQ plus WCT matches full-precision LoRA using 24.4% of inference memory.WCT updates only high-precision weak columns and preserves the OWQ format after fine-tuning.
  • Comparison with group-wise quantization: 3.01-bit OWQ has about 10% of grouped OPTQ’s storage overhead while achieving comparable or better perplexity and zero-shot accuracy.Fine-grained quantization provides negligible additional improvement because OWQ already substantially improves 3-bit model quality.

Conclusion

OWQ addresses quality degradation from low-precision weight quantization by accounting for activation-outlier-induced sensitivity in weight columns. It also provides WCT for task-specific adaptation with minimal memory overhead.

  • OWQ identifies activation outliers as a source of sensitivity in certain weight columns and quality degradation under low-precision quantization.The method uses this observation to design its quantization scheme.
  • OWQ improves quality over existing 3-bit quantization methods with negligible storage and computation overhead.
  • WCT enables task-specific adaptation with minimal memory overhead and outstanding performance.

1 Proof of Eq. (5) in the Main Manuscript

The proof decomposes quantization error across output channels and approximates each channel’s error using a Taylor expansion. At a local error minimum, the first-order term is ignored, leaving a second-order Hessian-based approximation.

  • Overall quantization error is separated into the summed errors of individual output channels.The objective of minimizing total error can therefore be treated as minimizing each channel error E_i.
  • The perturbation for each weight row is defined as ΔW_i,: = W_i,: − ˆW_i,: and paired with a Hessian of second derivatives.The Hessian captures second-order sensitivity for the weights in that row.
  • At a local minimum, the first-order gradient term can be ignored in the Taylor expansion of channel quantization error.Because E_i is quadratic, terms above third order become zero, while the remaining approximation is governed by second-order terms.
  • The resulting per-channel quantization error is represented by the second-order approximation in Eq. (10).

B Experiment Details

OWQ’s implementation is built on the official OPTQ (GPTQ) GitHub implementation.

  • OWQ is implemented using the official OPTQ (GPTQ) GitHub codebase.

1 Evaluation Settings

Evaluation measures language-modeling perplexity on WikiText-2, PTB, and C4, while comparing storage-oriented representations for weak columns. The storage overhead remains negligible in the reported 3.01-bit case.

  • Evaluation Settings: Language-modeling perplexity is evaluated on WikiText-2, PTB, and C4 using each model’s default HuggingFace tokenizer.Validation text is concatenated with dataset-specific separators before tokenization.
  • Storage Methods: The latency-favored representation stores a complete low-precision matrix with zero-filled weak columns plus additional fp16 weak columns.The storage-favored representation removes unnecessary zeros but adds operational overhead.
  • Evaluation Settings: The reported perplexity tables cover OPT and LLaMA evaluations on PTB and C4.The appendix identifies separate tables for OPT C4, OPT PTB, LLaMA PTB, and LLaMA C4 perplexity.
  • Storage Methods: The storage-favored method is used to calculate effective bit-width, while storage overhead is similar between methods because weak columns are few.
  • Storage Methods: 3.012-bit storage results when zero-filled matrix overhead is added to the 3.01-bit case, making the additional overhead negligible.

1 Additional Perplexity Results

The additional results examine evaluation conditions, layer-wise weak-column sensitivity, storage overhead, and learnable-parameter counts for WCT relative to QLoRA.

  • Evaluation Conditions: C4 perplexity measurements are not fully zero-shot because calibration data were sampled from the C4 training set.The OPTQ results additionally used true-sequential and act-order options, whereas OWQ did not.
  • Layer-wise Sensitivity: Preserving weak columns simultaneously in key and query weights gives the best accuracy in the OPT-6.7B layer-wise sensitivity analysis.Performance suffers significantly when both layers are not addressed together, especially when OWQ is applied only to queries.
  • Storage Overhead: OWQ with target bit-width 3.01 has overhead roughly equivalent to group-wise quantization with group size 1024.At target bit-width 3.1, OWQ closely matches the overhead of group size 128.
  • Learnable Parameters: For LLaMA 7B with r = 64 and k = 8, WCT has only 6.8% as many learnable parameters as QLoRA.The comparison uses d = 4096 and D = 11008, with r denoting adapter rank and k the number of weak columns.
Loading 2306.02272v4…