OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

Motivation

• GPT3-175B requires approximately 330 GB in fp16 merely to store parameters, making server-grade multi-GPU deployment costly.The paper identifies minimizing serving costs as crucial for widespread LLM adoption.Source: Introduction · p. 1
• 3-bit weight and fp16 activation matrix multiplication can outperform fp16 weight and activation across multiple GPUs on a single GPU.Low-precision weight quantization can reduce memory bottlenecks and communication costs.Source: Introduction · p. 1

Limitations of Existing Quantization

• The compressed 3-bit OPT-175B model occupies around 63 GB and outperforms the fp16 OPT-30B model, but 3-bit quantization still causes degradation.The degradation is more pronounced in smaller models, motivating improved accuracy preservation.Source: Introduction · p. 1

Task-Specific Adaptation

• QLoRA enables fine-tuning quantized LLMs with added high-precision low-rank tensors, but suboptimal dense-matrix quality can diminish fine-tuning benefits.The paper therefore treats high-quality weight quantization as essential for adaptation.Source: Introduction · p. 1

Proposed Approach

• OWQ uses mixed-precision quantization based on weight-column sensitivity to address degradation associated with activation outliers.The method is designed around LLM-specific activation-outlier characteristics.Source: Introduction · p. 1

Contributions

• 3.1-bit OWQ has comparable quality to the 4-bit OPTQ model.The paper also introduces WCT for fine-tuning with minimal memory overhead and fewer trainable parameters than existing methods.Source: Introduction · p. 2

Quantization Approaches

• Quantization exploits low-precision representations for size and performance benefits, but can degrade network quality.QAT and PTQ represent training-based and post-training strategies for managing this trade-off.Source: Quantization and LLMs · p. 2
• PTQ is favored for LLM quantization because QAT is less practical at extremely low precision due to the high cost of the training environment.The paper positions PTQ as an important approach for resource-intensive LLMs.Source: Quantization and LLMs · p. 2

LLM Quantization

• Sub-4-bit weight-only quantization is distinguished from approaches that quantize both activations and weights to int8.OWQ follows the weight-only direction.Source: Quantization and LLMs · p. 2

Activation Outliers

• LLM intermediate activations contain a few values significantly larger than others, creating a challenge for activation quantization.These activation outliers motivate examining their impact on weight quantization as well.Source: Int8 Quantization for Activation and… · p. 2
• Activation outliers increase the sensitivity of certain weight columns even when only weights are quantized.The paper identifies accounting for activation outliers as essential for accurate weight quantization.Source: Int8 Quantization for Activation and… · p. 2

OPTQ

• OPTQ uses Hessian-based layer-wise quantization and compensation, but accuracy decreases relative to fp16 baselines as model size or problem complexity increases.OPTQ parallelizes quantization across output-channel elements for rapid processing.Source: OPTQ: Weight Quantization for LLMs · pp. 2–3

OWQ

• OWQ selectively applies high precision to weights vulnerable to activation-outlier-induced quantization errors and applies modified OPTQ to the remaining weights.This design targets the accuracy limitations of uniform low-bit quantization.Source: OPTQ: Weight Quantization for LLMs · p. 3

Parameter-Efficient Fine-Tuning

• LoRA freezes pretrained weights and adds a small fraction of learnable parameters through low-rank decomposition.It exemplifies parameter-efficient fine-tuning for task adaptation.Source: Parameter-Efficient Fine-Tuning (PEFT) · p. 3

Problem Definition

• OWQ maps a full-precision weight matrix to low precision by minimizing the difference between pre- and post-quantization output activations.The formulation considers an input feature matrix with Cin input channels and N sequence length.Source: Problem Definition and Motivation · p. 3
• The quantization objective is constrained by a target compression ratio, with layer-wise quantization applied sequentially through the model.Embedding and head weights remain full precision.Source: Problem Definition and Motivation · p. 3

Motivation

• Weights linked to activation outliers are particularly susceptible to quantization.This relationship motivates the sensitivity-aware design of OWQ.Source: Layer-wise Quantization and Hessian of… · p. 3

Error Decomposition

• The layer-wise squared error decomposes into individual output-channel errors, with no Hessian interaction between output channels.The individual Hessians share the same input-channel dimensionality.Source: Layer-wise Quantization and Hessian of… · p. 3

Error Analysis

• Taylor expansion relates each output-channel error to the Hessian and the magnitude of weight perturbation.This connects quantization error to both curvature and the deviation between original and quantized weights.Source: Layer-wise Quantization and Hessian of… · p. 3

Sensitivity to Outliers

• Activation outliers create exceptionally large Hessian values, increasing the quantization sensitivity of corresponding weight channels.Equal bit-width quantization can therefore produce disproportionately large errors in channels associated with outliers.Source: Layer-wise Quantization and Hessian of… · p. 3
• A large portion of output error originates from a limited number of channels aligned with weak columns.Figure 2 sorts channel indices by their influence on output error and supports focusing special treatment on these channels.Source: Layer-wise Quantization and Hessian of… · p. 4, Figure 2 · p. 4

Weak-column selection

• OWQ selects the top-k weight columns by sensitivity, using Hessian-based column metrics, and excludes them from quantization.The sensitivity metric combines each column’s perturbation norm with the corresponding Hessian diagonal element.Source: OWQ: Outlier-aware Weight Quantization · p. 4

Low-precision quantization

• The remaining weights are integrated into OPTQ sequential column-wise quantization, with full-precision weak columns helping compensate quantization errors.Reordering the weak columns to the end allows their retained values to preserve compensation during OPTQ.Source: OWQ: Outlier-aware Weight Quantization · p. 4

Weak-column selection

• Unlike magnitude-based outlier methods, OWQ selects columns by their effect on output activation error rather than weight magnitude.Figure 3 contrasts weight ranges with the selected weak columns, while the paper explicitly distinguishes sensitivity-based selection from magnitude-based selection.Source: OWQ: Outlier-aware Weight Quantization · p. 4, Figure 3 · p. 4

Low-precision quantization

• OWQ modifies OPTQ with truncation-aware parameter search, using a 2D grid search over step size and zero point before applying OPTQ.The searched configuration uses nearest rounding with truncation to narrow the quantization range and reduce reconstruction error.Source: Quantization Configuration Search · pp. 4–5

Quantization quality

• Truncation-aware OWQ lowers WikiText-2 perplexity from 12.14 to 11.21 for OPT-6.7B, whereas applying truncation directly to conventional OPTQ raises it from 12.88 to 48.26.The paper attributes this contrast to exceptionally large values in weak key and query columns, which OWQ retains in full precision.Source: Quantization Configuration Search · p. 5

Weak Column Tuning

• WCT fine-tunes only OWQ’s full-precision weak columns, preserving the OWQ format and adding negligible memory overhead.The paper reports that WCT can support per-task weak columns while sharing the dense low-precision matrix across task-specific models.Source: PEFT with Weak Column Tuning · p. 5
• WCT surpasses leading fine-tuning methods in memory usage and output quality, which the paper attributes to OWQ’s denser representation and preselected sensitive update targets.These claims are presented as experiment-backed outcomes and are linked to the quality of OWQ’s low-precision representation.Source: PEFT with Weak Column Tuning · p. 5

Evaluation setup

• OWQ experiments evaluate OPT and LLaMA families using WikiText-2, PTB, C4, and few-shot ARC-challenge, Hellaswag, and MMLU tasks.Perplexity is used as an indicator of generative performance, while few-shot scores assess reasoning and general knowledge.Source: Experimental Setup · p. 5, Results of Perplexity Measure · p. 5, Results of Various Few-shot Tasks · p. 6

Perplexity and accuracy

• 3.01-bit OWQ mitigates the quality degradation of 3-bit OPTQ, while 3.1-bit OWQ achieves performance comparable to 4-bit OPTQ across LLM families and model sizes.OWQ 4.01-bit also produces noteworthy improvements, highlighting the value of treating weak columns.Source: Results of Perplexity Measure · p. 6
• OWQ consistently outperforms OPTQ with true-sequential and act-order options, while those options provide no additional benefit to OWQ.The results suggest sensitivity-aware quantization accounts for the benefit of act-order, whereas act-order alone cannot sufficiently address weak-column degradation.Source: Results of Perplexity Measure · p. 6

Real-device acceleration

• 3.21% latency overhead is measured for OWQ 3.01-bit versus 3-bit OPTQ acceleration on LLaMA 7B, with overhead generally amortized for larger models.A customized CUDA kernel selects weak-column activation channels on the fly while avoiding irregular memory access.Source: Acceleration on Real Device · p. 6

Quantization efficiency

• OWQ can quantize a 66B model in under 3 hours on an A100 GPU despite additional weak-column selection and hyperparameter-tuning operations.Sharing the Hessian with OPTQ reduces OWQ’s quantization overhead.Source: Quantization Speed · p. 7

WCT-based fine-tuning

• WCT with 8 weak columns uses 6.8% of QLoRA’s learnable parameters, while OWQ plus WCT matches full-precision LoRA using 24.4% of inference memory.WCT updates only high-precision weak columns and preserves the OWQ format after fine-tuning.Source: Results of WCT-based Fine-tuning · p. 7, Comparison of PTQ Methods used in WCT · p. 8

Comparison with group-wise quantization

• 3.01-bit OWQ has about 10% of grouped OPTQ’s storage overhead while achieving comparable or better perplexity and zero-shot accuracy.Fine-grained quantization provides negligible additional improvement because OWQ already substantially improves 3-bit model quality.Source: Comparison with Group-wise Quantization · p. 8

• OWQ identifies activation outliers as a source of sensitivity in certain weight columns and quality degradation under low-precision quantization.The method uses this observation to design its quantization scheme.Source: Conclusion · p. 8
• OWQ improves quality over existing 3-bit quantization methods with negligible storage and computation overhead.Source: Conclusion · p. 8
• WCT enables task-specific adaptation with minimal memory overhead and outstanding performance.Source: Conclusion · p. 8

• Overall quantization error is separated into the summed errors of individual output channels.The objective of minimizing total error can therefore be treated as minimizing each channel error E_i.Source: §1 · p. 10
• The perturbation for each weight row is defined as ΔW_i,: = W_i,: − ˆW_i,: and paired with a Hessian of second derivatives.The Hessian captures second-order sensitivity for the weights in that row.Source: §1 · p. 10
• At a local minimum, the first-order gradient term can be ignored in the Taylor expansion of channel quantization error.Because E_i is quadratic, terms above third order become zero, while the remaining approximation is governed by second-order terms.Source: §1 · p. 10
• The resulting per-channel quantization error is represented by the second-order approximation in Eq. (10).Source: §1 · p. 10

• OWQ is implemented using the official OPTQ (GPTQ) GitHub codebase.Source: B Experiment Details · p. 10

Evaluation Settings

• Language-modeling perplexity is evaluated on WikiText-2, PTB, and C4 using each model’s default HuggingFace tokenizer.Validation text is concatenated with dataset-specific separators before tokenization.Source: §1 · p. 10

Storage Methods

• The latency-favored representation stores a complete low-precision matrix with zero-filled weak columns plus additional fp16 weak columns.The storage-favored representation removes unnecessary zeros but adds operational overhead.Source: §2 · pp. 10–11

Evaluation Settings

• The reported perplexity tables cover OPT and LLaMA evaluations on PTB and C4.The appendix identifies separate tables for OPT C4, OPT PTB, LLaMA PTB, and LLaMA C4 perplexity.Source: Table 10 · p. 11, Table 9 · p. 11, Table 11 · p. 11

Storage Methods

• The storage-favored method is used to calculate effective bit-width, while storage overhead is similar between methods because weak columns are few.Source: §2 · p. 11
• 3.012-bit storage results when zero-filled matrix overhead is added to the 3.01-bit case, making the additional overhead negligible.Source: §2 · p. 11

Evaluation Conditions

• C4 perplexity measurements are not fully zero-shot because calibration data were sampled from the C4 training set.The OPTQ results additionally used true-sequential and act-order options, whereas OWQ did not.Source: §1 · p. 11

Layer-wise Sensitivity

• Preserving weak columns simultaneously in key and query weights gives the best accuracy in the OPT-6.7B layer-wise sensitivity analysis.Performance suffers significantly when both layers are not addressed together, especially when OWQ is applied only to queries.Source: §2 · p. 12

Storage Overhead

• OWQ with target bit-width 3.01 has overhead roughly equivalent to group-wise quantization with group size 1024.At target bit-width 3.1, OWQ closely matches the overhead of group size 128.Source: §1 · p. 12

Learnable Parameters

• For LLaMA 7B with r = 64 and k = 8, WCT has only 6.8% as many learnable parameters as QLoRA.The comparison uses d = 4096 and D = 11008, with r denoting adapter rank and k the number of weak columns.Source: §3 · p. 13