PaperlayerView exact revision on arXiv

Source-linked AI summary

OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting

Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, Sifan Zhou

arXiv:2501.13987v1cs.LGcs.AI

TL;DR

PTQ must compress LLMs despite deployment costs and data distributions that waste quantization precision, while existing transformations lack a metric for whole-space distribution optimization. The paper introduces QSUR and OSTQuant, combining learnable orthogonal and scaling transformations with KL-Top loss. OSTQuant outperforms existing methods, retaining over 99.5% of full-precision accuracy in W4A16 and at least 96% in W4A4KV4.

  • Problem

    Uneven, heavy-tailed distributions expand quantization ranges and reduce precision for most values, while prior transformations lack a metric for optimizing distributions across the entire quantization space.

  • Method

    OSTQuant uses learnable equivalent orthogonal and scaling transformations to optimize weights and activations, with KL-Top loss for limited-calibration optimization.

  • Results

    OSTQuant outperforms existing methods across models and benchmarks, retaining over 99.5% of full-precision accuracy in W4A16 and at least 96% in W4A4KV4.

  • Takeaways & Limitations

    Optimizing distribution utilization across the quantization space is associated with improved quantization performance, with QSUR positively correlated with accuracy.

  • Takeaways & Limitations

    The QSUR derivation neglects the mean vector when its magnitude is smaller than the largest eigenvalue.

Abstract

from arXiv · show

Post-training quantization (PTQ) has emerged as a widely adopted technique for compressing and accelerating Large Language Models (LLMs). The major challenge in LLM quantization is that uneven and heavy-tailed data distributions can expand the quantization range, thereby reducing bit precision for most values. Recent methods attempt to eliminate outliers and balance inter-channel differences by employing linear transformations; however, they remain heuristic and are often overlook optimizing the data distribution across the entire quantization space.In this paper, we introduce Quantization Space Utilization Rate (QSUR), a novel metric that effectively assesses the quantizability of transformed data by measuring the space utilization of the data in the quantization space. We complement QSUR with mathematical derivations that examine the effects and limitations of various transformations, guiding our development of Orthogonal and Scaling Transformation-based Quantization (OSTQuant). OSQuant employs a learnable equivalent transformation, consisting of an orthogonal transformation and a scaling transformation, to optimize the distributions of weights and activations across the entire quantization space. Futhermore, we propose the KL-Top loss function, designed to mitigate noise during optimization while retaining richer semantic information within the limited calibration data imposed by PTQ. OSTQuant outperforms existing work on various LLMs and benchmarks. In the W4-only setting, it retains 99.5\% of the floating-point accuracy. In the more challenging W4A4KV4 configuration, OSTQuant reduces the performance gap by 32\% on the LLaMA-3-8B model compared to state-of-the-art methods. \href{https://github.com/BrotherHappy/OSTQuant}{https://github.com/BrotherHappy/OSTQuant}.

1 INTRODUCTION

The paper frames PTQ as a response to LLM deployment costs, but uneven distributions and limited calibration data make accurate quantization difficult. It introduces QSUR, OSTQuant, and KL-Top to improve distribution utilization and quantization performance.

  • LLMs’ memory and computational demands limit practical deployment on resource-constrained edge devices and cloud GPUs.
  • Uneven and heteroscedastic data expand the quantization range, reducing available bit precision for most values.
  • Existing smooth- and rotation-based transformations improve distributions in specific quantization-space regions by redistributing difficulty or suppressing outliers.
  • QSUR measures the ratio of data volume to quantization-space volume, and experiments show positive correlation between QSUR and quantization accuracy.
  • OSTQuant learns orthogonal and scaling transformation pairs for LLM layers to optimize weight and activation distributions across the quantization space.
  • KL-Top uses top-k full-precision logits to retain richer semantic information and mitigate noise during optimization with limited calibration data.
  • In W4A16, OSTQuant retains over 99.5% of full-precision accuracy, while W4A4KV4 retains at least 96% of the model’s original performance.

2 RELATED WORK

Related work develops PTQ methods for weight-only and weight-activation quantization, while rotation-matrix optimization addresses the orthonormality constraints of learned transformations.

  • Weight-only PTQ methods reduce memory usage by minimizing quantization error or addressing activation outliers’ impact on weight quantization.
  • Weight-activation methods target inference speed by quantizing weights, activations, and sometimes the key-value cache.
  • Activation outliers dominate the quantization range, leaving few significant bits for most values and causing substantial errors.
  • Optimizing rotation matrices requires Riemannian optimization on the Stiefel manifold of orthogonal matrices.

3 QUANTIZATION SPACE UTILIZATION RATE

QSUR quantifies how efficiently data occupy their quantization hypercube. Its analysis links utilization to covariance geometry and motivates transformations that reduce eigenvalue disparities and outliers.

  • QSUR addresses the lack of a quantitative metric for assessing quantization difficulty and transformation effectiveness.
  • For X ∈ R^n×d, QSUR is the occupied hypervolume divided by the volume of a hypercubic quantization space defined by the maximum range across dimensions.
  • For Gaussian data, the occupied volume is modeled using the ellipsoid determined by the covariance matrix and mean vector.
  • The quantization hypercube is determined by the distribution’s extremal coordinate values, linked to principal-axis eigenvalues and eigenvectors.
  • The analysis neglects the mean vector when its magnitude is smaller than the largest eigenvalue, setting λmax = λmin = λ1.
  • QSUR increases with products of eigenvalue ratios relative to λ1 and decreases as the largest component of eigenvector q1 increases.
  • Scaling transformations reduce eigenvalue disparities but remain sensitive to outliers and uneven means; orthogonal transformations provide outlier reduction under suitable orientation.

4 METHODOLOGY

OSTQuant optimizes LLM weight and activation distributions using globally coordinated equivalent transformations while preserving computational equivalence. KL-Top focuses calibration on high-probability predictions to obtain more informative and efficient optimization signals.

  • 4.1 ORTHOGONAL AND SCALING TRANSFORMATION-BASED QUANTIZATION: Four equivalent transformation pairs operate within each block across FFN and self-attention layers, targeting distributions of weights, activations, and caches.Global residual-path rotation and attention-specific transformations are fused into weights while preserving the original computation when quantization is absent.
  • 4.1 ORTHOGONAL AND SCALING TRANSFORMATION-BASED QUANTIZATION: OSTQuant combines learnable orthogonal and scaling transformations to reshape weights and activations across the network for quantization.The transformations are organized as equivalent pairs assigned to fully connected layers and optimized across blocks.
  • 4.1 ORTHOGONAL AND SCALING TRANSFORMATION-BASED QUANTIZATION: Each transformation pair T = ΛO contains a diagonal scaling matrix Λ and a unit orthogonal matrix O.The diagonal inverse is computationally simple, while O can be optimized with gradient-based methods on the Stiefel manifold.
  • 4.2 KL-TOP LOSS: Small calibration sets make direct cross-entropy optimization risky because quantized models can show lower perplexity while losing zero-shot accuracy.The paper motivates distribution matching with KL divergence but restricts it to informative predictions through KL-Top.
  • 4.2 KL-TOP LOSS: KL-Top computes KL divergence only over the top-k highest-probability classes to reduce noise from negligible-probability vocabulary entries.The method is motivated by long-tail prediction distributions and large vocabularies; k = 1000 is given as an example that also reduces computation and memory costs.

5 EXPERIMENTS

Experiments evaluate OSTQuant across LLaMA models, quantization settings, baselines, efficiency measures, and ablations. Results report improved inference and training efficiency, with orthogonal transformations and KL-Top settings contributing materially to performance.

  • Overall Results: OSTQuant is evaluated across LLaMA-1, LLaMA-2, and LLaMA-3-8B using WikiText2 perplexity and up to nine zero-shot tasks.Baselines include RTN, SmoothQuant, GPTQ, Quarot, and SpinQuant for weight-only and weight-activation quantization.
  • Overall Results: 2× average inference speedup and over 3.5× memory savings are reported for OSTQuant’s 4-bit implementation versus FP16.Tests compare different LLaMA parameter sizes and sequence lengths on a Transformer block with batch size 4 on a 3090 GPU.
  • Overall Results: 5.3× faster training than OmniQuant is achieved in some settings, with 7B and 13B models optimized in about 20 minutes and 30B in 120 minutes.Optimization uses 150 iterations and a minimal number of learnable parameters.
  • Ablation Study: The global orthogonal transformation Rres provides the largest ablation improvement, followed closely by Rdown, while scaling further balances channel variance.The ablation uses LLaMA-2 7B under W4A4KV4 quantization and reports Wiki PPL and zero-shot9 scores.
  • Ablation Study: RiemannAdam delivers the best results with the fewest iterations among the evaluated manifold optimizers.CayleySGD typically requires a higher learning rate, while RiemannSGD needs more iterations.
  • Ablation Study: Setting k to 1,000 gives the best outcomes for the KL-Top loss experiments.Both excessively large and small k values negatively affect optimization; results cover W3-only and W4A4KV4 settings.

6 CONCLUSION

The paper presents OSTQuant as a PTQ method that uses QSUR-guided optimization of weight and activation distributions. Its transformations and KL-Top loss are associated with stronger quantization performance and more efficient deployment across tested LLMs.

  • Conclusion: QSUR measures data space utilization within the quantization space and is complemented by mathematical derivations for transformation optimization.The derivations provide theoretical guidance for optimizing data distributions across the quantization space.
  • Conclusion: OSTQuant learns orthogonal and scaling transformations to optimize weight and activation distributions across the quantization space.The transformation pair is equivalent and can be incorporated into the model’s fully connected layers.
  • Conclusion: KL-Top loss mitigates optimization noise while retaining richer semantic information with limited PTQ calibration data.The method uses the highest-probability logits from the full-precision model.
  • Conclusion: Extensive experiments show OSTQuant outperforms existing quantization methods across various LLMs and benchmarks.The paper frames these results as evidence for optimizing data distributions across the quantization space.

A.1 QUANTIZATION PRELIMINARIES

The preliminaries define uniform quantization as mapping floating-point tensors to discrete integer intervals and then reconstructing them through dequantization. Quantization step size is determined by range and bit width and affects model accuracy.

  • Quantization & Dequantization: Uniform quantization maps a floating-point number to a discrete interval represented by an integer number.The appendix considers only uniform quantization.
  • Quantization & Dequantization: X is the floating-point tensor, XI its quantized counterpart, and X′ the dequantized result.These quantities describe the quantization and reconstruction process.
  • Quantization & Dequantization: The quantization step size s is determined by xmin, xmax, and the number of bits nI.The appendix notes that nI can be 8 bits and that clamp performs truncation.
  • Quantization & Dequantization: The choice of s greatly affects quantized-model accuracy.Static quantization derives s from sample activations, whereas dynamic quantization derives it from runtime statistics.

A.2.1 THE COORDINATES OF THE EXTREMUM POINT AND THE INFLUENCE OF ROTATION MATRIX

This section analyzes how the eigenmatrix, eigenvectors, rotation, and mean vector affect the volume of the quantization hypercube. It identifies conditions associated with minimizing the quantization range and simplifying QSUR′.

  • Eigenmatrix and Volume: With eigenvalues and the mean vector fixed, the hypercube volume is determined by the eigenmatrix Q.The analysis considers each eigenvector qi in Q and its coordinate maximum.
  • Eigenmatrix and Volume: For each eigenvector qi, qimax is defined as the maximum among its coordinates, bounding every coordinate qij by qimax.This coordinate-wise bound is used to derive the subsequent inequality.
  • Rotation and Quantization Range: The derived inequality characterizes the quantization-space volume under the rotation-related conditions analyzed in this section.The supplied passage identifies the inequality but does not state its full expression.
  • Rotation and Quantization Range: When the relevant squared term is minimized, the quantization range is minimized and QSUR′ can be expressed accordingly.The passage links this condition to the subsequent QSUR′ expression.
  • Zero-Mean Case: When the mean vector µ equals zero, QSUR′ simplifies further.This is the zero-mean special case of the preceding expression.

A.2.2 THE BEST ORTHOGONAL MATRIX

This section derives an orthogonal transformation using the covariance structure of the data and a Hadamard matrix with ±1 entries. The transformed covariance is then analyzed to characterize its effect on quantization.

  • Distribution transformation: The analysis starts from a Gaussian input X ∼ N(µ, Σ) and examines how a linear transformation changes its distribution.The transformed distribution remains Gaussian, with transformed mean and covariance.
  • Orthogonal structure: The transformed eigenvectors remain orthonormal after applying the unitary transformation.This preserves the orthonormal structure needed for the subsequent matrix construction.
  • Matrix construction: The target orthogonal transformation is constructed from the eigenvectors of the covariance matrix and a Hadamard matrix.The Hadamard matrix is composed of ±1 entries, with dimensionality d determining its size.
  • Covariance analysis: The resulting transformation is substituted into the covariance expression to determine the covariance matrix after rotation.The derivation explicitly proceeds by solving for T and then evaluating the transformed covariance.

A.2.3 THE BEST TRANSFORM MATRIX

The section combines orthogonal and scaling transformations with WOMI initialization and KL-Top optimization to improve quantization distributions and performance. It also reports efficiency results and describes an extension toward full quantization, whose experiments remain future work.

  • The best transform matrix: OSTQuant uses a diagonal scaling transformation and an orthonormal transformation to optimize weight and activation distributions across the quantization space.The derivation states that the resulting covariance can achieve the maximum QSUR.
  • Initialization: WOMI initializes trainable orthogonal matrices with a Hadamard-based procedure that reduces weight outliers and activation inter-channel disparities.The paper reports smoother weight-channel differences, a smaller quantization space, and lower relative quantization error.
  • Initialization: WOMI achieves lower perplexity and higher few-shot accuracy than random Hadamard initialization under W4A4KV4, W4A16KV16, and W4-only settings.The reported performance gains are greater in W4-only quantization, where weight quantization errors are especially important.
  • Optimization loss: KL-Top alleviates calibration-set overfitting for OSTQuant, whereas SpinQuant alone provides little improvement and can cause degradation even with KL-Top.The comparison is reported on nine zero-shot tasks and WikiText2 perplexity.
  • Quantization trade-offs: Quantizing weights below 4 bits causes a significant accuracy drop, while larger quantized models can outperform smaller floating-point models in accuracy and parameter size.The comparison spans LLaMA models across bitwidths from 2 to 16 bits.
  • Optimization efficiency: OSTQuant requires about 150 optimization iterations, exceeding a 5x speedup over OmniQuant on LLaMA 7B and nearly 2x on LLaMA 70B.The paper attributes the training-time advantage to WOMI initialization and KL-Top loss.
  • Full quantization: The full-quantization extension inserts low-bit quantization nodes for all Transformer-block activation inputs and outputs and adds transformations around ROPE and SiLU.The paper states that full-quantization experiments will be conducted in future work.
Loading 2501.13987v1…