ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

Zhihang Yuan; Yuzhang Shang; Yue Song; Dawei Yang; Qiang Wu; Yan Yan; Guangyu Sun

Source-linked AI summary

ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models

Zhihang Yuan, Yuzhang Shang, Yue Song, Dawei Yang, Qiang Wu, Yan Yan, Guangyu Sun

arXiv:2312.05821v5Dec 10, 2023cs.CL

TL;DR

LLM low-rank decomposition must handle activation outliers and differing layer sensitivities without relying on costly retraining. The paper proposes training-free ASVD and related rank-searching procedures, achieving 10%-30% model compression and stable KV-cache compression at ratios above 40%.

Problem
Training-based low-rank decomposition is costly for LLMs, while post-training decomposition must address activation outliers and variation in layer sensitivity.
Method
ASVD incorporates activation distributions into weight decomposition, while Sensitivity-based Truncation Rank Searching selects layer-specific ranks using calibration.
Results
10%-30% model-size compression is achieved in LLaMA models, with 10%-90% rank reductions across layers.
Takeaways & Limitations
ASVD also compresses KV cache by storing low-rank intermediate activations, with perplexity remaining stable when the KV-cache ratio is above 40%.
Takeaways & Limitations
The rank-searching formulation assumes that layer effects are independent when minimizing perplexity impact under a parameter-size constraint.

Abstract

from arXiv · show

In this paper, we introduce a new post-training compression paradigm for Large Language Models (LLMs) to facilitate their wider adoption. We delve into LLM weight low-rank decomposition, and find that the challenges of this task stem from (1) the distribution variance in the LLM activations and (2) the sensitivity difference among various kinds of layers. To address these issues, we propose a training-free approach called Activation-aware Singular Value Decomposition (ASVD). Specifically, ASVD manages activation outliers by transforming the weight matrix based on the activation distribution. This transformation allows the outliers in the activation matrix to be absorbed into the transformed weight matrix, thereby enhancing decomposition accuracy. Additionally, we propose an efficient iterative calibration process to optimize layer-specific decomposition by addressing the varying sensitivity of different LLM layers. In this way, ASVD can compress a network by 10%-30%. Based on the success of the low-rank decomposition of projection matrices in the self-attention module, we further introduce ASVD to compress the KV cache. By reducing the channel dimension of KV activations, memory requirements for KV cache can be largely reduced. ASVD can further achieve 50% KV cache reductions without performance drop in a training-free manner.

1 INTRODUCTION

LLM low-rank decomposition is promising but difficult to apply after training because activations contain outliers and layers differ in decomposition sensitivity. ASVD addresses these issues without training, compressing model weights and extending decomposition to KV-cache storage.

Compatibility: The decomposition is designed to complement existing compression techniques, including quantization methods.The paper describes this compatibility as a plug-and-play property.
Motivation: Training-based low-rank decomposition is impractical for many LLMs because it requires training data and substantial computational resources.Post-training compression is presented as a more practical alternative.
Challenges: Activation outliers can intensify decomposition error, while differing layer sensitivities make uniform rank selection ineffective.These are identified as the two central challenges for post-training decomposition.
ASVD: ASVD scales weight columns according to input-activation distributions, directing decomposition capacity toward weights associated with activation outliers.The transformation incorporates activation statistics into weight decomposition.
Results: 10%-30% model-size compression is achieved in LLaMA models, while different layers receive 10%-90% rank reductions.The reported rank reduction varies by layer.
KV Cache Compression: ASVD applied to Key/Value projection matrices replaces the high-dimensional KV cache with low-dimensional intermediate activations.The replacement reduces KV-cache memory usage by up to 50%.

2 RELATED WORK

Prior LLM compression work emphasizes quantization, pruning, and distillation, while low-rank factorization has been less explored for LLMs. Existing low-rank methods generally require training, motivating ASVD as a training-free alternative.

LLM Compression: LLM compression research primarily includes weight quantization, network pruning, and knowledge distillation, while low-rank factorization remains less explored.The paper positions its contribution within this broader compression landscape.
Low-rank Decomposition: Fixed-rank decomposition methods use SVD or tensor decomposition and typically fine-tune the resulting factorized network.Fixed-rank approaches also treat decomposition rank as a hyperparameter requiring tuning.
Low-rank Decomposition: Rank-adaptive methods automatically determine or adjust low-rank structure, addressing the fixed-rank limitation.Prior approaches use heuristic search or learn rank structure.
ASVD: Existing low-rank methods had not worked in the LLM era because they required training, motivating ASVD as a post-training decomposition approach.ASVD is presented as the first training-free attempt to compress LLM weights through decomposition in the cited account.

3 METHOD

ASVD addresses two obstacles to post-training LLM decomposition: activation outliers can magnify output error, and layers differ in compression sensitivity. It transforms weights using activation information, applies truncated SVD, and searches layer-specific ranks while extending low-rank representations to KV-cache storage.

Naïve SVD: SVD approximates W with truncated factors Uk, Σk, and Vk^T, where k controls the compression–performance trade-off.The reconstructed matrix is Wk = UkΣkVk^T.
Activation-aware decomposition: Activation outliers can make small weight differences produce large activation-output differences, causing straightforward SVD decomposition to fail for LLMs.The paper evaluates decomposition by output variation rather than weight similarity.
Sensitivity-based truncation: Different layers require different truncation choices because their singular-value distributions imply different compression sensitivities.Layers with concentrated large singular values are generally easier to compress, whereas layers with more uniform smaller values require more tailored treatment.
ASVD: ASVD transforms W with an invertible matrix S derived from activation patterns, applies truncated SVD to WS, and reconstructs an approximation of W.The transformation makes the weight matrix more compatible with the input activation distribution.
Sensitivity-based truncation: STRS evaluates layer sensitivity across truncation ratios using calibration-set perplexity to select suitable singular-value truncations.The analysis reports noticeable sensitivity variation across layers and identifies lower parameter ratios as producing higher perplexity.
KV-cache compression: ASVD stores intermediate key and value activations in r dimensions instead of N dimensions, reducing KV-cache storage from two L × N matrices to two L × r matrices.Here r ≪ N, so the low-rank cache can reduce memory usage and support larger batches or longer sequences.

4 EXPERIMENTS

Experiments evaluate ASVD across LLaMA models, compression settings, KV-cache reduction, quantization compatibility, and decomposed-network structure. Results show that sensitivity-aware compression preserves performance in selected regimes while reducing model and cache parameters.

Experimental settings: ASVD compression is evaluated on LLaMA and LLaMA-2 models using Wikitext perplexity and MMLU tasks.The evaluation includes models ranging from 7 billion to 13 billion parameters and uses 32 Wikitext calibration samples of 2048 tokens each.
Weight compression: As the parameter ratio decreases, perplexity increases, with a plateau above 0.9 and a rapid increase below 0.85.STRS assigns different truncation levels according to layer sensitivity; lower ratios increasingly affect sensitive layers.
Weight compression: ASVD+ improves ASVD especially at high compression and outperforms SVD-LLM when the compression ratio is less than 30%.The comparison attributes the advantage to layer-specific compression ratios from Sensitivity-based Truncation Rank Searching.
KV-cache compression: KV-cache perplexity remains stable above a 40% cache ratio but decreases when the ratio falls below 40%.ASVD compresses the transformer k and v projections to reduce the KV-cache channel dimension.
Quantization compatibility: ASVD is combined with RTN, NF4, and AWQ quantization, with 8-bit quantization having negligible impact and 4-bit quantization causing further accuracy deterioration.The experiments first decompose the network with ASVD and then quantize the decomposed weights.
Decomposed-network analysis: MLP projections receive minimal compression, while q and k projections can be compressed substantially and the first layer can undergo substantial compression.The V projection is relatively less compressed, and most other layers show similar compression rates except two middle layers.
Decomposed-network analysis: ASVD’s computation ratio mirrors its weight-number compression ratio, reducing both deployment overhead and inference computation.The reported ratio is expressed as Ck/C and is equivalent to the parameter ratio.

5 CONCLUSION

The study presents ASVD and STRS as training-free techniques for more accurate and efficient LLM decomposition, including KV cache compression.

ASVD and STRS address activation outliers and varying layer sensitivities in training-free LLM compression.Together, they support more accurate and efficient decomposition while reducing memory usage and computational demands.
The techniques reduce memory usage and computational demands while maintaining model performance.
ASVD also extends to KV cache compression, supporting broader applicability in real-world scenarios.

A.1 IMPACT STATEMENTS AND LIMITATIONS

The method is intended to make LLMs more accessible, but its practical scope is constrained by quantization effects, MLP compression difficulty, and calibration cost.

Impact: ASVD improves LLM efficiency and accessibility by lowering deployment costs and hardware barriers, including for edge computing.
Limitations: 4-bit quantization can slightly decrease predictive accuracy, whereas 8-bit quantization has minimal effects on original and ASVD-compressed networks.
Limitations: ASVD struggles to compress MLP layers because their high-dimensional feature mappings create greater computational burdens.
Limitations: Evaluating each layer’s sensitivity requires forward propagation to calculate perplexity, demanding significant computational resources.

A.2 RELEASE SAFEGUARDS

Because compression can ease sharing and deployment of powerful models with misuse risks, the study implements access control and prohibits unethical applications.

ASVD’s compression capabilities could enable easier sharing and deployment of powerful models that carry misuse risks.
Users must agree to terms prohibiting unethical applications as part of the implemented access control.
The study does not claim that compression itself mitigates potential misuse by malicious actors.

A.3 INFERENCE COST WITH DECOMPOSED LLMS

ASVD changes a linear layer’s computation from a full matrix multiplication to a decomposed form, with the computation ratio matching the weight compression ratio.

The original linear layer computes Y = WX for input matrix X and weight matrix W.
After ASVD decomposition, the matrix multiplication is replaced by an approximate decomposed computation.
The decomposed computation uses Ck = tkm + tkn MAC operations.
The computation ratio mirrors the weight-number compression ratio, reducing weight storage, transfer overhead, and inference computation.

A.4 BINARY SEARCH FOR TRUNCATION RANKS

The truncation-rank search supports either a performance target or a parameter target. It iteratively adjusts layer ranks, using calibration perplexity for performance targeting but no calibration dataset for parameter targeting.

Target definitions: Performance targeting searches for truncation ranks that achieve a specified network performance, such as a perplexity threshold.Parameter targeting instead searches for ranks that meet a specified parameter budget.
Parameter-target search: During parameter-target search, each layer receives the minimum available truncation rank from the searched candidate range, after which total parameters are checked against the target.Layers without a qualifying candidate retain their existing truncation ranks.
Performance-target search: The performance-target algorithm uses binary search pointers over candidate configurations and adjusts each layer’s truncation rank around the middle pointer.For each layer, the selected rank is the smallest candidate rank to the right of the middle pointer.
Performance-target search: Each performance-target iteration evaluates the updated network on a calibration dataset using perplexity as the primary metric.If perplexity meets the threshold, the high pointer moves toward a lower-rank configuration; otherwise, the search moves toward higher ranks.
Parameter-target search: Parameter targeting sorts layer-rank candidates by sensitivity before searching for a configuration that meets the parameter target.The algorithm is specified in Algorithm 1 and does not require a calibration dataset.

A.5 DIFFERENCE WITH TENSORGPT.

ASVD differs from TensorGPT by targeting the full LLM weight spectrum with rank-adaptive decomposition rather than only compressing token embeddings at fixed ranks. Additional experiments select activation-aware hyperparameters and distribute singular values across both factors to improve quantization behavior.

Difference from TensorGPT: TensorGPT compresses the token embedding layer with Tensor-Train Decomposition, whereas ASVD targets the entire LLM weight spectrum.The comparison frames ASVD as addressing a broader portion of LLM compression than TensorGPT.
Difference from TensorGPT: ASVD supports rank-adaptive low-rank decomposition, contrasting with TensorGPT’s fixed or predetermined ranks.This distinction concerns how decomposition ranks are selected across the compressed model.
Comparison with FWSVD: FWSVD uses Fisher information to weight prediction-relevant parameters and requires training, while ASVD is compared against it on LLaMA-7b.The supplied comparison passage states that ASVD outperforms FWSVD comprehensively.
Hyperparameter exploration: The hyperparameter study on OPT-125m varies α across 0.1, 0.25, 0.5, 1, and 2 and compares absolute mean with absolute maximum activation significance.These choices determine the transform matrix S used in subsequent ASVD experiments.
Hyperparameter exploration: ASVD’s activation-aware methods outperform standard SVD+STRS, with α = 0.5 and Absolute Mean Value selected because mid-range α and mean-based significance perform best.The reported hyperparameter results are summarized for perplexity on Wikitext2.
Singular-value fusion: After decomposition, singular values are evenly absorbed into both factor matrices, producing more uniform factors and reducing channel disparity and quantization error.This fusion strategy is evaluated through perplexity on Wikitext-2 under different absorbing strategies.