PaperlayerView exact revision on arXiv

Source-linked AI summary

SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression

Xin Wang, Yu Zheng, Zhongwei Wan, Mi Zhang

arXiv:2403.07378v5cs.CLcs.LG

TL;DR

LLM compression is needed because model size creates substantial deployment resource demands, while existing SVD methods can incur compression loss from truncation and lack post-truncation updates. SVD-LLM combines truncation-aware data whitening with sequential low-rank parameter updates and outperforms prior methods across broad evaluations, especially at high compression ratios.

  • Problem

    Existing SVD-based LLM compression methods may incur higher compression loss when smaller singular values are truncated and do not update compressed weights afterward.

  • Method

    SVD-LLM uses truncation-aware data whitening to map singular values to compression loss and sequential low-rank approximation to update compressed parameters.

  • Results

    SVD-LLM outperforms state-of-the-art compression methods across 10 datasets, seven models, three LLM families, and three scales, especially at high compression ratios.

  • Takeaways & Limitations

    SVD-LLM provides a post-training route to strong compression performance without requiring retraining from scratch.

  • Takeaways & Limitations

    The truncation-aware whitening technique depends on singular-value distributions with suitable properties, motivating additional spectrum analysis.

Abstract

from arXiv · show

The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitates LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising solution for LLM compression. However, state-of-the-art SVD-based LLM compression methods have two key limitations: truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. In this work, we propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods. SVD-LLM incorporates a truncation-aware data whitening technique to ensure a direct mapping between singular values and compression loss. Moreover, SVD-LLM adopts a parameter update with sequential low-rank approximation to compensate for the accuracy degradation after SVD compression. We evaluate SVD-LLM on 10 datasets and seven models from three different LLM families at three different scales. Our results demonstrate the superiority of SVD-LLM over state-of-the-arts, especially at high model compression ratios. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

1 INTRODUCTION

LLM compression is motivated by substantial resource demands, but existing SVD-based methods can suffer severe degradation as compression increases. SVD-LLM addresses this through truncation-aware whitening and sequential parameter updates, and reports broad gains across models, datasets, and compression methods.

  • Motivation: LLMs’ substantial resource demands motivate post-training compression methods that avoid retraining from scratch.The broader compression landscape includes quantization, unstructured pruning, and structured pruning.
  • Limitations of prior SVD methods: Existing SVD methods can degrade severely at higher compression ratios because singular values lack a direct mapping to compression loss and truncated weights are not updated.The latter omission limits compensation for accuracy degradation, especially at high compression ratios.
  • SVD-LLM: SVD-LLM uses truncation-aware data whitening to identify singular values whose truncation incurs minimal compression loss.The technique is theoretically supported and establishes a direct mapping between singular values and compression loss.
  • SVD-LLM: SVD-LLM sequentially fine-tunes decomposed low-rank matrices to recover global accuracy after compression.This parameter-update strategy is designed to compensate for accuracy degradation caused by compression.
  • Evaluation and findings: Across 10 datasets, seven models, three LLM families, and three scales, SVD-LLM outperforms SVD, pruning, and quantization baselines.The evaluation also covers GPU and CPU deployment, and the reported findings include inference speedup and memory reduction.

2 RELATED WORK

Related work applies post-training compression through pruning, quantization, and low-rank approximation. SVD is attractive for reducing matrix size, but existing approaches face activation-memory, parameter-importance, and activation-distribution challenges.

  • Large Language Model Compression: Post-training LLM compression methods are commonly grouped into unstructured pruning, structured pruning, quantization, and low-rank approximation.These approaches avoid the resource-intensive retraining required by conventional compression for billion-parameter LLMs.
  • SVD for Language Model Compression: SVD compresses matrices by approximating them with two smaller low-ranking matrices.The method is widely used for model compression, including optimal compression for smaller language models such as BERT.
  • SVD for Language Model Compression: Applying SVD directly to LLM weight matrices can cause large compression loss when parameter importance is ignored.FWSVD addresses parameter importance with Fisher information but requires complex gradient calculations and substantial resources.

3 SVD-LLM

SVD-LLM combines truncation-aware data whitening with sequential low-rank parameter updates to reduce compression loss and recover accuracy after SVD truncation.

  • Overview: SVD-LLM uses calibration activations, Cholesky-derived whitening, SVD truncation, and sequential low-rank approximation for post-training compression.The method whitens activations before decomposing WS, then updates the compressed parameters after truncation.
  • Motivation: Existing SVD methods can incur higher compression loss when truncating smaller singular values because singular values are not directly aligned with loss.ASVD examples show truncating 0.1 produced loss 1.1, whereas truncating 0.9 produced loss 0.7.
  • Truncation-Aware Data Whitening: Whitening makes the compressed activation orthonormal, establishing a direct mapping between truncated singular values and compression loss.SVD-LLM derives the whitening matrix through Cholesky decomposition and applies SVD to WS.
  • Truncation-Aware Data Whitening: Theorem 3.2 states that truncating one singular value produces compression loss equal to that singular value.The result assumes S is the Cholesky decomposition of XXT.
  • Truncation-Aware Data Whitening: Truncating the smallest singular values yields the lowest compression loss because squared loss equals the sum of their squared values.For multiple truncated values, the loss is the square root of the sum of their squares.
  • Parameter Update with Sequential Low-Rank Approximation: SVD-LLM separately applies LoRA updates to the two compressed factors, sequentially freezing one factor while updating the other.The resulting low-rank updates are added to the compressed factors to form the final weights.

4 EXPERIMENTS AND ANALYSIS

SVD-LLM is evaluated broadly against SVD, pruning, and quantization baselines, with experiments spanning models, datasets, compression ratios, efficiency, and component ablations. Results show stronger accuracy and practical efficiency, particularly at high compression ratios.

  • Baselines: SVD-LLM is compared with vanilla SVD, FWSVD, ASVD, pruning methods, and quantization methods, including both post-training and training-required approaches.The SVD comparison includes SVD-LLM (W), which applies truncation-aware whitening without LoRA fine-tuning, for fair comparison with baselines lacking LoRA updates.
  • Experimental setup: Evaluation covers seven models from three LLM families, three scales, and 10 datasets, using GPU and CPU experiments.The models include LLaMA, OPT, Vicuna, and Mistral variants; calibration uses 256 WikiText-2 samples, while parameter updates use 50K Alpaca samples.
  • Performance under different compression ratios: At 40% compression and above, SVD-LLM reduces perplexity by more than 99% on two language-modeling datasets and achieves over 400% higher average accuracy on six classification datasets.Across compression ratios from 20% to 80%, SVD-LLM and SVD-LLM (W) consistently outperform vanilla SVD, FWSVD, and ASVD.
  • Performance under different compression ratios: At 60% compression and above, the three SVD baselines lose generation ability on TruthfulQA and GSM8K, whereas SVD-LLM still generates well at 80% compression.The baseline generation results are reported as zero, while SVD-LLM retains good generation under the highest tested compression ratio.
  • Generalization across models and scales: SVD-LLM consistently outperforms baselines across four LLMs at 20% compression and across LLaMA-13B and LLaMA-30B.The results also report more stable performance across different LLMs, especially relative to vanilla SVD and FWSVD.
  • Ablation study: Combining truncation-aware whitening with sequential low-rank parameter updates performs best, while whitening contributes more than the update component alone.All ablated variants outperform ASVD, and at compression ratios of 40% or higher they reduce perplexity by more than 99% relative to ASVD.
  • Ablation study: Changing calibration-data size, sampling seed, or source dataset changes final performance by no more than 3%.This indicates limited sensitivity to the tested calibration-data characteristics.

5 CONCLUSION

SVD-LLM is a post-training compression method that combines truncation-aware data whitening with sequential low-rank parameter updates. Evaluations across diverse models and datasets show superiority over state-of-the-art methods, especially at high compression ratios.

  • SVD-LLM uses truncation-aware data whitening to guide singular-value removal with minimal compression loss.It also applies sequential low-rank approximation to compensate for accuracy degradation caused by truncation.
  • SVD-LLM was evaluated on 10 datasets and seven models from three LLM families at three scales.
  • SVD-LLM outperforms state-of-the-art compression methods, particularly at high model compression ratios.

A.1 PSEUDOCODE OF SVD-LLM

The SVD-LLM pseudocode calibrates whitening matrices from sampled sentences, compresses weight matrices through whitened SVD truncation, and then updates the compressed parameters sequentially.

  • SVD-LLM: For each weight matrix, SVD-LLM decomposes the whitened matrix, truncates its smallest singular values, and forms two low-rank matrices.The decomposed matrices replace the original weight in the compressed model.
  • Parameter Update with Sequential Low-rank Approximation: After replacing the weights, SVD-LLM applies parameter update with sequential low-rank approximation to produce the final compressed model.The pseudocode passes the truncated model to the update procedure before returning it.
  • Truncation-Aware Data Whitening: SVD-LLM randomly collects calibration sentences and derives whitening matrices from weight activations using Cholesky decomposition.Algorithm 2 computes each whitening matrix from XX^T for the corresponding weight's calibration activation.

A.2 COMPRESSION LOSS OF ASVD

ASVD scales the weight matrix using activation-derived channel statistics before SVD truncation, but its compression error is not directly determined by the singular values.

  • ASVD formulation: ASVD formulates the linear layer as Y = (WS0)S−1 0 X using a diagonal scaling matrix that reflects input-channel significance.
  • ASVD truncation: ASVD compresses WS0 by retaining the largest m singular values.
  • Compression loss: ASVD's compression-loss expression remains a complex function involving the activation X, scaling matrix S0, and singular vectors.
  • Compression loss: Consequently, compression error is not directly related to singular values, so truncating the smallest values can produce suboptimal loss.

A.3 COMPRESSION LOSS OF SVD-LLM

SVD-LLM reformulates the linear layer with a whitening matrix so that squared compression loss equals the sum of squared truncated singular values. This makes removing the smallest singular values loss-minimizing.

  • SVD truncation: Keeping the largest m of r singular values defines compression, with loss determined by the discarded components.
  • SVD-LLM formulation: SVD-LLM formulates the linear layer as Y = (WS)S−1X using a whitening matrix S.
  • Compression loss: The squared loss L2 equals the sum of squared singular values, making truncation of the smallest singular values achieve the lowest compression loss.

A.4 SPECTRUM ANALYSIS OF SINGULAR VALUES DECOMPOSED BY SVD-LLM

SVD-LLM examines singular-value spectra across decomposed LLaMA-7B matrices to assess whether SVD compression is applicable.

  • The analysis studies singular-value spectra across layers for decomposed LLaMA-7B matrices.

A.5 COMPARISON WITH DRONE

SVD-LLM matches DRONE's theoretical compression optimum while reducing memory and computation requirements for LLM compression. Its incremental whitening-statistics computation avoids storing full activations and improves practical feasibility.

  • Optimality: SVD-LLM achieves the same theoretical compression loss as DRONE, equal to the rank-k SVD truncation loss of WX.
  • Memory efficiency: SVD-LLM incrementally updates XXT, requiring only 3.6GB for LLaMA-7B compression with 5,000 calibration samples.This avoids storing the full input activations required by DRONE.
  • Compression speed: SVD-LLM is approximately three times faster than DRONE when compressing randomly generated matrices at 50% compression.
  • Numerical stability: SVD-LLM has better numerical stability than DRONE and consequently lower empirical compression losses.The comparison attributes DRONE's instability to additional SVD and inverse operations on large matrices.

A.6 COMPARISON WITH FLAP

SVD-LLM is evaluated against FLAP and other compression configurations across perplexity, accuracy, speed, generation quality, and fine-tuning strategy. It generally retains stronger performance, while high compression and KV-cache use remain practical challenges.

  • Comparison with FLAP: SVD-LLM consistently outperforms FLAP on WikiText-2 perplexity for compressed LLaMA-7B, especially at high compression ratios.
  • Compression speed: SVD-LLM completes LLaMA-7B compression in 3.5 GPU hours versus 5.5 hours for ASVD, a 36% speedup.
  • Generation quality: At 80% compression, SVD-LLM can still generate text, whereas ASVD loses generation ability at 40% compression.
  • Ablation studies: Sequential LoRA fine-tuning consistently outperforms normal LoRA fine-tuning for SVD-LLM across compression ratios.The analysis links simultaneous optimization of both low-rank matrices with unstable fine-tuning and poorer performance.
  • Ablation studies: SVD-LLM with sequential LoRA fine-tuning consistently outperforms ASVD with the same fine-tuning strategy under various compression ratios.
  • Limitations: SVD-LLM still requires improved accuracy at high compression ratios, rather than relying on combination with quantization methods.
  • Limitations: Compressing the KV cache with SVD-LLM impacts inference speed because recovering original key and value states adds computation.
  • Limitations: SVD-LLM can generate repeated or otherwise low-quality content despite achieving low perplexity, even at low compression ratios.
Loading 2403.07378v5…