PaperlayerView exact revision on arXiv

Source-linked AI summary

SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression

Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, Mi Zhang

arXiv:2503.12340v1cs.CL

TL;DR

Large Language Models are costly to deploy, while existing SVD compression methods can retain high truncation loss. SVD-LLM V2 allocates heterogeneous compression ratios and performs loss-optimized truncation, outperforming state-of-the-art SVD-based methods across broad evaluations.

  • Problem

    LLMs’ immense sizes hinder practical deployment, and existing SVD-based compression methods remain limited by high truncation loss.

  • Method

    SVD-LLM V2 uses theoretical truncation loss to assign unique compression ratios across weight matrices and applies loss-optimized weight truncation.

  • Results

    SVD-LLM V2 consistently outperforms state-of-the-art SVD-based methods across ten datasets and five LLMs, with up to 28% lower perplexity than structured-pruning methods under a 7 GB budget.

  • Takeaways & Limitations

    SVD-LLM V2 achieves stronger compression performance and up to 2.71× inference throughput speedup over original LLMs on a single NVIDIA A100 GPU.

  • Takeaways & Limitations

    Under a 90% compression ratio, SVD-LLM V2 retains a small performance gap compared with state-of-the-art quantization methods.

Abstract

from arXiv · show

Despite significant advancements, the practical deployment of Large Language Models (LLMs) is often hampered by their immense sizes, highlighting the need for effective compression techniques. Singular Value Decomposition (SVD) is a promising LLM compression technique. However, existing SVD-based compression methods fall short in reducing truncation losses, leading to less competitive performance in compressed models. In this work, we introduce SVD-LLM V2, a SVD-based LLM compression method that optimizes singular value truncation in SVD compression with two techniques. First, SVD-LLM V2 proposes to use theoretical truncation loss of weight matrices to assign a unique compression ratio to each weight matrix at different layers to accommodate weight redundancy heterogeneity. Second, SVD-LLM V2 proposes loss-optimized weight truncation to ensure that the truncated singular values result in a lower and more stable truncation loss in practice. We evaluate SVD-LLM V2 on ten datasets and five LLMs at various scales. Our results show SVD-LLM V2 outperforms state-of-the-art SVD-based LLM compression methods. Our code is available at https://github.com/AIoT-MLSys-Lab/SVD-LLM

1 Introduction

LLM size limits accessibility, motivating post-training compression methods. SVD-LLM V2 addresses SVD-LLM’s truncation-loss limitations with heterogeneous compression ratios and loss-optimized truncation, achieving stronger reported results.

  • LLMs’ substantial resource requirements limit widespread accessibility, motivating model compression and post-training approaches that avoid resource-intensive retraining.
  • SVD compression is hardware-friendly, but existing SVD-based methods still seek to reduce truncation loss to preserve compressed-model accuracy.
  • SVD-LLM applies one compression ratio across weight matrices, overlooking heterogeneous redundancy across layers.
  • SVD-LLM V2 assigns each weight matrix a layer- and type-specific compression ratio using theoretical truncation loss, then replaces Cholesky truncation with two rounds of SVD.
  • SVD-LLM V2 consistently outperforms state-of-the-art SVD-based compression methods across ten datasets and five LLMs.
  • Up to 28% lower perplexity is achieved than structured-pruning methods under a 7 GB memory budget, while 1-bit quantization comparisons show 5% lower perplexity than PB-LLM.
  • Up to 2.71× throughput speedup over original LLMs is achieved on a single NVIDIA A100 GPU.

2 Related Work

Post-training LLM compression includes pruning, quantization, and low-rank approximation. SVD-based methods reduce truncation loss through techniques such as scaling, whitening, and activation-aware low-rank approximation.

  • Post-training compression avoids resource-intensive retraining and commonly includes unstructured pruning, structured pruning, quantization, and low-rank approximation.
  • SVD compresses matrices by truncating the smallest singular values and constructing two smaller lower-rank matrices.
  • Existing post-training SVD methods minimize truncation loss L = ||WX − W′X||F, where W is original, X is activation, and W′ is compressed low-ranking weight matrix.
  • ASVD scales weight matrices with a diagonal matrix to normalize input-channel impact, while whitening methods further mitigate that impact with a theoretical minimum-truncation guarantee.

3 SVD-LLM V2

SVD-LLM V2 combines heterogeneous compression-ratio allocation with loss-optimized weight truncation to reduce practical truncation loss while accommodating layer- and matrix-type differences.

  • Heterogeneous Compression Ratio Allocation: SVD-LLM V2 groups weight matrices by type, computes theoretical truncation losses, and assigns each matrix a distinct compression ratio.Groups include query and key matrices in attention blocks and Gate and Up matrices in MLP blocks.
  • Heterogeneous Compression Ratio Allocation: A query matrix in layer 27 has higher truncation loss than one in layer 1, motivating smaller compression for layer 27 and larger compression for layer 1.The comparison uses LLaMA-3 8B on WikiText-2 at a 50% compression ratio.
  • Heterogeneous Compression Ratio Allocation: The allocation uses inverse-log-normalized theoretical losses to determine compression ratios without measuring end-to-end perplexity.This uses truncation loss as an efficiently obtainable allocation signal.
  • Loss-optimized Weight Truncation: SVD-LLM V2 replaces Cholesky-based truncation with two SVD rounds on the activation covariance and transformed weights.The transformed matrix is W × Us × √Ss, followed by truncation of its singular values.
  • Loss-optimized Weight Truncation: The designed truncation is proven to achieve the theoretical minimum truncation loss at the assigned compression ratio.The compressed weight is reconstructed from the second SVD after truncating its singular values.
  • Loss-optimized Weight Truncation: On two LLaMA-3 8B matrices, SVD-LLM V2 shows better numerical stability and lower truncation loss than SVD-LLM.The comparison uses 256 C4 calibration samples and 20% and 60% compression ratios; SVD-LLM can fail when XX^T is not positive definite.

4 Experiments and Analysis

SVD-LLM V2 is evaluated across multiple LLMs, scales, datasets, compression ratios, and compression baselines. It consistently improves compression quality over SVD-based methods while also providing faster hardware inference and competitive results against pruning and quantization.

  • Experimental Setup: Experiments cover five LLMs at various scales and ten datasets spanning language modeling, classification, and generation tasks.The evaluation includes OPT-6.7B, LLaMA-7B, LLaMA-13B, LLaMA-30B, and LLaMA-3 8B; calibration uses 256 WikiText-2 samples.
  • Performance Comparison: SVD-LLM V2 consistently outperforms SVD-based baselines across three LLMs and ten datasets, including up to 42% perplexity reduction and 9% accuracy improvement on LLaMA-3 8B versus SVD-LLM.It also outperforms all baselines on LLaMA-13B and LLaMA-30B under 20% compression.
  • Performance Comparison: As compression increases from 20% to 80%, SVD-LLM V2 remains ahead of competing SVD methods and its performance gain over the best baseline increases.Figure 4 reports perplexity on WikiText-2 and average accuracy across six classification datasets.
  • Compression Speed: 18 minutes are required for SVD-LLM V2 to compress LLaMA-7B, compared with about 15 minutes for SVD-LLM, 5.5 GPU hours for ASVD, and 6 GPU hours for FWSVD.FWSVD's longer time is attributed to gradient calculation.
  • Inference Speedup: 1.29x, 1.63x, 2.08x, and 2.71x speedups are achieved at 20%, 40%, 60%, and 80% compression ratios, respectively, on a single NVIDIA A100 GPU.The evaluation fixes batch size at 4, prefill length at 1024, and decoding length at 256.
  • Comparisons with Other Compression Methods: 28% lower perplexity and 13% higher average accuracy are achieved than structured pruning methods under a 7 GB memory budget.SVD-LLM V2 also achieves 5% lower perplexity than PB-LLM and 69% lower perplexity than BiLLM when combined with 2-bit quantization.

5 Conclusion

SVD-LLM V2 is a post-training SVD-based compression method that reduces truncation loss through heterogeneous ratio allocation and loss-optimized weight truncation. Its evaluation shows superiority over state-of-the-art SVD-based post-training compression methods.

  • SVD-LLM V2 balances truncation loss across weight matrices using heterogeneous compression ratio allocation.
  • SVD-LLM V2 introduces loss-optimized weight truncation to achieve lower and more stable truncation loss.
  • SVD-LLM V2 outperforms state-of-the-art SVD-based post-training LLM compression methods.

6 Limitations

SVD-LLM V2 outperforms existing SVD-based LLM compression methods, but a small performance gap remains compared with state-of-the-art quantization methods at a 90% compression ratio.

  • At a 90% compression ratio, SVD-LLM V2 has a small performance gap compared with state-of-the-art quantization methods.
  • The authors identify closing this gap as a direction for future work.
Loading 2503.12340v1…