PaperlayerView exact revision on arXiv

Source-linked AI summary

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

Qinsi Wang, Jinghan Ke, Masayoshi Tomizuka, Yiran Chen, Kurt Keutzer, Chenfeng Xu

arXiv:2502.02723v1cs.LG

TL;DR

Dobi-SVD addresses the difficulty of compressing LLMs with SVD while retaining pretrained-model performance and reducing inference cost. It directly truncates activations, differentiably learns truncation positions, reconstructs weights with IPCA, and uses remapping to overcome SVD’s truncation limitation. At a 0.4 compression ratio, it compresses LLaMA-7B to a WikiText2 PPL of 9.07 and reports substantial inference acceleration.

  • Problem

    SVD-based LLM compression suffers from difficult truncation-position selection, inadequate weight updates, and information loss caused by truncation, limiting its performance and practical use.

  • Method

    Dobi-SVD directly truncates activations, learns truncation positions with differentiable optimization, updates weights using the Eckart-Young-Mirsky theorem and IPCA, and applies quantized memory remapping.

  • Results

    At a 0.4 compression ratio, Dobi-SVD compresses LLaMA-7B to a 9.07 PPL on WikiText2, a 78.5% improvement over the previous state-of-the-art SVD compression method.

  • Takeaways & Limitations

    Dobi-SVD makes SVD-based compression competitive for LLMs and extends the approach to vision-language models and hardware acceleration.

  • Takeaways & Limitations

    Traditional SVD compression remains constrained by truncation-induced information loss; for square matrices, matching the original model size can require truncating half the singular values.

Abstract

from arXiv · show

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression.

1 INTRODUCTION

Dobi-SVD presents an SVD-based approach to reduce LLM inference cost while preserving performance by truncating activations, reconstructing weights, and remapping compressed representations.

  • The method targets expensive LLM inference caused by growing parameter counts and limited hardware memory, especially for resource-constrained and latency-sensitive applications.The paper motivates compression for robotics, edge devices, and interactive entertainment while seeking to preserve pretrained-model performance.
  • The paper identifies three SVD-compression challenges: selecting layer-specific truncation positions, updating weights without losing activation information, and overcoming truncation-induced information loss.Different weight matrices have different sensitivities, direct weight truncation degrades performance, and conventional square-matrix compression may require truncating half the singular values.
  • Dobi-SVD addresses these challenges with differentiable truncation-position training, stable SVD backpropagation, IPCA-based weight updates, and remapping based on singular-value concentration.The approach directly truncates activations rather than merely minimizing activation distance, then reconstructs weights and establishes a bijection between truncation position and compression ratio.
  • Dobi-SVD achieves a PPL of 9.07 on WikiText2 when compressing LLaMA-7B to a 0.4 compression ratio, outperforming prior SVD and pruning methods.This corresponds to a 78.5% improvement over the previous state-of-the-art SVD method and a 9.83% improvement over the best pruning methods requiring post-training and fine-tuning.
  • Dobi-SVD reaches 25.97 tokens/second on an NVIDIA TITAN Xp 12GB GPU, providing a 12.4× speedup over the original LLaMA-7B model.The method is described as hardware-agnostic and generalizes across different hardware targets.
  • Dobi-SVD also improves compressed vision-language models, increasing LLAVA V1.5-7B throughput by 1.2 times and accelerating OpenVLA by 17.6% at a 0.4 compression ratio.The OpenVLA result maintains nearly lossless performance on an NVIDIA A100, while LLAVA performance improves on the Pope-random dataset.

2 PRELIMINARIES

SVD compression decomposes weight matrices, retains leading singular values, and reconstructs them as two smaller matrices. Dobi-SVD instead identifies activation truncation as the theoretically and empirically preferable paradigm, then optimizes truncation values through differentiable training.

  • Mathematical Expression of SVD for LLM: SVD decomposes W into UΣV^T, retains the top k singular values, and reconstructs the compressed matrix as W1 = U√Σk and W2 = √ΣkV.The resulting factors have dimensions m×k and k×n, with compression ratio k(m + n)/(m × n).
  • Mathematical Expression of SVD for LLM: SVD compression can reduce a matrix to two low-rank factors after decomposition and truncation, with retained singular values controlling the trade-off.The compression ratio is determined by k relative to the original matrix dimensions.
  • Basic Propositions: The Eckart-Young-Mirsky theorem characterizes the best rank-k approximation, while rank multiplication bounds constrain the rank of compressed products.These propositions provide mathematical foundations for the truncation and reconstruction analysis.
  • Differentiable Algorithm for Finding Optimal k: Dobi-SVD finds layer-specific truncation values by smoothly truncating activation singular values and optimizing k with a task loss plus compression-ratio penalty.The algorithm freezes other network parameters while updating the truncation values toward the target compression ratio.
  • Motivation: Truncate Weights or Activations?: Directly truncating activations produces the optimal rank-k approximation and outperforms weight truncation and existing activation-aware methods.The paper reports this conclusion at both module and model levels, with Tables 1 and 2 supporting the comparisons.

3 DOBI-SVD METHOD

Dobi-SVD compresses LLMs by differentiably selecting activation truncation positions, reconstructing rank-k weights with IPCA, and remapping compression ratios to avoid information loss from conventional SVD truncation.

  • 3.1 Q1: HOW TO GET THE OPTIMAL TRUNCATION POSITION?: Only 224 parameters are trainable for Llama-7B, requiring 8 GPU hours for truncation-position optimization.
  • 3.1 Q1: HOW TO GET THE OPTIMAL TRUNCATION POSITION?: Differentiable SVD stabilizes backpropagation and enables end-to-end optimization of each layer’s truncation position.The method smooths discrete positions with tanh and trains truncation parameters using task loss plus a compression-ratio penalty.
  • 3.2 Q2: HOW TO UPDATE WEIGHTS OPTIMALLY?: IPCA reconstructs the theoretical optimal rank-k weight matrix from sequentially processed projected activations while reducing PCA memory usage.The reconstruction avoids infeasible inversion because input matrices are generally not full-rank square matrices.
  • 3.3 HOW TO OVERCOME THE LONG-OVERLOOKED TRUNCATION LIMITATION?: Traditional SVD can lose information even when model size is unchanged because its compression-ratio mapping forces truncation below the matrix rank.For square matrices, setting compression ratio r = 1 truncates half the singular values.
  • 3.3 HOW TO OVERCOME THE LONG-OVERLOOKED TRUNCATION LIMITATION?: Dobi-SVD remaps compression ratios and truncation positions bijectively, using r = k/rank(W), and implements this through quantized storage.
  • 3.2 Q2: HOW TO UPDATE WEIGHTS OPTIMALLY?: Direct activation truncation yields the optimal k-rank approximation and is theoretically shown to minimize training loss better than weight truncation.
  • 3.4 CONCLUSION: TWO NEW PERSPECTIVES OF DOBI-SVD: Together, activation-to-weight reconstruction and limitation-remapping define Dobi-SVD’s two new perspectives for SVD-based compression.

4 EXPERIMENTS

Experiments show that Dobi-SVD preserves language-model and reasoning performance at aggressive compression, while reducing training and inference costs across models and hardware. Component analyses support differentiable truncation, IPCA-based updates, and quantized remapping as key design choices.

  • 4.1 MAIN RESULTS: At 0.4 compression ratio, Dobi-SVD achieves 9.70 PPL on Wikitext-2, versus 43,104 for ASVD and 458 for SVD-LLM.The result is reported for LLaMA-7B and shows substantially better in-domain performance than prior SVD methods.
  • 4.1 MAIN RESULTS: At 0.4 compression ratio, Dobi-SVD retains 40% average accuracy across seven zero-shot datasets, versus 29% for ASVD and 31% for SVD-LLM.At 0.8 compression ratio, its average performance drop is 3.14%.
  • 4.1 MAIN RESULTS: Dobi-SVD outperforms FLAP on all four compared tasks while training 224 truncation-position parameters instead of LLM-Pruner’s additional 1.2 billion parameters.The comparison places Dobi-SVD and pruning methods at comparable compression ratios and evaluates their performance–memory trade-off.
  • 4.1 MAIN RESULTS: Across LLaMA-2-7B and LLaMA-3.1-8B, Dobi-SVD remains consistent with LLaMA-7B results and achieves higher accuracy than pruning methods on five commonsense tasks.On LLaMA-3.1-8B at 0.8 compression, its perplexity drops 8.7%, compared with 33% for SliceGPT and LLM-Pruner.
  • 4.2 ANALYSIS: Differentiable truncation performs comparably with batch size 16 and 256, while IPCA keeps memory usage nearly constant as matrix dimensionality increases.The guided-truncation experiments evaluate layer 29 alone and layers 29–31; the memory comparison contrasts IPCA with full PCA.

5 CONCLUSION

The paper concludes that Dobi-SVD makes SVD-based compression viable by combining activation truncation with differentiable position optimization, efficient weight updates, and quantized remapping. It reports low performance loss at aggressive compression alongside substantial hardware acceleration.

  • 5 CONCLUSION: Dobi-SVD addresses truncation-position selection, efficient weight updates, and information loss in SVD-based LLM compression.Its solutions use differentiable optimization, an Eckart-Young-Mirsky-based update method, and quantized memory remapping.
  • 5 CONCLUSION: At 0.4 compression ratio, Dobi-SVD reaches 9.95 PPL on WikiText2 and outperforms advanced SVD-based and pruning methods.The conclusion reports this result for LLaMA-7B.
  • 5 CONCLUSION: Dobi-SVD provides a 12.4× speedup on an NVIDIA Titan Xp 12GB GPU with negligible loss.The conclusion characterizes the method as hardware-agnostic and adaptable.

A APPENDIX

The appendix adds methodological, analytical, experimental, and limitation-focused material beyond the main paper. It covers IPCA theory, differentiable SVD, truncation analyses, additional results, and sentence-generation demonstrations.

  • A APPENDIX: Appendix A.2 discusses limitations and potential solutions, while A.3 provides experimental-setting details.These sections extend the main paper with implementation and evaluation context.
  • A APPENDIX: Appendix A.4 presents a path from activation to weight, including theoretical support for updating weights with IPCA.The appendix separately identifies IPCA as the theoretical basis for efficient weight updates.
  • A APPENDIX: Appendix A.5 addresses a long-overlooked limitation of SVD for data compression, while A.6 develops a robust differentiable SVD algorithm for general matrices.These sections correspond to the paper’s remapping and general-matrix algorithmic perspectives.
  • A APPENDIX: Additional appendix analyses examine quantization precision loss, differentiable-training effectiveness, differentiable k changes, and truncation sensitivity.These analyses are listed under Appendix A.7.
  • A APPENDIX: Additional experiments cover more models, quantization combinations, comparisons with smaller uncompressed models, and real sentence generation.The appendix identifies these materials as Sections A.8 and A.9.
  • A APPENDIX: Appendix A.10 analyzes directly truncating activations rather than weights.This appendix section further examines the paper’s central activation-truncation perspective.

A.1 RELATED WORK

Related work covers conventional LLM compression through pruning, quantization, and low-rank decomposition, then focuses on SVD-based and activation-aware methods. Prior approaches commonly require retraining or suffer substantial performance degradation at high compression.

  • A.1 RELATED WORK: LLM compression methods include pruning, quantization, and low-rank decomposition, targeting models with billions of parameters.These approaches seek to make inference feasible on resource-constrained hardware, with pruning removing weights or structured components.
  • A.1 RELATED WORK: SVD-based compression has evolved from embedding-layer compression toward weight-matrix compression as model sizes have increased.Earlier and later studies apply SVD in different parts of neural networks, with recent methods often requiring fine-tuning.
  • A.1 RELATED WORK: Activation-aware methods minimize the difference between original and compressed activations, using scaling or whitening to identify important components.ASVD scales weights according to input-channel impact, while SVD-LLM uses truncation-aware whitening.
  • A.1 RELATED WORK: Other low-rank methods apply SVD to quantization errors or training sample handling, while non-SVD approaches such as CALDERA use alternative low-rank decompositions.These methods broaden low-rank compression beyond the specific SVD formulations discussed earlier.

A.2 LIMITATIONS AND POTENTIAL SOLUTIONS

Dobi-SVD identifies three limitations involving SVD computation, low-bit remapping, and inference dequantization, and proposes corresponding directions for mitigation.

  • Dobi-SVD’s current implementation is time-consuming and memory-intensive because Python supports only fp32 SVD; low-precision SVD could alleviate this.
  • Further quantization becomes difficult during memory remapping, with performance dropping at 2-bit precision.
  • Quantization adds inference-time dequantization, which could be reduced by quantizing larger matrices or using more advanced libraries.
  • Future work includes applying Dobi-SVD to vision-language models and robotics.

A.3 EXPERIMENTAL SETTING DETAILS

The experiments evaluate Dobi-SVD across language-model tasks, baselines, hardware, and differentiable-training configurations, using Llama-7B and specified remapping and optimization procedures.

  • Models, Datasets and Metric: Experiments use Llama-7B across C4, Wikitext2, PTB, and seven common-sense reasoning datasets to assess task generalization.Perplexity is used on the three in-domain datasets, with lower values indicating better performance.
  • Baselines: Dobi-SVD is compared with activation-aware SVD methods ASVD and SVD-LLM, alongside several pruning methods.
  • Hardware: Hardware evaluation uses an 80GB NVIDIA A100 and a 12GB NVIDIA Titan Xp to represent high- and low-performance GPUs.
  • Training Procedure: The training set contains 256 randomly selected WikiText2 samples of 2048 tokens for LLMs and 256 TextQA samples of 660 tokens for VLMs.
  • Training Procedure: Training freezes model parameters except each matrix’s truncation position k, while using β = 10, γ = 1e-10, and K = 10.
  • Weight Update: Weight updates reuse the same 256 training samples to collect truncated activations, then apply IPCA to calculate each new weight matrix without extra data or training.
  • Memory Remapping: Memory remapping quantizes matrices with bnb-8bit and concatenates two quantized 8-bit matrices into one 16-bit matrix.

A.4 DOBI-SVD’S NEW PERSPECTIVE 1: A NOVEL PATH FROM ACTIVATION TO WEIGHT

Dobi-SVD shifts SVD-based compression from truncating weights or scaled weights to directly truncating activations, then reconstructs weights through IPCA.

  • Existing approaches either truncate weights directly or apply activation-aware SVD to WS before reconstructing with S−1, whose inversion can fail theoretically and numerically.
  • Dobi-SVD performs SVD directly on activations A = xW, using the EYM theorem to establish a path from activations to weights.
  • The perspective spans direct weight truncation, activation-aware WS methods, and direct activation truncation, with the latter representing the most radical shift.
  • Direct activation truncation is closest to activations and farthest from weights, making weight reconstruction difficult despite its reported performance advantage.
  • IPCA compresses high-dimensional orthogonal matrices and reconstructs weights after activation truncation without additional data or fine-tuning.

A.4.1 THEORETICAL SUPPORT FOR UPDATING WEIGHTS USING IPCA

The theoretical formulation seeks a rank-k weight approximation closest to projected weight matrices and uses PCA, implemented incrementally, to determine the relevant orthogonal subspace.

  • The objective seeks a rank-k matrix fW closest to the set of projected weight matrices Wp.
  • Assuming fW = WVV^T, the optimization is converted into a Frobenius-norm objective over the projection matrix V.
  • Because ||W|| is fixed, the objective can be reformulated using the orthogonality of Vi and V and the resulting Frobenius norm relation.
  • The resulting goal is to maximize the trace-based objective over the projection components.
  • PCA solves for the orthogonal components V1, V2, ..., Vn, while IPCA provides a memory-efficient implementation for the high-memory setting.

A.5 DOBI-SVD’S NEW PERSPECTIVE 2: FULLY UNLOCKING SVD’S POTENTIAL FOR DATA COMPRESSION BY ADDRESSING A LONG-OVERLOOKED LIMITATION

Dobi-SVD addresses a fundamental storage limitation in truncated SVD by remapping compressed representations and using mixed-precision quantization to retain broader rank choices.

  • Storage limitation: Traditional truncated SVD requires k < (m · n)/(m + n) for compression, excluding some valid singular-value ranks.The conventional storage ratio is k · (m + n)/(m · n).
  • Prior limitation: Before Dobi-SVD, SVD-based methods had not addressed this storage issue, which had been used to critique their practicality.The cited non-SVD-based work treated the limitation as a primary criticism of SVD-based compression.
  • Dobi-SVD’s new perspective: Dobi-SVD remaps compressed storage from k · (m + n) to k · max(m, n), allowing k across the full range below min(m, n).This remapping ensures a storage ratio below 1 while preserving the available rank range.
  • Mixed-precision implementation: Mixed-precision quantization exploits Gaussian properties of U and V to reduce storage while bridging the two compressed-storage formulations.The method uses an 8-bit quantizer for selected rows and stores the result as a 16-bit mixed-precision matrix.

A.6 DOBI-SVD’S ROBUST AND EFFICIENT DIFFERENTIABLE SVD ALGORITHM FOR GENERAL MATRICES

Dobi-SVD develops a differentiable SVD procedure for general large matrices and uses it to study rank sensitivity, activation truncation, and compression behavior across LLM layers and settings.

  • Differentiable SVD algorithm: Dobi-SVD extends Taylor-expansion-based SVD backpropagation to general matrices and LLM compression with a more efficient parallelized algorithm for large dimensions.The implementation provides custom low-rank forward and backward passes with numerical-stability handling.
  • Layer sensitivity: At compression ratio 0.4, later layers can be truncated more aggressively because they tend to suffer less performance loss under low-rank decomposition.Earlier layers tend to retain higher k values, whereas later layers tend toward lower rank.
  • Layer sensitivity: Across compression ratios, layer-wise truncation exhibits similar trends, supporting the observed differences in sensitivity between matrix types and layer positions.The reported evolution was analyzed at a compression ratio of 0.4 and illustrated further at other ratios.
  • Rank adjustment: Dobi adjusts ranks at 0.024% granularity and operates end-to-end without fine-tuning, unlike search-based methods with a coarse minimum adjustment of 10%.The supplied comparison reports substantial performance loss for larger adjustment ratios and search-based methods.
  • Large-model evaluation: At compression rate 0.8, Dobi-SVD reduced performance by less than 3% on both Llama-13b and Llama 2-13b, outperforming all baselines.The result is reported for the larger-scale model experiments.
  • Hardware and quantization evaluation: Dobi-SVD can be combined with quantization, and its compressed model exceeded the speed of an unoptimized quantized model despite having a larger model size.The reported explanation cites fewer FLOPs and avoiding dequantization time.
  • Practical evaluation: A Dobi-SVD-compressed large model outperformed the original smaller model in both accuracy and hardware metrics.This comparison was presented as evidence of practical value in real-world scenarios.
  • Activation truncation: Direct activation truncation is more effective than weight truncation because the reported inequality ΔL_A ≤ ΔL_W holds for most tested LLM activations.The comparison kept ∂L/∂A constant while evaluating activation and weight truncations.
Loading 2502.02723v1…