PaperlayerView exact revision on arXiv

Source-linked AI summary

Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

Ruoling Qi, Yirui Liu, Xuaner Wu, Xiangyu Wang, Ming Li, Chen Chen, Jian Chen, Yin Chen, Qizhen Weng

arXiv:2604.01609v2cs.CL

TL;DR

LLM inference is constrained by the memory demands of static weights and growing KV caches, while existing layer-wise rank allocation can be inefficient or suboptimal. Swift-SVD uses activation-aware closed-form spectral compression with dynamic rank allocation, achieving state-of-the-art performance alongside 3–70× faster end-to-end compression.

  • Problem

    LLM inference faces memory demands from massive static weights and sequence-growing KV caches, while layer-wise rank allocation lacks efficient loss estimation.

  • Method

    Swift-SVD performs training-free activation-aware closed-form low-rank compression using a single eigenvalue decomposition and dynamic rank allocation across layers.

  • Results

    3–70× end-to-end compression speedups are reported while Swift-SVD maintains state-of-the-art performance across diverse architectures.

  • Takeaways & Limitations

    Swift-SVD reconciles theoretical optimum with practical efficiency for low-rank LLM compression.

  • Takeaways & Limitations

    Combining Swift-SVD with LoRA for further performance recovery is left as future work.

Abstract

from arXiv · show

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3-70X speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

1. Introduction

Swift-SVD addresses the memory demands of static LLM weights and dynamic KV caches with an activation-aware, training-free low-rank compression framework. It combines a closed-form optimal projection with efficient layer-wise rank allocation and demonstrates strong empirical performance across diverse models and datasets.

  • Motivation: LLM inference faces memory pressure from massive model parameters and sequence-length-dependent cached Key–Value states.The KV cache grows during autoregressive decoding, creating a distinct runtime-memory challenge.
  • Background: Post-training low-rank compression reduces the intrinsic dimensionality of linear layers without retraining, complementing quantization and pruning.It formulates compression as a matrix approximation problem.
  • Limitations: Existing SVD-based methods are limited by activation-insensitive reconstruction or computationally costly decompositions, while heuristic rank allocation can be suboptimal.Non-uniform layer compression lacks efficient loss estimation for exhaustive rank searches.
  • Method: Swift-SVD jointly compresses static weights and KV caches using activation-aware, training-free low-rank approximation with a single eigenvalue decomposition.Its direct spectral solution avoids repeated SVD operations and targets low overhead, efficiency, and numerical stability.
  • Evaluation: 6 LLMs and 8 datasets are used to evaluate Swift-SVD with perplexity and QA accuracy, alongside time, memory, compression-ratio, and dataset-scale analyses.The experiments report consistent advantages over existing low-rank compression baselines, including gradient-based methods.

2. Preliminary

Low-rank compression replaces a weight matrix with rank-k factors, reducing weight and KV-cache memory. While conventional SVD is optimal under weight-space Frobenius loss, activation-aware compression instead minimizes output reconstruction error using input activations.

  • Low-Rank Compression: Rank-k compression factorizes Wk as AkBk, replacing W while reducing the parameter representation from m×n to k(m+n).The factors have dimensions Ak ∈ Rm×k and Bk ∈ Rk×n.
  • Low-Rank Compression: KV-cache memory decreases by caching intermediate latents XAk ∈ Rl×k instead of output activations XW ∈ Rl×n when k < n.
  • Conventional SVD: Under Frobenius loss ||W − Wk||F, truncating W to its top k singular values and vectors gives the optimal rank-k approximation.
  • Activation-Aware Compression: Direct SVD truncation ignores input activations X and can therefore cause significant practical performance degradation.Activation-aware losses instead seek to match the original output Y = XW.
  • Activation-Aware Compression: The activation-aware objective defines Wk among rank-k matrices and selects W ∗k with the corresponding minimal reconstruction loss.The output activation is Y = XW.

3. Method

Swift-SVD provides training-free, activation-aware low-rank compression through a closed-form optimal solution computed efficiently from aggregated output-activation covariance. Its dynamic compression strategy allocates ranks using local effective rank and end-to-end layer importance, then selects among candidates by lightweight validation.

  • Framework overview: Swift-SVD compresses arbitrary weight matrices in two stages: optimal activation-aware low-rank compression followed by dynamic compression.The framework applies uniformly to query, key, value, and other weight matrices.
  • Optimal activation-aware compression: Theorem 3.1 states that for target rank k, the optimal compressed weights are characterized by the top-k right singular vectors of Y = XW.The resulting rank-k approximation corresponds to the truncated SVD of Y and minimizes reconstruction loss.
  • Efficient spectral computation: Swift-SVD computes the solution by aggregating the covariance C = Y^T Y and performing a single eigenvalue decomposition to obtain Y’s singular values and right singular vectors.This requires only n × n covariance storage and avoids Cholesky factorization and multiple SVDs, improving efficiency and numerical stability.
  • Dynamic rank allocation: Lower effective rank indicates stronger intrinsic low-rank structure and higher local compressibility, while lower layer importance indicates higher end-to-end compressibility.Figure 3 reports a clear negative correlation between normalized effective rank and normalized layer importance across seven weight matrices.
  • Dynamic rank allocation: Swift-SVD generates candidate rank allocations from layer importance and local compression loss, then selects the best candidate through lightweight validation without retraining.Each candidate is compressed using the closed-form optimal solution and evaluated on a validation set for end-to-end performance.

4. Experiments and Analysis

Experiments evaluate Swift-SVD across diverse LLMs, datasets, compression ratios, and metrics against five SVD-based baselines. Results show strong accuracy, cross-architecture generalization, activation sensitivity, efficient compression, and benefits from constrained dynamic rank allocation.

  • Experimental Setup: Swift-SVD is compared with five SVD-based methods: FWSVD, ASVD, SVD-LLM, SVD-LLM v2, and Dobi-SVD.Evaluation spans LLaMA-7B, LLaMA2-7B, OPT-6.7B, Mistral-7B, and Qwen3-4B/8B, using language-modeling and zero-shot reasoning benchmarks.
  • Compression with Different Methods: Swift-SVD achieves the highest average accuracy across all compression levels while yielding the lowest reported metric in Table 1.Baseline results are cited from original papers, and each method follows the corresponding baseline’s calibration dataset, sample size, and protocol.
  • Compression with Different Methods: 0.34 average accuracy is achieved by training-free Swift-SVD at compression ratio 0.4 on LLaMA-7B, versus 0.30 after LoRA fine-tuning SVD-LLM.SVD-LLM reaches 0.11 immediately after compression under the same cited setting; Swift-SVD low-rank matrices are compatible with LoRA initialization.
  • Performance across Different LLMs: Both Swift-SVD and Swift-SVD* outperform SVD-LLM(W) and ASVD across OPT-6.7B, LLaMA2-7B, and Mistral-7B.The comparison uses uniform compression and demonstrates stability and universality across these three architectures.
  • Impact of Aware Samples Size: N = 256 is adopted because performance improves rapidly with few calibration samples but shows diminishing returns as sample size increases.Larger samples continue to provide marginal gains, and the standard setting supports fair comparison with baselines.
  • End-to-end Compression Time: 3.8× speedup over SVD-LLM(W) and 76.9× over Dobi-SVD(w/o) are achieved in end-to-end compression time.Swift-SVD’s efficiency comes from incremental activation-covariance aggregation followed by a single eigen-decomposition, whereas baselines perform more costly procedures.

5. Related Work

Prior work studies rank structure in language models for efficiency, expressivity, and modeling capacity, while recent compression methods target low-rank weights and joint KV caches. Activation-aware approaches address evidence that transformer activations are typically lower-rank than weights, which weight-only SVD methods overlook.

  • Rank Analysis in Language Models: Rank-analysis studies use low-rank structure for efficiency, prevent rank collapse, or maximize rank utilization in language models.The cited works examine transformer weights or representations in relation to model performance.
  • Low-Rank Model Compression: Recent work explores low-rank joint KV cache compression to facilitate scalable inference.
  • Low-Rank Model Compression: MHA2MLA and PALU use SVD to reformulate Multi-Head Attention into Multi-head Latent Attention, but their weight-only designs overlook activation rank.Transformer weights typically exhibit higher rank than corresponding activations, motivating activation-aware compression.

6. Conclusion

Swift-SVD is a training-free, activation-aware compression framework that combines theoretical optimality with practical efficiency through closed-form eigenvalue decomposition. It also uses layer-wise compressibility and importance for dynamic rank allocation, while leaving LoRA-based performance recovery as future work.

  • Core contributions: Swift-SVD is a training-free, activation-aware framework that reconciles theoretical optimum with practical efficiency.It formulates compression as a closed-form eigenvalue decomposition problem.
  • Core contributions: The closed-form formulation eliminates numerical instability and computational bottlenecks.This supports efficient compression without training.
  • Core contributions: Swift-SVD uses layer-wise compressibility and importance to allocate ranks dynamically.The strategy accounts for both properties when assigning compression ranks.
  • Experimental outcome: 3–70× end-to-end compression speedups are achieved in extensive experiments.The reported speedups characterize Swift-SVD’s practical compression efficiency.
  • Future work: Combining Swift-SVD with LoRA fine-tuning is identified as a future direction for further performance recovery.The compressed model is directly compatible with LoRA fine-tuning as initialization.

A. Additional Experimental Details and Analysis · A.1. Calibration Data

The appendix details calibration-data construction for dynamic compression experiments. Calibration uses 256 randomly selected samples, with dataset-specific sequence formatting and lengths.

  • A. Additional Experimental Details and Analysis: The appendix provides additional details on calibration data and analyzes hyperparameters for dynamic compression experiments.
  • A.1. Calibration Data: Calibration data are used during the incremental statistic aggregation phase.
  • A.1. Calibration Data: 256 calibration samples are randomly selected for incremental statistic aggregation.
  • A.1. Calibration Data: WikiText-2 and C4 samples are processed with a fixed sequence length of 2048 tokens.
  • A.1. Calibration Data: Text datasets use fixed-length processing during calibration.
  • A.1. Calibration Data: Conversational datasets such as Alpaca are formatted from individual raw entries according to their official prompt templates.
  • A.1. Calibration Data: Zero-shot common-sense benchmarks such as PIQA are likewise constructed from individually formatted raw data entries.
  • A.1. Calibration Data: Formatted conversational and benchmark samples are concatenated as discrete, individually separated units.

A.2. Hyperparameter Analysis in Dynamic Compression · B. Additional Experiment Results

Swift-SVD’s dynamic rank allocation is governed by scaling factor α and preserved ratio δ, balancing reconstruction loss with layer importance while stabilizing performance. Its cached spectral components enable efficient hyperparameter-grid exploration, and both tested configurations show U-shaped performance trends.

  • A.2. Hyperparameter Analysis in Dynamic Compression: The dynamic rank allocation strategy has two primary hyperparameters: scaling factor α and preserved ratio δ, each ranging from 0 to 1.These hyperparameters define the allocation behavior examined in the analysis.
  • A.2. Hyperparameter Analysis in Dynamic Compression: α controls the trade-off between reconstruction loss ϵ and layer importance β.α = 0 yields purely loss-driven allocation, whereas α = 1 exclusively prioritizes layer importance.
  • A.2. Hyperparameter Analysis in Dynamic Compression: δ functions as a performance stabilizer in dynamic rank allocation.The passage specifically identifies δ = 0 as the condition without a preserved ratio, before describing unrestricted dynamic allocation.
  • A.2. Hyperparameter Analysis in Dynamic Compression: Swift-SVD avoids redundant SVD operations by performing one eigenvalue decomposition of activation statistics.This decomposition is performed once after the relevant statistics are aggregated, as described in the supplied passage.
  • A.2. Hyperparameter Analysis in Dynamic Compression: The method caches spectral components Σ and V for reuse during direct truncation and model reconstruction.These cached components support reconstruction under dynamic rank assignments.
  • A.2. Hyperparameter Analysis in Dynamic Compression: The resulting computational efficiency supports expanding the hyperparameter search grid.The passage connects reduced decomposition overhead with broader exploration of hyperparameter settings.
  • A.2. Hyperparameter Analysis in Dynamic Compression: Both configurations in Figure A.1 exhibit a U-shaped impact of α and δ on model performance.The supplied figure caption identifies this shared trend for the two configurations.

B.1. Accuracy on Specific Task … B.6. Throughput

Across supplementary evaluations, Swift-SVD benefits from task-aligned or unified calibration, outperforms competing methods on Qwen3-4B and Qwen-32B, and improves throughput as compression increases. Its dynamic allocation is motivated by uneven singular spectra and negative correlation between layer compressibility and importance.

  • B.1. Accuracy on Specific Task: Each calibration achieves the highest accuracy, while unified All calibration closely matches Each and general C4 calibration performs worst.The observed hierarchy is Each ≳ All > C4.
  • B.2. Comparison with Basis Sharing: Swift-SVD achieves lower C4 perplexity and higher average accuracy than Basis Sharing under the matched FP32 LLaMA-7B protocol.Basis Sharing results come from its original paper, while Swift-SVD uses the same calibration dataset and sample size.
  • B.3. Comparison on Qwen3-4B: Swift-SVD outperforms SVD-LLM(W) and Dobi-SVD(w/o) under uniform compression, while Swift-SVD* achieves the highest accuracy under dynamic allocation.Swift-SVD* jointly considers layer-wise loss and layer importance; all comparisons are training-free.
  • B.4. Scalability to Larger Models: On Qwen-32B using a single H800, Swift-SVD completes compression while activation-aware SVD-LLM and Dobi-SVD fail with OOM errors.Swift-SVD also improves average accuracy from 0.42 to 0.59 at compression ratio 0.8.
  • B.5. Singular Value Distribution: Key and Value modules show highly uneven singular spectra, with dominant singular values reaching 10^5 while median values are substantially lower.This disparity makes Frobenius-norm-only rank allocation ineffective because large singular values disproportionately influence the loss.
  • B.5. Singular Value Distribution: Layer-wise compressibility is highly negatively correlated with layer importance, motivating dynamic allocation that accounts for both properties.Frobenius-based methods can under-allocate rank to layers with smaller spectral norms.
  • B.6. Throughput: As compression ratio increases across batch sizes, Swift-SVD boosts throughput while alleviating HBM pressure.The throughput and memory-efficiency evaluation uses generated sequences of length 1024.

B.7. Serving Throughput Comparison Across Methods

On LLaMA-2-7B, uniform compression methods deliver comparable serving throughput at equal compression ratios, while Swift-SVD’s dynamic allocations preserve throughput but Dobi-SVD(w) substantially reduces it. The degradation is attributed to increased k_proj and v_proj ranks, which enlarge KV-cache dimensions and memory traffic during decoding.

  • Experimental setup: Throughput comparisons use tokens/sec on an H800 GPU with batch size 16, prompt length 32, and output length 1024.The evaluated methods include uniform SVD-LLM(W), Dobi-SVD(w/o), Swift-SVD, and dynamic Swift-SVD* and Swift-SVD(C).
  • Uniform compression: Uniform SVD-LLM(W), Dobi-SVD(w/o), and Swift-SVD achieve nearly identical throughput at the same compression ratio.Equal compression ratios produce equal throughput because uniform allocation gives all layers identical ranks and GEMM dimensions.
  • Dynamic compression — Swift-SVD: Swift-SVD* and Swift-SVD(C) retain throughput close to their respective uniform baselines without significant degradation.Dynamic allocations may remain close enough to uniform ranks that GEMM-dimension variation does not observably affect throughput.
  • Dynamic compression — Dobi-SVD(w): Dobi-SVD(w) exhibits substantially lower throughput than its uniform counterpart.Its inter-type allocation may raise k_proj and v_proj ranks, enlarging KV-cache dimensions and increasing memory traffic during memory-bandwidth-bound decoding.

B.8. Throughput on Longer Generation Tasks

Swift-SVD remains efficient for longer generation tasks because up-projecting cached KV latents adds O(L) overhead per decoding step, matching the O(L) memory-bandwidth savings from compressed-cache reads. On Mistral-7B, throughput gains at 8192 tokens are comparable to or exceed those at 1024 tokens across compression ratios.

  • Scalability: Up-projecting all L cached KV latents adds O(L), rather than O(L^2), computational overhead per decoding step because the up-projection matrix has fixed dimensions.The additional cost per cached token remains constant as sequence length grows.
  • Scalability: The compressed KV cache provides O(L) memory-bandwidth savings per decoding step, offsetting the linear overhead of up-projecting cached latents.Both the additional computation and the bandwidth savings scale linearly with the number of cached tokens.
  • Evaluation: Throughput was measured on Mistral-7B at generation lengths of 1024 and 8192 tokens using an H800 GPU with batch size 16 and prompt length 32.The H800 was used because longer sequences exceeded the RTX 5090's memory capacity due to total KV-cache size.
  • Results: At length 8192, Swift-SVD's throughput gain is comparable to or exceeds its gain at length 1024 for every compression ratio.Throughput decreases with sequence length for both compressed and uncompressed models, consistent with autoregressive decoding requiring increasingly large KV-cache reads.

B.9. Layer-wise NER and Importance Results · B.10. Contents Generated From Compressed LLMs

The appendix reports layer-wise NER and importance comparisons, then examines AI assistant responses generated by compressed LLMs across datasets and compression ratios.

  • B.9. Layer-wise NER and Importance Results: The paper presents Part I of a layer-wise comparison of NER and importance.The results are provided in Figure B.3.
  • B.10. Contents Generated From Compressed LLMs: Figure B.3 continues the layer-wise NER and importance comparison in Part II.This figure appears in the compressed-LLM content section.
  • B.10. Contents Generated From Compressed LLMs: B.10 includes the second part of the layer-wise NER and importance comparison.It also covers content generated from compressed LLMs.
  • B.10. Contents Generated From Compressed LLMs: The appendix compares AI assistant responses across different datasets and compression ratios.These comparisons are summarized in Table B.8.
  • B.10. Contents Generated From Compressed LLMs: Table B.8 evaluates generated assistant content under varying datasets and compression ratios.The table is labeled as a comparison of AI assistant responses.
Loading 2604.01609v2…