PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing SVD-based LLM Compression via Layer-Wise Error Model Search

Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Manoj Rohit Vemparala, Christos Anagnostopoulos, Pierpaolo Mori, Nael Fasfous, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann

arXiv:2610.32878v2cs.AI

TL;DR

SVD-based LLM compression is limited by rank-allocation methods that miss error propagation and Fisher-based decompositions that can suffer rank collapse. The paper introduces LEMS for rank allocation and KFAC-SVD for decomposition. Across model families, it reports LEMS accuracy gains and improved perplexity with KFAC-SVD.

  • Problem

    SVD compression faces global rank-allocation challenges from uncalibrated proxies that neglect error propagation, and Fisher-based decomposition can suffer rank deficiencies.

    Source: §1 · p. 1

  • Method

    LEMS uses a calibrated layer-wise error surrogate and an ILP for global rank allocation, while KFAC-SVD estimates Fisher curvature using token-wise statistics.

    Source: §1 · p. 2

  • Results

    Across Mistral, Qwen3, and Llama 3, KFAC-SVD improves WikiText2 perplexity by an average of 19%, while LEMS gains 2.1-4.7 p.p. over the best existing rank-allocation methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LEMS gains are reported across model sizes and activation- and Fisher-based SVD, including a 4.0 p.p. improvement over uniform rank allocation at 70B parameters.

    Source: §1 · p. 2, §6 · p. 9

    The token-wise approximation omits cross-token second moments.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. However, existing methods face two key limitations: (i) global rank allocation, where uncalibrated error proxies fail to capture complex error propagation, and (ii) decomposition quality, where Fisher-based estimators suffer from severe rank collapse. In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias, enabling effective global rank allocation via an ILP formulation. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches. Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.7 p.p. that generalize to model sizes of 70B parameters, while KFAC-SVD achieves an average perplexity improvement of 19%. Project page and code: https://lems-svd.github.io

1. Introduction

The paper addresses two limitations of SVD compression: global rank allocation and decomposition quality. It introduces LEMS and KFAC-SVD, reporting gains across model families and rank-allocation settings.

1. Introduction

  • Existing SVD methods face global rank allocation challenges from uncalibrated importance proxies and decomposition-quality challenges from rank deficiencies in Fisher-based approaches.The allocation proxies may neglect nonlinear error propagation in deep networks.Source: §1 · p. 1
  • LEMS combines local sensitivity, empirical calibration, and a propagation bias in a layer-wise error model, then uses an ILP to search global rank allocations.The model explicitly accounts for layer-wise error accumulation.Source: §1 · pp. 1–2
  • KFAC-SVD uses token-wise Kronecker-factor statistics to mitigate rank deficiency in prior Fisher-based SVD and improve decomposition fidelity.It accumulates token-level loss-curvature information rather than relying on sequence-based statistics.Source: §1 · p. 2
  • 19% average WikiText2 perplexity improvement over baselines from KFAC-SVD, while LEMS gains 2.1-4.7 p.p. over the best existing rank-allocation methods across Mistral, Qwen3, and Llama 3.LEMS also generalizes to 70B parameters, achieving a 4.0 p.p. improvement over uniform rank allocation.Source: §1 · p. 2

2. Related Work

Related SVD work differs in its reconstruction objective and rank-search strategy. The paper positions LEMS as global error minimization and KFAC-SVD as a token-wise Fisher approximation.

2. Related Work

  • Standard SVD minimizes weight reconstruction error, while activation-based methods target layer-output error; SVD-LLM uses Cholesky whitening to optimize that output objective.Other approaches reuse low-rank bases, alter activations, or remap decomposed components.Source: §2 · p. 2
  • Fisher-weighted methods target loss impact: diagonal-Fisher weighting and Kronecker-factored approaches such as GFWSVD model parameter importance and curvature.GFWSVD uses two-sided whitening for its rank-r solution under the Kronecker-factored approximation.Source: §2 · p. 2
  • KFAC-SVD estimates Kronecker factors from token-wise second moments to make Fisher-based truncation viable and stable at LLM scale.This contrasts with activation-based methods described as robust but loss-agnostic and sequence-gradient Fisher fitting described as sample-limited.Source: §2 · p. 2
  • Rank searches include greedy updates, balanced reconstruction-error heuristics, learned allocation, and global sparsity-allocation formulations.Examples include MRCS and FastLRD; ASVD and Flar-SVD; ComCat, ARS, and DOBI-SVD; and MDP's ILP formulation.Source: §2 · pp. 2–3
  • LEMS directly minimizes total error under a global budget by solving an ILP over discrete per-layer rank candidates using a layer-wise proxy model.The approach is inspired by ILP-based sparsity allocation and analyses of error propagation.Source: §2 · p. 3

3. Preliminaries

The preliminaries formulate Fisher-weighted SVD as minimizing compression-induced loss increase and explain how Kronecker approximations yield a whitened Frobenius-norm objective. They relate this objective to activation-based SVD and describe the low-rank implementation.

3. Preliminaries

  • Fisher-weighted SVD seeks a rank-r weight approximation that minimizes compression-induced loss increase using a second-order Taylor expansion with the Fisher matrix approximating the Hessian.The parameter change is represented by the vectorized difference between the original and compressed weights.Source: §3 · p. 3
  • Because the full layer-wise Fisher matrix is intractable at LLM dimensions, the method uses Kronecker-factored approximations to obtain a Frobenius-norm objective in whitened coordinates.The factors A and G are positive semidefinite.Source: §3 · p. 3
  • The whitened objective generalizes SVD-LLM; setting G = I recovers its input-covariance-weighted layer-output reconstruction objective.The optimal rank-r approximation is obtained by truncating the SVD in whitened coordinates.Source: §3 · p. 3
  • The rank-r approximation is implemented as two consecutive linear layers by symmetrically splitting singular values, while any bias terms remain unchanged.The forward pass is y = C(Bx).Source: §3 · p. 3

4. Methodology

The methodology pairs KFAC-SVD’s token-wise Fisher estimation with LEMS, which models layer-wise degradation and allocates ranks under a parameter budget. KFAC-SVD addresses rank collapse in prior Fisher-based SVD, while LEMS accounts for spectral shape, empirical sensitivity, and depth-dependent error propagation.

4.1. Decoupled Token-Wise Estimation (KFAC-SVD)

  • Prior sequence-gradient Fisher estimation is rank- and conditioning-limited when the parameter dimension greatly exceeds feasible k, producing spectral collapse and greater Fisher degradation after truncation.The paper gives k≤256 as an example of the feasible sample count.Source: §4.1 · p. 3
  • Token-wise second moments estimate KFAC factors from token instances rather than a small set of sequence gradients, avoiding the k-limited rank bottleneck.For a linear layer, each token’s weight gradient is δx⊤; the method approximates the expectation of the Kronecker product by the Kronecker product of expectations.Source: §4.1 · p. 4

4.2. Global Rank Allocation via Layer-wise Error Modeling (LEMS)

  • Simple sums of layer-wise errors underestimate global degradation because early-layer errors propagate and amplify through network depth.LEMS uses a soft depth-dependent bias rather than rigid block-level constraints to model this interaction.Source: §4.2 · pp. 4–5
  • LEMS selects one rank configuration per linear layer by minimizing estimated degradation subject to a parameter budget.The ILP identifies the optimal allocation for its proxy objective in seconds using off-the-shelf solvers.Source: §4.2 · pp. 4–5
  • LEMS models estimated global degradation with a layer-additive surrogate combining spectral shape, empirical calibration, and a depth-dependent bias.The bias addresses the propagation of early-layer errors; the spectral tail-energy curve supplies rank granularity, while calibration maps it to task loss.Source: §4.2 · pp. 4–5

5. Evaluation

Across Mistral, Llama 3, and Qwen3, the evaluation compares LEMS rank allocation and KFAC-SVD decomposition with existing methods, then examines scalability, component contributions, calibration, and estimator quality. LEMS improves rank-allocation results across evaluated settings, while KFAC-SVD lowers perplexity and its token-wise Fisher estimate shows better held-out curvature fit than GFWSVD.

5.1. Rank Allocation Results

  • 2.14–4.67 p.p. higher accuracy than the second-best search and 26.06–75.40% lower mean perplexity than uniform allocation across six settings.The comparison covers WikiText2, C4, and PTB; LEMS has the highest accuracy in all six settings.Source: Table 1 · p. 6, §5.1 · p. 6

5. Evaluation

  • The evaluation compares search and SVD baselines across Mistral, Llama 3, and Qwen3 using language-modeling and zero-shot benchmarks.Search methods share the decomposition pipeline, calibration data, and candidate ranks; experiments use WikiText2 calibration data and a single NVIDIA H200 GPU.Source: §5 · pp. 5–6, §5.1 · p. 6

5.2. SVD-Compression Results

  • At 0.7 compression, KFAC-SVD improves WikiText2 perplexity by 49% over DOBI-SVD, and by 66% when combined with LEMS.Across three model families, KFAC-SVD consistently yields the lowest perplexity; under mild compression, LEMS recovers near-lossless accuracy (61.8% vs. 63.3% original).Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3.1. SCALABILITY OF LEMS

  • Across all budgets and model sizes, LEMS reduces perplexity by 17% for KFAC-SVD and 34% for SVD-LLM, while increasing zero-shot accuracy by 7.2 and 9.1 p.p., respectively.On Llama 3.1-70B, LEMS lowers WikiText2 perplexity from 7.9 to 4.5 and raises accuracy from 62.6 to 66.6.Source: §5.3.1 · p. 7

5.3.3. LEMS ERROR CALIBRATION

  • On Qwen3-4B, KL calibration improves zero-shot accuracy over MSE by up to 1.6 p.p., while MSE has slightly lower perplexity.The authors select cref=0.1 for lower perplexity (14.5 versus 14.7 at cref=0.3), despite a 0.9 p.p. zero-shot reduction.Source: Figure 3 · p. 8, §5.3.3 · p. 8

5.3.2. INDIVIDUAL CONTRIBUTIONS TO LEMS

  • At 0.6 retention, KL calibration lowers perplexity below uniform allocation by 19% for Llama 3 and 13% for Qwen3; cross-layer bias adds further reductions of 42% and 3%.KL calibration adds 17–22 minutes, while bias correction adds 2 minutes; initial LEMS runs take 20–25 minutes on the evaluated 8B models.Source: Table 3 · p. 8, §5.3.2 · p. 8

5.3.4. KFAC-SVD ESTIMATION QUALITY

  • Across truncation ranks, KFAC’s token-wise estimator yields lower Fisher degradation on held-out data than GFWSVD, whose eigenspectrum decays rapidly.The analysis associates GFWSVD’s spectral collapse with truncation discarding curvature information.Source: Figure 4 · p. 8, §5.3.4 · pp. 8–9

6. Conclusion and Limitations

The paper presents LEMS for calibrated layer-wise rank allocation and KFAC-SVD for improved Fisher-based decomposition. It reports gains across model families while noting computational and scaling limitations.

6. Conclusion and Limitations

  • LEMS gains are reported across model sizes and activation- and Fisher-based SVD, while KFAC-SVD improves decomposition quality through token-wise Fisher estimation.The conclusion says KFAC-SVD still requires gradient computation, and LEMS uses a one-time calibration step that scales linearly with model depth.Source: §6 · p. 9
  • The authors identify formal error-propagation bounds and evaluation on vision-language and mixture-of-experts models as future work.Source: §6 · p. 9
  • KFAC-SVD’s gradient-computation requirement limits its computational efficiency, while LEMS calibration scales linearly with model depth.Source: §6 · p. 9

AI Acknowledgement

The acknowledgement credits AI assistance with arranging and proofreading the paper and supporting implementation of added related approaches. The authors retain responsibility for the research and manuscript content.

AI Acknowledgement

Glossary

The glossary expands abbreviations and names used in the paper, including decomposition methods, compression approaches, and model or dataset terms.

Glossary

  • ASVD means activation-aware singular value decomposition, and BasisSharing means Basis Sharing: Cross-Layer Parameter Sharing for LLM Compression.Source: Glossary · p. 12
  • BitStack is Any-Size Compression of LLMs in Variable Memory Environments.Source: Glossary · p. 12
  • ComCat means Compression and customization of attention-based vision models; FastLRD means Fast low rank decomposition; and FLAR-SVD means Fast and latency-aware SVD.Source: Glossary · p. 12
  • KFAC means Kronecker-factored approximate curvature, and KFAC-SVD means Kronecker-factored approximate curvature singular value decomposition.Source: Glossary · p. 12
  • LEMS means Layer-wise error model search, and MDP means Multidimensional vision model pruning with latency constraint.Source: Glossary · p. 12
  • MSE means mean squared error, and SVD means singular value decomposition.Source: Glossary · pp. 12–13
  • SVD-LLMv2 means Optimizing SVD for LLM compression; VLM means vision-language models; and MoE means mixture-of-experts.Source: Glossary · pp. 12–13
  • WikiText2 is listed as a dataset name in the glossary.Source: Glossary · p. 13

A. Extended Experimental Setup

The extended setup evaluates LEMS and KFAC-SVD across model families and scales, and describes comparison and calibration choices. It also outlines supplementary analyses of LEMS’s bias, scale, and component contributions.

A. Extended Experimental Setup

  • The evaluation spans Qwen3 models from 1.7B to 32B, Llama variants including Llama 3.1 70B, Llama 3.2 1B and 3B, and Mistral-7B.The passage also includes Llama 2-7B.Source: A. Extended Experimental Setup · p. 14
  • Search baselines use shared decomposition and candidate-rank settings where applicable; several methods were reimplemented or adapted, while ASVD used its public codebase.The described baselines include MRCS, SVD-LLMv2, ARS, and ATP.Source: A. Extended Experimental Setup · p. 14
  • LEMS bias parameters were fit with 10 Optuna trials, with bounded bias magnitude and decay shape and scale calibration at a reference compression of 0.1.Source: A. Extended Experimental Setup · p. 14

B.1. Extended Explanations and Ablations for LEMS

  • The supplementary LEMS analyses examine the bias term, compare importance metrics and calibration points for scale, and assess the combined contribution of its components.Source: §B.1 · pp. 14–15

B.1.1. BIAS FUNCTION IN ERROR MODELING

The section motivates a depth-dependent bias because summing local compression errors underestimates total degradation, then describes calibrated importance metrics and global rank allocation. Its cross-ablation finds the strongest result when the calibrated proxy and ILP search are combined.

B.1.1. BIAS FUNCTION IN ERROR MODELING

  • The measured total error exceeds the sum of individual block errors, especially for early perturbations in Llama 3.2-3B, motivating a depth-dependent bias.Underestimation is also present in Qwen3-4B, although the additive approximation holds better there.Source: §B.1.1 · p. 15, Figure 5 · p. 15
  • Any tested bias correction outperforms the uncorrected search; the best settings reach 42.27 perplexity for Llama 3.2-3B and 30.32 for Qwen3-4B.The optimal κ and γ vary by model, so the authors fit them with a lightweight TPE-based bilevel search.Source: §B.1.1 · p. 16

B.1.2. CORRECT CROSS-LAYER SCALING (IMPORTANCE)

  • KL divergence provides the selected importance calibration: unlike noisy perplexity, it yields stable sensitivity trends and slightly better search results than MSE.The calibration point is cref = 0.1, chosen for high-compression calibration and performance close to cref = 0.3.Source: §B.1.2 · p. 18

B.1.3. DETAILED COMPARISON BETWEEN LEMS, ATP AND ASVD

  • ATP’s uniform compression yields 13.48 perplexity, while ASVD’s layer-specific compression improves to 12.06; LEMS globally trades off layer importance and parameter count, achieving a superior architectural trade-off and better performance.Slightly increasing compression of larger up- and down-projection layers frees budget to preserve smaller, functionally critical layers at higher fidelity.Source: §B.1.3 · pp. 18–19

B.1.4. HOW DO COMPONENTS CONTRIBUTE TO LEMS PERFORMANCE?

  • Neither calibrated proxies nor global ILP optimization alone accounts for the full gain; the best performance comes from combining the calibrated proxy with global allocation.The ILP can be misled by uncalibrated energy, while retrofitting MRCS or ASVD with hybrid scores does not match the combined method.Source: §B.1.4 · pp. 19–20
  • 8.04 perplexity with ILP search, versus 9.79 with ASVD search and 10.71 with MRCS search, was the best cross-ablation result.The comparison uses the full hybrid+bias proxy at 0.8 parameter retention.Source: §B.1.4 · pp. 19–20, Table 5 · p. 21

B.2. Additional Explanations and Ablations on KFAC-SVD

These supplementary comparisons test KFAC-SVD’s Fisher truncation, LEMS across model sizes and search methods, and their behavior with alternative decompositions. The results highlight KFAC’s lower held-out Fisher degradation, LEMS’s gains in large-model and concurrent-method comparisons, and trade-offs in BitStack and BasisSharing.

B.2. Additional Explanations and Ablations on KFAC-SVD

  • At rank 128, KFAC achieves ≈1.4× to 3× lower held-out Fisher degradation than GFWSVD across model and layer groups, alongside higher effective rank.The authors attribute sequence-gradient spectral collapse to few sequence-level samples concentrating curvature in a low-dimensional subspace; this collapse correlates with worse truncation.Source: §B.2 · p. 21, Table 6 · p. 22

B.3. Llama 2-7B Results

  • The supplementary Llama 2-7B results are presented as further evidence for KFAC-SVD and LEMS.The cited table covers methods across multiple uniform compression rates and SVD baselines.Source: §B.3 · p. 22, Table 11 · p. 25

B.4. Llama-7B Results

  • The Llama-7B comparisons report decomposition results at 0.9 and 0.7 retention and search comparisons at 0.8 and 0.6 retention.The setup uses calibration seed 3 and the main-section protocol; concurrent-work comparisons also note that some decomposition gaps are small and may reflect implementation deviations.Source: §B.4 · p. 22, Table 7 · p. 22, Table 8 · p. 23

B.5. Comparison with Concurrent Work

  • Across four models, LEMS substantially improves results in almost all cases versus concurrent rank searches applied to KFAC-SVD.The comparison covers 0.8 and 0.6 parameter retentions; ZS-SVD is second-best on all 0.8 results, while Swift-SVD is significantly better than uniform on Llama 3 at 0.6 (26.77 vs. 33.97 perplexity).Source: §B.5 · p. 24, Table 10 · p. 25

B.6. Llama 3.1-70B Results

  • On Llama 3.1-70B, LEMS improves average zero-shot accuracy by 4.0 p.p. over uniform allocation and lowers WikiText2 perplexity from 7.94 to 4.53.LEMS improves or matches uniform on every zero-shot benchmark; ASVD trails by 2.5 p.p. and costs nearly 3.5× as much computationally.Source: §B.6 · p. 24, Table 12 · p. 25

B.7.1. BITSTACK

  • For Llama 3-8B, KFAC-SVD and LEMS achieve a better memory–accuracy trade-off than BitStack, even when BitStack’s memory footprint is reported under favorable AWQ assumptions.BitStack requires full-weight reconstruction before computation and its low-rank structure is incompatible with standard quantization kernels.Source: §B.7.1 · p. 26, Table 13 · p. 26

B.7.2. BASISSHARING

  • On Llama 2-7B, applying the updated search to BasisSharing reduces WikiText2 perplexity from 8.12 to 6.25, while PTB shows substantial degradation.The authors describe the PTB result as an outlier; their 32-bit reproduction matches the reference, while the paper’s fp16 pipeline shows calibration-data sensitivity.Source: §B.7.2 · p. 27, Table 14 · p. 27
  • With matched memory and compute budgets, KFAC-SVD outperforms BasisSharing in perplexity and zero-shot comparisons without search.The authors also report that their LEMS formulation improves BasisSharing results, including WikiText2 perplexity from 8.12 to 6.25 and a 9.3 p.p. zero-shot accuracy gain over uniform.Source: §B.7.2 · p. 27, Table 15 · p. 27, Table 14 · p. 27

B.8. Calibration Data Ablations

The supplementary evaluations examine sensitivity to calibration data, inference speed, and example generations. More calibration samples improve perplexity with diminishing returns, while LEMS speedups grow with compression; the qualitative table compares outputs across compression ratios.

B.8. Calibration Data Ablations

  • Going from 64 to 512 calibration samples improves perplexity from 20.54 to 17.54, with diminishing returns as sample counts rise.Doubling from 64 to 128 samples gains 1.98 perplexity, while doubling from 256 to 512 gains 0.46.Source: §B.8 · p. 28, Figure 12 · p. 28
  • Changing the calibration set to FineWeb 16M improves PTB, C4, and classification performance but worsens WikiText2, while LEMS continues to outperform baselines.The analysis uses 7B and 8B models; the random seed has less impact than sample count, with a largest observed difference of 1.05 perplexity.Source: §B.8 · p. 28

B.9. Inference Acceleration

  • LEMS generation speedup on Qwen3-8B rises from 1.14× at 20% compression to 2.01× at 80% compression.Measurements used one NVIDIA A100 GPU, batch size 4, 1024-token prefill, and 256-token decoding; qualitative examples compare response coherence, formatting, and content fidelity across compression ratios.Source: §B.9 · p. 28, Figure 12 · p. 28, §B.10 · pp. 28–29

C. Algorithms and Implementation Details

The algorithms separate decomposition, global rank allocation, and reconstruction, allowing the decomposition stage to be swapped independently of LEMS. KFAC-SVD estimates token-wise curvature factors, while LEMS calibrates an error proxy and searches rank assignments under a parameter budget.

End-to-End Compression Pipeline

  • The compression pipeline decomposes layers, allocates ranks globally with LEMS, then reconstructs the model using low-rank factors.The decomposition stage is modular and can use KFAC-SVD or another SVD variant that provides a valid singular-value spectrum; recalibration during reconstruction is optional and was not used for the reported results.Source: C. Algorithms and Implementation Details · p. 31

KFAC Fisher-Weighted SVD Decomposition

  • KFAC-SVD accumulates token-wise input and output-gradient statistics, applies shrinkage toward scaled identities, then uses Cholesky factors for whitening and decomposition.The implementation uses Cholesky factors for numerical efficiency; the text states this is isometric to the symmetric-square-root formulation for the Frobenius-norm objective.Source: C. Algorithms and Implementation Details · p. 32

Global Rank Allocation

  • LEMS calibrates each layer’s error scale at a reference rank, then searches propagation-bias parameters and solves an ILP for ranks under the target budget.Candidate assignments are validated against measured KL loss, and the bias function gives higher penalties to earlier blocks; its parameters are optimized rather than fixed across architectures.Source: C. Algorithms and Implementation Details · p. 33

D. Detailed Result Tables

The supplementary tables provide expanded results for decomposition, search, and model-size comparisons. Their captions identify the models, retention rates, metrics, and allocation comparisons represented; the supplied passages do not give the table-cell values.

Detailed Search Tables

Detailed SVD Tables

Numerical Results for Figure 2

  • Table 22 compares uniform and LEMS rank allocation for KFAC-SVD and SVD-LLM across five Qwen3 sizes from 1.7B to 32B and compression ratios from 0.5 to 0.9.Its caption says bold marks the better result between uniform and search-based allocation for each method.Source: D. Detailed Result Tables · p. 34, Table 22 · p. 37

Related papers

Loading 2610.32878v2…