PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

Artur Zagitov, Alexander Miasnikov, Maxim Krutikov, Vladimir Aletov, Gleb Molodtsov, Nail Bashirov, Artem Tsedenov, Aleksandr Beznosikov

arXiv:2606.03465v1cs.LGcs.AI

TL;DR

Large language model compression can reduce storage and deployment costs, but existing tensor-decomposition studies leave its effectiveness under realistic large-scale deployment unclear. The paper evaluates matrix and tensor methods across dense and MoE models and develops a theoretical analysis of tensor decomposition failures. It reports that tensor formats do not outperform matrix counterparts at matched compression ratios across the evaluated settings.

  • Problem

    Existing tensor-decomposition studies use evaluation protocols that do not reflect full-scale deployment constraints, leaving practical utility unclear.

    Source: §1 · p. 2

  • Method

    The paper conducts a large-scale empirical study of matrix and tensor decompositions across dense and MoE architectures and formalizes three chained theoretical obstructions.

    Source: §1 · p. 2

  • Results

    Across the evaluated settings, tensor formats fail to outperform matrix counterparts at matched compression ratios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis identifies three chained obstructions to Frobenius-optimal tensor decompositions serving as effective Transformer-weight compressors without significant fine-tuning.

    Source: §1 · p. 2

    The evaluation covers GPT-J 6B, LLaMA 2 7B, Qwen3-30B-A3B, and GPT-OSS-20B, and the conclusions may not transfer without verification to substantially different attention, normalization, or expert-routing architectures.

    Source: §7 · p. 10

Abstract

from arXiv · show

Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment. The code is available at https://github.com/brain-lab-research/TT-LLM.

1 Introduction

The introduction frames post-training compression as a way to reduce LLM deployment costs while preserving quality, but says tensor decompositions’ practical value under realistic deployment conditions remains unclear. The paper addresses this gap with large-scale dense and MoE evaluations and a theoretical analysis of why Frobenius-optimal tensor decompositions may fail without significant fine-tuning.

1 Introduction

  • Compression seeks to reduce model redundancy while preserving functional behavior, but pruning, quantization, and distillation typically require extensive fine-tuning or data-driven calibration for state-of-the-art performance.Source: §1 · p. 1
  • Matrix and tensor decompositions reduce parameters by replacing dense layers with smaller factors; Tucker uses a compressed core and per-mode factors, while Tensor Train chains three-dimensional cores.Source: §1 · p. 1
  • Existing tensor-decomposition studies leave practical utility unclear because their evaluation protocols do not reflect full-scale deployment constraints.Source: §1 · p. 2
  • Across the evaluated settings, tensor formats fail to outperform matrix counterparts at matched compression ratios, and the study diagnoses a consistent failure mechanism.The evaluation spans dense and MoE architectures, perplexity, downstream accuracy, and lightweight LoRA repair.Source: §1 · p. 2
  • The paper formalizes three chained obstructions to Frobenius-optimal tensor decompositions serving as effective Transformer-weight compressors without significant fine-tuning.Source: §1 · p. 2

2 Related Work

Related work covers low-rank matrix compression and tensor decomposition for LLM weights, alongside separate tensor-based approaches to fine-tuning. The cited studies span direct truncation, data- or structure-aware matrix methods, and tensor formats applied to attention, MoE, or cross-layer weights.

2 Related Work

  • Truncated SVD is presented as a globally optimal low-rank approximation, while LASER [32] applies it directly to selected matrices without retraining.Source: §2 · p. 2
  • Matrix-compression approaches include rotation-based SliceGPT [2], whitening and closed-form updates in SVD-LLM [45], calibration-based Dobi-SVD [44], and sparse-plus-low-rank HASSLE-free [25].Source: §2 · p. 2
  • Tensor work applies Tucker to attention projections [16] [21], stacks MoE expert weights for Tucker, or uses CP/Tucker-style decomposition across layers [23]; separate methods use TT for fine-tuning [49] [1] [48].Source: §2 · p. 2

3 Compression Strategies

Across dense and MoE models, tensor decompositions show poor compression–quality trade-offs against matrix methods. The analyses link these results to residual-stream geometry, critical weight structure, and rank truncation.

3.1 Pruning

  • Pruning exploits intermediate-layer redundancy, but leaves the remaining weights unreparameterized and limits maximum compression because most parameters remain intact.The paper describes pruning as a naïve baseline for more structurally advanced decompositions.Source: §3.1 · p. 3

3.2 Matrix Decompositions

  • On GPT-J 6B, for compression of the same middle-to-late block ranges, calibration- or fine-tuning-based methods degrade less than decomposition-only methods; LASER achieves ≈14% compression with a +8 PPL increase.SoLA keeps a dense subset of top-k highest-activation-norm neurons at full precision and allocates ranks across layers via integer programming.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 Tensor decompositions for Dense Architectures

  • Attention-only tensor compression preserves perplexity but saves little size, while FFN compression saves more at the cost of sharp degradation; LoRA repair only partly recovers quality.Across tested practical ratios, tensor methods still show poor compression-quality trade-offs. The dense-model experiments use Tucker on attention, TT on FFNs, or both.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4 Tensor decompositions for Mixture-of-Experts

  • On Qwen3-30B-A3B, TD-MoE trails MoBE; on GPT-OSS-20B, preserving expert rank beats compressing it at equal storage.The expert-rank ablation indicates that, for GPT-OSS-20B, expert directions are not interchangeable under the tested Tucker allocation.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 Structural Mismatch of Tensor Decompositions in LLMs

  • Across tested runs, higher perplexity accompanies greater residual-stream angular drift, norm shrinkage, or both, consistent with progressive distortion of activation geometry.The diagnostic compares original and compressed residual-stream activations after the last decomposed block, measuring mean angle and norm ratio.Source: §3.5 · pp. 5–6, Figure 5 · p. 6

3.6 Important Weights for Activation Restoration

  • Rank 1500 is required to restore the super-weight alone, while recovering all HASSLE-free-selected critical weights requires rank near the original matrix.A no-super-weight control showed qualitatively similar restoration results, so super-weight restoration alone is not sufficient for activation recovery.Source: §3.6 · pp. 6–7, Figure 6 · p. 6

4 Why Tensorization Fails: An Operator Analysis Perspective

Tensorization preserves Frobenius geometry but not the operator geometry that governs functional preservation; under heavy-tailed spectra, Frobenius-optimal tensor truncations can therefore have much larger spectral error than matrix-optimal truncation.

Operator-norm mismatch

  • Tensorization is a Frobenius isometry, but for d≥3 its tensor spectral norm uses a strict subset of the operator-norm ball, so Frobenius and operator optima need not coincide.The two norms can be polynomially separated, with the upper bound generically attained on matrices with delocalized singular vectors.Source: §4 · p. 7

Spectral suboptimality

  • On heavy-tailed spectra, HOSVD and TT-SVD spectral error can exceed the rank-k matrix-optimal error by a factor growing polynomially with layer width.The comparison is between the Eckart–Young rank-k optimum and the unfolding of a Frobenius-optimal tensor truncation.Source: §4 · pp. 7–8
  • For power-law spectra, suboptimality grows as Θ(N^(1/2−α)) when α<1/2, is Θ(log(N/k)) at α=1/2, and is bounded for α>1/2.The tail is dimension-dominated, rather than cutoff-dominated, in the heavy-tailed regime.Source: §4 · p. 8

Empirical relevance

  • Measured Qwen3-30B-A3B expert-tensor exponents all fall below 1/2, with expert and output modes routinely below 0.1 and the input mode below 0.4.Table 1 reports fitted power-law exponents for the three mode unfoldings of those expert tensors.Source: §4 · p. 8, Table 1 · p. 8

Implications

  • The operator residual may be up to min(m,n) times the matrix-optimal residual, misaligning Frobenius optimality with residual-stream preservation.Residual-stream propagation is governed by operator rather than Frobenius norms.Source: §4 · p. 8

Reconciliation with empirical findings

  • Angular drift and norm shrinkage track operator error, while recovery curves reflect critical singular directions deep in the spectrum; expert-axis tensorization compounds the gap.The expert mode has fitted exponents around 0.03–0.05 at intermediate layers, and neither reweighting Frobenius loss nor reordering modes mitigates these failures.Source: §4 · pp. 8–9

5 Comparison with the Quantization Baseline

The comparison benchmarks tensor- and matrix-based decompositions against 4- and 8-bit RTN. RTN leads the compression-quality comparison, while a Tucker(MHA)+LASER(FFN) hybrid outperforms the TT-based combinations at matched mid-range compression.

5 Comparison with the Quantization Baseline

  • The results position training-free decompositions as complementary to, rather than substitutes for, quantization at deployment-relevant compression ratios.Within decomposition methods, the passage attributes tensor methods’ disadvantage to the heavy-tailed spectra of FFN weights.Source: §5 · p. 9
  • RTN-4bit and RTN-8-bit track the dense baseline without perceptible perplexity degradation up to approximately 18% bits saved, dominating the tested compression-quality frontier.Among decompositions, HASSLE-free remains competitive only at low compression; beyond that regime, the methods incur perplexity penalties avoided by RTN.Source: Figure 7 · p. 9, §5 · p. 9
  • At approximately 17% bits saved, Tucker(MHA)+LASER(FFN) at rank 1024 achieves PPL approximately 19, while TT-based combinations exceed PPL 28.The hybrid outperforms Tucker(MHA)+TT(FFN) and TT(All) at matched compression ratios in the approximately 15–20% range.Source: §5 · p. 9

6 Conclusion

Tensor decompositions fail as standalone post-training LLM compressors because their optimization geometry does not preserve the spectral subspaces carrying model function. The resulting perturbations accumulate through depth and appear as activation drift and downstream degradation.

6 Conclusion

  • Tensor decompositions preserve Frobenius mass while distorting head-, neuron-, and expert-specific spectral subspaces, causing activation drift and downstream degradation to accumulate across depth.The paper identifies this geometry mismatch as the reason they fail as standalone post-training compressors.Source: §6 · p. 9

7 Limitations

The conclusions are bounded by the architectures evaluated, the looseness and scope of the theoretical analysis, and the deliberately limited LoRA repair. The paper also confines its evaluation to public models and research use.

7 Limitations

  • The evaluation covers GPT-J 6B, LLaMA 2 7B, Qwen3-30B-A3B, and GPT-OSS-20B, so transfer to substantially different attention, normalization, or routing designs remains unverified.The authors specifically note models trained with Muon or other second-order optimizers as potentially different because of their weight spectra.Source: §7 · p. 10
  • The operator-norm and spectral-suboptimality bounds are sufficient conditions for the observed failure, not tight characterizations, and practical gaps depend on individual-layer singular-value profiles.The fitted power-law exponents come from MoE expert tensors and may not generalize to all dense layer types.Source: §7 · p. 10
  • The rank-16, 100-step LoRA repair is a lower bound on recoverability, not a best-effort fine-tuning baseline, so the study does not establish full-fine-tuning recovery limits.Stronger post-hoc adaptation could narrow the reported quality gap.Source: §7 · p. 10
  • The evaluation uses publicly available models and benchmarks solely for research and does not fine-tune on sensitive data or target deployment outputs.The authors state that the studied post-training weight compression adds no capabilities raising safety concerns beyond those already in the evaluated models.Source: §7 · p. 10

B Additional information and experiments for Dense models

The appendix details the dense-model compression protocol and tests layerwise sensitivity on LLaMA 2 7B. Sensitivity varies substantially by depth: early layers are especially fragile, while multi-block compression still degrades quality as errors accumulate.

B.1 GPT-J and LLaMA 2 Tensor-Decomposition Protocol

  • GPT-J 6B compression progresses from block 14 to 20, and LLaMA 2 7B from block 16 to 23, adding one consecutive block at a time against a fixed dense baseline.The experiments report perplexity on WikiText-2 and C4 and calculate storage from compressed representations, excluding embeddings.Source: §B.1 · pp. 15–16
  • Attention projections use partial Tucker factorization with the head mode explicit, while FFN and TT-all weights use twelve paired input-output modes and TT ranks capped at 64.The Tucker ranks are capped at 64 for input features, 4 for head dimension, and 2 for projection type; present biases remain dense.Source: §B.1 · p. 15
  • Rank-16 LoRA adapters are trained after decomposition with modules frozen, using WikiText-2 for 100 optimizer steps to test limited data-dependent recovery rather than full fine-tuning.The protocol uses scaling factor 32, AdamW, learning rate 2 × 10^-4, and gradient accumulation over 8 micro-batches.Source: §B.1 · p. 15

B.2 Single-Layer Decomposition on LLaMA 2 7B

  • Early LLaMA 2 7B layers are especially fragile: first-layer attention Tucker and per-head TT cause large perplexity spikes, while FFN TT is most unstable in the first two layers.The single-layer protocol modifies one layer at a time across all 32 layers and measures WikiText-2 and C4 perplexity.Source: Figure 8 · p. 16, §B.2 · p. 16
  • Away from early layers, attention decompositions usually stay closer to baseline but save little model size; FFN decompositions save more, with effects growing toward final layers, especially for TT.The layerwise findings support the progressive middle-to-late schedule, but quality still degrades when errors accumulate across multiple compressed blocks.Source: §B.2 · p. 16
  • Early layers near the super-weight layer, Layer 1, show worse perplexity degradation than under pruning, indicating dominant singular components alone do not recover the original weight structure.Source: §B.2 · p. 16

B.3 Outlier Restoration Without Known Super-Weights

Without a super-weight, activation-weighted entries still remain below full recovery until retained rank nears the full matrix rank, while top-magnitude entries recover more smoothly.

B.3 Outlier Restoration Without Known Super-Weights

  • The diagnostic repeats the layer-1 MLP output-projection test on the corresponding layer-17 projection to check whether the rank-restoration barrier depends on one coordinate.The layer-1 test concerns a known super-weight; the layer-17 test checks the pattern without one.Source: §B.3 · p. 16
  • Activation-weighted entries remain substantially below full recovery until retained rank approaches full matrix rank, while top-magnitude entries recover more smoothly.Figure 9 tracks random inliers, top-magnitude entries, and activation-weighted top-k entries against retained SVD rank.Source: §B.3 · p. 16, Figure 9 · p. 17
  • The single-layer decomposition study reports bits saved over non-embedding parameters, with each run compressing one layer at a time.Source: Table 3 · p. 17

B.4 Detailed GPT-J and LLaMA 2 Results

Dense-Sparse TT substantially improves on plain TT for GPT-J 6B in the largest compressed range, but is nearly indistinguishable from TT on LLaMA 2 7B; the section also specifies the comparison protocols for detailed results and quality–compression trade-offs.

B.4 Detailed GPT-J and LLaMA 2 Results

  • Table 4 reports direct and rank-16 LoRA-repaired results across middle-to-late block ranges, with Angle and Norm measuring residual-stream activation geometry after the last compressed block.Lower Angle and Norm closer to 1 are better; the maximum-L table gives each model’s most compressed setting.Source: §B.4 · p. 17, Table 4 · p. 19

B.5 Quality–Compression Trade-offs

  • The quality–compression evaluation reports WikiText-2 perplexity and zero-shot accuracy drops on five LM-Eval tasks, with macro drop defined as their unweighted average in percentage points.Figure 10 plots perplexity against bits saved; Figure 11 plots macro accuracy drop against bits saved, with lower values better.Source: §B.5 · p. 17, Figure 10 · p. 18, Figure 11 · p. 18

B.6 Dense-Sparse TT with Outlier Preservation

  • At the largest compressed range, both Dense-Sparse TT variants have substantially lower C4 perplexity than plain TT on GPT-J 6B, but are nearly indistinguishable from TT on LLaMA 2 7B.Figure 12 compares TT with magnitude-selected and activation-weighted sparse corrections; the benefit is model-dependent and does not remove the compression–quality trade-off.Source: §B.6 · p. 18

C.1 Qwen3-30B-A3B and GPT-OSS-20B Tensor-Decomposition Protocol

The experiments apply joint Tucker decomposition to MoE layers and progressively compress layers in Qwen3-30B-A3B and GPT-OSS-20B. The reported architecture-dependent behavior is consistent with Qwen3’s greater expert redundancy, while GPT-OSS raw-text perplexity is not a reliable quality measure.

C.1 Qwen3-30B-A3B and GPT-OSS-20B Tensor-Decomposition Protocol

  • The progressive middle-out schedule compresses up to 24 MoE layers on Qwen3-30B-A3B and 12 on GPT-OSS-20B, using per-layer ratios ρ∈{0.2, 0.4}.Both experiments are evaluated against a fixed dense baseline for the same model.Source: §C.1 · p. 20
  • TD-MoE stacks a layer’s expert weights into an expert-by-input-by-output tensor and applies a joint Tucker factorization whose latent modes span experts, input features, and output features.The core tensor couples the three latent modes.Source: §C.1 · p. 20
  • Qwen3-30B-A3B’s 128 experts and 8 active experts per token leave more expert-mode redundancy than GPT-OSS-20B’s 32 experts and 4 active experts, which may help explain their different downstream behavior under compression.The proposed explanation is that moderate compression may denoise less-critical Qwen3 subspaces, whereas GPT-OSS has less redundancy to exploit before losing functional diversity.Source: §C.1 · p. 20
  • GPT-OSS-20B raw-text perplexity is anomalously high and unreliable because the model targets the harmony chat format, so downstream accuracy is treated as the reliable quality signal.The paper reports perplexity only for completeness.Source: §C.1 · pp. 20–21

C.2.1 Expert-Mode Comparison

  • For GPT-OSS-20B, preserve keeps all experts as distinct latent directions, whereas compress reduces the expert-mode rank and reallocates budget to higher-rank input and output factors.At the same target compression ratio, compress trades expert-mode capacity for the ability to exploit cross-expert redundancy, at the cost of reduced expert diversity.Source: §C.2.1 · p. 21

C.2.2 Comparison Against MoBE

At matched bits saved, MoBE perturbs GPT-OSS-20B less than TD-MoE according to the reported perplexity ordering and activation geometry, while raw-text perplexity is unreliable in absolute terms. The associated MoE experiments required about 60 H100 GPU-hours.

C.2.2 Comparison Against MoBE

  • GPT-OSS-20B’s uncompressed raw-text perplexity baseline is miscalibrated, so absolute perplexity is not a reliable quality signal and downstream accuracy and activation geometry are used instead.The relative perplexity ordering is reported as informative despite the unreliable absolute values.Source: §C.2.2 · p. 22, Figure 17 · p. 22

D Computational Costs

  • The Qwen3-30B-A3B and GPT-OSS-20B tensor-decomposition experiments, including the MoBE comparison, required approximately 60 H100 GPU-hours.GPU-hours are wall-clock runtime multiplied by the number of GPUs used.Source: D Computational Costs · p. 23

C.2.2 Comparison Against MoBE

  • At matched bits saved, MoBE produces smaller MoE-output angles and norm ratios closer to one than TD-MoE, consistent with the reported relative perplexity ordering.Figure 19 encodes block-schedule settings by marker, decomposition family by shape, bits saved by marker area, and C4 perplexity by colour.Source: §C.2.2 · p. 22, Figure 19 · p. 23, Figure 18 · p. 23

Related papers

Loading 2606.03465v1…