Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression

1 Introduction

• Per-matrix whitened SVD reaches the Eckart–Young optimum for its local objective, yet yields WikiText-2 perplexity 42.1 versus 5.68 for the dense LLaMA-7B model at ρ=0.6.Errors from projections interact through block nonlinearities and compound across the residual stream.Source: §1 · pp. 1–2
• The three-level chain widens optimization from individual matrices through Transformer blocks to the full-model language-modeling loss, using 256 calibration sequences and no added inference-time parameters.WikiText-2 validation is used only for L3 early stopping.Source: §1 · p. 2
• On LLaMA-7B at 60% compression, WikiText-2 perplexity falls from 42.1 after per-matrix SVD to 19.1 after block optimization and 11.4 after full-model optimization.The final result is below SVD-LLM + Sequential LoRA (15.0), despite using 256 samples rather than 50K.Source: §1 · p. 2
• Perplexity gains hold across 20–80% compression, PTB and C4, and five architectures; skipping the block stage worsens OOD perplexity by 24 points, a gap extra end-to-end training did not close.The reported chain also outperforms SVD-LLM + Sequential LoRA (15.0) while using 256 rather than 50K samples.Source: §1 · p. 2

2 Background and Related Work

• Per-matrix methods improve factorization objectives, compensate sequential error, or allocate ranks, but do not jointly optimize sub-layers within a Transformer block.Examples include SVD-LLM, ASVD, and Dobi-SVD for objectives; SAES-SVD for compensation; and ARA, Swift-SVD, and DipSVD for rank allocation.Source: §2 · pp. 2–3
• Broader-scope methods include per-block distillation, layer-output anchoring, and external-data LoRA recovery, but generally retain local reconstruction objectives or add a data dependency.The paper identifies few fixed-budget comparisons across matrix, block, and full-model scopes, and no prior study of the block stage as a regularizer for end-to-end refinement.Source: §2 · p. 3
• The table’s LA and LM columns mark, respectively, next-block vocabulary-shift penalties and subsequent end-to-end language-modeling refinement.The proposed stage’s stated distinction is next-block vocabulary lookahead followed by factor-constrained LM-loss refinement under the same small calibration contract.Source: Table 1 · p. 3, §2 · p. 3

3 Three-Level Optimization

• The three-level chain warm-starts successive scopes—individual weights, each Transformer block, then the full model—using the same calibration set without external data.L1 supplies a per-matrix baseline, L2 captures cross-sub-layer interactions, and L3 corrects cross-block accumulation.Source: §3 · p. 3

3.1 Per-Matrix Whitened SVD (L1)

• L1 applies activation-whitened SVD to each weight matrix at a rank set by the removal ratio, optimizing its activation-aware Frobenius distance independently.Despite that per-matrix optimum, propagated errors interact through the block’s nonlinear forward pass, motivating L2.Source: §3.1 · p. 4

3.2 Block-Level Joint Optimization (L2)

• L2 jointly optimizes a block’s low-rank factors to redistribute errors across attention and MLP components, combining reconstruction error with a next-block lookahead term.A trained vocabulary-space probe compares the following block’s token distribution with the full model’s distribution; the final block omits lookahead, and multi-block cascades remain for L3.Source: §3.2 · pp. 4–5

3.3 End-to-End LM-Loss Refinement (L3)

• L3 warm-starts the low-rank factors from L2 and updates them end-to-end using standard autoregressive next-token cross-entropy, without extra parameters or data.This extends optimization beyond L2’s sequential blocks with detached gradients, which do not capture cross-block error accumulation.Source: §3.3 · p. 5

4.1 Setup

• All methods use 256 WikiText-2 training-split calibration sequences; evaluation covers WikiText-2, PTB, and C4 perplexity plus seven zero-shot tasks.The SVD-family baseline numbers come from a unified protocol, while the paper’s rows use the same pipeline and hyperparameters except for L2/L3.Source: §4.1 · p. 5

4.2 Overall Performance

• Across every tested ratio, L3 achieves the lowest WikiText-2 perplexity and highest zero-shot average among SVD-family methods, using 256 calibration sequences.At ρ=0.6, L3 nearly halves SAES-SVD’s perplexity without external data; SVD-LLM + Sequential LoRA uses 50K Alpaca samples and reaches 15.0.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Ratio Sensitivity

• At ρ=0.6, L2 reduces L1’s perplexity by ∼2× and L3 closes half the remaining gap; at ρ=0.8, all three levels are needed.L1 captures most of the gain at ρ=0.2; L3 is most worthwhile when OOD robustness matters or compression exceeds 50%.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

4.4 Cross-Architecture Portability

• At ρ=0.6, the chain improves all five models, but per-level contributions vary; architecture-specific configurations prevent attributing differences among models to architecture alone.LLaMA-2-7B is an outlier: WikiText-2 and C4 improve, while PTB remains high.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Compute and Deployment Efficiency

• The chain trades offline compute for lower inference memory: parameter count falls ∼58% and peak GPU memory 49–56%, while batch-1 decode is 6–14% slower.L1 takes 17.3 min, L2 13.76 h, and L3 42.4 min; timers exclude loading, serialization, and final perplexity evaluation, and hardware-matched break-even is not established.Source: §4.5 · p. 7, Table 4 · p. 8

4.6 Ablation Study

• Adding the lookahead probe amplifies L2’s perplexity reduction 2–3× over target choice alone; CF+LA is the default, though SI+LA wins on Mistral-7B.With the probe, the Mistral variants reduce WikiText-2 perplexity by 53–59% relative to L1, versus 20–23% without it.Source: §4.6 · p. 8
• L2’s OOD advantage rises from ∆=22 at 256 samples to ∆=72 at 32; with a 1024-sequence pool, skip-L2 is competitive on WikiText-2 and PTB, but not C4.The fixed-budget comparison reports that enlarging the calibration pool lets skip-L2 match or beat the full chain on WikiText-2 and PTB, while remaining worse on C4.Source: §4.6 · p. 8, Figure 3 · p. 9, Table 6 · p. 9

Compression Error at Multiple Interaction Scales

• L1→L2 accounts for more than half of the total perplexity reduction on LLaMA-7B, while L2→L3 closes much of the remaining gap.The balance is architecture-dependent: L3 provides the larger gain on Mistral-7B, but L2 helps on all five models.Source: §5 · p. 9

L2 as OOD Regularizer

• Skipping L2 matches in-distribution perplexity but worsens PTB by 24 points, with the penalty tripling under data scarcity.Block-level MSE preserves hidden-state directions that calibration-token LM loss alone does not reward.Source: §5 · p. 9

When to Use Each Level

• At ρ ≤0.2, L1 captures most of the gain; at ρ=0.4–0.6, L2 captures most robustness gains; at ρ ≥0.8, the full chain is needed.Under severe calibration scarcity, L2 still protects PTB but no longer dominates on WikiText-2 or C4; these guidelines derive from LLaMA-7B and may shift for other architectures.Source: §5 · p. 9

6 Conclusion

• The three-level chain widens SVD-based compression from individual matrices to Transformer blocks to the full model, using only 256 calibration sequences.The block-level stage matters for OOD robustness under this budget: skipping it costs 24 PTB points, and the penalty grows as calibration data shrink.Source: §6 · p. 9
• On LLaMA-7B at 60% compression, the chain reduces WikiText-2 perplexity from 42.1 to 11.4, below SVD-LLM + 50K-sample LoRA (15.0).The five-architecture runs use architecture-specific configurations rather than one common protocol.Source: §6 · p. 9
• With a larger calibration pool, skipping the block-level stage becomes competitive, so its benefit is a trade-off between offline compute and calibration data.The reported conclusions are restricted to perplexity and compression fidelity.Source: §6 · p. 9

Limitations

• The evaluation covers five models at 7B–13B, not 70B models, instruction-following, safety, or long-context settings.The authors claim only perplexity and compression fidelity, not preservation of broad capability, reasoning, instruction following, safety, or open-ended generation.Source: Limitations · p. 10
• Block-level optimization stores full hidden states per sample, limiting calibration to 256 samples; its lookahead probe captures only one block ahead.Multi-block cascades are handled by L3 rather than L2.Source: Limitations · p. 10
• The factored representation slows decode throughput by 9–14% in the eager, unfused implementation, and no hardware-matched compute comparison or break-even point is reported.The measured slowdown reflects the implementation rather than low-rank arithmetic itself; fused AB-kernels are not yet implemented.Source: Limitations · p. 10
• Downstream accuracy does not uniformly improve despite perplexity gains, and at 60% removal every compressed state scores 0% strict exact match on GSM8K.Held-out SAMSum summaries were degenerate and highly repetitive; the authors also caution that calibration distribution determines which capabilities survive compression.Source: Limitations · p. 10
• Compression can unevenly degrade safety-related behaviors, so deploying compressed models requires dedicated safety evaluation.Safety settings were not tested in the evaluation.Source: Limitations · p. 10

A LiLlama Comparison and Spectral Analysis

• Table 7 compares LiLlama-style per-block distillation with the authors’ method at ρ=0.6 using 256 calibration samples.Its archived wall-clock readings lack hardware metadata and do not support a compute-matched comparison.Source: Table 7 · p. 12

Re-Extraction Control

• On LLaMA-7B, merge→re-extract→F-stage changes WikiText-2 perplexity from 11.4 with saved factors to 11.37 with re-extracted factors.The authors conclude that re-extraction with QR+SVD re-balancing is benign or mildly beneficial, and associate LiLlama’s divergence with spectral amplification instead.Source: A LiLlama Comparison and Spectral… · p. 12

OOD Transfer

• On LLaMA-7B, L3 improves both WikiText-2 and C4 perplexity, while LiLlama’s L3 worsens C4 perplexity by 21% despite improving WikiText-2.The authors attribute the difference to headroom: LiLlama’s L2 is already near the dense WikiText-2 baseline, whereas their weaker L2 leaves room for gains that generalize to held-out corpora.Source: A LiLlama Comparison and Spectral… · p. 12

Warm-Start Quality

• Figure 4 shows that better L2 warm starts monotonically correspond to better L3 WikiText-2 perplexity on LLaMA-7B, but not on Mistral-7B, where LiLlama diverges.LiLlama has the best L2 result on Mistral-7B despite diverging at L3.Source: A LiLlama Comparison and Spectral… · p. 12
• LiLlama’s L3 diverges on Mistral-7B despite its best L2 warm start, whereas the best L2 warm start corresponds to better L3 on LLaMA-7B.The LLaMA-7B relationship is monotonic; the Mistral-7B relationship breaks.Source: A LiLlama Comparison and Spectral… · p. 12

Spectral Stability

• Under the 256-sample budget, Mistral-7B MLP sub-layers show 40–95% leading-singular-value amplification with scale-invariant L2, correlating with gradient explosion during subsequent L3 training.The authors report that their MSE-based L2 keeps singular values stable across architectures and data budgets; the instability may be specific to low-data settings.Source: A LiLlama Comparison and Spectral… · p. 12

• At ρ=0.8, WikiText-2 perplexity falls from 659 with L1 to 30.5 with L3.Source: B Full Results Tables · p. 12
• Each optimization level improves perplexity at every tested compression ratio on LLaMA-7B.Source: B Full Results Tables · p. 12
• Dense PTB/Wiki ratios range from 1.2× for OPT-6.7B to 7.2× for LLaMA-2-13B; high compressed-model PTB partly reflects domain sensitivity, while amplification after compression remains a limitation.Source: B Full Results Tables · p. 12

• At 32 calibration samples, skipping L2 worsens PTB perplexity by 72 points (161 vs. 89), compared with 22 points at 256 samples.Source: C Low-Data Ablation and Hyperparameters · p. 12
• The results indicate that L2’s block-level regularization is most valuable under data scarcity.Source: C Low-Data Ablation and Hyperparameters · p. 12

• On Mistral-7B, LiLlama has the best L2 result but diverges at L3 alongside 40–95% MLP singular-value amplification; CF+LA preserves baseline spectral structure.Source: Figure 4 · p. 13, Figure 5 · p. 13
• L2 improves all seven zero-shot tasks over L1, and L3 improves six of seven over L2, while compressed states remain far below dense; the reported averages are not a decomposition of Table 2.Source: D Robustness and Diagnostic Results · p. 13
• A bounded implementation study reports 1.537× block-stage and 1.302× complete-call speedups, but does not establish full-stage or hardware-independent gains.Source: D Robustness and Diagnostic Results · p. 14
• Skip-L2 reaches lower validation PPL (11.98 vs. 12.07) but overfits past step 90; its WikiText-2 test PPL is similar (11.4 vs. 11.9), while PTB differs by 24 points (62.8 vs. 86.9).Source: D Robustness and Diagnostic Results · p. 15, Figure 7 · p. 16
• L2’s hidden-state matching provides ∼16M per-token constraints that regularize factored parameters before L3 fine-tuning.Source: D Robustness and Diagnostic Results · p. 15
• The L1→L2→L3 perplexity chain generalizes across five 7B–13B models, although per-level gains depend on the model.Source: Figure 6 · p. 15
• Across three pipeline seeds, the full chain is lower on WikiText-2, PTB, and C4; effect size varies, and the check resamples pipelines rather than calibration subsets.Source: D Robustness and Diagnostic Results · pp. 12–13, Table 12 · p. 15
• At 60% removal, compressed states hit the 128-token ceiling on every SAMSum example with 50.9%–72.8% repeated-4gram rates; all score 0% on GSM8K versus 9.17% for dense.Source: D Robustness and Diagnostic Results · pp. 13–14, Table 14 · p. 15