PaperlayerView exact revision on arXiv

Source-linked AI summary

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

arXiv:2606.07098v1cs.CLcs.LG

TL;DR

SVD-based LLM compression is limited by poorly performing naïve decompositions and underexplored learned scaling strategies. SigmaScale learns activation-aware row and column scaling before truncated SVD, achieving competitive benchmark performance, especially under mild-to-moderate compression.

  • Problem

    Learning scaling matrices for SVD-based LLM compression remains underexplored, while analytical scaling and naïve Frobenius-norm SVD have limitations.

  • Method

    SigmaScale learns row- and column-wise scaling transformations before truncated SVD, followed by final compression and post-compression fine-tuning.

  • Results

    SigmaScale performs on par with similar state-of-the-art methods, with strongest gains in mild-to-moderate compression and effective-rank entropy strongly correlating with compression loss.

  • Takeaways & Limitations

    Learned scaling provides a competitive alternative to analytically derived scaling while adapting the intrinsic rank of individual weight matrices.

  • Takeaways & Limitations

    SigmaScale degrades sharply, especially on Llama 3.1, and should not be considered a complete solution for extreme low-rank compression.

Abstract

from arXiv · show

We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes two sets of vectors that define diagonal row and column scaling transformations under an activation-aware compression loss. We show that learned scaling lowers the effective intrinsic rank of weight matrices, as reflected by reductions in effective-rank entropy, and that this reduction is strongly correlated with compression loss. Experiments on Llama 3.1 8B Instruct and Qwen3-8B show that SigmaScale is competitive with closely related state-of-the-art SVD-based compression methods across perplexity and zero-shot benchmarks. By using learned activation-aware transformations, SigmaScale explores a more flexible route to low-rank LLM compression by adapting to the structure of individual model weights. The advantage observed in specific tasks makes our approach a valid option for applications requiring a reduced LLM-inference computing cost.

1 Introduction and Background

LLM growth has increased deployment demands, motivating compression methods including quantization, pruning, knowledge distillation, and low-rank decomposition. Although truncated SVD offers an analytical low-rank approximation, its computational limitations and poor performance under naïve Frobenius-norm compression motivate activation-aware objectives and learned scaling matrices.

  • Motivation: LLM parameter growth increases technological accessibility challenges, motivating research into model-compression methods.The surveyed approaches include quantization, pruning, knowledge distillation, and low-rank decomposition.
  • Low-Rank Compression: Low-rank decomposition approximates W ∈ R^m×n with L ∈ R^m×k and R ∈ R^k×n, where k ≪ min(m, n), without specialized hardware.It can also be deployed alongside quantization and pruning.
  • SVD Background: For Frobenius-norm minimization, the Eckart–Young–Mirsky theorem identifies truncated SVD as the optimal analytical solution.Retaining the top k singular values and corresponding singular vectors discards lower-energy modes.
  • SVD Limitations: SVD has computational cost O(n^3) for square matrices, and its unstable derivative makes repeated optimization-step decomposition poorly scalable.Taylor expansion-based approximations have been used to approximate its gradients.
  • Activation-Aware Compression: Naïve Frobenius-norm SVD compression performs poorly on neural-network weights, partly because activation outliers are not represented.Prior work therefore uses activation-aware losses ||Wx − W′x||F and linear invertible scaling matrices S to optimize functionality and address outliers.
  • Research Gap: Learning scaling matrices for SVD improvement is underexplored because prior methods derive S analytically, while knowledge distillation is suggested for post-compression performance recovery.Compressed LLMs are difficult to realign because faithful distribution recovery would ideally require the original post-training datasets.

2 Methodology

SigmaScale compresses each model layer through sensitivity-based rank selection, learned row and column scaling before truncated SVD, and post-compression fine-tuning. The method optimizes activation-aware transformations of individual weight matrices before replacing them with low-rank factors.

  • Compression pipeline: The pipeline probes layer and module sensitivity, learns scaling matrices before truncated SVD, performs final compression, and realigns weights through post-compression fine-tuning.Experiments are based on Llama 3.1.
  • Sensitivity probing: Sensitivity probing evaluates compression ratios c ∈ {0.1, 0.2, . . . , 0.9} to determine truncated-SVD target ranks k.The ratios represent the percentage of parameters retained after decomposition.
  • Sensitivity probing: The method searches ranks for isolated MLP and attention matrices across layers, minimizing perplexity increases while achieving a global target compression ratio.Rank selection uses the binary-search algorithm introduced in ASVD.
  • Learned scaling: For each weight matrix, SigmaScale initializes row and column vectors with Gaussian noise scaled by the matrix standard deviation.The initialization is dr,c = (0.1) σW ϵr,c, with ϵr and ϵc sampled from identity-covariance Gaussian distributions.
  • Learned scaling: Exponentiated vectors define positive diagonal row and column scalings, after which truncated SVD, inverse scaling, and normalized activation-aware loss guide learning.The procedure uses Sr = diag(exp(dr)) and Sc = diag(exp(dc)) and computes loss using calibration activations X.
  • Final compression: After learning the scaling vectors, the method forms W′ from low-rank factors by absorbing singular values and inverse transformations, then fine-tunes the compressed model.The compressed matrix satisfies W′ = LR and replaces the original weight matrix.

3 Experimental setup

The experiments evaluate Qwen3-8B and Llama 3.1-8B-Instruction on English-focused perplexity and post-compression fine-tuning tasks. The setup uses Wikitext2 data, Alpaca fine-tuning, selective weight freezing, and comparisons with SVD-LLM and ASVD+.

  • The study evaluates Qwen3-8B and Llama 3.1-8B-Instruction with a focus on English language.
  • Perplexity uses 141 Wikitext2-raw-v1 test samples with sequence length 2048, while calibration uses 32 Wikitext training samples of the same length.
  • Post-compression fine-tuning uses Alpaca data and freezes unmodified weight matrices, comparing supervised fine-tuning with knowledge distillation from an uncompressed teacher.
  • Compression excludes token embeddings, layer normalizations, and the language modeling head, and comparisons include SVD-LLM and ASVD+.

4 Results and Analysis

Results show SigmaScale is most effective under mild-to-moderate compression, improving perplexity and often recovering zero-shot performance, while degrading more sharply at aggressive compression. Analysis further finds a strong correlation between compression loss and effective-rank entropy in compressed weight matrices.

  • Compression-regime results: At 0.90x retention, SigmaScale substantially improves perplexity over SVD-LLM for both models while recovering much zero-shot performance.The comparison covers Llama 3.1-8B-Instruction and Qwen3-8B.
  • Compression-regime results: At 0.75x retention, SigmaScale generally improves several zero-shot benchmarks, but perplexity gains are marginal.
  • Compression-regime results: At 0.50x retention, SigmaScale degrades more sharply, particularly for Llama 3.1-8B-Instruction.The retained subspace may become too small for learned scaling alone to compensate for discarded singular directions.
  • Compression-regime results: The method is most effective when reshaping the singular-value spectrum preserves dominant weight-matrix components.
  • Effective-rank analysis: A strong correlation exists between compression loss and effective-rank entropy of compressed weight matrices’ Σ components during scaling-vector optimization.The analysis aggregates mean compression-loss and effective-rank-entropy decreases; similar patterns are observed for Qwen3.

5 Conclusions

SigmaScale demonstrates effective SVD-based LLM compression by learning scaling matrices rather than deriving them analytically. The learned transformations manipulate weight-matrix intrinsic rank, with effective-rank entropy changes correlated with compression loss, while achieving performance on par with similar state-of-the-art methods.

  • 5 Conclusions: SigmaScale performs on par with the most similar state-of-the-art SVD-based compression methods.Its approach differs fundamentally by learning scaling matrices S instead of deriving them analytically, as in SVD-LLM or ASVD.
  • 5 Conclusions: Learned scaling matrices manipulate a weight matrix’s intrinsic rank, as reflected by changes in effective-rank entropy.These entropy changes correlate with compression loss.

Limitations

SigmaScale has substantial computational and evaluation limitations. It requires costly exact SVD updates, degrades sharply at extreme low ranks, and has unverified robustness across calibration distributions and task types.

  • Computational cost: O(n^3) SVD computation at every scaling-matrix update creates substantial computational expense, and faster approximate SVD methods are unexplored.The method computes SVD while learning S, rather than using approximations.
  • Compression range: The method degrades sharply, especially for Llama 3.1, and is not a complete solution for extreme low-rank compression.This limitation is reported in the paper’s Section 4 results.
  • Evaluation scope: Evaluation is limited to perplexity and selected zero-shot benchmarks, without examining longer-form generation or coding tasks.The passage specifically identifies longer-form generation and coding as unexplored effects.
  • Calibration robustness: Robustness to different calibration distributions has not been formally verified, while Wikitext was used mainly for consistency with SVD-LLM and ASVD comparisons.The authors anticipate that Wikitext is a subpar calibration choice.

Ethical Considerations · A Core Experiment Variations

The paper reports no additional ethics review requirement, sensitive-data or human testing concerns, licensing violations, or significant direct risks. Core experiment variations test Alpaca- and Wikitext2-based post-compression fine-tuning, with differing effects on perplexity and zero-shot performance.

  • Ethical Considerations: The authors state that the work does not require an additional ethics review.They report no human testing and no use of sensitive data.
  • Ethical Considerations: The paper reports that its asset licenses are summarized in Appendix I and that no licensing terms are violated.
  • Ethical Considerations: The authors foresee no additional significant ethical, societal, or environmental risks arising directly from the work.They urge downstream users to cross-check and verify their model int
  • A Core Experiment Variations: Additional experiments use a custom Alpaca dataset whose outputs are generated from Llama 3.1 8B Instruction.The dataset provides three output generations per instruction to introduce variance.
  • A Core Experiment Variations: Post-compression fine-tuning compares one answer per instruction over three epochs with three answers per instruction over one epoch.The results are reported in Table 3, with tests showing minor improvements using the Llama-Alpaca dataset.
  • A Core Experiment Variations: Wikitext2 post-compression fine-tuning uses the continued pretraining task, with results reported in Table 4.
  • A Core Experiment Variations: Wikitext2 fine-tuning improves perplexity but decreases overall zero-shot benchmark performance across the board.

B Implementation Details

SigmaScale uses Wikitext-based calibration and evaluation, Alpaca fine-tuning, and grid-searched scaling-learning hyperparameters. The implementation also specifies knowledge-distillation settings and handles unstable SVD derivatives by skipping problematic updates.

  • Data: Wikitext2-raw-v1-Train calibrates Llama 3.1 and Qwen3, while Wikitext-Test is used for all perplexity evaluations.Wikitext2raw-v1 supplies the Wikitext subsplits, and the cited dataset links identify the data sources.
  • Post-compression fine-tuning: Original Alpaca 2 supports post-compression fine-tuning, which runs for 1 epoch with cross-entropy computed only over the response span.The learning rate is 10^-6 with cosine scheduling and a 0.1 warmup step ratio for SVD-LLM and SigmaScale.
  • Scaling Matrix Learning: Scaling-matrix learning uses grid-search hyperparameter optimization, with the selected configuration reported in Table 5.Algorithm 1 describes the training loop, and Algorithm 2 describes constructing the final compressed W ′.
  • Knowledge distillation: Knowledge distillation combines KL divergence between student and teacher logits with student cross-entropy on ground-truth labels, using α = 0.7 by default.The loss function is specified in Eq. 6.
  • SVD derivative during training: SVD derivative instability is addressed by skipping updates when the σ_i − σ_j denominator approaches zero and produces NaN values.Despite skipped updates, the loss often converges and triggers early stopping, though the workaround is not necessarily robust.

C Compute Budget · D Custom Alpaca Dataset

The appendices describe the computational setup and supporting data resources for SigmaScale experiments. They specify GPU-based processing costs, Alpaca data variations, and benchmark or training-data references.

  • C Compute Budget: Compute uses Nvidia and AMD GPUs, with approximate times and GPU counts reported for each processing stage.The figures are reported per experimental condition, including model and compression ratio.
  • C Compute Budget: Compute reporting is conditioned on the model and corresponding compression ratio for each experimental setting.The passage frames these quantities as per-condition measurements rather than aggregate totals.
  • C Compute Budget: The compute-budget materials reference OpenBookQA, ARC-Easy, WinoGrande, PIQA, and HellaSwag.These benchmark names are listed together in the supplied passage.
  • C Compute Budget: Table 3 covers Llama-Alpaca variations using 3-parameter, 1-epoch and 1-parameter, 3-epoch configurations, alongside Vanilla Alpaca with 3 training epochs.The table annotations distinguish the listed training variations.
  • C Compute Budget: Table 4 reports benchmark results when Wikitext-Train serves as post-compression training data for Llama 3.1 8B Instruct.The passage identifies both the training-data source and target model.
  • D Custom Alpaca Dataset: The custom Alpaca dataset is created from Llama 3.1 8B Instruct by generating 3 output versions for each datapoint row.The stated purpose is to introduce data variance for weight realignment.
  • D Custom Alpaca Dataset: Dataset creation reruns inference 3 times with specified generation settings.The supplied passage indicates that the generation settings are presented in a following table.

E Investigating Scaling Matrix S Training Paradigms

The section investigates isolated and aggregated row/column scaling training for S, finding that scaling both rows and columns decreases compression loss. It also specifies how learned scaling is incorporated into low-rank factors and motivates joint training across scaling matrices.

  • Low-Rank Reconstruction: After symmetric row/column scaling, the method maps the low-rank factors back to the original parameter space using inverse row and column scaling.Algorithm 2 outputs L = S_r^-1 L_scaled and R = R_scaled S_c^-1.
  • Scaling Training Paradigms: Scaling both rows and columns clearly decreases compression loss in the Llama 3.1 8B layer 30 Key matrix test case.The analysis also examines the MLP_down module through loss curves.
  • Scaling Training Paradigms: Sequentially training row then column scaling is compared with jointly training both scaling matrices for a single module.The comparison informs the core methodology’s joint training of all scaling matrices.

F Further Analysis on Sigma Values … J Alpaca Prompt Template

The further analysis examines how learned scaling matrices alter sigma-value distributions and relate to compression loss and effective-rank entropy. The merged sections also document generative-AI use, resource licensing, and Alpaca-style prompt formatting for Llama and Qwen models.

  • F Further Analysis on Sigma Values: SigmaScale’s further analysis expands the compression-loss versus sigma-value effective-rank-entropy study to Qwen3-8B.Table 10 reports loss and effective-rank-entropy decreases across all seven modules.
  • F Further Analysis on Sigma Values: Compression loss is analyzed for row and column scaling matrices trained separately and together on Llama 3.1 Key and MLP_down matrices.Figures 2 and 3 cover layer 30 Key and layer 14 MLP_down matrices, respectively.
  • F Further Analysis on Sigma Values: Applying scaling matrix S to weight W shifts the sigma-value distribution by amplifying higher sigma values and slightly reducing lower values.This downstream distributional effect is illustrated in the accompanying analysis figures.
  • H Generative AI Disclosure: The study used generative AI as a coding assistant and writing aid, with all outputs human cross-checked and validated.The assistance included refining text and cross-checking grammar.
  • I Used Resources Licensing Overview: Licenses for the models and datasets used in the study are summarized in Table 11.The licensing overview covers the resources used as part of the study.
  • J Alpaca Prompt Template: The prompt follows a template similar to the original Alpaca dataset and uses a Hugging Face tokenizer for automatic Llama and Qwen formatting.The system prompt contains overall instructions, the user prompt contains task-specific instructions and optional input, and the assistant section contains the expected output.
  • J Alpaca Prompt Template: Listing 1 specifies the Llama 3.1 Alpaca-style chat template, including an instruction field and an assistant output field.The displayed template places the instruction in the user message and the output in the assistant message.
  • J Alpaca Prompt Template: Figures 4 and 5 compare sigma values after SVD on scaled and unscaled Llama 3.1 weights using logarithmic and linear x-axis views.They show layer 30 Query and layer 14 Key matrices, while the dashed Rank line marks the SVD truncation rank.
Loading 2606.07098v1…