LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

1 Introduction

• At high sparsity, pruning risks removing expressive neurons, while low-rank approximation misses incoherent parts that may be expressive and performance-critical.The introduction notes that light pruning can enhance generalization, whereas heavy pruning can severely hurt performance.Source: §1 · pp. 2–3
• LoSparse approximates a weight matrix as the sum of a low-rank matrix and a sparse matrix, separating coherent from incoherent neuron components.The low-rank part captures coherent expressive bases, while the sparse part removes non-expressive information in incoherent components.Source: §1 · p. 3
• LoSparse achieves more than 2.0% higher accuracy than existing baseline methods on MNLI, 3.3 points in F1 score over other pruning methods on SQuADv1.1 at extreme low remaining ratio, and 2.99 points in Rouge-1 score over current methods on XSum.The paper also reports that LoSparse significantly outperforms existing pruning methods on GLUE tasks.Source: §1 · p. 3
• LoSparse is orthogonal to current knowledge-distillation methods and could be integrated with them to improve performance.Source: §1 · p. 3

2 Background

• The background reviews transformer language models and pruning as context for the methods that follow.Source: §2 · p. 4

2.1 Transformer Models

• A transformer layer combines multi-head self-attention with a feed-forward network, with query, key, value, and output projection matrices in the attention module.The feed-forward network uses two linear transformations and an activation, followed by residual connection and layer normalization.Source: §2.1 · p. 4

2.2 Importance Scores for Pruning

• Pruning scores estimate parameter importance; sensitivity measures the loss change when a parameter is zeroed, though stochastic mini-batch sampling can make estimates uncertain.The paper also describes smoothing importance scores with an exponential moving average.Source: §2.2 · p. 4

2.3 Structured Pruning

• Because sensitivity targets unstructured pruning, the paper extends it to structured pruning by defining importance scores for weight-matrix neuron columns.The scores for all columns are collected into a vector Γ(W).Source: §2.3 · p. 5

3 Method

• LoSparse represents each weight matrix as UV + S, assigning coherent neuron structure to the low-rank term and incoherent components to the sparse term.The two terms are used in parallel in the forward pass.Source: §3 · p. 5, §3.1 · p. 6

3.1 Approximation by Low-rank and Sparse Matrices

• Rapidly declining singular-value spectra in language-model weight matrices indicate shared neuron subspaces that even a small-rank approximation can capture.The low-rank term approximates coherent parts, while the sparse term can capture residual incoherent parts.Source: §3.1 · pp. 5–6
• After decoupling, the method prunes the sparse term by neuron importance scores, since many incoherent components have low scores.The remaining incoherent parts are added as S and pruned to remove non-expressive components.Source: §3.1 · pp. 6–7, §3.2 · p. 8

3.2 Algorithm

• For each pretrained weight matrix, U and V are initialized from its top-r singular components before the matrix is represented as UV + S.The algorithm computes an SVD of the pretrained weights and initializes U and V from the leading singular vectors and values.Source: §3.2 · pp. 7–8
• The compressed forward pass computes (XU)V + XS instead of multiplying the input by the original weight matrix.The method applies this decomposition to every model weight matrix.Source: §3.2 · p. 7
• Each iteration updates U, V, and S, scores neurons in S, and prunes according to a remaining-neuron schedule that decays cubically.The schedule defines p_t as the percentage of neurons remaining at iteration t.Source: §3.2 · pp. 7–8

4.1 Natural Language Understanding

• At a 10% remaining ratio, LoSparse achieves 81.7% accuracy on MNLI-m, surpassing the best-performing baseline, ITP, by 2%; on BERT-base, it outperforms all baselines across MNLI, RTE, and QNLI at every sparsity level.Source: §4.1 · pp. 10–11

4.2 Question Answering

• On SQuAD v1.1, LoSparse consistently surpasses baseline methods across all remaining ratios in EM and F1, and exceeds ITP by 3.0% on F1 at low remaining ratios.For BERT-base at 10% remaining ratio, LoSparse outperforms ITP by 2.6% on F1.Source: §4.2 · p. 12
• On SQuAD v1.1, ITP achieves the best result over LoSparse and movement pruning at 50% remaining ratio; at 40%, LoSparse and ITP both achieve 84.2 F1.Source: Table 3 · p. 11, §4.2 · p. 12

4.3 Natural Language Generation

• On summarization, LoSparse consistently surpasses baseline methods at all tested remaining-weight ratios in ROUGE; on XSum, it outperforms ITP by 2.99 ROUGE-1, with gain increasing from 0.76 at 50% to 2.99 at 30%.Source: §4.3 · p. 12
• Figure 5 reports that LoSparse outperforms the tested low-rank approximation variants, indicating that adding sparse approximation can improve performance.The comparison spans different tasks and remaining ratios.Source: Figure 5 · p. 13, §4.3 · p. 13

4.4 Analysis

• LoSparse outperforms both low-rank variants, and Low-rank II performs better than Low-rank I.The variants discard sparse matrices and fine-tune only low-rank matrices (Low-rank I), or initialize as in (8) and gradually prune the sparse component to zero (Low-rank II).Source: §4.4 · p. 13
• Performance on NLU tasks stays stable as the allocation between low-rank and sparse approximations changes, suggesting LoSparse is not sensitive to the low-rank ratio.The analysis varies the proportion of low-rank matrices while holding the remaining ratio fixed.Source: Figure 6 · p. 14, §4.4 · p. 14

4.5 Combination with Knowledge Distillation

• With distillation, LoSparse outperforms ITP, while distillation further improves LoSparse and other compression methods.For BERT-base models, LoSparse with distillation achieves on-par or better results than most distillation methods, including PKD and Low-Rank BERT Feature Distillation + KD.Source: §4.5 · p. 14, Table 5 · p. 15, Table 6 · p. 15

4.6 Embed with Other Compression Method

• LoSparse improves CoFi on all datasets, with accuracy gains of around 1% on MRPC, RTE, and SST-2.The authors describe the improvement as evidence that LoSparse is complementary to existing compression methods.Source: Table 7 · p. 16, §4.6 · p. 16

5 Discussion

• Low-rank II performs much better than Low-rank I, and sparse matrices outperform fine-tuning a singular-value-thresholded low-rank matrix.The discussion says this result suggests LoSparse can enhance low-rank approximation.Source: §5 · p. 16
• Low-rank I's initialization can lose pretrained knowledge and hurt downstream performance; LoSparse bridges that gap and uses sparse approximation to guide training.The method bridges low-rank initialization and pretrained weights to retain their original knowledge.Source: §5 · p. 17
• Low-rank approximation ignores neuron diversity, an intrinsic drawback that the authors say LoSparse is crucial to remedy.Source: §5 · p. 17
• Earlier CNN approaches used parallel low-rank and sparse convolutions rather than directly approximating one matrix; their lower-rank kernels diminish efficacy at high compression.The discussion names Yu et al. (2017) and Hawkins et al. (2021).Source: §5 · p. 17
• Unlike DSEE's incremental-matrix compression and attention-head masking, LoSparse compresses the full model and prunes neurons.The discussion identifies DSEE as Chen et al. (2021).Source: §5 · p. 17

• LoSparse combines low-rank approximation with structured sparse approximation to compress transformer models.Source: §6 · p. 17
• Across natural language understanding, question answering, and natural language generation, LoSparse significantly surpasses previous compression approaches.Source: §6 · p. 17
• LoSparse is particularly effective in natural language generation and at extremely high sparsity levels.Source: §6 · p. 17
• LoSparse is generic and complementary to other compression methods; experiments show it can improve CoFi and conventional iterative pruning with knowledge distillation.Source: §6 · p. 17

A GLUE Dataset Statistics

• The GLUE statistics table organizes datasets by corpus, task, train, development, test, label, and metric columns.Source: A GLUE Dataset Statistics · p. 23, Table 8 · p. 23
• The table includes single-sentence classification tasks in GLUE.Source: A GLUE Dataset Statistics · p. 23
• The table also includes pairwise text classification tasks in GLUE.Source: A GLUE Dataset Statistics · p. 23

B.1 Training Details

• The implementation is based on the publicly available HuggingFace codebase.Source: §B.1 · p. 23
• The proportion of parameters allocated to all low-rank matrices is selected from 1%, 2%, 3%, or 5%, with the best final ratio reported.Source: §B.1 · p. 23
• Importance scores can vary across batches and training dynamics; LoSparse uses an exponential moving average and large batches, which help most GLUE tasks.Source: §B.1 · p. 23
• The pruning schedule follows Zhang et al. (2022), with warm-up steps adjusted for batch-size changes and minor β changes for some CoLA settings.Source: §B.1 · pp. 23–24
• Tables 9 and 10 summarize task hyperparameters for pruning DeBERTaV3-base and BERT-base, respectively.Source: §B.1 · p. 24, Table 9 · p. 24, Table 10 · p. 24

C.1 Dataset

• SQuAD v1.1 is used to evaluate LoSparse on question answering.Source: §C.1 · p. 25

C.2 Training Details

• Question-answering experiments use batch size 16, 10 fine-tuning epochs, AdamW, and a learning rate of 5 × 10^-5 across sparsities.Source: §C.2 · p. 25
• The experiments follow Zhang et al. (2022) for initial and final warm-up steps and use Table 11 settings for both DeBERTaV3-base and BERT-base.Source: §C.2 · p. 25, Table 11 · p. 25

D.1 Training Details

• Training used batch size 32, 10 epochs, Adam, warm-up steps 12800 and 51200, and β=0.85.Source: §D.1 · p. 25
• The selected learning rate was 4 × 10−5 and the best sparse approximation ratio was 10%.Source: §D.1 · p. 25

E.1 Teacher Models

• DeBERTaV3-base teachers followed He et al. (2021)’s official-repository hyperparameters, while BERT-base teachers came from Huggingface.Source: §E.1 · pp. 25–26

E.2 Training Details

• Training prunes the model before layerwise distillation, using distillation coefficient α=15; hyperparameter setups are tabulated for both teacher-model settings.Source: §E.2 · p. 26, Table 12 · p. 26, Table 13 · p. 26

F.1 Teacher Models

• Teacher models came from Text Attack except for MNLI, for which BERT-base was fine-tuned as the teacher.Source: §F.1 · p. 26

F.2 Training Details

• LoSparse and CoFi compression ratios multiply; settings of 0.5 and 0.2, respectively, yield 10% total compression.Source: §F.2 · p. 27