PaperlayerView exact revision on arXiv

Source-linked AI summary

Language model compression with weighted low-rank factorization

Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, Hongxia Jin

arXiv:2207.00112v1cs.LGcs.AIcs.CL

TL;DR

Standard SVD compresses learned language-model weights by minimizing reconstruction error, but that objective can misalign with task performance. The paper introduces Fisher-Weighted SVD, which uses task-based parameter importance, and reports closer task accuracy, effective compression of compact models, and reduced dependence on generic pre-training.

  • Problem

    Standard SVD may reconstruct task-critical parameters poorly because it optimizes matrix reconstruction error rather than task performance.

  • Method

    FWSVD weights SVD using estimated Fisher information computed from squared task-objective gradients over training data.

  • Results

    FWSVD achieves an average GLUE and NER score of 83.6 with fine-tuning, beating all other path-1 and path-2 methods without expensive generic pre-training.

  • Takeaways & Limitations

    FWSVD can compress already compact models and further reduce parameters while retaining task accuracy, while reusing the SVD solver and linear-layer implementation.

  • Takeaways & Limitations

    FWSVD requires a task objective and target-task training dataset, making it more suitable for task-specific than generic pre-trained models.

Abstract

from arXiv · show

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

1 INTRODUCTION

Transformer language models can be overlarge for restricted target tasks, motivating compression methods that avoid costly generic pre-training while preserving task performance. This work identifies a mismatch between standard factorization and task objectives and proposes Fisher-weighted SVD.

  • Large transformer models can be overkill for restricted target tasks and difficult to deploy on resource-constrained hardware.
  • 384 NVIDIA V100 GPU hours are required to pre-train TinyBERT on the Wiki corpus, making generic pre-training costly.
  • Low-rank factorization can directly inherit knowledge from a trained model without generic pre-training.
  • Standard SVD minimizes reconstruction error but may lose most task performance because it does not prioritize task-critical parameters.
  • Fisher-weighted SVD assigns parameter importance using Fisher information and is evaluated across varied language tasks.
  • The work reports that FWSVD can compress already compact models and achieve comparable compression and performance to methods requiring expensive generic pre-training.

2 BACKGROUND

Low-rank approximation replaces a weight matrix with smaller factors by truncating small singular values, reducing parameters and enabling two smaller linear layers. Fisher information provides a task-based parameter-importance signal, while prior work had not applied it to SVD compression.

  • 2.1 MODEL COMPRESSION WITH LOW-RANK APPROXIMATION: SVD approximates W with USV^T and obtains target rank r by setting singular values σr+1 through σk to zero.
  • 2.1 MODEL COMPRESSION WITH LOW-RANK APPROXIMATION: Factorization implements the approximation with two smaller linear layers containing Nr and Mr parameters, reducing model size when r < 0.5N for full-rank square matrices.
  • 2.1 MODEL COMPRESSION WITH LOW-RANK APPROXIMATION: SVD has been applied to transformer layers, but prior work did not investigate its poor results without fine-tuning.
  • 2.2 FISHER INFORMATION: Empirical Fisher information accumulates squared gradients over training data using the target task objective, such as cross-entropy or mean squared error.
  • 2.2 FISHER INFORMATION: Parameters with large absolute gradients are treated as important because they change the task objective more and should be reconstructed better.
  • 2.2 FISHER INFORMATION: Although Fisher-based importance has supported continual learning and structured pruning, prior work had not explored using it to assist SVD compression.

3 MODEL COMPRESSION WITH SVD MAY LOSE PERFORMANCE QUICKLY

The analysis tests whether SVD’s smallest singular-value groups are least important by truncating them and measuring task-performance drops. In a BERT model, the smallest-value group can hurt performance as much as a high-value group, exposing SVD’s misaligned importance ordering.

  • Brute-force truncation analysis: The brute-force analysis truncates singular values individually, reconstructs the weights, and evaluates the resulting model performance.
  • Grouped truncation: Singular values are divided into 10 groups, with group 1 containing the largest 10% and group 10 the smallest 10%.
  • Grouped truncation: Figure 1 compares the ideal red-dashed trend with actual blue-bar performance drops for grouped truncation.
  • Vanilla SVD dilemma: Small-singular-value parameters can significantly affect task performance yet be poorly reconstructed by vanilla SVD.
  • Grouped truncation: Groups are truncated across all layers together to observe a smoothed summed impact on performance.
  • Observed performance impact: In STS-B-trained 12-layer BERT, truncating the smallest-value 10th group caused a performance drop as large as truncating the 2nd group.

4 FISHER-WEIGHTED LOW-RANK APPROXIMATION

FWSVD modifies low-rank approximation so reconstruction errors are weighted by parameter importance derived from Fisher information, while retaining an SVD-based solution. The resulting factorization reserves r ranks after removing the weighting matrix from the factorized matrices.

  • FWSVD addresses SVD’s task-misaligned objective by introducing Fisher information into the low-rank approximation objective.The weighted objective accounts for parameters’ differing effects on task performance.
  • The method simplifies element-wise weighting by assigning each row of W a shared importance equal to the row’s summed Fisher information.This row-sharing assumption makes the weighted optimization easier to deploy and analyze.
  • Standard SVD on the Fisher-weighted matrix yields the solution after removing the information matrix from the factorized matrices.The resulting factors are A = Î^-1U*S* and B = V*T.
  • FWSVD truncates U*, S*, and V* to retain only r ranks when compressing W.The rank r controls the number of retained components in the factorization.
  • Because FWSVD shares standard SVD’s optimization process, its observed advantages are attributed directly to the Fisher-information weighting.The paper names this weighted method Fisher-Weighted SVD, or FWSVD.

5 EXPERIMENTS

The experiments compare three compression paths across eight language tasks, including GLUE and named entity recognition. They evaluate direct factorization, optional fine-tuning, further compression of compact models, and compatibility with parameter sharing.

  • THE PATHS TO A COMPRESSED LANGUAGE MODEL: The experiments examine three paths: compact-model distillation, direct factorization of task-specific large models, and compression of already compact models.These paths are denoted S → Sg → St, Lg → Lt → Ltf, and St → Stf.
  • THE PATHS TO A COMPRESSED LANGUAGE MODEL: Path-2 directly compresses task-specific large models and avoids generic pre-training for the compressed model.The comparison is designed to assess whether it can match path-1 performance at the same compression rate with less training.
  • LANGUAGE TASKS AND DATASETS: The evaluation covers eight natural language tasks from GLUE and named entity recognition on CoNLL-2003.Metrics include task-specific correlation, accuracy, F-1, and Pearson-Spearman correlation measures.
  • IMPLEMENTATION DETAILS AND THE BASELINE MODELS: All experiments use the same training configuration without hyperparameter screening, and results are reported directly on dataset development sets.The implementation uses HuggingFace Transformers defaults for unspecified settings.

5.3 PATH-1 VERSUS PATH-2

FWSVD outperforms other compact-model strategies on the evaluated tasks without expensive generic pre-training, while performance is examined across reserved-rank ratios without post-factorization fine-tuning.

  • 83.6 average score: FWSVD with fine-tuning beats all other path-1 and path-2 methods on GLUE and NER.This result does not require expensive generic pre-training.
  • FWSVD with fine-tuning consistently retains performance across tasks, unlike path-1 methods that show larger task-to-task variance.DistilBERT and MiniLMv2 each perform unevenly across CoLA and STS-B, whereas FWSVD+fine-tuning has no obvious shortcoming.
  • Figure 5 compares FWSVD and SVD as the reserved-rank ratio changes, including full-rank reconstruction at ratio 1.0.The plotted models receive no fine-tuning after factorization.

5.4 COMPRESSING AN ALREADY COMPACT MODEL

FWSVD can further compress already compact models while preserving task performance, including substantial parameter reductions for DistillBERT, TinyBERT, and ALBERT-large.

  • 30% parameter reduction: FWSVD with fine-tuning removes redundancy from DistillBERT.Under the same setting without fine-tuning, SVD reduces DistillBERT to zero accuracy while FWSVD retains a score of 67.9.
  • 18% parameter reduction: FWSVD with fine-tuning compresses TinyBERT without performance loss despite its 14.4M parameters.Compressed TinyBERT, MiniLM, and DistillBERT-MRPC slightly exceed their original performance in the reported results.
  • 84.3% versus 77.1%: FWSVD outperforms SVD on ALBERT-large when both reduce parameters by 14%.The ALBERT-large baseline average score is 84.7%.

5.5 FWSVD VERSUS SVD

Across rank ratios and grouped truncation, FWSVD retains task accuracy more reliably than SVD, even though it often produces larger reconstruction errors.

  • 17.5% average absolute improvement: FWSVD without fine-tuning outperforms SVD across all evaluated tasks.FWSVD retains substantial performance on CoNLL and STS-B, where SVD completely fails.
  • FWSVD performance increases more steadily with rank ratio, whereas SVD fluctuates and can abruptly drop, including on STS-B at ratio 0.3 versus 0.2.The comparison uses models without fine-tuning after factorization.
  • 5.5.1 REVISIT THE BRUTE FORCE ATTACK: FWSVD shows smaller performance drops for truncated small-singular-value groups, especially the tail groups truncated first.On STS-B, group 10 contains the smallest singular values.
  • 5.5.1 REVISIT THE BRUTE FORCE ATTACK: FWSVD often incurs larger reconstruction errors than SVD, particularly for rank groups 5 to 10 that are truncated first.Its objective therefore sacrifices reconstruction error while aligning more closely with the task objective.

6 LIMITATION AND FUTURE WORK

FWSVD improves task-specific compression but has narrower applicability than vanilla SVD and uses a simplified importance matrix that does not fully exploit Fisher information.

  • FWSVD requires a target task objective and training dataset, making it more suitable for task-specific models than generic pre-trained models.This task dependence trades method applicability for target-task performance; vanilla SVD applies more broadly.
  • FWSVD assigns the same importance to parameters in the same row of W, so it does not fully utilize element-wise Fisher information.The paper proposes directly seeking an element-wise factorization solution as future work.

7 CONCLUSION

FWSVD uses estimated Fisher information to weight parameter importance during SVD-based compression, alleviating mismatched optimization objectives. It significantly compresses already compact models while remaining simple to implement and deploy.

  • FWSVD weights parameters with estimated Fisher information to alleviate the mismatch between low-rank approximation and target-task objectives.The method reuses the existing SVD solver while incorporating parameter importance.
  • FWSVD achieves significant success when compressing an already compact model.
  • FWSVD implements factorized matrices with linear layers, making the method simple to implement and deploy.

A SUPPLEMENTARY

The supplementary figures examine grouped rank truncation across model settings and language tasks. In BERTbase experiments, FWSVD generally shows smaller performance drops when groups are truncated, while SVD is vulnerable when the smallest-singular-value group is removed first.

  • Figure 7 repeats the grouped rank truncation experiment from Figure 6a using the ALBERTbase model with 11.7M parameters.
  • FWSVD has a smaller performance drop than SVD when groups 5 to 10 are truncated first across all 8 BERTbase language tasks.
  • SVD usually shows a significant performance drop when group 10, containing the smallest singular values, is truncated first.
  • FWSVD does not show the same group-10 truncation issue in the reported BERTbase cases.
  • Figure 9 displays only groups 3 to 10 from Figure 8 to better visualize groups associated with smaller performance drops.
Loading 2207.00112v1…