PaperlayerView exact revision on arXiv

Source-linked AI summary

DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models

Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Hassan Awadallah, Zhangyang Wang, Yu Cheng

arXiv:2111.00160v3cs.LGcs.CL

TL;DR

Large pretrained models make fine-tuning costly, while task-specific models retain the full pretrained size. DSEE combines sparse-aware low-rank updates with update-derived masks; experiments report efficiency gains with competitive downstream performance.

  • Problem

    Full fine-tuning of growing pretrained models requires substantial memory and computation, while parameter-efficient methods still retain full pretrained weights for inference.

    Source: §1 · pp. 1–2

  • Method

    DSEE combines sparsity-aware low-rank weight updates with sparse masks derived from those updates and applied to pretrained weights.

    Source: §1 · p. 2

  • Results

    Across models and datasets, DSEE reports competitive downstream performance with parameter and inference efficiency; on BERT it uses 0.5% trainable parameters and saves about 25% inference FLOPs while losing less than 2% performance.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The framework jointly targets parameter efficiency during fine-tuning and resource efficiency in the resulting model, including through unstructured or structured sparsity.

    Source: §1 · p. 2

    The studied models are at the 100–300M parameter scale and evaluation focuses on GLUE, E2E, WebNLG, and DART; unstructured sparsity may yield limited speedup because it is less hardware-friendly.

    Source: §5 · p. 8

Abstract

from arXiv · show

Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the fine-tuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many fine-tuned models will be deployed in resource-constrained environments. To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning - by enforcing sparsity-aware low-rank updates on top of the pre-trained weights; and (ii) resource-efficient inference - by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structured sparse patterns in pre-trained language models via a unified approach. Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, RoBERTa, and GPT-2) on dozens of datasets, consistently demonstrate impressive parameter-/inference-efficiency, while maintaining competitive downstream performance. For instance, DSEE saves about 25% inference FLOPs while achieving comparable performance, with 0.5% trainable parameters on BERT. Codes are available in https://github.com/VITA-Group/DSEE.

1 Introduction

DSEE targets the high cost of fine-tuning and deploying large pretrained models by combining sparse-aware low-rank updates with sparse final weights. Its reported experiments show parameter and inference savings while retaining competitive performance.

1 Introduction

  • Full fine-tuning becomes less feasible as models grow, and storing a full-sized model for each task complicates deployment in resource-constrained settings.The passage gives examples ranging from 110M trainable parameters for BERTBASE to 175B for GPT-3.Source: §1 · p. 1
  • Earlier parameter-efficient methods can reduce trainable parameters but retain full pretrained weights for inference, while sparse-only or low-rank-only update assumptions may restrict updates.The introduction contrasts these limitations with DSEE's combined sparsity and low-rank approach.Source: §1 · p. 2
  • DSEE uses sparsity-aware low-rank updates for parameter-efficient fine-tuning, then derives masks from those updates to sparsify pretrained weights for inference efficiency.The approach targets both weight updates and the final model weights.Source: §1 · p. 2
  • The framework investigates unstructured and structured sparsity, deriving final-weight masks from updates without extra parameters or access to pretrained weights.The authors describe this as jointly optimizing parameter efficiency during fine-tuning and resource efficiency in the fine-tuned model.Source: §1 · p. 2
  • On BERT, DSEE uses 0.5% trainable parameters and saves about 25% inference FLOPs while losing less than 2% performance.On GPT-2, the reported BLUE scores are 69.5, 54.9, and 47.5 on E2E, WebNLG, and DART, respectively, with 0.1% trainable parameters and 20% of pretrained parameters removed.Source: §1 · p. 2

2 Related Work

Related work covers pruning and sparsification, low-rank methods, and parameter-efficient adaptation. The paper positions DSEE at the intersection of sparse and low-rank structures for efficient fine-tuning.

Pruning and Sparsification

  • Pruning can reduce model parameters and improve training and inference efficiency, but prior pruning and sparsification methods do not provide parameter-efficient updates.Examples include attention-head pruning, criteria based on weight change, low-rank factorization with ℓ0 regularization, and structured Butterfly sparsification.Source: §2 · p. 2

Low-rank decomposition

  • Low-rank approximation has broad machine-learning applications, and deep learned over-parameterized models can have approximately low-rank weight structures.The cited examples include robust principal component analysis and analyses of learned weights [Oymak et al., 2019].Source: §2 · p. 2
  • Prior work has used low-rank decomposition to shrink models and reduce trainable parameters, but the paper identifies sparse-plus-low-rank structures for pretrained-language-model fine-tuning as unstudied.The passage also notes earlier work imposing low-rank constraints during training and work on low-rank adaptation.Source: §2 · p. 3

Parameter-efficient adaptation

  • Parameter-efficient adaptation methods train adapters, sparse updates, prompts, biases, or low-rank factors, but their fine-tuned models retain the pretrained model's full weight count.The paper distinguishes reducing trainable parameters from building sparse models.Source: §2 · p. 3

3 Methodology

The methodology combines sparse and low-rank representations for parameter-efficient updates and considers structured and unstructured pruning for resource savings. It introduces preliminaries and an algorithm for selecting sparse residual locations.

3 Methodology

  • DSEE's methodology is organized around sparsity generation, parameter-efficient fine-tuning, and the proposed dually sparsity-embedded algorithms.The section outline places the preliminaries in Section 3.1 and the algorithms in Sections 3.2 and 3.3.Source: §3 · p. 3

3.1 Preliminaries

  • Unstructured pruning saves memory by reducing nonzero weights, while structured pruning can also reduce computation through smaller sparse weight structures.The preliminaries define pruning as applying a binary mask to weights; parameter-efficient tuning represents ΔW with fewer trainable parameters.Source: §3.1 · p. 3
  • The paper combines sparsifying weight updates with low-rank decomposition to represent ΔW using fewer trainable parameters.This combines approaches previously described separately as sparse updates and low-rank decompositions.Source: §3.1 · p. 3

3.2 Sparsity-Embedded Parameter-Efficient Fine-tuning

  • Algorithm 1 decomposes pretrained matrices into low-rank factors plus sparse residuals, then retains the largest-magnitude residual entries to form index sets.Its inputs include a nonzero-element count and a number of weights to decompose; the listed procedure thresholds S′ and records selected locations in Ωi.Source: §3.2 · p. 4

5 end

The method represents each update as a low-rank component plus a sparse residual, then uses update-derived masks to sparsify pretrained weights. It supports both unstructured and structured pruning while maintaining parameter-efficient updates.

5 end

  • ΔW combines a low-rank component UV with a sparse residual PΩ(S), reducing trainable parameters to (m + n) × r + card(Ω) for an m × n matrix.This is below full fine-tuning's m × n trainable parameters when (m+n) is approximately no more than 0.5 min{m,n}; r is described as small relative to m and n.Source: §5 · p. 4
  • The sparse residual locations Ω are motivated by the hypothesis that residuals outside the low-dimensional subspace of pretrained weights may also lie outside the update's low-rank component.The authors formulate residual discovery as robust principal component analysis and use GreBsmo, an SVD-free algorithm, to solve it.Source: §5 · p. 4

3.3 Dually Sparsity-Embedded Efficient Tuning (DSEE)

  • DSEE derives sparse masks from parameter-efficient updates and applies them to pretrained weights, requiring neither extra sparsification parameters nor access to the underlying pretrained weights.The passage presents this as a route to memory or computational resource efficiency beyond the parameter savings of the updates alone.Source: §3.3 · p. 4
  • Unstructured pruning keeps positions with the largest update magnitudes, while structured pruning removes attention heads with the lowest summed update magnitudes.For structured pruning, the method adjusts V and ΔWs to the new shape while keeping U intact.Source: §3.3 · p. 5
  • With a fixed parameter budget, adapting more modules reduces parameters per module; the authors report Wq and Wv as the optimal modules to adapt.Modules not adapted are pruned separately by the magnitudes of their pretrained weights, followed by several recovery fine-tuning epochs.Source: §3.3 · p. 5

4 Experiment Results

Across BERT, GPT-2, and RoBERTa experiments, DSEE combines sparse weight updates with sparse final weights to reduce model size while retaining competitive downstream performance. Ablations identify effective pruning criteria and adaptation choices, with outcomes depending on sparsity level and task.

4 Experiment Results

  • Experiments evaluate BERTBASE and RoBERTaLARGE on GLUE and GPT-2 on E2E, WebNLG, and DART, using task metrics and parameter, FLOPs, and model-size measures.Comparisons include full fine-tuning, pruning, and parameter-efficient baselines such as LoRA and adapters.Source: §4 · p. 5

4.1 Efficient Tuning with DSEE

  • At approximately 600K trainable parameters, 384 sparse residual entries improve validation by 0.23%–1.09% on all but CoLA; at 0.20M parameters, they raise all reported metrics across three GPT-2 tasks.At 0.39M parameters, sparse residuals improve all metrics on WebNLG and DART and slightly boost E2E NIST.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 7
  • With 50% of pretrained weights retained, GPT-2 DSEE achieves performance comparable to full fine-tuning on E2E and DART.The experiments also report competitive performance against baselines with fewer parameters at 80% of total parameters.Source: §4.1 · p. 6, Table 4 · p. 8
  • At 80% and 70% of total parameters, BERT DSEE retains comparable performance; at 50%, CoLA and RTE drop more, while QQP stays within a <1.5% gap.The comparison uses unstructured pruning on GLUE benchmarks.Source: §4.1 · p. 6, Table 3 · p. 7

4.2.1 Different criteria for sparse masks

  • Pruning by |∆W| preserves performance with both unstructured and structured masks; structured pruning by this criterion performs best on RTE and SST-2.Unstructured pruning without pretrained weights is comparable to pruning by final adapted weights, though slightly weaker.Source: §4.2.1 · p. 7

4.2.2 Different choices of modules to adapt

  • At equal trainable-parameter counts and 30% sparsity, adapting Wq and Wv performs best; DSEE's Ω selection beats magnitude and random selection across sizes, with 16 optimal on SST-2.The default Ω size is 16 for simplicity.Source: §4.2.2 · p. 7, Figure 2 · p. 8

4.1 Efficient Tuning with DSEE

  • On RoBERTaLARGE, DSEE reaches comparable performance to full fine-tuning, LoRA, and Adapter across four tasks, with a maximum 1% gap after removing 30% of parameters.The four datasets are CoLA, SST-2, MRPC, and RTE.Source: §4.1 · p. 6, Table 5 · p. 8
  • With three attention heads removed, structured-pruned BERT has comparable or better performance than full fine-tuning on several tasks while reducing inference FLOPs.Removing six heads lowers performance; removing nine yields performance comparable to EarlyBERT with 0.6M versus 66M trainable parameters.Source: §4.1 · p. 6, Table 6 · p. 8

5 Conclusion

DSEE jointly targets parameter-efficient fine-tuning and resource-efficient inference, with competitive transfer performance reported across models and datasets. The authors note hardware and evaluation-scope limitations.

5 Conclusion

  • DSEE jointly optimizes fine-tuning parameter efficiency and the resource efficiency of the fine-tuned model.The conclusion describes this as the first attempt to jointly optimize both objectives.Source: §5 · p. 8
  • Across BERT, GPT, and RoBERTa datasets, DSEE reports parameter and inference efficiency while preserving competitive downstream transfer performance.The authors describe this pattern across several datasets and tasks.Source: §5 · p. 8
  • Unstructured sparsity is less hardware-friendly than structured sparsity, so its implementation may limit speedup.The evaluated models are at the 100–300M parameter level, and datasets focus on GLUE, E2E, WebNLG, and DART.Source: §5 · p. 8

6 Ethical and Broader Impacts

DSEE may reduce the cost of saving new weights when tuning large-scale language models for downstream tasks; the authors report no obvious ethical impacts.

6 Ethical and Broader Impacts

  • Reducing trainable parameters can help save the cost of storing new weights.The authors identify this as potentially helpful for companies fine-tuning large-scale language models on downstream tasks.Source: §6 · p. 8
  • The authors describe potential positive broader impact from this use in companies tuning language models across downstream tasks.They also state that the work has no obvious ethical impacts because it focuses on model tuning.Source: §6 · pp. 8–9
  • The authors state that the work has no obvious ethical impacts because it focuses on model tuning.Source: §6 · p. 9

A More Implementation Details

The appendix reports DSEE hyperparameters, experimental hardware, and evaluation protocols for E2E, WebNLG, and DART.

A.1 Hyper-parameters

  • The reported DSEE hyperparameters include learning rates, batch sizes, and maximum sequence lengths in Table 11.Source: §A.1 · p. 12
  • Experiments used NVIDIA GeForce GTX 1080 Ti, GeForce RTX 2080 Ti, Titan RTX, and A6000 devices.Source: §A.1 · p. 12
  • Evaluation protocols for E2E, WebNLG, and DART follow Hu et al. (2021).Source: §A.1 · p. 12

A.2 Decomposition Method

The appendix identifies GreBsmo as the solver for the Robust PCA-like decomposition and gives iterative update rules for U, V, and S.

A.2 Decomposition Method

  • GreBsmo is used to solve Robust PCA-like methods through iterative optimization of U, V, and S.Source: §A.2 · p. 12
  • The update rules form U from a QR decomposition, update V using the transposed Q, and update S with soft-thresholding.The passage defines X as the original dense matrix, QR(·) as QR decomposition, Sλ(·) as the soft-threshold function, and k as the optimization step.Source: §A.2 · p. 12
  • The iterative rules specify the optimization steps for U, V, and S.Source: §A.2 · p. 12

A.3 Statistics and Usage of Datasets

Table 8 reports dataset statistics, and the experiments retain the datasets’ conventional splits.

A.3 Statistics and Usage of Datasets

  • Table 8 reports statistics for the datasets used in the experiments.Source: §A.3 · p. 12
  • For GLUE, the reported dataset sizes cover train, dev, and test sets.Source: §A.3 · p. 12
  • For non-GLUE tasks, sizes cover train, validation (dev), and test sets, using conventional splits without modification.Source: §A.3 · p. 12

B.1 Ablation Studies

The ablations report same-level performance with LoRA and MAM Adapter for sparse models, alongside findings on adapted weights, Ω sizes, and pruning strategy.

B.1 Ablation Studies

  • Adapting Wq and Wv leads to the highest performance among the tested adapted weights.Source: §B.1 · p. 12, Table 9 · p. 12
  • Ω size 16 performs best on STSB and QNLI and achieves comparable test accuracy on MRPC.Source: §B.1 · p. 12, Table 12 · p. 13
  • Tables 8 and 11 report dataset statistics and dataset/architecture hyperparameters, respectively.Source: Table 8 · p. 12, Table 11 · p. 13
  • Sparse models achieve same-level performance with LoRA and MAM Adapter in comparisons on SST-2, QNLI, and RTE.Source: §B.1 · p. 12, Table 13 · p. 13
  • On RTE, iterative magnitude pruning with repeated 10%-of-remaining-weight pruning and fine-tuning does not improve over one-shot pruning.Source: §B.1 · p. 12, Table 14 · p. 13
  • Table 10 compares GPT-2 decomposition variants across weight-update terms and reports standard deviations from five runs.Source: Table 10 · p. 13

Related papers

Loading 2111.00160v3…