PaperlayerView exact revision on arXiv

Source-linked AI summary

Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization

Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen

arXiv:2105.12002v2cs.LGcs.CL

TL;DR

Prior work on winning tickets in pre-trained language models largely focused on highly compressed subnetworks, leaving lightly compressed cases underexplored. The paper studies how compression affects their generalization and proposes adaptively sharing threshold-selected super tickets across tasks. It reports improved GLUE task-average scores for single-task fine-tuning and benefits from ticket sharing in multi-task learning.

  • Problem

    Prior work on winning tickets in pre-trained language models mainly searched for highly compressed subnetworks, while their behavior under light compression remained largely overlooked.

    Source: §1 · p. 1

  • Method

    The paper studies winning-ticket generalization across compression levels, identifies threshold-selected super tickets, and proposes adaptively sharing them across tasks.

    Source: §1 · pp. 1–2

  • Results

    On GLUE, super tickets improve single-task fine-tuning by 0.9 points over BERT-base and 1.0 points over BERT-large in task-average score; the transition is more pronounced for larger models and less training data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings indicate that pruning can improve generalization under light compression, while adaptive super-ticket sharing benefits multi-task fine-tuning.

    Source: §1 · p. 2, §10 · p. 9

    The approach uses one-shot pruning and an output-sensitivity importance measure; other measures and pruning schedules are left for future work.

    Source: §9 · p. 9

Abstract

from arXiv · show

The Lottery Ticket Hypothesis suggests that an over-parametrized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can match the performance of the full model. In this paper, we study such a collection of tickets, which is referred to as ``winning tickets'', in extremely over-parametrized models, e.g., pre-trained language models. We observe that at certain compression ratios, the generalization performance of the winning tickets can not only match but also exceed that of the full model. In particular, we observe a phase transition phenomenon: As the compression ratio increases, generalization performance of the winning tickets first improves then deteriorates after a certain threshold. We refer to the tickets on the threshold as ``super tickets''. We further show that the phase transition is task and model dependent -- as the model size becomes larger and the training data set becomes smaller, the transition becomes more pronounced. Our experiments on the GLUE benchmark show that the super tickets improve single task fine-tuning by $0.9$ points on BERT-base and $1.0$ points on BERT-large, in terms of task-average score. We also demonstrate that adaptively sharing the super tickets across tasks benefits multi-task learning.

1 Introduction

Prior work on winning tickets in pre-trained language models focused on highly compressed subnetworks, leaving lightly compressed tickets largely overlooked. This paper identifies a phase transition in their generalization and uses super tickets to improve single-task and multi-task fine-tuning.

1 Introduction

  • Winning tickets’ test accuracy improves with increasing compression until a threshold, then deteriorates while remaining above random tickets; at extreme compression, training collapses.The authors call tickets selected at the threshold “super tickets.”Source: §1 · pp. 1–2
  • The authors interpret the transition as a bias–variance trade-off: pruning first reduces variance without increasing bias, but further pruning raises bias and eventually breaks training down.Their account distinguishes non-expressive, lightly expressive, and highly expressive tickets.Source: §1 · p. 2
  • On GLUE, super tickets improve single-task fine-tuning by 0.9 points over BERT-base and 1.0 points over BERT-large in task-average score.The phase transition is more pronounced for larger models and tasks with less training data.Source: §1 · p. 2
  • For multi-task learning, the authors select super tickets during single-task fine-tuning and adaptively share them across tasks.Tickets sharing improves task-average score by 0.9 points over MT-DNNBASE and 1.0 points over MT-DNNLARGE; the resulting model also shows lower sensitivity to downstream fine-tuning randomness.Source: §1 · p. 2
  • The paper identifies a phase transition in pruning large neural language models and shows that pruning can improve generalization when models are lightly compressed.It presents this lightly compressed regime as overlooked by previous work.Source: §1 · p. 2

2 Background

This background outlines Transformer components and prior lottery-ticket work, including structured-ticket evidence and its remaining compression-range question.

2.1 Transformer Architecture

  • Each Transformer layer places multi-head attention before a two-layer feed-forward network, with a residual connection around each module.Attention heads operate in parallel, and their outputs are aggregated.Source: §2.1 · p. 3

2.2 Structured and Unstructured LTHs

3 Finding Super Tickets

The method identifies structured winning tickets in BERT by scoring attention-head and FFN masks, pruning low-scoring structures, and evaluating rewound subnetworks.

3 Finding Super Tickets

  • Winning tickets are formed by structured pruning of BERT attention heads and feed-forward layers, using binary masks where 0 prunes the corresponding structure.Source: §3 · p. 3
  • Importance scores measure expected sensitivity of model outputs to mask variables; head scores receive layer-wise ℓ2 normalization, and scores are averaged over training data in practice.Source: §3 · p. 3
  • Low-importance structures have small output contributions and low expressive power, whereas high-importance structures imply higher expressive power.Source: §3 · p. 4
  • One backward pass at fine-tuning's end supplies scores for one-shot pruning of equal percentages of the lowest-scoring heads and feed-forward layers across compression ratios.Source: §3 · p. 4
  • Winning tickets are reset to pretrained weights and fine-tuned with the original schedule; the best rewinding validation performance selects super tickets.Source: §3 · p. 4

4 Multi-task Learning with Tickets Sharing

The paper proposes sharing task-specific super tickets across tasks and reports a phase transition in which winning tickets first improve and then lose accuracy as compression increases. The transition is more pronounced for larger models and smaller datasets.

4 Multi-task Learning with Tickets Sharing

  • Tickets sharing jointly updates structures selected as super tickets by multiple tasks, updates singly selected structures for their task, and prunes the rest.Task-specific masks determine which parameters are updated; the approach uses structured tickets from pretrained weights and does not require Multi-task Warmup, unlike Sparse Sharing.Source: §4 · p. 4

5.1 Data

  • GLUE supplies the nine NLU tasks used to evaluate generalization in this section.The benchmark includes tasks such as question answering and sentiment analysis.Source: §5.1 · p. 4

5.2 Models & Training

  • SuperT is selected from eight head-and-FFN sparsity levels using rewinding validation performance, with 10% of the GLUE development set used for selection.The models append a task-specific fully connected layer to BERT.Source: §5.2 · p. 5

5.3 Generalization of the Super Tickets

  • SuperT improves task-average GLUE development performance by around 0.9 over ST-DNNBASE and 1.0 over ST-DNNLARGE.The authors report that the improvement is consistent across tasks.Source: §5.3 · p. 5
  • Super-ticket gains are larger on smaller tasks: RTE gains 3.3 points with 83% of weights remaining, versus QQP gains of 0.4/0.3 with 93% remaining for SuperTBASE.The authors interpret these observations as consistent with greater variance in small tasks and variance reduction from removing non-expressive tickets.Source: §5.3 · p. 5
  • Moving from SuperTBASE to SuperTLARGE shrinks the percentage of weights retained across tasks while generalization gains persist.The authors infer that larger models allow more non-expressive tickets to be pruned without performance degradation.Source: §5.3 · p. 5

5.4 Phase Transition

  • Winning-ticket accuracy rises, then falls past a compression threshold and eventually intersects random-ticket accuracy; the transition is more pronounced in larger models and smaller tasks.The paper reports the pattern across GLUE tasks and notes a stronger transition for BERT-large than BERT-base and for smaller tasks or reduced training subsets.Source: Figure 4 · p. 6, Figure 5 · p. 6, §5.4 · p. 7

6 Multi-task Learning Experiments

The multi-task experiments apply tickets sharing to MT-DNN and compare it with MT-DNN and its single-task fine-tuned variants. Ticket-Share improves task-average scores, including after downstream fine-tuning.

6.1 Model & Training

  • The experiments use MT-DNN shared layers initialized from pretrained BERT-base/large and task-specific layers, with Ticket-Share initialized from the union of task super tickets.The reported models include MT-DNN and Ticket-Share variants with and without single-task fine-tuning.Source: §6.1 · p. 7

6.2 Experimental Results

  • Ticket-ShareBASE/LARGE improves task-average score by 0.9/1.0 over MT-DNNBASE/LARGE, respectively.On small tasks it is better than or on par with MT-DNN plus fine-tuning; on large tasks it performs equally well with the full model.Source: §6.2 · p. 7
  • Ticket-ShareBASE/LARGE plus fine-tuning gains 1.0/0.7 task-average points over the corresponding MT-DNN plus fine-tuning models.The authors state that reducing variance in the multi-task model benefits downstream fine-tuning.Source: §6.2 · p. 7

7 Domain Adaptation

The paper evaluates whether super tickets transfer to out-of-domain NLI datasets under few-shot adaptation. SuperT and Ticket-Share adapt better than their corresponding baselines, especially with little training data.

7.1 Data & Training

  • SciTail is derived from the science question-answering dataset SciQ, while SNLI contains 570k sentence pairs with premises drawn from Flickr captions.The hypotheses in SciTail are created from science questions.Source: §7.1 · p. 8

7.2 Experimental Results

  • The domain-adaptation experiments transfer MNLI-trained SuperT and ST-DNN models, and Ticket-Share and MT-DNN shared embeddings, to SNLI and SciTail.The study evaluates both single-task and multi-task super-ticket variants.Source: §7.2 · p. 8
  • SuperT and Ticket-Share adapt better to SNLI and SciTail than ST-DNN and MT-DNN, especially in few-shot settings.The evaluation uses 0.1%, 1%, 10%, and 100% training subsets and scores on the datasets’ development sets.Source: §7.2 · p. 8

8 Analysis

The analysis reports lower random-seed sensitivity for super-ticket models and examines how ticket importance and task shares vary across GLUE tasks.

8 Analysis

  • SuperTLARGE and Ticket-ShareLARGE show smaller validation-result standard deviations across random seeds in the evaluated fine-tuning settings.The analysis focuses on fitting small tasks with highly over-parameterized models.Source: §8 · p. 9, Table 5 · p. 9
  • Many tickets have nearly equal task shares across more than five of eight tasks, while some are dominated by one task or jointly by CoLA and STS-B.The authors connect the broadly shared tickets to knowledge sharing and note that they are non-expressive for SST-2; this helps explain why SST-2 benefits little from ticket sharing.Source: §8 · p. 9

9 Discussion

The discussion reports positive results for structured lottery tickets and describes sensitivity-based selection with one-shot pruning. The conclusion summarizes the phase transition and ticket sharing, while the broader-impact statement claims improved energy efficiency through compression.

9 Discussion

  • Structured lottery tickets in weight matrices such as FFN layers and attention heads can match full-model performance, with practical relevance because structured matrices are better optimized for hardware acceleration.The discussion contrasts this positive result with earlier work reporting no structured winning tickets in highly compressed models.Source: §9 · p. 9
  • Super tickets are selected by output sensitivity to structure-mask variables, and the paper uses one-shot pruning for computational simplicity.Other importance measures and pruning schedules are left for future work as possible ways to find better-generalizing super tickets.Source: §9 · p. 9

10 Conclusion

  • The conclusion reports a phase transition in structured lottery-ticket generalization and proposes sharing tickets to improve multi-task fine-tuning.The authors frame their analysis as informing understanding of the connection between model compression and generalization.Source: §10 · p. 9

Broader Impact

  • The broader-impact statement says the method improves performance through model compression, rendering it energy efficient, and foresees no direct social consequences or ethical issues.It also states that the study neither introduces social or ethical bias to the model nor amplifies bias in the data.Source: Broader Impact · p. 9

A Appendix

The appendix describes GLUE’s task coverage, single-task training settings, and how evaluation results and variability are reported. GLUE comprises nine NLU tasks spanning several task types.

A.1.1 Data

  • GLUE covers nine NLU tasks, including question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference.The appendix points to a table for task statistics and evaluation metrics.Source: §A.1.1 · p. 13

A.1.2 Training

  • Single-task training uses Adamax, a linear learning-rate decay schedule with 0.1 warm-up, gradient clipping at 1, and dropout of 0.1 except 0.3 for MNLI and 0.05 for CoLA.Texts are wordpiece-tokenized and truncated to spans of at most 512 tokens; experiments use Nvidia V100 GPUs.Source: §A.1.2 · p. 13

A.1.3 Evaluation Results Statistics

  • GLUE development results average super-ticket rewinding scores across five experiment sets, while test-set predictions use one seed selected for the best task-average validation results.The appendix reports development-set standard deviations and selected-ticket weight-remaining statistics over five seeds; the test-set seed count is limited by the evaluation server’s submission cap.Source: §A.1.3 · p. 13

A.2.1 Multi-task Model Training

The appendix specifies optimization settings for multi-task training and downstream fine-tuning, reports a matched-learning-rate observation for Ticket Share, and describes SNLI/SciTail domain-adaptation experiments.

A.2.1 Multi-task Model Training

  • Multi-task model training adopts the MT-DNN code base, Adamax at 5 × 10−5 with batch size 32, at most five epochs with early stopping, and task-specific dropout settings.The appendix also specifies linear decay with 0.1 warm-up, gradient clipping at 1, and a maximum input span of 512 tokens.Source: §A.2.1 · p. 13
  • Ticket Share’s task-specific super tickets are selected using the same 5 × 10−5 learning rate as multi-task training; matched settings usually outperform mismatched settings in rewinding.The authors relate this observation to prior evidence that unstructured winning tickets depend on both weight initialization and the optimization path.Source: §A.2.1 · p. 13

A.2.2 Multi-task Model Downstream Fine-tuning

  • Downstream fine-tuning varies learning rates from 1 × 10−5 to 2 × 10−4 and task-specific dropout from 0.05 to 0.3.The stated candidate learning rates are 1 × 10−5, 2 × 10−5, 5 × 10−5, 1 × 10−4, and 2 × 10−4.Source: §A.2.2 · p. 13

A.3.1 Data

  • SNLI and SciTail are used for NLI domain adaptation from MNLI, with the single-task learning rate selected from 5 × 10−5, 1 × 10−4, and 5 × 10−4.SciTail uses science-question hypotheses and web-sentence premises, with high premise–hypothesis lexical similarity described as a challenge.Source: §A.3.1 · p. 13, §A.3.2 · p. 13

A.4 Sensitivity Analysis

The sensitivity appendix reports a Ticket ShareLARGE robustness finding and describes additional GLUE phase-transition plots. It also specifies how seed-based variability was assessed.

A.4.1 Randomness Analysis

  • Single-task results vary random seeds while holding other hyperparameters fixed, and validation standard deviations are reported over five rewinding trials.For multi-task downstream fine-tuning, standard deviations are reported over five trials, with each result averaged across three learning rates.Source: §A.4.1 · p. 14

A.4.2 Hyper-parameter Analysis

  • Ticket ShareLARGE exhibits stronger robustness to downstream learning-rate changes in the analyzed GLUE tasks.The analysis varies learning rate over 5 × 10−5, 1 × 10−4, and 2 × 10−4, averaging each result over five random seeds; Table 8 reports standard deviations.Source: §A.4.2 · p. 14, Table 8 · p. 14

A.5 Phase Transition on GLUE Tasks

  • Figure 7 presents single-task winning-ticket results across sparsity levels for GLUE tasks absent from Figure 4, using the same experimental settings as Figure 4.The supplied passages identify the figure’s scope but do not state a specific outcome from its plots.Source: §A.5 · p. 14, Figure 7 · p. 14
Loading 2105.12002v2…