Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

1 Introduction

• Finetuned models can require separate copies of millions of parameters for each task, making their memory footprint a deployment obstacle.BERT-large, for example, has 340 million parameters to finetune.Source: §1 · p. 1
• Instead of updating pretrained weights, masking learns task-specific binary selections of important weights and discards irrelevant ones.Only 1-bit masks need to be stored per task, rather than all 32-bit finetuned parameters, reducing memory use when deploying multiple tasks.Source: §1 · p. 1
• Masking BERT, RoBERTa, and DistilBERT achieves performance comparable to finetuning across tagging, classification, and reading-comprehension tasks.The paper also reports extensive analysis of masking across eleven diverse NLP tasks.Source: §1 · pp. 1–2
• The paper analyzes factors affecting masking performance and studies masked-model representations and the loss landscape connecting masked and finetuned models.These analyses address factors critical to performance on eleven diverse NLP tasks.Source: §1 · pp. 1–2

2 Related Work

• Earlier efficiency approaches include projected attention with multi-task learning and adapter modules, which alter BERT’s forward pass and require careful initialization.Adapters are initialized close to identity.Source: §2 · p. 2
• The paper learns selective binary masks end to end for pretrained language models, keeping pretrained parameters untouched while solving downstream tasks with minimal overhead.This differs from Mallya et al. (2018), who update weights in a task-specific classifier layer; this work learns masks for both the pretrained model and randomly initialized classifier.Source: §2 · p. 2
• The method uses the straight-through estimator to train selective masks for pretrained language-model parameters.The estimator had been used to train efficient binarized neural networks.Source: §2 · p. 2
• Masking draws on pruning and fixed-model architecture search, and is interpreted as implicit neural architecture search for pretrained language models.Gaier and Ha (2019) searched architectures without explicit weight training, inspiring this approach.Source: §2 · p. 2
• Unlike prior binary-mask work on CNNs, this paper studies pretrained language models in NLP and analyzes factors associated with good performance.Radiya-Dixit and Wang (2020) analyze BERT finetuning, whereas this work aims to provide an efficient alternative.Source: §2 · p. 2

3.1 Background on Transformer and finetuning

• Standard finetuning adds a randomly initialized task classifier and updates all parameters, while the method instead keeps pretrained weights fixed and selects them with binary masks.The Transformer blocks contain attention and feed-forward linear layers; the pooler and task classifier are stacked for sequence classification.Source: §3.1 · pp. 2–3, §3.2 · p. 3

3.2 Learning the mask

• Binary masks threshold learned real-valued matrices and multiply pretrained linear-layer weights elementwise to select task-relevant parameters.The threshold is a global hyperparameter, and the masks are trained end-to-end for the task.Source: §3.2 · p. 3
• Because hard thresholding gives zero gradients almost everywhere, mask updates use a noisy estimator of the loss gradient with respect to each mask.The task objective trains the masks, and the resulting binary masks are saved for inference.Source: §3.2 · p. 3
• Binary masks take about 3% of the memory of 32-bit finetuned parameters, with further savings possible by leaving some layers unmasked.The embedding layer is always selected rather than masked.Source: §3.2 · p. 3, §3.3 · p. 4

3.3 Configuration of masking

• Initial mask sparsity affects optimization behavior and final performance, while deciding which layers to mask is difficult because their encoded linguistic information differs.The paper investigates initial sparsity and layer selection as configuration choices.Source: §3.3 · pp. 3–4
• The embedding layer is left unmasked because it contains context-independent word meanings and comprises almost 21% of BERT-base-uncased parameters; biases and layer-normalization parameters are also left unmasked.The authors report no positive performance effect from learning masks for biases and layer-normalization parameters.Source: §3.3 · p. 4

• Experiments evaluate POS tagging on Penn Treebank and NER on CoNLL-2003.The study also includes sequence classification and reading comprehension tasks.Source: §4 · p. 4
• Sequence-classification benchmarks include five GLUE tasks and the public-test-set datasets TREC, AG News, and SemEval-2016 4B.Source: §4 · p. 4
• Reading comprehension is evaluated on SWAG; the reported metrics are micro-F1 for NER, Matthew’s correlation coefficient for CoLA, and accuracy for other tasks.Source: §4 · p. 4
• Due to resource limitations, experiments use BERT-base-uncased, RoBERTa-base, and DistilBERT-base-uncased.BERT and RoBERTa have 12 transformer blocks; DistilBERT has 6.Source: §4 · p. 4
• Inputs are limited to 128 wordpieces, and the Adam learning rate is tuned for each setup using an expanded search grid when needed.For POS and NER, the length cap means some word-tag annotations are excluded.Source: §4 · p. 4

5.1 Initial sparsity of binary masks

• 3%–10% initial sparsity generally yields reasonable results; 95% hurts performance, and 1% also hurts by retaining too many pretrained parameters.Large datasets such as SST2 are less sensitive to initial sparsity than small datasets such as RTE.Source: Figure 1 · p. 5, §5.1 · p. 5

5.2 Layer-wise behaviors

• Top-down masking usually outperforms bottom-up at fixed sparsity and layer count; adding masked layers helps bottom-up, while top-down CoLA improves and MRPC/RTE remain insensitive.Bottom-up masks blocks from the bottom of BERT; top-down masks from the top, with WP and WT always masked.Source: §5.2 · p. 5, Figure 2 · p. 6

5.3 Comparing finetuning and masking

• Masked BERT, RoBERTa, and DistilBERT achieve performance comparable to finetuning overall, with a reported performance drop on RoBERTa-RTE.The comparison covers dev-set results across eleven NLP tasks.Source: Table 1 · p. 6, §5.3 · p. 6
• NER results are slightly worse than Devlin et al.’s, possibly because this study uses sentence-level context capped at 128 tokens rather than maximal document context.The authors describe this as a possible explanation for the difference.Source: §5.3 · p. 6
• 474MB for masking versus 1752MB for finetuning, with comparable performance in an ensemble setting.The memory comparison uses BERT-base-uncased as its example.Source: Figure 3 · p. 6, §5.3 · pp. 6–7
• Masking also yields test-set results comparable to finetuning on SEM, TREC, AG, POS, and NER.Table 2 reports test error rates and model-size comparisons for single models and four-model ensembles.Source: §5.3 · p. 6, Table 2 · p. 7

6.1 Intrinsic evaluations

• On SST2 dev examples, masked and finetuned top-layer [CLS] representations are almost linearly separable, unlike pretrained representations.The t-SNE visualization compares pretrained, finetuned, and masked BERT/RoBERTa representations.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 Properties of the binary masked models

• ≈20% improvement over the majority-vote baseline is reported when SEM-trained finetuned and masked models are evaluated on SST2.Table 3 reports generalization on dev, with training datasets in rows and evaluation datasets in columns.Source: Table 3 · p. 7, §6.2 · p. 8
• 84.79% accuracy for masking and 85.25% for finetuning on IMDB both exceed the 50% baseline after training on SST2.The passage presents these results as successful transfer to another movie-review dataset.Source: §6.2 · p. 8
• SST2-trained models do not generalize to SEM for either masking or finetuning, unlike SEM-trained models evaluated on SST2.The authors hypothesize that the more specific Twitter domain and its emojis or symbols may explain the asymmetry.Source: §6.2 · p. 8
• Mask dissimilarity is larger in higher BERT layers, task masks tend to differ even for similar tasks, and different-seed masks can achieve similar performance.The comparison concerns masks across layers and downstream tasks; similar-performing subnetworks are also noted in prior BERT lottery-ticket evaluations.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 Loss landscape

• Interpolating between finetuned and masked minima on MRPC and SST2 yields a high-accuracy path, indicating that the minima lie on a connected low-loss manifold.The experiments use the best-performing BERT and RoBERTa models with matching seeds and training epochs.Source: §6.3 · p. 9, Figure 6 · p. 9

7 Conclusion

• Masking trains one set of binary masks per task to select critical pretrained parameters rather than updating those parameters.The conclusion names BERT, RoBERTa, and DistilBERT as example pretrained models.Source: §7 · p. 9
• Across a series of NLP tasks, masking yields performance comparable to finetuning and is more memory-efficient when several tasks must be solved.The pretrained parameters remain unchanged, and only task-specific masks are trained.Source: §7 · p. 9
• Intrinsic evaluations indicate that binary masked models extract valid and generalizable representations for downstream tasks.Source: §7 · p. 9
• The minima obtained by masking and finetuning can be connected by a line segment.The conclusion presents this as evidence supporting masking's effectiveness for pretrained language models.Source: §7 · p. 9

A.1 Computing infrastructure

• Experiments used Tesla V100, GeForce GTX 1080 Ti, and GeForce GTX 1080 GPUs with per-GPU batch size 32.QNLI and AG comparisons used four GPUs; other tasks used one.Source: §A.1 · p. 14

A.2 Number of parameters

• The checklist directs readers to Table 8 for numerical comparisons of parameter counts and memory consumption between finetuning and masking.Source: §A.2 · p. 14

A.3 Validation performance

• Validation reporting uses Matthew’s correlation coefficient for CoLA, micro-F1 for NER, and accuracy for other tasks.The text says Table 2 dev performance is covered in Table 1.Source: §A.3 · p. 14

A.4 Hyperparameter search

• Learning rate was the only hyperparameter searched for both masking and finetuning.Optimal values are listed in Table 4.Source: §A.4 · p. 14

A.5 Datasets

• GLUE tasks use official benchmark datasets; TREC and AG use datasets developed by Zhang et al. (2015), and SEM uses its official SemEval dataset.Source: §A.5 · p. 14
• NER, POS, and SWAG use their stated official sources, while POS uses WSJ sections 0–18 for training, 19–21 for development, and 22–24 for testing.Other datasets use their official train, development, and test splits.Source: §A.5 · p. 14
• WordPiece preprocessing with maximum sequence length 128 excludes annotated words from some long POS and NER examples.The appendix reports 5 development and 87 test annotations excluded for POS, and 27 development and 8 test annotations for NER.Source: §A.5 · p. 14

B More on Mode Connectivity

• The appendix trains a 3-bend Bézier curve between finetuned and masked endpoints by minimizing expected loss over t sampled uniformly from [0, 1].Monte Carlo estimates the gradient of the expectation-based objective.Source: B More on Mode Connectivity · p. 14

C More Empirical Results

• Table 6 covers single-model and ensemble test-set results for RoBERTa and DistilBERT on SEM, TREC, AG, POS, and NER.Single results average four random-seed models; the ensemble combines those four models.Source: C More Empirical Results · p. 14
• Table 4 reports optimal learning rates by task for BERT, RoBERTa, and DistilBERT, using 10-epoch runs and early stopping after 2 epochs.Source: Table 4 · p. 15
• Figure 7 tracks MRPC development accuracy along the curve connecting the finetuning minimum at γ = 0 and masking minimum at γ = 1.The caption does not report the accuracy values along the curve.Source: Figure 7 · p. 15
• Table 5 gives dev and test example counts by task, counting words rather than examples for POS and NER.Source: Table 5 · p. 15

D.1 Layer-wise behaviors

• Table 7 provides numerical values for the layer-wise behavior experiment, which uses 10 epochs, mini-batches of 32, and learning rates tuned on mean results across four random seeds.Source: §D.1 · p. 15, Table 7 · p. 16

D.2 Memory consumption

• Table 8 compares masking and finetuning model sizes using BERT-base-uncased and gives numerical values corresponding to Figure 3.Source: §D.2 · p. 15, Table 8 · p. 16
• The masking scheme permits parameter sharing across tasks when they have the same output dimension, allowing them to use the same classifier layer.Source: Table 8 · p. 16