PaperlayerView exact revision on arXiv

Source-linked AI summary

Should You Mask 15% in Masked Language Modeling?

Alexander Wettig, Tianyu Gao, Zexuan Zhong, Danqi Chen

arXiv:2202.08005v3cs.CLcs.LG

TL;DR

The paper asks whether MLMs should universally mask 15% of tokens, given that this convention was adopted across model sizes and masking strategies with limited exploration. It evaluates masking rates and strategies, then separates corruption from prediction effects. Larger models generally benefit from higher rates: 40% outperforms 15% for BERT-large models, while 80% retains more than 95% of fine-tuning performance; optimal rates also vary by masking strategy.

  • Problem

    15% masking has been used ubiquitously despite limited investigation across model sizes, masking strategies, and optimization recipes.

  • Method

    The paper comprehensively studies masking rates across model sizes and masking strategies, disentangles corruption and prediction rates, and analyzes BERT’s 80-10-10 strategy.

  • Results

    Larger models favor higher masking rates: 40% outperforms 15% for BERT-large models, and 80% masking preserves more than 95% of fine-tuning performance.

  • Takeaways & Limitations

    Masking rates should be considered jointly with model size and masking strategy, while corruption and prediction have opposing effects in MLM training.

  • Takeaways & Limitations

    The analysis focuses on MLMs, English experiments, and a relatively small set of downstream tasks, leaving other methods, languages, factors, and domain-specific knowledge for further study.

Abstract

from arXiv · show

Masked language models (MLMs) conventionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations; this masking rate has been widely used, regardless of model sizes or masking strategies. In this work, we revisit this important choice of MLM pre-training. We first establish that 15% is not universally optimal, and larger models should adopt a higher masking rate. Specifically, we find that masking 40% outperforms 15% for BERT-large size models on GLUE and SQuAD. Interestingly, an extremely high masking rate of 80% can still preserve 95% fine-tuning performance and most of the accuracy in linguistic probing, challenging the conventional wisdom about the role of the masking rate. We then examine the interplay between masking rates and masking strategies and find that uniform masking requires a higher masking rate compared to sophisticated masking strategies such as span or PMI masking. Finally, we argue that increasing the masking rate has two distinct effects: it leads to more corruption, which makes the prediction task more difficult; it also enables more predictions, which benefits optimization. Using this framework, we revisit BERT's 80-10-10 corruption strategy. Together, our results contribute to a better understanding of MLM pre-training.

1 Introduction

The paper revisits the conventional 15% MLM masking rate, finding that optimal rates depend on model size and masking strategy. Higher rates can improve large-model training, while even 80% masking preserves most downstream performance.

  • 15% masking has been widely retained from BERT despite limited study across model sizes, masking strategies, and optimization recipes.
  • 40% masking outperforms 15% for BERT-large size models fine-tuned on GLUE and SQuAD.
  • 80% masking preserves more than 95% of fine-tuning performance relative to 15% masking and retains considerable linguistic-probing performance.
  • Uniform masking benefits more from higher masking rates than span or PMI masking, while uniform masking remains competitive at each strategy’s optimal rate.
  • The paper separates masking into corruption and prediction rates, which respectively increase task difficulty and training signals.
  • The results characterize masking rate as an overlooked MLM pre-training factor with opposing effects from corruption and prediction.

2 Background

MLMs restore masked tokens from bidirectional corrupted context, but their standard 15% rate limits how many tokens contribute to training. The paper studies masking rates using an efficient BERT-large recipe and downstream fine-tuning evaluation.

  • 2.1 Masked Language Modeling: Masked language modeling trains a model to restore a corrupted input sequence by independently predicting masked tokens from the remaining context.
  • 2.1 Masked Language Modeling: The masking rate m specifies the percentage of tokens removed from the original sentence and predicted from the corrupted context.
  • 2.1 Masked Language Modeling: Uniform masking samples tokens randomly, span masking selects contiguous text spans, and PMI masking selects words or spans with high pointwise mutual information.
  • 2.1 Masked Language Modeling: MLMs provide bidirectional context but incur substantial computational cost because standard training predicts only 15% of sequence tokens.
  • 2.2 Experiment Setup: The efficient 24hBERT recipe uses BERT-large architecture, larger learning rate and batch size, shorter sequences, and fewer training steps to accelerate pre-training.
  • 2.2 Experiment Setup: The study measures MLM quality with downstream fine-tuning performance because fine-tuning predominates in downstream use, while perplexity correlates poorly with that performance.

3 Larger Models Can Benefit From Higher Masking Rates

Experiments across model sizes show that larger MLMs favor higher optimal masking rates. For BERT-large models, 40% generally outperforms 15% and can reach comparable performance with substantially less training.

  • 40% is optimal on average for large models, compared with 20% for base models and 15% for medium models.
  • Larger models favor higher optimal masking rates under the efficient pre-training recipe.
  • 40% masking consistently outperforms 15% during training on most tasks, and QNLI and QQP reach 15% baseline performance with almost half the training time.
  • SST-2 performs better with 15% masking at the end of training, showing that the optimal rate can depend on the task.
  • With longer or RoBERTa-style training recipes, 40% masking still performs well and achieves similar performance to 15% masking.

4 MLMs in High-Masking Regimes

Very high masking rates still produce useful representations in large MLMs, despite making input reconstruction extremely difficult. At 80% masking, downstream and linguistic-probing performance remain substantial, though some linguistic categories deteriorate.

  • 80% masking produces validation perplexity above 1,000, indicating that independent token predictions cannot effectively reconstruct corrupted inputs.
  • 80% masking retains 95% of fine-tuning performance relative to the 15% baseline across a range of downstream tasks.
  • 80% masking preserves 90% of the 15% baseline’s average BLiMP probing accuracy.Most linguistic phenomena are acquired evenly from 15% to 60% masking, but filler-gap dependencies and island effects deteriorate at higher rates.
  • The findings suggest that useful linguistic knowledge can be learned from a patchy training signal.

5 Masking Rates vs. Masking Strategies

Masking strategy changes which masking rate is optimal. Uniform masking benefits more from higher rates, while span and PMI masking reach their optima at lower rates; at tuned rates, uniform masking remains competitive.

  • Higher masking rates create longer spans under uniform sampling, whereas T5-style span masking keeps average span length at 3 when enforceable.
  • All three masking strategies have optimal rates above 15%, while span and PMI masking prefer lower optimal rates than uniform masking.
  • At their respective optimal rates, uniform masking achieves results similar to or better than span and PMI masking.
  • With 15% masking, increasing the rate can improve SQuAD performance more effectively than switching from uniform to a more advanced strategy.
  • Raising uniform masking from 15% to 40% increases the chance of completely masking a PMI n-gram eightfold.The 40% uniform setting reaches a value comparable to PMI masking at 15%.

6 Understanding Masking As Corruption and Prediction

The masking rate jointly controls corruption and prediction, which have opposing effects during MLM pre-training. Ablations show that more predictions help, more corruption hurts, and the balance determines whether higher masking improves performance.

  • The masking rate m determines both corruption rate m_corr and prediction rate m_pred in MLM pre-training.m_corr measures erased input content, while m_pred measures tokens contributing to the cross-entropy loss.
  • Higher corruption makes prediction harder by leaving fewer context tokens, whereas higher prediction supplies more training signals and benefits optimization.
  • With m_corr fixed at 40%, reducing m_pred from 40% to 20% consistently lowers downstream performance.
  • With m_pred fixed at 40%, reducing m_corr consistently improves performance, with smaller gains when reducing corruption from 10% to 5%.
  • Comparing equal 20% and 40% corruption-prediction settings shows that the benefit of more predictions can outweigh the drawback of more corruption.

7 Revisiting BERT’s Corruption Strategy

The paper revisits BERT’s 80-10-10 corruption rule by separating same-token predictions from random-token corruptions. Both generally hurt downstream performance relative to using alone, except that same-token predictions help on SST-2.

  • Same-token predictions contribute little learning signal and should count neither toward corruption nor toward prediction.Their loss is described as a small auxiliary regularization effect.
  • Random-token corruptions count toward both corruption and prediction because they alter the input and create a non-trivial prediction task.Their loss is slightly higher than for tokens.
  • Same-token predictions and random-token corruptions deteriorate performance on most downstream tasks.
  • The 80-10-10 rule performs worse than using only replacements, except on SST-2, where same-token predictions are beneficial.
  • In the fine-tuning paradigm, models can adapt to full uncorrupted sentences regardless of alternative corruption strategies.

8 Related Work

Related work studies masking rates and strategies across language models and modalities, while also showing that linguistically implausible objectives can retain useful downstream performance.

  • Dynamic masking-rate sampling from 0% to 100% can improve downstream performance and generative ability.
  • Span and PMI masking represent extensively explored alternatives to uniform masking in pre-training and intermediate pre-training.
  • Linguistically implausible objectives, including shuffled word order, random sequences, or predicting only first characters, can still achieve competitive or non-trivial downstream performance.
  • Sequence-to-sequence models also corrupt text with masking rates but use autoregressive decoders and different fine-tuning procedures.
  • ELECTRA predicts on 100% of tokens while its average corruption rate is roughly 7% near the end of training, leaving its connection to the paper’s framework for future work.
  • Images and videos use much higher masking rates than language models, such as 75% for images and 90% for videos, motivated by information redundancy.
  • Speech masked modeling has used a masking rate of around 50%.

9 Conclusion & Discussion

The paper argues that masking-rate choices should account for model size, masking strategy, and the separate effects of corruption and prediction. It also identifies higher masking rates and architectural separation as routes toward more efficient MLM pre-training.

  • 15% is not universally optimal: larger models should use higher masking rates, and masking strategies should be considered jointly with those rates.
  • Implications on higher masking rates: 40% masking can achieve comparable results to a 15% baseline on several tasks with half the training time, supporting better sample efficiency for larger models.
  • Separating masked and unmasked tokens: Separating masked and unmasked tokens could reduce training cost by using a shorter encoder input when masking rates are high.
  • Disentangling corruption and prediction: Lower corruption rates and higher prediction rates yield better model performance, but standard MLMs tie both factors to one masking rate.
  • Disentangling corruption and prediction: Encoding a sequence once and efficiently predicting many small mask sets could substantially accelerate MLM pre-training.

Limitations

The study’s conclusions are scoped by its focus on MLM, English, a limited task set, and computational constraints that prevented training multiple models across multiple seeds.

  • The analysis covers masked language modeling but leaves masking rates in seq2seq models and ELECTRA for future work.
  • Optimal masking rates may also depend on vocabulary size, corpus, language family, and linguistic structure beyond the studied factors.
  • The experiments use English and a relatively small set of downstream tasks that exclude domain-specific knowledge and advanced reasoning skills.
  • Expensive pre-training prevented training multiple pre-trained models over multiple random seeds.

Ethical Considerations

The ethical discussion highlights that masking-rate changes do not address societal biases in language-model training, while the study uses an efficient but computationally expensive pre-training setup and standard downstream evaluations.

  • Ethical Considerations: Stereotypes encoded in training data are not detected by standard GLUE or SQuAD evaluation.
  • Ethical Considerations: Simple masking-rate modifications are not expected to solve societal-bias problems in large language models.
  • Ethical Considerations: Language-model pre-training has significant environmental costs and makes reproduction and follow-up research difficult in academic settings.
  • Implementation: The implementation uses fairseq and integrates the DeepSpeed Transformer kernel to speed up pre-training.
  • Implementation: The models use a RoBERTa tokenizer, English Wikipedia and BookCorpus, and omit BERT’s 80-10-10 rule.
  • Evaluation: Evaluation covers GLUE and SQuAD, reporting task-specific metrics including accuracy, F1, Matthew’s correlation, and Spearman’s correlation.
  • Results: Table 7 compares 15%, 40%, and 80% masking for large models using the efficient pre-training recipe on GLUE and SQuAD development results.
  • Results: Table 9 compares BERT WordPieces with RoBERTa BPE tokenizers under large-model, 40%-masking conditions.

D Longer Training

Longer-training experiments largely preserve the advantage of 40% over 15% masking, while a shorter-sequence, larger-learning-rate recipe matches RoBERTa’s results in less time. Additional experiments extend these findings to smaller models, masking strategies, French XNLI, and information flow under 80-10-10 corruption.

  • Longer Training: The longer-training comparison modifies the efficient pre-training recipe and also evaluates a recipe from RoBERTa.The major differences include a much larger learning rate and sequence length 128.
  • Longer Training: 40% masking remains better than 15% on most tasks when models are trained longer.The advantage is larger when training steps are limited.
  • Longer Training: 40% masking achieves comparable results to 15% on most tasks under the RoBERTa recipe.The train-longer recipe uses shorter sequences and larger learning rates yet achieves comparable results to RoBERTa with much shorter time.
  • Additional Experiments: Figure 10 compares uniform, T5-style span, and PMI masking on downstream tasks, while Figures 8 and 9 report base- and medium-model results.These experiments provide complementary results for earlier comparisons.
  • Additional Experiments: 40% masking is better than 15% on French XNLI after pre-training on 2020 French Wikipedia.Accuracy is averaged over 4 seeds.
  • 80-10-10 Analysis: Mutual-information analysis shows initial source-token information loss followed by contextual information gain, with same-token predictions producing a later reconstruction stage.The analysis examines the 80-10-10 corruption strategies.
Loading 2202.08005v3…