Source-linked AI summary
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov
TL;DR
Comparing language-model pretraining methods is difficult because training is expensive and datasets vary in size. This study replicates and systematically evaluates BERT pretraining choices, finding that RoBERTa matches or exceeds competing methods across major benchmarks.
Problem
Comparing pretraining methods is difficult because training is expensive and datasets vary in size, limiting measurement of which choices drive performance.
Method
The study replicates BERT pretraining while evaluating hyperparameters, training-data size, and an improved recipe using longer training, more data, and revised objectives.
Results
RoBERTa achieves state-of-the-art results on GLUE, RACE, and SQuAD, including a GLUE score of 88.5.
Takeaways & Limitations
The results show that overlooked pretraining design choices can make BERT’s masked-language-model objective competitive with recently proposed alternatives.
Takeaways & Limitations
The study does not determine whether competing methods could also improve with additional tuning, leaving that exploration for future work.
Abstract
from arXiv · showhide
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.
1 Introduction
The paper replicates BERT pretraining to evaluate hyperparameter tuning and training-set size, finding BERT significantly undertrained. RoBERTa’s revised training procedure matches or exceeds post-BERT methods and achieves strong GLUE, SQuAD, and RACE results.
- Motivation: BERT pretraining is difficult to compare because training is computationally expensive and uses private datasets of varying sizes.These constraints limit hyperparameter tuning and the ability to measure the effects of training-data size.
- Approach: The replication study finds that BERT was significantly undertrained and proposes RoBERTa as an improved training recipe.The study carefully evaluates hyperparameter tuning and training-set size, and reports that RoBERTa can match or exceed all post-BERT methods.
- Results: 88.5 on the public GLUE leaderboard matches the 88.4 reported by Yang et al. (2019).The model was trained longer over additional data.
- Results: RoBERTa establishes a new state-of-the-art on 4/9 GLUE tasks: MNLI, QNLI, RTE and STS-B.When controlling for training data, the improved procedure also improves upon published BERT results on GLUE and SQuAD.
- Contributions: Using more pretraining data further improves downstream-task performance, and masked language-model pretraining remains competitive under the right design choices.The paper introduces CC-NEWS and reports matching state-of-the-art results on SQuAD and RACE.
2 Background
BERT preprocesses paired token segments into a single sequence, then pretrains on unlabeled text before fine-tuning on labeled downstream tasks. Its pretraining combines masked language modeling with next sentence prediction, using specified masking and segment-pair construction procedures.
- Input representation: BERT concatenates two token segments, separated by [SEP] and bounded by [CLS] and [EOS], with M + N < T.T controls the maximum sequence length during training.
- Training pipeline: BERT is pretrained on a large unlabeled text corpus and subsequently fine-tuned with labeled end-task data.
- Pretraining objectives: BERT uses masked language modeling and next sentence prediction as its two pretraining objectives.MLM predicts randomly selected masked tokens, while NSP classifies whether two segments are consecutive in the original text.
- Masked language modeling: 15% of input tokens are selected for possible replacement: 80% become [MASK], 10% remain unchanged, and 10% become random vocabulary tokens.The MLM objective is a cross-entropy loss on predicting the selected tokens.
- Next sentence prediction: NSP samples positive consecutive-sentence pairs and negative cross-document pairs with equal probability.NSP is a binary classification loss intended to support downstream tasks requiring reasoning about sentence relationships.
3 Experimental Setup
The study reimplements BERT in FAIRSEQ and evaluates pretraining across five English corpora totaling over 160GB, using GLUE, SQuAD, and RACE downstream benchmarks. Its setup largely follows BERT while tuning selected optimization parameters and using full-length sequences up to 512 tokens.
- Implementation: The replication study reimplements BERT in FAIRSEQ and primarily follows the original BERT optimization hyperparameters.The peak learning rate and warmup steps are tuned separately for each setting.
- Implementation: Training uses full-length sequences of at most T = 512 tokens, without randomly injected short sequences or an initial reduced-length phase.The setup also uses mixed-precision arithmetic on DGX-1 machines with 8 × 32GB Nvidia V100 GPUs.
- Pretraining Data: Over 160GB of uncompressed text spans five English-language corpora with varying sizes and domains.The corpora include the original BOOKCORPUS plus English Wikipedia, CC-NEWS, OPENWEBTEXT, and STORIES.
- Evaluation: Models are evaluated on GLUE, SQuAD, and RACE downstream benchmarks.GLUE contains 9 datasets; SQuAD includes V1.1 and V2.0, while RACE contains more than 28,000 passages and nearly 100,000 questions.
- Evaluation: For the replication study, results use development sets after single-task fine-tuning without multi-task training or ensembling.The fine-tuning procedure follows the original BERT paper; later test-set leaderboard results use task-specific modifications.
4 Training Procedure Analysis
The analysis isolates the effects of masking, input formatting and NSP, batch size, and tokenization while holding the BERTBASE architecture fixed. Dynamic masking, document-level inputs without NSP, and large batches improve or preserve performance, while encoding differences are small.
- Scope: The study keeps the BERTBASE architecture fixed, leaving architectural changes, including larger architectures, for future work.The baseline configuration is BERTBASE with L = 12, H = 768, A = 12, and 110M parameters.
- Masking: Dynamic masking is comparable or slightly better than static masking, while avoiding repeated use of identical masks and becoming crucial for longer training or larger datasets.Static masking duplicated each sequence 10 times, so each sequence saw the same mask four times across 40 epochs; dynamic masking generates a pattern at every presentation.
- Input format and NSP: Individual sentence pairs hurt downstream performance relative to segment pairs, likely because shorter inputs limit learning of long-range dependencies.Both formats retain the NSP loss, but SENTENCE-PAIR+NSP uses single natural sentences while SEGMENT-PAIR+NSP permits multiple sentences per input.
- Input format and NSP: Removing NSP matches or slightly improves downstream performance, and DOC-SENTENCES outperforms the originally published BERTBASE results.DOC-SENTENCES restricts packed sequences to a single document, whereas FULL-SENTENCES may cross document boundaries; DOC-SENTENCES performs slightly better but has variable batch sizes.
- Batch size: Large batches improve masked-language-modeling perplexity and end-task accuracy while also simplifying distributed data-parallel training.The experiments subsequently use batches of 8K sequences, while further exploration of batches up to 32K is left for future work.
- Encoding: BPE encoding choices produce only slight differences, with the Radford et al. (2019) encoding slightly worse on some tasks; the authors nevertheless use it thereafter.The authors prioritize the advantages of a universal encoding scheme over the minor performance degradation.
5 RoBERTa
RoBERTa combines several BERT pretraining improvements with larger and more diverse data and longer training. It achieves state-of-the-art performance across GLUE, SQuAD, and RACE benchmarks.
- Model and training: RoBERTa combines dynamic masking, full-sentence training without NSP, large mini-batches, and a larger byte-level BPE.These modifications are aggregated from the preceding analysis of BERT pretraining choices.
- Model and training: 160GB of text yields further downstream improvements across all tasks, validating the importance of pretraining data size and diversity.RoBERTa is trained for 100K steps over the combined datasets, compared with 16GB in the earlier configuration.
- Model and training: 500K training steps produce significant downstream gains, with the 300K- and 500K-step models outperforming XLNetLARGE across most tasks.The longest-trained model does not appear to overfit and would likely benefit from additional training.
- GLUE: RoBERTa achieves state-of-the-art results on all 9 GLUE development sets while using BERTLARGE’s architecture and masked language modeling objective.It consistently outperforms both BERTLARGE and XLNetLARGE, highlighting the impact of dataset size and training details.
- SQuAD and RACE: RoBERTa achieves state-of-the-art results on both SQuAD settings and both RACE test settings.On SQuAD v2.0, it improves over XLNet by 0.4 points in EM and 0.6 points in F1; on SQuAD v1.1, it matches XLNet’s state-of-the-art.
6 Related Work
Prior pretraining methods use diverse objectives, while recent approaches commonly combine masked language model pretraining with task-specific fine-tuning.
- Training objectives: Pretraining objectives include language modeling, machine translation, and masked language modeling.The passage cites representative work for each objective.
- Fine-tuning: Recent papers commonly fine-tune pretrained models separately for each downstream task.
- Common recipe: A basic recent recipe combines task-specific fine-tuning with a variant of masked language model pretraining.
7 Conclusion
The study identifies several pretraining choices that substantially improve BERT and presents RoBERTa, which achieves state-of-the-art results on GLUE, RACE, and SQuAD. It also uses the CC-NEWS dataset and releases models and code for pretraining and finetuning.
- 7 Conclusion: RoBERTa improves BERT pretraining through longer training, larger batches, more data, removal of next sentence prediction, longer sequences, and dynamic masking.The study reports these design choices as substantially improving performance.
- 7 Conclusion: RoBERTa achieves state-of-the-art results on GLUE, RACE, and SQuAD without multi-task finetuning.
- 7 Conclusion: The study uses the CC-NEWS dataset and releases models and code for pretraining and finetuning.The release is provided through the cited fairseq repository.
Appendix for “RoBERTa: A Robustly Optimized BERT Pretraining Approach” · A Full results on GLUE
Table 8 reports the complete development-set results for RoBERTa on GLUE, covering both LARGE and BASE configurations aligned with BERTLARGE and BERTBASE, respectively.
- A Full results on GLUE: Table 8 presents the full set of development-set results for RoBERTa.These are GLUE development-set results.
- A Full results on GLUE: The reported results concern the GLUE benchmark.The appendix section is titled “A Full results on GLUE.”
- A Full results on GLUE: The appendix reports results for a LARGE configuration.This configuration follows BERTLARGE.
- A Full results on GLUE: The LARGE configuration follows BERTLARGE.Its configuration is explicitly described as following BERTLARGE.
- A Full results on GLUE: The appendix also reports results for a BASE configuration.This configuration follows BERTBASE.
- A Full results on GLUE: The BASE configuration follows BERTBASE.Its configuration is explicitly described as following BERTBASE.
B Pretraining Hyperparameters
The section presents the pretraining hyperparameters used for RoBERTaLARGE and RoBERTaBASE.
- Table 9 describes the pretraining hyperparameters for RoBERTaLARGE and RoBERTaBASE.
C Finetuning Hyperparameters
This section reports RoBERTa finetuning hyperparameters for RACE, SQuAD, and GLUE. The selected values are based on the median across five random seeds for each task.
- C Finetuning Hyperparameters: 5 random seeds determine the selected hyperparameter values for each task via their median.The selection procedure is applied separately to RACE, SQuAD, and GLUE.
- C Finetuning Hyperparameters: Table 10 lists finetuning hyperparameters for RACE, SQuAD, and GLUE.The table is specifically for finetuning RoBERTaLARGE.
- C Finetuning Hyperparameters: The reported finetuning configurations cover three evaluation settings: RACE, SQuAD, and GLUE.Table 10 is titled as the hyperparameter table for finetuning RoBERTaLARGE on these tasks.