BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

1 Introduction

• Feature-based methods incorporate pre-trained representations into task-specific architectures, whereas fine-tuning adapts all pre-trained parameters with minimal task-specific additions.ELMo (Peters et al., 2018a) exemplifies feature-based methods, while OpenAI GPT exemplifies fine-tuning. Both approaches use unidirectional language-model objectives during pre-training.Source: §1 · p. 1
• Unidirectional language models restrict pre-training architectures, which the authors argue is suboptimal for sentence-level tasks and potentially harmful for token-level question answering.Question answering crucially requires context from both directions.Source: §1 · p. 1
• BERT’s masked language-model objective enables deep bidirectional pre-training by predicting randomly masked tokens using both left and right context.An additional next sentence prediction task jointly pre-trains text-pair representations.Source: §1 · pp. 1–2
• BERT contrasts with unidirectional pre-training in Radford et al. (2018) and shallow concatenation of independently trained directional language models in Peters et al. (2018a).Masked language modeling enables BERT to learn deep bidirectional representations instead.Source: §1 · p. 2
• Eleven NLP tasks receive state-of-the-art results from BERT, which outperforms many task-specific architectures across sentence-level and token-level tasks.The authors identify BERT as the first fine-tuning-based representation model to achieve state-of-the-art performance across a large suite of both task types.Source: §1 · p. 2

2.1 Unsupervised Feature-based Approaches

• Pre-trained word embeddings improve NLP systems over embeddings learned from scratch, building on decades of non-neural and neural representation research.Training objectives include left-to-right language modeling and discriminating correct from incorrect words using left and right context (Mikolov et al., 2013).Source: §2.1 · p. 2
• Prior work has generalized representations to coarser granularities, including sentence embeddings (Kiros et al., 2015; Logeswaran and Lee, 2018) and paragraph embeddings (Le and Mikolov, 2014).Sentence-training objectives include ranking candidate next sentences (Jernite et al., 2017; Logeswaran and Lee, 2018), generating next-sentence words left-to-right given a previous-sentence representation (Kiros et al., 2015), and denoising auto-encoder-derived objectives (Hill et al., 2016).Source: §2.1 · p. 2
• ELMo and its predecessor (Peters et al., 2017, 2018a) concatenate left-to-right and right-to-left language-model representations to provide context-sensitive token features.Integrated into task-specific architectures, ELMo advances question answering, sentiment analysis, and named entity recognition benchmarks (Peters et al., 2018a). Melamud et al. (2016) predict words from both contexts using feature-based, non-deeply-bidirectional LSTMs; Fedus et al. (2018) use cloze training to improve text-generation robustness.Source: §2.1 · p. 2

2.2 Unsupervised Fine-tuning Approaches

• Unsupervised fine-tuning pre-trains contextual sentence or document encoders on unlabeled text, leaving few parameters to learn from scratch for downstream tasks.Early approaches pre-trained only word embeddings, while later encoders used left-to-right language modeling and auto-encoder objectives (Howard and Ruder, 2018; Dai and Le, 2015). OpenAI GPT achieved previously state-of-the-art results on many sentence-level GLUE tasks (Wang et al., 2018a).Source: §2.2 · pp. 2–3

2.3 Transfer Learning from Supervised Data

• Supervised transfer also succeeds using large natural language inference datasets (Conneau et al., 2017) and machine translation datasets (McCann et al., 2017).Computer vision provides another effective transfer-learning recipe: fine-tuning models pre-trained with ImageNet.Source: §2.3 · p. 3

3 BERT

• BERT uses a multi-layer bidirectional Transformer encoder based on Vaswani et al. (2017), unlike GPT’s left-context-only self-attention.BERTBASE has 12 layers, hidden size 768, 12 attention heads, and 110M parameters; BERTLARGE has 24 layers, hidden size 1024, 16 heads, and 340M parameters. BERTBASE matches OpenAI GPT’s model size for comparison.Source: §3 · p. 3
• BERT represents single text spans or text pairs in one sequence, summing token, segment, and position embeddings for each token.WordPiece embeddings (Wu et al., 2016) use a 30,000-token vocabulary; [SEP] and learned A/B segment embeddings distinguish paired spans. Every sequence starts with [CLS], whose final hidden state supplies the aggregate representation for classification.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Pre-training BERT

• 15% of WordPiece positions are randomly selected for masked language modeling, predicting only their original tokens to train bidirectional representations.Selected tokens become [MASK] 80% of the time, random tokens 10%, and remain unchanged 10%, mitigating the mismatch because [MASK] is absent during fine-tuning. Pre-training uses BooksCorpus (800M words) and English Wikipedia (2,500M words), retaining Wikipedia text passages rather than lists, tables, or headers and requiring document-level corpora for long contiguous sequences.Source: §3.1 · pp. 4–5
• 50% of next sentence prediction examples use the actual next sentence and 50% a random sentence, teaching sentence relationships beneficial for QA and NLI.The [CLS] vector predicts IsNext versus NotNext, with the final model achieving 97%-98% NSP accuracy. This vector is not a meaningful sentence representation without fine-tuning.Source: §3.1 · p. 4, §3 · p. 4

3.2 Fine-tuning BERT

• BERT fine-tunes all parameters end-to-end on labeled downstream data, initializing a separate model for each task from the same pre-trained parameters.Self-attention jointly encodes concatenated text pairs with bidirectional cross-attention; task-specific output layers use token representations for tagging or question answering and [CLS] for classification. Starting from the same pre-trained model, all reported results can be replicated in at most 1 hour on a single Cloud TPU or a few hours on a GPU.Source: §3 · p. 3, §3.2 · p. 5

4.1 GLUE

• GLUE fine-tuning uses the final [CLS] representation and adds only classification-layer weights, trained with a standard classification loss.The same representation handles both single-sentence and sentence-pair inputs.Source: §4.1 · p. 5
• 80.5 is BERTLARGE’s official GLUE score, compared with OpenAI GPT’s 72.8.Excluding WNLI, BERTBASE and BERTLARGE improve average accuracy over prior state of the art by 4.5% and 7.0%, respectively. BERT gains 4.6% absolute accuracy on MNLI; BERTLARGE outperforms BERTBASE across all tasks, especially those with little training data.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 SQuAD v1.1

• SQuAD v1.1 fine-tuning packs the question and passage together and introduces only start and end vectors to predict answer spans.Token–vector dot products produce start and end probabilities, trained using the correct endpoints’ log-likelihoods. Prediction selects the highest-scoring span whose end is not before its start.Source: §4.2 · p. 6
• +1.5 F1 for ensembling and +1.3 F1 for a single system are BERT’s improvements over the corresponding top SQuAD v1.1 leaderboard systems.A single BERT model also surpasses the top ensemble; BERT’s ensemble combines seven systems with different pre-training checkpoints and fine-tuning seeds. Preliminary fine-tuning on TriviaQA (Joshi et al., 2017) contributes only 0.1-0.4 F1, with BERT still outperforming existing systems without it.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 SQuAD v2.0

• +5.1 F1 over the previous best system is BERT’s improvement on SQuAD v2.0.Unanswerable questions use the [CLS] token as both span endpoints; prediction compares the best answer span against this null span using a development-set threshold maximizing F1. This model uses no TriviaQA data.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 SWAG

• +27.1% over ESIM+ELMo and 8.3% over OpenAI GPT are BERTLARGE’s SWAG accuracy improvements.Each of four candidate continuations is paired with the given sentence; one task-specific vector scores each [CLS] representation, and softmax normalizes the choice scores.Source: Table 4 · p. 7, §4.4 · p. 7

5.1 Effect of Pre-training Tasks

• Removing NSP significantly hurts QNLI, MNLI, and SQuAD 1.1; replacing masked language modeling with left-to-right modeling worsens every tested task.The BERTBASE variants use identical pre-training data, fine-tuning schemes, and hyperparameters; the left-to-right variant omits NSP and retains its left-context constraint during fine-tuning. Its largest performance drops occur on MRPC and SQuAD.Source: Table 5 · p. 8, §5.1 · p. 8
• Adding a randomly initialized BiLSTM improves left-to-right BERT on SQuAD but remains far below pre-trained bidirectional models and hurts GLUE performance.The added BiLSTM attempts to compensate for left-to-right token representations lacking right-side context.Source: §5.1 · p. 8
• Twice as expensive as one bidirectional model, separate left-to-right and right-to-left models, as in ELMo, cannot jointly use both contexts at every layer.Concatenating their token representations also leaves the right-to-left model unable to condition question-answering answers on the question.Source: §5.1 · p. 8

5.3 Feature-based Approach with BERT

• Fixed-feature BERT supports task-specific architectures and allows expensive representations to be precomputed once for repeated experiments with cheaper downstream models.In contrast, the fine-tuning approach adds a simple classification layer and jointly updates all model parameters.Source: §5.3 · p. 9

5.2 Effect of Model Size

• Larger BERT models strictly improve accuracy across every tested GLUE dataset, including small-data MRPC.Average Dev Set accuracy is measured over five fine-tuning restarts across four datasets; MRPC has only 3,600 labeled training examples. The authors emphasize sufficient pre-training and hypothesize that direct fine-tuning with few new parameters enables small-data tasks to exploit larger representations.Source: §5.2 · pp. 8–9, Table 6 · p. 9

5.3 Feature-based Approach with BERT

• 0.3 F1 behind full-model fine-tuning, the best feature-based BERT variant concatenates the top four hidden layers on CoNLL-2003 NER.Frozen BERT activations feed a randomly initialized two-layer, 768-dimensional BiLSTM before classification. BERTLARGE performs competitively with state-of-the-art methods.Source: Table 7 · p. 9, §5.3 · p. 9

• The main contribution extends language-model transfer learning to deep bidirectional architectures, enabling one pre-trained model to successfully tackle a broad set of NLP tasks.Source: §6 · p. 9
• Empirical improvements from language-model transfer learning demonstrate that rich, unsupervised pre-training is integral to many language understanding systems.Source: §6 · p. 9
• Prior language-model transfer-learning results show that even low-resource tasks can benefit from deep unidirectional architectures.Source: §6 · p. 9

A Additional Details for BERT

• Appendix A presents additional implementation details for BERT.Source: Appendix for “BERT: Pre-training of… · p. 12

Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”

• Appendix B presents additional details for the experiments.Source: Appendix for “BERT: Pre-training of… · p. 12
• Appendix C presents additional ablation studies, including training-step counts and different masking procedures.Source: Appendix for “BERT: Pre-training of… · p. 12

Masked LM and the Masking Procedure

• The word is replaced with [MASK] 80% of the time, replaced with a random word 10%, and kept unchanged 10%, aiming to bias representations toward the actual observed word.Source: §A.1 · p. 12
• Uncertainty about which words require prediction or were randomly replaced forces the encoder to maintain a distributional contextual representation of every input token.1.5% of all tokens undergo random replacement, which does not appear to harm language understanding.Source: §A.1 · p. 12
• Masked LM converges marginally slower than left-to-right training, but its empirical improvements outweigh the increased training cost.15% of tokens receive predictions in each masked-LM batch, whereas left-to-right training predicts every token.Source: §A.1 · pp. 12–13

A.2 Pre-training Procedure

• 50% of training pairs use the actual next text span, and 50% use a random span for next sentence prediction.The spans receive A/B embeddings and together contain at most 512 tokens; these “sentences” need not correspond to individual sentences. Masking occurs uniformly at 15% after WordPiece tokenization, without special treatment of partial word pieces.Source: §A.2 · p. 13
• 1,000,000 pre-training steps use batches of 256 sequences, approximately covering 40 epochs over the 3.3 billion-word corpus.The training loss sums the mean masked language-model likelihood and the mean next sentence prediction likelihood.Source: §A.2 · p. 13
• Pre-training uses Adam with a learning rate of 1e-4, 10,000 warmup steps, linear learning-rate decay, and dropout of 0.1.Adam uses β1 = 0.9, β2 = 0.999, and L2 weight decay of 0.01. Following OpenAI GPT, BERT uses gelu instead of relu.Source: §A.2 · p. 13
• 90% of pre-training steps use sequence length 128; the remaining 10% use length 512 to learn positional embeddings.This schedule reduces training expense because attention cost grows quadratically with sequence length.Source: §A.2 · p. 13

A.3 Fine-tuning Procedure

• Fine-tuning searches batch sizes of 16 or 32, Adam learning rates of 5e-5, 3e-5, or 2e-5, and epoch counts of 2, 3, or 4.Most other hyperparameters remain unchanged from pre-training, including dropout of 0.1. Exhaustive search selects the best development-set model; datasets with 100k+ labeled examples were much less sensitive to hyperparameter choices than smaller datasets.Source: §A.3 · pp. 13–14

A.4 Comparison of BERT, ELMo ,and OpenAI GPT

• BERT and GPT differ in training data, pre-trained special embeddings, batch size, and fine-tuning learning-rate selection despite deliberately similar designs.GPT uses BooksCorpus (800M words), while BERT also uses Wikipedia (2,500M words); both train for 1M steps, with batch sizes of 32,000 and 128,000 words respectively. BERT learns [SEP], [CLS], and A/B embeddings during pre-training rather than introducing special tokens only at fine-tuning, and selects task-specific development-set learning rates instead of GPT’s fixed 5e-5.Source: §A.4 · p. 14
• Ablations attribute most improvements to BERT’s two pre-training tasks and their enabled bidirectionality, rather than other training differences from OpenAI GPT.GPT uses a left-to-right Transformer language model; BERT and GPT are fine-tuning approaches, whereas ELMo is feature-based.Source: §A.4 · p. 14

A.5 Illustrations of Fine-tuning on Different Tasks

• BERT supports sequence-level and token-level tasks by adding one output layer, minimizing the parameters learned from scratch.[CLS] supplies the classification output, while [SEP] separates non-consecutive token sequences.Source: §A.5 · p. 14

B.1 Detailed Descriptions for the GLUE Benchmark Experiments.

• MNLI classifies sentence pairs as entailment, contradiction, or neutral; RTE is binary entailment with less training data.MNLI is a large-scale, crowdsourced task.Source: §B.1 · pp. 14–15
• QQP identifies semantically equivalent Quora questions, while MRPC labels semantic equivalence in news sentence pairs.MRPC sentence pairs are automatically extracted from online news and annotated by humans.Source: §B.1 · pp. 14–15
• QNLI converts SQuAD question answering (Rajpurkar et al., 2016) into binary classification of whether a sentence contains the answer (Wang et al., 2018a).Positive question–sentence pairs contain the correct answer; negative pairs use sentences from the same paragraph that do not.Source: §B.1 · p. 14
• SST-2 is binary single-sentence sentiment classification using movie-review sentences with human sentiment annotations.Source: §B.1 · p. 15
• CoLA is binary single-sentence classification predicting whether an English sentence is linguistically acceptable (Warstadt et al., 2018).Source: §B.1 · p. 15
• 1 to 5 semantic-similarity scores annotate STS-B sentence pairs from news headlines and other sources.The scores indicate how similar the sentences are in semantic meaning.Source: §B.1 · p. 15
• 65.1 majority-class baseline accuracy exceeded every trained GLUE submission on WNLI, which BERT excludes for fairness to OpenAI GPT.WNLI is a small inference dataset with construction issues noted by the GLUE webpage.Source: §B.1 · p. 15
• Reported results use only single-task fine-tuning, although multitask training with MNLI produced substantial improvements on RTE.The authors suggest multitask fine-tuning could improve performance further.Source: §B.1 · p. 15

C.1 Effect of Number of Training Steps

• The experiment measures MNLI Dev accuracy after fine-tuning checkpoints pre-trained for different numbers of steps.Figure 5 presents this comparison.Source: §C.1 · p. 16
• Almost 1.0% additional MNLI accuracy is achieved by BERTBASE when pre-trained for 1M steps rather than 500k steps.The full pre-training schedule uses 128,000 words per batch for 1,000,000 steps.Source: §C.1 · p. 16
• MLM pre-training converges slightly slower than LTR pre-training but begins outperforming it in absolute accuracy almost immediately.MLM predicts only 15% of words in each batch, whereas LTR predicts every word.Source: §C.1 · p. 16

C.2 Ablation for Different Masking Procedures

• The masking strategies aim to reduce pre-training–fine-tuning mismatch because [MASK] never appears during fine-tuning.The authors expect greater mismatch for feature-based approaches because the model cannot adjust its representations.Source: §C.2 · p. 16
• Evaluation reports MNLI and NER Dev results, including both fine-tuning and feature-based approaches for NER.The feature-based approach concatenates BERT’s last four layers, following the best approach identified in Section 5.3.Source: §C.2 · p. 16
• Fine-tuning is robust to different masking strategies, whereas feature-based NER is sensitive to MASK-only pre-training.Using only RND also performs much worse than BERT’s mixed strategy.Source: Table 8 · p. 16, §C.2 · p. 16
• 80% MASK, 10% SAME, and 10% RND constitute BERT’s mixed masking strategy for MLM pre-training.MASK replaces the target with [MASK], SAME keeps it unchanged, and RND replaces it with another random token.Source: §C.2 · p. 16