PaperlayerView exact revision on arXiv

Source-linked AI summary

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

arXiv:1810.04805v1cs.CL

TL;DR

Existing unidirectional pre-training restricts language representations, especially for tasks requiring context from both directions. BERT addresses this with masked-language-model pre-training and achieves state-of-the-art performance across eleven NLP tasks.

  • Problem

    Unidirectional language models restrict pre-training architectures and cannot fully incorporate both-directional context, limiting fine-tuning for sentence- and token-level tasks.

  • Method

    BERT uses masked language modeling to learn deep bidirectional representations, then fine-tunes them jointly with task-specific output layers.

  • Results

    BERT advances the state of the art on eleven NLP tasks, spanning sentence-level and token-level evaluation.

  • Takeaways & Limitations

    Deep bidirectional pre-training enables one pre-trained model to successfully tackle a broad set of NLP tasks.

  • Takeaways & Limitations

    The paper does not investigate which linguistic phenomena BERT captures.

Abstract

from arXiv · show

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement) and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement), outperforming human performance by 2.0%.

1 Introduction

BERT addresses the limitations of unidirectional language-model pre-training with masked language modeling, enabling deep bidirectional representations and simple fine-tuning across sentence-level and token-level NLP tasks. It achieves state-of-the-art performance on eleven NLP tasks while showing that bidirectionality is its most important contribution.

  • Background and problem: Prior language-model pre-training improved diverse NLP tasks, but existing feature-based and fine-tuning strategies imposed limitations on downstream representations and architecture choices.Feature-based methods such as ELMo (Peters et al., 2018) add representations to task-specific architectures, whereas fine-tuning methods such as OpenAI GPT use minimal task-specific parameters.
  • Problem and contribution: BERT uses masked language modeling to fuse left and right context, overcoming the unidirectional constraint of standard language models such as OpenAI GPT (Radford et al., 2018).The objective randomly masks input tokens and predicts their original vocabulary identities from context, enabling deep bidirectional Transformer pre-training.
  • Contributions and results: Bidirectional pre-training is BERT’s central contribution, contrasting with Radford et al. (2018)’s unidirectional models and Peters et al. (2018)’s shallow concatenation of separate directional LMs.The authors report extensive ablations identifying bidirectionality as the single most important new contribution.
  • Contributions and results: BERT achieves state-of-the-art performance on eleven NLP tasks, including sentence-level and token-level tasks, without heavily engineered task-specific architectures.The paper presents BERT as a fine-tuning-based representation model that outperforms systems built with task-specific architectures.

2 Related Work

Prior work spans pre-trained word, sentence, and paragraph representations, contextual word features, and language-model transfer learning. BERT differs from ELMo and OpenAI GPT by jointly conditioning a bidirectional Transformer on left and right context in every layer.

  • Word and sentence representations: Pre-trained word embeddings became integral to modern NLP systems and substantially improved performance over embeddings learned from scratch.Research included both non-neural methods (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et al., 2006) and neural methods (Mikolov et al., 2013).
  • Word and sentence representations: Learned representations were extended from words to sentences and paragraphs, but were typically used as features in downstream models.These approaches include sentence embeddings (Kiros et al., 2015; Logeswaran and Lee, 2018) and paragraph embeddings (Le and Mikolov, 2014).
  • Contextual representations: ELMo extracts context-sensitive features from a language model and improves several NLP benchmarks when integrated with task-specific architectures.Reported applications include question answering on SQuAD, sentiment analysis, and named entity recognition (Peters et al., 2018; Rajpurkar et al., 2016; Socher et al., 2013; Tjong Kim).
  • Transfer learning: Language-model transfer learning pre-trains a model on a language-model objective before fine-tuning it for supervised downstream tasks, reducing parameters learned from scratch.This approach was explored by Dai and Le (2015), Howard and Ruder (2018), and Radford et al. (2018), with OpenAI GPT achieving state-of-the-art results on many sentence-level tasks.
  • Pre-training architectures: Only BERT jointly conditions representations on left and right context in all layers; OpenAI GPT is left-to-right, while ELMo concatenates separately trained directional LSTMs.BERT uses a bidirectional Transformer, whereas OpenAI GPT uses a left-to-right Transformer and ELMo combines independently trained left-to-right and right-to-left LSTMs.

3 BERT · 3.1 Model Architecture

BERT is introduced as a multi-layer bidirectional Transformer encoder, with this section covering its architecture and input representation before later sections address pre-training and fine-tuning. Its key architectural distinction from GPT is bidirectional self-attention rather than left-context-only attention.

  • 3 BERT: The section introduces BERT’s architecture and input representation, while reserving pre-training, optimization procedures, fine-tuning, and GPT differences for later sections.The pre-training tasks are identified as the paper’s core innovation and are introduced in Section 3.3.
  • 3.1 Model Architecture: BERT is introduced as a multi-layer bidirectional Transformer encoder based on the original Transformer implementation.The implementation is described as effectively identical to the original implementation released in the tensor2tensor library.
  • 3.1 Model Architecture: BERT denotes the number of Transformer blocks as L, hidden size as H, and self-attention heads as A.These symbols define the model configurations reported in the paper.
  • 3.1 Model Architecture: The feed-forward/filter size is set to 4H in all BERT configurations.This corresponds to 3072 when H = 768 and 4096 when H = 1024.
  • 3.1 Model Architecture: The paper primarily reports results for two BERT model sizes.The supplied passage introduces these model sizes but does not include their full specifications.
  • 3.1 Model Architecture: Unlike GPT’s constrained self-attention, BERT uses bidirectional self-attention so tokens can attend beyond only their left context.BERTBASE was selected to match OpenAI GPT’s model size for comparison, while retaining this different attention structure.

3.2 Input Representation

BERT represents single sentences or sentence pairs in one sequence by summing token, segment, and position embeddings. Special tokens and learned sentence-type embeddings support sequence classification and distinguish paired inputs.

  • 3.2 Input Representation: BERT constructs each input representation by summing token, segment, and position embeddings, supporting either a single sentence or a sentence pair in one sequence.Figure 2 depicts this summed embedding representation.
  • 3.2 Input Representation: BERT uses WordPiece embeddings with a 30,000-token vocabulary and learned positional embeddings supporting sequences up to 512 tokens.Split word pieces are marked with ##.
  • 3.2 Input Representation: The first token is [CLS], whose final Transformer hidden state serves as the aggregate sequence representation for classification and is ignored for other tasks.
  • 3.2 Input Representation: Sentence pairs are packed into one sequence, separated by [SEP], and assigned learned sentence A or sentence B embeddings; single-sentence inputs use only sentence A embeddings.

3.3 Pre-training Tasks

BERT is pre-trained with two unsupervised tasks rather than traditional directional language modeling: masked language modeling and next sentence prediction. MLM enables deep bidirectional representations, while next sentence prediction targets sentence relationships important for QA and NLI.

  • Masked LM: Masked language modeling predicts randomly selected tokens from bidirectional context, with 15% of WordPiece tokens masked per sequence.Only masked words are predicted, rather than reconstructing the entire input.
  • Masked LM: To reduce pre-training–fine-tuning mismatch, selected tokens are replaced by [MASK] 80% of the time, random words 10%, and left unchanged 10%.This also biases representations toward observed words while preserving contextual information about every token.
  • Masked LM: MLM predicts only 15% of tokens per batch and converges marginally slower than a left-to-right model, but its empirical improvements outweigh the increased training cost.The comparison is reported in Section 5.3.
  • Next Sentence Prediction: Next sentence prediction pre-trains a binarized task because QA and NLI require understanding relationships between two sentences, which language modeling does not directly capture.The task can be generated trivially from any monolingual corpus.
  • Next Sentence Prediction: The final pre-trained model achieves 97%-98% accuracy on next sentence prediction, which is beneficial to both QA and NLI despite the task’s simplicity.NotNext sentences are chosen completely at random.

3.4 Pre-training Procedure

BERT’s pre-training uses a document-level corpus combining BooksCorpus and English Wikipedia, with paired text spans supporting next sentence prediction. Training uses large-batch optimization for 1,000,000 steps on TPU configurations, with specified learning-rate scheduling, regularization, and activation choices.

  • Corpus: The 3.3-billion-word corpus combines BooksCorpus (800M words) and English Wikipedia (2,500M words), retaining Wikipedia text passages while excluding lists, tables, and headers.The procedure requires a document-level corpus to extract long contiguous sequences rather than shuffled sentences.
  • Input construction: Training inputs contain two sampled text spans assigned A and B embeddings; B is the actual successor of A 50% of the time and random otherwise for next sentence prediction.The spans are called “sentences” although they are typically much longer than single sentences and can sometimes be shorter.
  • Optimization: BERT trains with batches of 256 sequences, 1,000,000 steps, Adam at 1e-4, 0.01 L2 weight decay, 10,000-step warmup, linear decay, 0.1 dropout, and gelu activation.Each batch contains 128,000 tokens, and training covers approximately 40 epochs over the corpus.
  • Compute: BERTBASE uses 4 Cloud TPUs in Pod configuration, while BERTLARGE uses 16 Cloud TPUs; each pre-training run takes 4 days.These configurations correspond to 16 TPU chips for BERTBASE and 64 TPU chips for BERTLARGE.

3.5 Fine-tuning Procedure

BERT fine-tuning adds only a task-specific classification layer for sequence classification while jointly updating all model parameters. Task-specific hyperparameter search is practical because fine-tuning is fast, and large labeled datasets are less sensitive to these choices.

  • Sequence-level classification: For sequence classification, BERT uses the [CLS] token’s final hidden state as a pooled representation and adds only a softmax classification layer, jointly fine-tuning all parameters.The classifier has K labels and parameters W ∈ R^K×H; training maximizes the correct label’s log-probability.
  • Hyperparameters: Fine-tuning retains most pre-training hyperparameters, but task-specific batch size, learning rate, and epoch count are tuned while dropout remains fixed at 0.1.Effective ranges were batch sizes 16 or 32, Adam learning rates 5e-5, 3e-5, or 2e-5, and 3 or 4 epochs.
  • Hyperparameter sensitivity: Large datasets with 100k+ labeled examples are less sensitive to hyperparameter choice than small datasets.This makes exhaustive search over the specified settings especially reasonable for selecting the best development-set model.
  • Hyperparameter selection: Fine-tuning is typically very fast, enabling exhaustive searches over the recommended hyperparameter combinations using development-set performance.The search covers batch size, Adam learning rate, and training epochs.

3.6 Comparison of BERT and OpenAI GPT

BERT was designed for a close comparison with OpenAI GPT, differing in corpus, pre-training treatment of special tokens and sentence embeddings, batch size, and fine-tuning learning-rate selection. Ablations show that the new pre-training tasks account for most of BERT’s improvements.

  • Comparison and ablation: BERT uses BooksCorpus and Wikipedia, whereas GPT uses only BooksCorpus; BERT also pre-trains [SEP], [CLS], and sentence A/B embeddings.GPT introduces these special tokens and embeddings only during fine-tuning.
  • Comparison and ablation: Both models train for 1M steps, but BERT uses a 128,000-word batch size compared with GPT’s 32,000-word batch size.The comparison holds training duration constant while differing in batch size.
  • Comparison and ablation: GPT uses a fixed 5e-5 fine-tuning learning rate, whereas BERT selects a task-specific rate that performs best on the development set.These choices were included among the differences isolated by the ablation experiments.
  • Comparison and ablation: Ablation experiments demonstrate that the majority of BERT’s improvements over GPT come from its new pre-training tasks.The experiments isolate the effects of the methodological differences between the two models.

4 Experiments

BERT fine-tuning achieves substantial improvements across GLUE, SQuAD, CoNLL-2003 NER, and natural-language-inference evaluation tasks. The experiments use minimal task-specific parameters, while reporting single-task fine-tuning and excluding WNLI from the GLUE average.

  • GLUE: BERTBASE and BERTLARGE outperform all existing GLUE systems on every task, improving average accuracy by 4.4% and 6.7%, respectively.On MNLI, BERT achieves a 4.7% absolute accuracy improvement over the state of the art.
  • Evaluation scope: The GLUE average excludes WNLI because submitted systems performed worse than its 65.1 majority-class baseline, and the reported results use single-task fine-tuning.The authors note that multitask fine-tuning could improve results further, including on RTE.
  • Fine-tuning setup: GLUE fine-tuning uses the [CLS] representation with only a newly introduced classification layer, while SQuAD adds only start and end vectors.GLUE uses 3 epochs and batch size 32, selecting learning rates on the development set; unstable BERTLARGE runs use random restarts.
  • SQuAD: BERT’s best SQuAD system surpasses the top leaderboard system by +1.5 F1 with ensembling and +1.3 F1 as a single system.A single BERT model also outperforms the top ensemble system in F1; fine-tuning only on SQuAD reduces performance by 0.1–0.4 F1 but retains a wide margin over existing systems.
  • CoNLL-2003 NER: BERTLARGE outperforms Cross-View Training with multi-task learning by +0.2 on the CoNLL-2003 NER Test.
  • Natural-language inference: BERTLARGE outperforms the authors’ ESIM+ELMo baseline by +27.1%.The experiment fine-tunes for 3 epochs with a learning rate of 2e-5 and batch size 16.

5 Ablation Studies

The ablations show that BERT’s masked bidirectional pre-training and next sentence prediction are central to its performance, while scaling, longer pre-training, and feature extraction provide additional benefits. Larger models improve even small-task accuracy, and fixed contextual features nearly match full fine-tuning on NER.

  • Pre-training Tasks: Removing NSP significantly hurts QNLI, MNLI, and SQuAD, while left-to-right pre-training performs worse than MLM on every task, especially MRPC and SQuAD.Adding a randomly initialized BiLSTM improves SQuAD for the left-to-right model but remains far worse than pre-trained bidirectional models and hurts all four GLUE tasks.
  • Model Size: Larger BERT models produce strict accuracy improvements across all four selected GLUE datasets, including MRPC despite only 3,600 labeled training examples.BERTBASE has 110M parameters and BERTLARGE has 340M; the results support large improvements on small tasks when models are sufficiently pre-trained.
  • Training Steps: 1.0% additional MNLI accuracy is achieved by BERTBASE after 1M rather than 500k pre-training steps, while MLM begins outperforming LTR almost immediately despite slightly slower convergence.The comparison uses MNLI Dev accuracy after fine-tuning from checkpoints pretrained for k steps.
  • Feature-Based Approach: 0.3 F1 separates concatenated top-four-layer contextual features from full-model fine-tuning on CoNLL-2003 NER, demonstrating effectiveness in both feature-based and fine-tuning settings.The feature-based approach feeds frozen BERT activations into a randomly initialized two-layer 768-dimensional BiLSTM.

6 Conclusion

BERT extends the benefits of unsupervised language-model pre-training to deep bidirectional architectures, enabling one pre-trained model to address a broad set of NLP tasks. Despite strong empirical results, future work should examine which linguistic phenomena BERT captures.

  • 6 Conclusion: BERT generalizes transfer-learning gains from language-model pre-training to deep bidirectional architectures for language understanding.This extends earlier success with deep unidirectional architectures.
  • 6 Conclusion: A single pre-trained BERT model can successfully tackle a broad set of NLP tasks, including low-resource tasks benefiting from deep architectures.
  • 6 Conclusion: Future work should investigate which linguistic phenomena BERT captures, despite empirical results that in some cases surpass human performance.
Loading 1810.04805v1…