PaperlayerView exact revision on arXiv

Source-linked AI summary

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova

arXiv:1810.04805v2cs.CL

TL;DR

Existing unidirectional language models limit contextual representations for sentence- and token-level NLP tasks. BERT uses masked-language-model pre-training with bidirectional context and achieves state-of-the-art performance across eleven NLP tasks, including a GLUE score of 80.5 versus 72.8 for OpenAI GPT.

  • Problem

    Unidirectional language models restrict pre-trained representations by preventing simultaneous use of left and right context, especially for sentence-level and token-level tasks.

  • Method

    BERT pre-trains deep bidirectional representations using masked language modeling and next-sentence prediction, then fine-tunes them for downstream tasks.

  • Results

    80.5 GLUE score versus 72.8 for OpenAI GPT, with BERT achieving state-of-the-art performance across all eleven evaluated NLP tasks.

  • Takeaways & Limitations

    Deep bidirectional pre-training generalizes transfer learning to a broad set of NLP tasks while reducing the need for heavily engineered task-specific architectures.

  • Takeaways & Limitations

    Masked-language-model pre-training creates a mismatch with fine-tuning because the [MASK] token does not appear during fine-tuning.

Abstract

from arXiv · show

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

1 Introduction

BERT addresses the unidirectionality of standard language-model pre-training with masked language modeling, enabling deep bidirectional representations. It reduces reliance on task-specific architectures and advances state of the art across eleven NLP tasks.

  • Problem and contribution: BERT addresses the unidirectional limitation of prior language-model pre-training, unlike OpenAI GPT’s left-to-right model and ELMo’s shallow concatenation of independent directional models.
  • Contribution: BERT uses masked language modeling to fuse left and right context, enabling pre-training of deep bidirectional Transformer representations.The objective randomly masks input tokens and predicts their original vocabulary identities from context.
  • Method: BERT jointly pre-trains text-pair representations with a next sentence prediction task in addition to masked language modeling.
  • Contribution: BERT reduces the need for heavily engineered task-specific architectures through a fine-tuning-based representation model.The paper describes BERT as achieving state-of-the-art performance on a large suite of sentence-level and token-level tasks while outperforming many task-specific architectures.
  • Results: BERT advances the state of the art for eleven NLP tasks.

2 Related Work

Prior work progressed from pretrained word embeddings to sentence- and paragraph-level representations, contextual token features, and pretrained encoders fine-tuned for downstream tasks. These approaches established the value of transfer from unlabeled and supervised data while using diverse language-modeling, denoising, and auto-encoding objectives.

  • Word and sentence representations: Pretrained word embeddings, developed through non-neural and neural methods, became integral to NLP systems and improved performance over embeddings learned from scratch.Representative approaches include left-to-right language-modeling objectives and earlier methods that pretrained only word-embedding parameters from unlabeled text.
  • Word and sentence representations: Sentence and paragraph representations extended pretraining beyond words using next-sentence ranking, left-to-right generation, and denoising auto-encoder objectives.These methods targeted increasingly coarse-grained representations while preserving transfer from unlabeled text.
  • Contextual representations: ELMo and its predecessor produced contextual token features by concatenating left-to-right and right-to-left language-model representations, improving several major NLP benchmarks.Their contextual embeddings were integrated with existing task-specific architectures.
  • Pretrained encoders: Pretrained sentence or document encoders later produced contextual token representations and were fine-tuned on supervised downstream tasks, reducing parameters learned from scratch.These models used language-modeling or auto-encoder objectives; OpenAI GPT achieved state-of-the-art results on many GLUE sentence-level tasks.
  • Transfer learning: Transfer learning also proved effective from supervised natural-language-inference and machine-translation datasets, while computer vision demonstrated the value of fine-tuning ImageNet-pretrained models.These results broadened the evidence for transferring knowledge from large pretrained models across tasks.

3 BERT

BERT is a bidirectional Transformer encoder pretrained with masked language modeling and next sentence prediction, then fine-tuned end-to-end for downstream tasks. Its unified architecture represents single sentences and sentence pairs in one sequence and requires minimal task-specific modification.

  • Fine-tuning: Fine-tuning initializes separate downstream models with BERT’s pre-trained parameters and updates all parameters end-to-end while swapping task-specific inputs and output layers.Token representations support token-level tasks, whereas the [CLS] representation supports classification.
  • Model Architecture: BERT uses a multi-layer bidirectional Transformer encoder, unlike GPT’s constrained self-attention, and reports BERTBASE with 110M and BERTLARGE with 340M parameters.BERTBASE has L=12, H=768, A=12; BERTLARGE has L=24, H=1024, A=16.
  • Input/Output Representations: BERT packs one or two sentences into a single sequence using WordPiece tokens, [CLS] and [SEP] markers, segment embeddings, position embeddings, and summed token representations.The [CLS] state serves as the aggregate representation for classification tasks.
  • Pre-training Tasks: Masked language modeling randomly masks 15% of WordPiece tokens and predicts them from bidirectional context, while varying replacements reduces the mismatch between pre-training and fine-tuning.The masking procedure addresses the absence of [MASK] tokens during fine-tuning.
  • Pre-training Tasks: Next sentence prediction pre-trains BERT to model relationships between sentence pairs, targeting relationships needed for tasks such as question answering and natural language inference.The task is a binarized classification objective generated from monolingual corpora.
  • Pre-training data: Pre-training uses BooksCorpus with 800M words and English Wikipedia with 2,500M words, retaining document-level text from Wikipedia while excluding lists, tables, and headers.The document-level corpus supports extracting long coherent sequences.

4 Experiments

BERT fine-tuning achieves substantial gains across GLUE, SQuAD 1.1, SQuAD 2.0, and SWAG. The experiments use lightweight task-specific output layers while retaining strong performance without extensive architectural modification.

  • GLUE: BERTBASE and BERTLARGE outperform all systems on every GLUE task, improving average accuracy over prior state of the art by 4.5% and 7.0%, respectively.On MNLI, BERT achieves a 4.6% absolute accuracy improvement.
  • SQuAD 1.1: +1.5 F1 with ensembling and +1.3 F1 as a single system over the top SQuAD 1.1 leaderboard system, with the single BERT model surpassing its top ensemble.Without TriviaQA fine-tuning data, performance drops only 0.1–0.4 F1 while remaining ahead of existing systems.
  • SWAG: BERTLARGE outperforms the ESIM+ELMo baseline by +27.1% and OpenAI GPT by 8.3% on SWAG.SWAG fine-tuning scores each of four sentence-pair continuations using a task-specific vector applied to the [CLS] representation.

5 Ablation Studies

Ablations show that BERT’s deep bidirectional pre-training, larger model capacity, and fine-tuning strategy each materially improve downstream performance. Bidirectional models outperform left-to-right alternatives, scaling helps even on small tasks, and feature-based use remains competitive with fine-tuning.

  • Pre-training task ablations: Removing NSP significantly hurts QNLI, MNLI, and SQuAD 1.1, while left-to-right pre-training performs worse than masked language modeling on every evaluated task.The left-to-right model shows especially large drops on MRPC and SQuAD.
  • Pre-training task ablations: Adding a BiLSTM improves the left-to-right model on SQuAD but remains far worse than bidirectional pre-training and hurts GLUE performance.The left-to-right model lacks right-context information in token-level hidden states, motivating the added BiLSTM.
  • Model size ablation: Larger BERT models produce strict accuracy improvements across all four selected GLUE datasets, including MRPC despite only 3,600 labeled training examples.The results use average development-set accuracy from five random fine-tuning restarts; the authors characterize scaling to extreme model sizes as improving very small tasks when sufficiently pre-trained.
  • Feature-based approach: BERTLARGE performs competitively for CoNLL-2003 NER, with top-four-layer feature concatenation only 0.3 F1 behind full-model fine-tuning.This demonstrates effectiveness in both fine-tuning and feature-based settings.

6 Conclusion

The paper argues that rich unsupervised pre-training is integral to language understanding systems and extends its benefits from deep unidirectional to deep bidirectional architectures across broad NLP tasks.

  • 6 Conclusion: Rich unsupervised pre-training has become an integral component of many language understanding systems.
  • 6 Conclusion: Deep unidirectional architectures enable transfer-learning benefits even for low-resource language understanding tasks.
  • 6 Conclusion: BERT generalizes these findings to deep bidirectional architectures, enabling one pre-trained model to tackle a broad set of NLP tasks.

Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” · A Additional Details for BERT

The appendix is organized into three sections covering additional BERT implementation details, experimental details, and ablation studies. The ablations include training-step effects and different masking procedures.

  • Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”: The appendix is divided into three sections.
  • A Additional Details for BERT: Appendix A presents additional implementation details for BERT.
  • Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”: Appendix B provides additional details about the experiments.
  • Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”: Appendix C presents additional ablation studies for BERT.
  • Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”: The additional ablation studies examine the effect of the number of training steps.
  • Appendix for “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”: The additional ablation studies also compare different masking procedures.

A.1 Illustration of the Pre-training Tasks

This section illustrates BERT’s masked language modeling procedure and next sentence prediction task. Masked LM hides or alters selected tokens to force contextual representations, converges marginally slower than left-to-right training, but yields improvements that outweigh the added cost.

  • Masked LM and the Masking Procedure: The masking procedure forces the Transformer encoder to maintain distributional contextual representations for every input token because it cannot identify prediction or random-replacement positions.Only 1.5% of all tokens are randomly replaced, which the paper says does not seem to harm language understanding capability.
  • Masked LM and the Masking Procedure: Masked LM predicts only 15% of tokens per batch, so it may require more pre-training steps and converges marginally slower than a left-to-right model, but its empirical improvements outweigh the increased training cost.The comparison is against a left-to-right model that predicts every token.
  • Next Sentence Prediction: Next sentence prediction is illustrated with examples containing [MASK] tokens and [SEP] sentence boundaries.The supplied passage introduces examples but does not specify the task’s sampling procedure or objective.

A.2 Pre-training Procedure … B Detailed Experimental Setup

BERT pre-training samples paired spans for masked language modeling and next sentence prediction, using a staged sequence-length schedule and large-scale optimization. Fine-tuning adapts BERT to sequence- and token-level tasks with one additional output layer, while comparisons with ELMo and GPT emphasize bidirectionality, pre-training tasks, and task-specific adaptation.

  • A.2 Pre-training Procedure: BERT samples paired spans with sentence-A/B embeddings, making the second span the true successor 50% of the time and a random sentence 50% of the time for next sentence prediction.The paired spans are sampled so their combined length does not exceed the model’s sequence limit.
  • A.2 Pre-training Procedure: Pre-training runs for 1,000,000 steps with batches of 256 sequences, totaling 128,000 tokens per batch, over approximately 40 epochs of a 3.3-billion-word corpus.Optimization uses Adam with learning rate 1e-4, warmup for 10,000 steps, linear decay, dropout 0.1, GELU activations, and a loss combining masked-LM and next-sentence likelihoods.
  • B Detailed Experimental Setup: 90% of pre-training steps use sequence length 128 and the remaining 10% use length 512, reducing quadratic attention cost while learning positional embeddings for long sequences.Longer sequences are disproportionately expensive because attention scales quadratically with sequence length.
  • A.3 Fine-tuning Procedure: Fine-tuning generally retains pre-training hyperparameters except batch size, learning rate, and epochs, with dropout fixed at 0.1.Across tasks, useful ranges are batch size 16 or 32, Adam learning rate 5e-5, 3e-5, or 2e-5, and 2, 3, or 4 epochs.
  • A.3 Fine-tuning Procedure: Fine-tuning is less sensitive to hyperparameters on large datasets with 100k+ labeled examples, so exhaustive development-set search is practical because fine-tuning is typically fast.The recommended search covers the listed batch sizes, learning rates, and epoch counts.
  • A.4 Comparison of BERT, ELMo ,and OpenAI GPT: Compared with ELMo’s feature-based approach and GPT’s fine-tuning approach, BERT is also a fine-tuning method whose pre-training incorporates sentence structure and task-specific learning rates.BERT uses BooksCorpus and Wikipedia, learns [SEP], [CLS], and sentence A/B embeddings during pre-training, and selects the fine-tuning learning rate on development data.
  • A.4 Comparison of BERT, ELMo ,and OpenAI GPT: BERT uses bidirectional representations and two pre-training tasks, which the paper identifies as the main sources of improvement over GPT’s left-to-right language modeling.Ablations are used to isolate the effects of bidirectionality and the two pre-training tasks.
  • A.5 Illustrations of Fine-tuning on Different Tasks: Task-specific models add one output layer to BERT, supporting both sequence-level and token-level tasks while learning only a minimal number of parameters from scratch.The [CLS] representation provides classification output in the illustrated sequence-level setup.

B.1 Detailed Descriptions for the GLUE Benchmark Experiments. · C Additional Ablation Studies

The GLUE experiments span diverse entailment, equivalence, sentiment, acceptability, similarity, and paraphrase tasks, while excluding WNLI and reporting only single-task fine-tuning results. The supplied passages do not describe the merged ablation-study section.

  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: The reported GLUE results were obtained from the GLUE leaderboard and an OpenAI language-unsupervised blog source.The passage introduces these sources before listing dataset descriptions summarized from Wang et al. (2018a).
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: The benchmark includes MNLI, a three-way entailment classification task predicting entailment, contradiction, or neutrality from sentence pairs.MNLI is described as large-scale and crowdsourced.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: QQP and MRPC evaluate whether sentence pairs are semantically equivalent, using Quora questions for QQP and online-news sentences for MRPC.QQP is binary classification, while MRPC uses human annotations of equivalence.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: QNLI converts SQuAD into binary classification of whether a question–sentence pair contains the correct answer.Negative examples come from the same paragraph but do not contain the answer.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: SST-2, CoLA, and STS-B respectively measure movie-review sentiment, English-sentence acceptability, and semantic similarity scored from 1 to 5.These tasks use single sentences or sentence pairs drawn from reviews, linguistic examples, news headlines, and other sources.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: RTE is a binary entailment task similar to MNLI but with substantially less training data.The passage identifies RTE as Recognizing Textual Entailment.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: WNLI was excluded because its construction has known issues and submitted systems underperformed the 65.1 majority-class baseline; the submission always predicted the majority class.This exclusion was intended to ensure a fair comparison with OpenAI GPT.
  • B.1 Detailed Descriptions for the GLUE Benchmark Experiments.: Only single-task fine-tuning results are reported, although multitask fine-tuning could improve performance, including substantial observed gains on RTE from MNLI training.The passage states that multitask improvements could potentially push performance further.

C.1 Effect of Number of Training Steps

BERT benefits substantially from extended pre-training: BERTBASE gains almost 1.0% MNLI accuracy at 1M versus 500k steps. Although MLM converges slightly slower than LTR, it surpasses LTR in absolute accuracy almost immediately.

  • C.1 Effect of Number of Training Steps: Almost 1.0% additional MNLI accuracy is achieved by BERTBASE at 1M versus 500k pre-training steps.Figure 5 reports MNLI Dev accuracy after fine-tuning checkpoints pretrained for k steps.
  • C.1 Effect of Number of Training Steps: MLM converges slightly slower than LTR, but begins outperforming it in absolute accuracy almost immediately.The comparison reflects that MLM predicts only 15% of words in each batch, whereas LTR predicts every word.

C.2 Ablation for Different Masking Procedures

The ablation evaluates masking strategies intended to reduce pre-training–fine-tuning mismatch, reporting MNLI and NER Dev results for fine-tuning and feature-based approaches. Fine-tuning is robust across strategies, but feature-based NER performs poorly with only MASK, while only RND also underperforms BERT’s mixed strategy.

  • C.2 Ablation for Different Masking Procedures: The masking ablation tests the mixed MLM strategy against alternative procedures to reduce mismatch because [MASK] never appears during fine-tuning.NER includes both fine-tuning and feature-based evaluation, with mismatch expected to be amplified for feature-based representations that cannot adjust during fine-tuning.
  • C.2 Ablation for Different Masking Procedures: Fine-tuning is robust to different masking strategies, whereas feature-based NER suffers with only MASK and only RND performs worse than BERT’s mixed strategy.The comparison uses Table 8’s Dev results for MNLI and NER; the feature-based NER representation concatenates BERT’s last four layers.
  • C.2 Ablation for Different Masking Procedures: MASK replaces the target with [MASK], SAME keeps it unchanged, and RND replaces it with another random token during MLM pre-training.BERT’s mixed strategy uses probabilities of 80% MASK, 10% SAME, and 10% RND.
Loading 1810.04805v2…