PaperlayerView exact revision on arXiv

Source-linked AI summary

ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding

Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

arXiv:2010.12148v2cs.CLcs.LG

TL;DR

BERT-style MLM and contiguous n-gram masking give limited attention to coarse-grained linguistic information and can miss dependencies within n-grams. ERNIE-Gram predicts explicit n-gram identities and uses a generator to model relations; it outperforms prior models across NLU benchmarks, including XLNet and RoBERTa.

  • Problem

    BERT’s token-level MLM rarely considers coarse-grained information, while contiguous n-gram masking independently predicts tokens and neglects intra-dependencies.

    Source: §1 · p. 1

  • Method

    ERNIE-Gram masks n-grams and predicts their explicit identities, with a generator supplying plausible n-gram identities for relation modeling.

    Source: §1 · pp. 1–2, §3.4 · p. 4

  • Results

    Across NLU benchmarks, ERNIE-Gram outperforms XLNet and RoBERTa by a large margin and achieves state-of-the-art results; on base-scale GLUE it averages 1.7 points above TUPE.

    Source: §4.3 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    The results support ERNIE-Gram as an approach to incorporating coarse-grained linguistic information into pretraining across English and Chinese NLU tasks.

    Source: §1 · p. 2, §5 · p. 8

    The authors argue that n-gram length information can harm representation learning because it may arbitrarily prune semantic content.

    Source: §3.3 · p. 3

Abstract

from arXiv · show

Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT's Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, n-grams are masked and predicted directly using explicit n-gram identities rather than contiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n-gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive n-gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

1 Introduction

BERT’s token-level MLM gives limited attention to coarse-grained linguistic information, while contiguous n-gram masking can miss dependencies among tokens in a masked n-gram. ERNIE-Gram predicts explicit n-gram identities and adds complementary prediction and relation-modeling mechanisms.

1 Introduction

  • BERT’s MLM focuses on words or subwords in English and characters in Chinese, rarely considering coarse-grained information such as named entities, phrases, or Chinese words.Source: §1 · p. 1
  • Contiguous n-gram masking predicts each masked token independently, neglecting intra-dependencies and requiring recovery in a huge, sparse prediction space.The paper contrasts this with predicting the n-gram as a unit.Source: §1 · p. 1
  • ERNIE-Gram masks n-grams with single [MASK] symbols and predicts explicit n-gram identities in a small, dense prediction space.The n-gram identities come from a prior lexicon.Source: §1 · p. 1
  • ERNIE-Gram predicts masked n-grams both as explicit identities and through their contained token identities, using designed attention masks.Source: §1 · p. 2
  • A generator samples plausible n-gram identities for masking; the model recovers the original n-grams and uses their pair relation to model relationships.The method also uses replaced token detection to distinguish original n-grams from plausible ones and enhance interactions with fine-grained contextual tokens.Source: §1 · p. 2
  • ERNIE-Gram was pretrained on 16GB and 160GB corpora and fine-tuned on 13 English and 6 Chinese NLU tasks, outperforming previous well-performing models across benchmarks.Source: §1 · p. 2

2 Related Work

Related work covers self-supervised pretraining objectives and approaches to incorporating coarse-grained linguistic information. ERNIE-Gram is distinguished from methods that add explicit n-gram representations to both pretraining and fine-tuning.

2.1 Self-Supervised Pre-Training for NLU

  • GPT uses unidirectional language modeling, BERT uses bidirectional MLM, XLNet uses permutation language modeling, and ELECTRA uses replaced token detection.Source: §2.1 · p. 2

2.2 Coarse-grained Linguistic Information Incorporating for Pre-Training

  • ZEN (Diao et al., 2020) and AMBERT (Zhang and Li, 2020) add explicit n-gram representations through extra transformer encoders and computation in both pretraining and fine-tuning.Source: §2.2 · p. 2
  • Li et al., 2019 report that explicit n-gram representations can be unreliable because of data sparsity and out-of-vocabulary n-grams; ERNIE-Gram removes its auxiliary n-gram classifier and embedding weights before fine-tuning, retaining BERT’s parameters and computation.Source: §2.2 · p. 2

3 Proposed Method

ERNIE-Gram replaces contiguous-token masking with explicit n-gram identity prediction, then adds joint coarse- and fine-grained prediction and generator-based relation modeling.

3.1 Background

  • The background formulation masks 15% of n-gram starting boundaries and predicts the resulting contiguous masked tokens from the sequence after masking.The masked n-grams are represented as contiguous spans of tokens.Source: §3.1 · p. 3

3.2 Explicitly N-gram Masked Language Modeling

  • Explicit n-gram MLM masks each n-gram with one symbol and predicts its identity, reducing the target space relative to token-sequence prediction.Targets are explicit n-gram identities rather than contiguous sequences of tokens.Source: §3.2 · p. 3

3.3 Comprehensive N-gram Prediction

  • Comprehensive prediction jointly predicts explicit n-gram identities and contained tokens from the same context, using attention controls to prevent n-gram length leakage.Distinct mask symbols query context for each contained token, while the attention mask prevents the masked queries from attending to each other.Source: §3.3 · pp. 3–4, Figure 2 · p. 3

3.4 Enhanced N-gram Relation Modeling

  • A small generator samples plausible n-gram identities to replace masked spans, while the standard model recovers originals; sampled n-grams may differ in length.The model also uses replaced-token detection to distinguish fake n-grams from the mixed-grained context.Source: §3.4 · p. 4

3.5 N-gram Extraction

  • T-test scores rank corpus n-grams for the lexicon, and maximum matching selects the shortest valid lexicon paths as n-gram boundaries.The method takes the top k_l l-grams by t-statistic score to construct the lexicon.Source: §3.5 · pp. 4–5

4 Experiments

The experiments compare ERNIE-Gram across English and Chinese benchmarks and examine its training setup, component ablations, and entity modeling. Results favor ERNIE-Gram across the reported tasks, with ablations and case studies probing its n-gram mechanisms.

4.1 Pre-training Text Corpora / 4.2 Pre-training Setup

  • English experiments use 16GB or 160GB corpora, while Chinese experiments use ERNIE2.0 corpora; the setup extracts 200K bi-grams and 100K tri-grams and uses a 110M-parameter base model.Training settings vary by corpus; auxiliary explicit n-gram embedding weights are removed during fine-tuning.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Results on GLUE Benchmark

  • ERNIE-Gram’s GLUE average score rises 1.7 points over RoBERTa and 0.6 over ELECTRA with large-scale corpora.With base-scale corpora, it exceeds TUPE by 1.7 points and F-TFM by 1.3 points on average.Source: §4.3 · p. 5, Table 1 · p. 6

4.4 Results on Question Answering (SQuAD)

  • ERNIE-Gram performs better than strong baselines on SQuAD with both base-scale and large-scale pre-training corpora.The comparison uses base-size models on SQuAD development sets, evaluated by Exact-Match and F1.Source: Table 2 · p. 6, §4.4 · p. 6

4.5 Results on RACE and Text Classification Tasks

  • ERNIE-Gram consistently outperforms previous models on RACE, IMDb, and AG, including tasks involving long text and reasoning.The reported comparisons use base-size models on the test sets.Source: Table 3 · p. 6, §4.5 · p. 6

4.6 Results on Chinese NLU Tasks

  • ERNIE-Gram achieves state-of-the-art results among base-size models across six Chinese NLU tasks and beats larger models on XNLI, LCQMC, and CMRC2018.The six tasks include inference, machine reading comprehension, named entity recognition, and semantic similarity.Source: Table 4 · p. 7, §4.6 · p. 7

4.7 Ablation Studies

  • Explicit n-gram MLM achieves lower n-gram perplexity and better downstream performance than contiguous MLM.The compared models use the same settings and a 300K n-gram lexicon.Source: Figure 4 · p. 7, §4.7 · p. 7
  • A 300K lexicon performs best, 400K declines, and removing comprehensive prediction, relation modeling, or both reduces performance.The authors attribute the 400K decline to unnecessarily learning more low-frequency n-grams.Source: §4.7 · pp. 7–8, Table 5 · p. 8

4.8 Case Studies

  • ERNIE-Gram’s whole-entity recall declines more slowly as entity length grows, and its attention maps show within-entity blocks rather than token self-focus.The attention comparison is between ERNIE-Gram and contiguously MLM.Source: Figure 6 · p. 8, §4.8 · p. 8

5 Conclusion

ERNIE-Gram uses explicit n-gram masking and comprehensive prediction and relation modeling to incorporate coarse-grained linguistic information into pre-training. On NLU benchmarks, it outperforms XLNet and RoBERTa by a large margin and achieves state-of-the-art results on various benchmarks.

5 Conclusion

  • ERNIE-Gram outperforms XLNet and RoBERTa by a large margin and achieves state-of-the-art results on various benchmarks.Source: §5 · p. 8
  • ERNIE-Gram explicitly masks and predicts n-grams to incorporate coarse-grained linguistic information, then uses comprehensive n-gram prediction and relation modeling to enhance semantic n-gram learning.Source: §5 · p. 8
  • Future work includes constructing a more comprehensive n-gram lexicon with n>3 and pre-training a large-size model for more downstream tasks.Source: §5 · p. 9

A Hyperparameters for Pre-Training

The appendix identifies the pre-training hyperparameter settings and the fine-tuning procedure. Base-scale English settings follow BERTBASE, while large-scale settings follow RoBERTa except for batch size.

A Hyperparameters for Pre-Training

B Hyperparameters for Fine-Tuning

B.1 GLUE benchmark

The GLUE fine-tuning hyperparameters are provided in Table 7.

B.1 GLUE benchmark

B.2 SQuAD benchmark and RACE dataset

The appendix points to task-specific fine-tuning hyperparameter tables for SQuAD and RACE, and for IMDb and AG’news. For text longer than 512 tokens, fine-tuning uses the first 512 tokens.

B.2 SQuAD benchmark and RACE dataset

B.3 Text Classification tasks

  • IMDb and AG’news hyperparameters are listed in Table 9; texts longer than 512 tokens are processed using their first 512 tokens for fine-tuning.Source: §B.3 · p. 12, Table 9 · p. 12

B.4 Chinese NLU tasks

Table 10 presents fine-tuning hyperparameters for Chinese NLU tasks, with layerwise learning-rate decay used across tasks.

B.4 Chinese NLU tasks

  • The listed tasks are XNLI, LCQMC, DRCD, DuReader, CMRC2018, and MSRA-NER.Source: §B.4 · p. 12
  • All Chinese NLU tasks use layerwise learning-rate decay with a decay rate of 0.8.Source: Table 10 · p. 12

C Detailed Results for Ablation Studies

The appendix describes ablation results on GLUE and SQuAD and compares ERNIE-Gram variants on GLUE.

C Detailed Results for Ablation Studies

D More cases on CoNLL2003 Dataset

CoNLL2003 examples contrast attention patterns for ERNIE-Gram and contiguous MLM, while the appendix also identifies ablation tables on model variants and relative position bias.

D More cases on CoNLL2003 Dataset

Loading 2010.12148v2…