PaperlayerView exact revision on arXiv

Source-linked AI summary

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang

arXiv:1907.12412v2cs.CL

TL;DR

Existing pre-training often emphasizes word and sentence co-occurrence despite additional lexical, syntactic, and semantic information in corpora. ERNIE 2.0 incrementally builds diverse tasks through continual multi-task learning, and it outperforms BERT and XLNet across 16 English and Chinese tasks.

  • Problem

    Current pre-training focuses on co-occurrence while valuable lexical, syntactic, and semantic information such as named entities and discourse relations remains to be examined.

  • Method

    ERNIE 2.0 incrementally constructs and learns diverse unsupervised pre-training tasks through continual multi-task learning.

  • Results

    ERNIE 2.0 significantly improves over BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.

  • Takeaways & Limitations

    The framework supports learning lexical, syntactic, and semantic representations through incrementally added pre-training tasks.

Abstract

from arXiv · show

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

Introduction

ERNIE 2.0 addresses the limits of co-occurrence-focused pre-training by incrementally constructing diverse tasks through continual multi-task learning. The framework targets lexical, syntactic, and semantic information and improves performance across English and Chinese language understanding tasks.

  • Motivation: Existing pre-training commonly models word and sentence co-occurrence, while training corpora also contain valuable lexical, syntactic, and semantic information.Examples include named entities, sentence order, sentence proximity, semantic similarity, and discourse relations.
  • Approach: ERNIE 2.0 incrementally builds and trains a large variety of pre-training tasks through continual multi-task learning.The framework supports introducing new tasks while retaining previously learned knowledge.
  • Contributions: The framework supports customized training tasks and continual multi-task learning in an incremental way.This is the stated core contribution of ERNIE 2.0.
  • Results: ERNIE 2.0 achieves significant improvements over BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.The authors evaluate the framework across both English and Chinese language understanding settings.
  • Availability: Fine-tuning code and models pre-trained on English corpora are released through the PaddlePaddle/ERNIE repository.The release supports reproduction and downstream use of the framework.

Related Work

Prior work established the value of large-scale pre-training, contextualized representations, and continual learning. ERNIE 2.0 extends these directions by combining diverse pre-training objectives with continual task learning.

  • Contextualized Pre-training: Pre-training on large amounts of unannotated data has improved language representations beyond traditional fixed word embeddings based on word co-occurrence.Contextualized approaches such as ELMo, GPT, BERT, and related models learn context-dependent representations.
  • ERNIE 2.0: ERNIE 2.0 is presented as a framework that continually introduces a large variety of pre-training tasks to learn lexical, syntactic, and semantic representations.Its framework combines the pre-training-and-fine-tuning architecture with continual multi-task learning.
  • Contextualized Pre-training: BERT combines masked language modeling with next sentence prediction, while earlier contextualized models use other language-modeling or Transformer-based strategies.These methods motivate expanding the objectives used during pre-training.
  • Continual Learning: Continual learning trains models on several tasks in sequence while aiming to retain previously learned tasks.Its motivation is cumulative acquisition of information and skills.

The ERNIE 2.0 Framework

The ERNIE 2.0 framework constructs unsupervised tasks incrementally and updates a shared model through continual multi-task learning. It then fine-tunes the resulting representation model for downstream language understanding tasks.

  • Framework: Continual pre-training first constructs unsupervised tasks using large data and prior knowledge, then incrementally updates ERNIE through continual multi-task learning.The framework can add word-aware, structure-aware, and semantic-aware tasks over time.
  • Continual Multi-task Learning: The framework addresses forgetting and efficiency by initializing new-task training from prior parameters, training new and original tasks together, and allocating iterations across stages.This design preserves previously learned knowledge while distributing training effort across tasks.
  • Continual Multi-task Learning: Each learning stage uses shared encoding layers whose parameters are updated across tasks, with recurrent neural networks or deep Transformers as possible encoders.Sentence-level and token-level losses support the pre-training tasks.
  • Fine-tuning: Fine-tuning adapts the pre-trained model to downstream tasks such as question answering, natural language inference, and semantic similarity.Each downstream task receives its own fine-tuned model.
  • Task Construction: Different pre-training datasets can generate different tasks, and one dataset type can correspond to multiple pre-training tasks.This dataset-task relationship supports incremental task construction.

ERNIE 2.0 Model

The ERNIE 2.0 model uses a Transformer encoder and task-aware input representations to support multiple word-, structure-, and semantic-aware pre-training tasks. These tasks target lexical information, document structure, sentence relations, and short-text relevance.

  • Model Structure: ERNIE 2.0 uses a multi-layer Transformer as its basic encoder to produce contextual embeddings through self-attention.The input sequence includes the special classification embedding [CLS].
  • Model Structure: Task embeddings encode task identity alongside token, segment, and position embeddings, allowing task-specific inputs during pre-training and fine-tuning.Each task id is assigned a unique task embedding.
  • Pre-training Tasks: The model constructs word-aware, structure-aware, and semantic-aware tasks to capture lexical, syntactic, and semantic information.The three task categories organize the different pre-training objectives.
  • Structure-aware Tasks: Sentence reordering randomly permutes paragraph segments and trains the model to reconstruct their order as a k-class classification problem.The task is intended to learn relationships among sentences in a document.
  • Structure-aware Tasks: Sentence distance classifies whether two sentences are adjacent in one document, nonadjacent in one document, or from different documents.The task uses three document-level distance classes.
  • Semantic-aware Tasks: The short-text relevance task classifies query-title pairs as strongly relevant, weakly relevant, or irrelevant using commercial search-engine logs.Strong relevance corresponds to titles clicked after users entered the query; weak relevance corresponds to displayed but unclicked titles.

Experiments

The experiments compare ERNIE 2.0 with established pre-training models on English GLUE tasks and Chinese datasets, while also examining pre-training dataset sizes.

  • ERNIE 2.0 is compared with BERT and XLNet on English GLUE tasks, and with BERT and ERNIE 1.0 on several Chinese datasets.
  • The experiments also compare ERNIE 2.0 with multi-task learning and traditional continual learning.
  • Table 2 reports the sizes of the pre-training datasets.

Pre-training and Implementation

The pre-training setup combines English and Chinese corpora with Transformer configurations aligned to BERT, using substantial GPU resources and fixed optimization settings.

  • Pre-training Data: English data include Wikipedia, BookCorpus, Reddit, and Discovery discourse-relation data, while Chinese data span encyclopedia, news, dialogue, retrieval, and discourse-relation sources.
  • Model Settings: The base Transformer has 12 layers, 12 self-attention heads, and 768-dimensional hidden states; the large model has 24 layers, 16 heads, and 1024-dimensional hidden states.
  • Fine-tuning Settings: Tables 3 and 4 provide the fine-tuning experiment settings for GLUE and Chinese datasets.
  • Training Resources: ERNIE 2.0 uses 48 V100 GPUs for base models and 64 V100 GPUs for large models in both languages.
  • Optimization: Adam optimization uses β1 = 0.9, β2 = 0.98, and a batch size of 393216 tokens, with language-specific learning rates and 4,000 warmup steps.

Fine-tuning Tasks

Evaluation covers GLUE and nine Chinese NLP tasks spanning comprehension, recognition, inference, similarity, sentiment, and question answering, alongside continual-pre-training comparisons.

  • English Evaluation: GLUE provides a multi-task benchmark for evaluating natural language understanding performance.
  • Chinese Evaluation: The Chinese evaluation includes nine NLP tasks across machine reading comprehension, named entity recognition, inference, semantic similarity, sentiment analysis, and question answering.
  • Results and Analysis: Tables 5 and 6 report results on GLUE and nine common Chinese NLP tasks, while Table 7 compares continual-pre-training methods.
  • Chinese Evaluation: Chinese machine reading comprehension uses CMRC 2018, DRCD, and DuReader, while semantic similarity uses LCQMC and BQ Corpus.
  • Chinese Evaluation: Named entity recognition, inference, sentiment analysis, and question answering use MSRA-NER, XNLI, ChnSentiCorp, and NLPCC-DBQA, respectively.

Implementation Details for Fine-tuning

Fine-tuning settings for the English and Chinese tasks are documented separately in Tables 3 and 4.

  • English Tasks: Table 3 presents the detailed fine-tuning experimental settings for English tasks.
  • Chinese Tasks: Table 4 presents the detailed fine-tuning experimental settings for Chinese tasks.

Experimental Results

ERNIE 2.0 consistently improves over prior pre-trained models across English GLUE and Chinese NLP evaluations. Its large model reaches new state-of-the-art results on the reported Chinese tasks.

  • ERNIE 2.0BASE outperforms BERTBASE on all 10 GLUE tasks and obtains a score of 80.6.
  • 83.6 on the GLUE test set gives ERNIE 2.0LARGE a 3.1% improvement over BERTLARGE.
  • ERNIE 2.0LARGE consistently outperforms BERTLARGE and XLNetLARGE on most GLUE dev tasks, except MNLI-m.
  • ERNIE 2.0 significantly outperforms BERTBASE on all nine classical Chinese NLP tasks.The evaluation covers nine classical Chinese NLP tasks.
  • ERNIE 2.0LARGE achieves the best performance and new state-of-the-art results on the Chinese NLP tasks.

Comparison of Different Learning Methods

The paper compares continual multi-task learning with alternative training strategies under matched iteration budgets, but the supplied passage does not report the comparison outcome.

  • The comparison evaluates continual multi-task learning against multi-task learning and traditional continual learning.
  • All methods are evaluated with the same training iterations for each task.Each task receives 50k iterations, for 200k iterations across all tasks in the stated setting.

Conclusion

ERNIE 2.0 incrementally builds and learns pre-training tasks through continual multi-task learning, and the resulting model improves over BERT and XLNet across evaluated benchmarks.

  • ERNIE 2.0 incrementally builds and learns pre-training tasks through continual multi-task learning.
  • The framework covers different aspects of language and produces a model described as more competent in language representation.
  • ERNIE 2.0 obtains significant improvements over BERT and XLNet on GLUE benchmarks and various Chinese tasks.
  • Future work will add more pre-training tasks and investigate other sophisticated continual learning methods.
Loading 1907.12412v2…