PaperlayerView exact revision on arXiv

Source-linked AI summary

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang

arXiv:1907.12412v2cs.CL

TL;DR

Existing pre-training often emphasizes word and sentence co-occurrence despite additional lexical, syntactic, and semantic information in corpora. ERNIE 2.0 incrementally builds diverse tasks through continual multi-task learning, and it outperforms BERT and XLNet across 16 English and Chinese tasks.

  • Problem

    Current pre-training focuses on co-occurrence while valuable lexical, syntactic, and semantic information such as named entities and discourse relations remains to be examined.

    Source: Introduction · p. 1

  • Method

    ERNIE 2.0 incrementally constructs and learns diverse unsupervised pre-training tasks through continual multi-task learning.

    Source: Introduction · p. 1, Continual Multi-task Learning · p. 3

  • Results

    ERNIE 2.0 significantly improves over BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.

    Source: Introduction · p. 1, Conclusion · p. 7

  • Takeaways & Limitations

    The framework supports learning lexical, syntactic, and semantic representations through incrementally added pre-training tasks.

    Source: Introduction · p. 1, Conclusion · p. 7

Abstract

from arXiv · show

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

Introduction

ERNIE 2.0 addresses the limits of co-occurrence-focused pre-training by incrementally constructing diverse tasks through continual multi-task learning. The framework targets lexical, syntactic, and semantic information and improves performance across English and Chinese language understanding tasks.

Motivation

  • Existing pre-training commonly models word and sentence co-occurrence, while training corpora also contain valuable lexical, syntactic, and semantic information.Examples include named entities, sentence order, sentence proximity, semantic similarity, and discourse relations.Source: Introduction · p. 1

Approach

  • ERNIE 2.0 incrementally builds and trains a large variety of pre-training tasks through continual multi-task learning.The framework supports introducing new tasks while retaining previously learned knowledge.Source: Introduction · p. 1

Contributions

  • The framework supports customized training tasks and continual multi-task learning in an incremental way.This is the stated core contribution of ERNIE 2.0.Source: Introduction · p. 1

Results

  • ERNIE 2.0 achieves significant improvements over BERT and XLNet on 16 tasks, including English GLUE benchmarks and several Chinese tasks.The authors evaluate the framework across both English and Chinese language understanding settings.Source: Introduction · p. 1

Availability

  • Fine-tuning code and models pre-trained on English corpora are released through the PaddlePaddle/ERNIE repository.The release supports reproduction and downstream use of the framework.Source: Introduction · p. 1

Related Work

Prior work established the value of large-scale pre-training, contextualized representations, and continual learning. ERNIE 2.0 extends these directions by combining diverse pre-training objectives with continual task learning.

Contextualized Pre-training

  • Pre-training on large amounts of unannotated data has improved language representations beyond traditional fixed word embeddings based on word co-occurrence.Contextualized approaches such as ELMo, GPT, BERT, and related models learn context-dependent representations.Source: Unsupervised Learning for Language… · pp. 1–2

ERNIE 2.0

  • ERNIE 2.0 is presented as a framework that continually introduces a large variety of pre-training tasks to learn lexical, syntactic, and semantic representations.Its framework combines the pre-training-and-fine-tuning architecture with continual multi-task learning.Source: Figure 1 · p. 2

Contextualized Pre-training

  • BERT combines masked language modeling with next sentence prediction, while earlier contextualized models use other language-modeling or Transformer-based strategies.These methods motivate expanding the objectives used during pre-training.Source: Unsupervised Learning for Language… · p. 2

Continual Learning

  • Continual learning trains models on several tasks in sequence while aiming to retain previously learned tasks.Its motivation is cumulative acquisition of information and skills.Source: Continual Learning · p. 2

The ERNIE 2.0 Framework

The ERNIE 2.0 framework constructs unsupervised tasks incrementally and updates a shared model through continual multi-task learning. It then fine-tunes the resulting representation model for downstream language understanding tasks.

Framework

Continual Multi-task Learning

  • The framework addresses forgetting and efficiency by initializing new-task training from prior parameters, training new and original tasks together, and allocating iterations across stages.This design preserves previously learned knowledge while distributing training effort across tasks.Source: Continual Multi-task Learning · p. 3
  • Each learning stage uses shared encoding layers whose parameters are updated across tasks, with recurrent neural networks or deep Transformers as possible encoders.Sentence-level and token-level losses support the pre-training tasks.Source: Continual Multi-task Learning · p. 3

Fine-tuning

  • Fine-tuning adapts the pre-trained model to downstream tasks such as question answering, natural language inference, and semantic similarity.Each downstream task receives its own fine-tuned model.Source: Fine-tuning for Application Tasks · pp. 3–4

Task Construction

  • Different pre-training datasets can generate different tasks, and one dataset type can correspond to multiple pre-training tasks.This dataset-task relationship supports incremental task construction.Source: Table 1 · p. 4

ERNIE 2.0 Model

The ERNIE 2.0 model uses a Transformer encoder and task-aware input representations to support multiple word-, structure-, and semantic-aware pre-training tasks. These tasks target lexical information, document structure, sentence relations, and short-text relevance.

Model Structure

  • ERNIE 2.0 uses a multi-layer Transformer as its basic encoder to produce contextual embeddings through self-attention.The input sequence includes the special classification embedding [CLS].Source: Transformer Encoder · p. 4
  • Task embeddings encode task identity alongside token, segment, and position embeddings, allowing task-specific inputs during pre-training and fine-tuning.Each task id is assigned a unique task embedding.Source: Task Embedding · p. 4

Pre-training Tasks

  • The model constructs word-aware, structure-aware, and semantic-aware tasks to capture lexical, syntactic, and semantic information.The three task categories organize the different pre-training objectives.Source: Pre-training Tasks · p. 4

Structure-aware Tasks

  • Sentence reordering randomly permutes paragraph segments and trains the model to reconstruct their order as a k-class classification problem.The task is intended to learn relationships among sentences in a document.Source: Sentence Reordering Task · p. 4
  • Sentence distance classifies whether two sentences are adjacent in one document, nonadjacent in one document, or from different documents.The task uses three document-level distance classes.Source: Sentence Distance Task · p. 4

Semantic-aware Tasks

  • The short-text relevance task classifies query-title pairs as strongly relevant, weakly relevant, or irrelevant using commercial search-engine logs.Strong relevance corresponds to titles clicked after users entered the query; weak relevance corresponds to displayed but unclicked titles.Source: IR Relevance Task · p. 5

Experiments

The experiments compare ERNIE 2.0 with established pre-training models on English GLUE tasks and Chinese datasets, while also examining pre-training dataset sizes.

  • ERNIE 2.0 is compared with BERT and XLNet on English GLUE tasks, and with BERT and ERNIE 1.0 on several Chinese datasets.Source: Experiments · p. 5
  • The experiments also compare ERNIE 2.0 with multi-task learning and traditional continual learning.Source: Experiments · p. 5
  • Table 2 reports the sizes of the pre-training datasets.Source: Table 2 · p. 5

Pre-training and Implementation

The pre-training setup combines English and Chinese corpora with Transformer configurations aligned to BERT, using substantial GPU resources and fixed optimization settings.

Pre-training Data

  • English data include Wikipedia, BookCorpus, Reddit, and Discovery discourse-relation data, while Chinese data span encyclopedia, news, dialogue, retrieval, and discourse-relation sources.Source: Pre-training Data · p. 5

Model Settings

  • The base Transformer has 12 layers, 12 self-attention heads, and 768-dimensional hidden states; the large model has 24 layers, 16 heads, and 1024-dimensional hidden states.Source: Pre-training Settings · p. 5

Fine-tuning Settings

Training Resources

Optimization

  • Adam optimization uses β1 = 0.9, β2 = 0.98, and a batch size of 393216 tokens, with language-specific learning rates and 4,000 warmup steps.Source: Pre-training Settings · p. 5

Fine-tuning Tasks

Evaluation covers GLUE and nine Chinese NLP tasks spanning comprehension, recognition, inference, similarity, sentiment, and question answering, alongside continual-pre-training comparisons.

English Evaluation

  • GLUE provides a multi-task benchmark for evaluating natural language understanding performance.Source: English Task · p. 5

Chinese Evaluation

  • The Chinese evaluation includes nine NLP tasks across machine reading comprehension, named entity recognition, inference, semantic similarity, sentiment analysis, and question answering.Source: Chinese Tasks · pp. 5, 7

Results and Analysis

Chinese Evaluation

  • Chinese machine reading comprehension uses CMRC 2018, DRCD, and DuReader, while semantic similarity uses LCQMC and BQ Corpus.Source: Chinese Tasks · p. 7
  • Named entity recognition, inference, sentiment analysis, and question answering use MSRA-NER, XNLI, ChnSentiCorp, and NLPCC-DBQA, respectively.Source: Chinese Tasks · p. 7

Implementation Details for Fine-tuning

Fine-tuning settings for the English and Chinese tasks are documented separately in Tables 3 and 4.

English Tasks

Chinese Tasks

Experimental Results

ERNIE 2.0 consistently improves over prior pre-trained models across English GLUE and Chinese NLP evaluations. Its large model reaches new state-of-the-art results on the reported Chinese tasks.

Comparison of Different Learning Methods

The paper compares continual multi-task learning with alternative training strategies under matched iteration budgets, but the supplied passage does not report the comparison outcome.

Conclusion

ERNIE 2.0 incrementally builds and learns pre-training tasks through continual multi-task learning, and the resulting model improves over BERT and XLNet across evaluated benchmarks.

  • ERNIE 2.0 incrementally builds and learns pre-training tasks through continual multi-task learning.Source: Conclusion · p. 7
  • The framework covers different aspects of language and produces a model described as more competent in language representation.Source: Conclusion · p. 7
  • ERNIE 2.0 obtains significant improvements over BERT and XLNet on GLUE benchmarks and various Chinese tasks.Source: Conclusion · p. 7
  • Future work will add more pre-training tasks and investigate other sophisticated continual learning methods.Source: Conclusion · p. 7

Related papers

Loading 1907.12412v2…