Source-linked AI summary
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, Haifeng Wang
TL;DR
Large-scale language models often learn from plain text with auto-regressive objectives, leaving explicit linguistic and world knowledge insufficiently represented for language understanding. ERNIE 3.0 combines auto-regressive and auto-encoding networks, trains a 10-billion-parameter model on text and a knowledge graph, and reports strong results across broad Chinese NLP benchmarks and SuperGLUE. Its NLI gains are limited on translated XNLI, which the authors associate with dataset quality.
Problem
Large-scale models lack explicit linguistic and world knowledge because they are trained on plain texts, while auto-regressive models perform relatively poorly on downstream language understanding with traditional fine-tuning.
Method
ERNIE 3.0 fuses auto-regressive and auto-encoding networks and pre-trains a 10-billion-parameter model on 4TB of plain texts and a large-scale knowledge graph.
Results
ERNIE 3.0 outperforms state-of-the-art pre-trained models across 54 benchmarks and achieves first place on SuperGLUE.
Takeaways & Limitations
The unified framework supports natural language understanding and generation through zero-shot learning, few-shot learning, or fine-tuning.
Takeaways & Limitations
XNLI improvement is limited, which the authors suggest may reflect the poor quality of its English-to-other-language translation.
Abstract
from arXiv · showhide
Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).
1 Introduction
Large-scale pre-trained models improve generalization but typically rely on plain text and auto-regressive training, limiting explicit knowledge integration and downstream language understanding. ERNIE 3.0 addresses these issues with a unified knowledge-enhanced framework that combines auto-regressive and auto-encoding networks.
- Motivation: 175 billion parameters gave GPT-3 strong few-shot and zero-shot performance, extending a trend toward increasingly large pre-trained models.T5 used 10 billion parameters, while Switch-Transformer reached one trillion parameters.
- Motivation: Large-scale models trained on plain texts lack explicit linguistic and world knowledge, while auto-regressive models show poorer downstream understanding after traditional fine-tuning.GPT-3 is described as training on 570GB of filtered Common Crawl texts.
- Approach: ERNIE 3.0 fuses auto-regressive and auto-encoding networks and trains on plain texts plus a large-scale knowledge graph.The training corpus contains 4TB of data.
- Approach: The unified framework supports natural language understanding and generation through zero-shot learning, few-shot learning, or fine-tuning.Customized tasks can be added and trained jointly through multi-task learning.
- Results: 10 billion-parameter ERNIE 3.0 models outperform state-of-the-art models across 54 benchmarks and rank first on SuperGLUE.The contribution passage reports these results without specifying the SuperGLUE score in this section.
2 Related Work
Prior work established large-scale pre-training as a powerful paradigm, while knowledge-enhanced models sought to address the limited world knowledge captured from text alone. ERNIE 3.0 extends these directions with massive multilingual or Chinese-focused model development and knowledge-graph-based pre-training.
- Large-scale Pre-trained Models: BERT helped establish pre-trained language models as a new NLP paradigm, and scaling models such as T5 and GPT-3 improved broad few-shot and zero-shot performance.GPT-3 is described as an auto-regressive model with 175 billion parameters.
- Large-scale Pre-trained Models: Recent non-English efforts include the 2.6-billion-parameter Chinese CPM and the 204-billion-parameter Korean HyperCLOVA.HyperCLOVA used Korean data reported as 6,500 times larger than GPT-3’s.
- Knowledge-enhanced Pre-training: Knowledge-enhanced pre-training addresses the gap between syntactic or semantic knowledge captured from text and missing world knowledge.Knowledge graphs commonly represent entities and relations for integration into pre-trained models.
- Knowledge-enhanced Pre-training: Existing knowledge-enhanced approaches integrate entity and relation information from knowledge graphs or replace entity mentions with other entities during pre-training.The passage names several knowledge-graph integration approaches and WKLM.
3 ERNIE 3.0
ERNIE 3.0 is a continual multi-paradigm unified framework that shares a large universal representation network while separating task-specific representations for understanding and generation. It uses Transformer-XL components to support shared semantic learning and practical task adaptation.
- 3 ERNIE 3.0: ERNIE 3.0 spans natural language understanding, generation, and knowledge extraction through collaborative multi-task pre-training.The framework is described as a Continual Multi-Paradigms Unified Pre-training Framework.
- 3.1 Overview of ERNIE 3.0 Framework: A shared Universal Representation Module extracts universal semantic features, while Task-specific Representation Modules learn features for individual task paradigms.The shared module’s parameters are used across task paradigms, whereas task-specific parameters follow task objectives.
- 3.1.1 Universal Representation Module: The universal backbone is a multi-layer Transformer-XL shared across task paradigms, with recurrence memory used for longer-text modeling.The memory module is valid only for natural language generation tasks.
- 3.1.2 Task-specific Representation Module: Task-specific modules use manageable base-model-sized Transformer-XL networks to capture top-level semantic representations without substantially expanding the large shared model.The design supports fine-tuning only task-specific parameters.
- 3.1.2 Task-specific Representation Module: ERNIE 3.0 contains separate NLU-specific and NLG-specific representation modules, using bidirectional and unidirectional modeling respectively.The passage describes these as two task-specific representation modules.
3.2 Pre-training Tasks
ERNIE 3.0 combines pre-training tasks for lexical, syntactic, semantic, generative, and knowledge-aware learning. Its central knowledge-text task jointly masks graph relations or sentence words to connect structured knowledge with textual context.
- Task Overview: The pre-training task suite targets understanding, generation, reasoning, and knowledge extraction across multiple task paradigms.The framework constructs several tasks to capture different information types from training corpora.
- Knowledge-aware Tasks: Knowledge masked language modeling uses phrase and named-entity masking to learn dependencies across local and global contexts.This task was introduced in ERNIE 1.0.
- Generation Tasks: Document language modeling provides the generative pre-training task, while ERNIE 3.0 uses a traditional language model to reduce network complexity.The task supports natural language generation.
- Structure-aware Tasks: Sentence reordering learns relationships among paragraph segments by asking the model to restore randomly permuted segments.The task is formulated as k-class classification, with k = P_m.
- Structure-aware Tasks: Sentence distance is a three-class task distinguishing adjacent sentences, nonadjacent sentences from the same document, and sentences from different documents.It extends the traditional next sentence prediction task.
- Knowledge-aware Tasks: Universal Knowledge-Text Prediction jointly uses knowledge-graph triples and corresponding encyclopedia sentences, masking either a relation or sentence words.Relation prediction uses textual mentions of head and tail entities, while word prediction incorporates knowledge context.
3.3 Pre-training Process
ERNIE 3.0’s pre-training uses a large, diverse corpus and Transformer-XL-based modules totaling 10 billion parameters. Data processing includes filtering, segmentation, deduplication, and dataset reweighting.
- Corpus construction: 4TB of Chinese corpora spans 11 categories and includes web, question-answering, domain-specific, and knowledge-graph data.The corpus incorporates a Baidu knowledge graph containing more than 50 million facts.
- Data preprocessing: Deduplication operates at character, paragraph, and document levels to reduce repeated content.Character-level processing collapses consecutive identical characters, while paragraph-level processing removes repeated consecutive paragraphs.
- Data preprocessing: Sentences shorter than 10 words are filtered because they may be incomplete and contain limited semantic information.
- Data preprocessing: Sentence segmentation and word segmentation help the model learn sentence boundaries and named-entity knowledge.
- Data preprocessing: Each dataset is multiplied by a user-defined factor after truncation to increase data diversity for NLU-network pre-training.
- Model architecture: The universal and task-specific modules use Transformer-XL backbones, with 48 and 12 layers respectively, totaling 10 billion parameters.The universal module has 4096 hidden units and 64 heads; task-specific modules have 768 hidden units and 12 heads.
4 Experiments
The experiments compare ERNIE 3.0 with four state-of-the-art pre-training models across fine-tuning tasks for language understanding and generation, as well as zero-shot learning.
- Experimental design: ERNIE 3.0 is compared with four state-of-the-art pre-training models using fine-tuning on understanding and generation tasks and zero-shot learning.
4.1 Evaluation Tasks
The evaluation covers 54 NLP tasks, spanning 14 types of natural language understanding and seven types of natural language generation across 54 datasets.
- Scope: 54 NLP tasks evaluate the models’ fine-tuning and zero-shot learning performance.
- Natural language understanding: 45 datasets cover 14 types of natural language understanding tasks, including sentiment analysis, inference, similarity, extraction, and classification.
- Natural language understanding: The understanding-task suite includes sentiment analysis, opinion and event extraction, inference, relation extraction, similarity, reading comprehension, and named-entity recognition.
- Natural language generation: 9 datasets cover seven types of natural language generation tasks, including summarization, question generation, question answering, mathematics, advertising, translation, and dialogue.
4.2 Experiments on Fine-tuning Tasks
Fine-tuning results show ERNIE 3.0 outperforming baselines across a broad range of Chinese understanding and generation tasks, with improvements reported in classification, extraction, comprehension, summarization, question generation, mathematics, advertising, and translation.
- Natural Language Understanding: ERNIE 3.0 substantially improves sentiment-analysis accuracy across four datasets and outperforms SKEP on the reported benchmarks.
- Natural Language Understanding: ERNIE 3.0 outperforms the current state-of-the-art system on all three opinion-extraction datasets.
- Natural Language Understanding: 3.9 and 0.7 accuracy points are reported improvements on OCNLI and XNLI, respectively, with limited improvement on XNLI attributed to translated-data quality.
- Natural Language Understanding: 25.7 points is the reported improvement on the Winograd Schemas Challenge.
- Natural Language Understanding: 2.8 points is the average accuracy improvement across six Chinese news-classification datasets.
- Natural Language Understanding: At least 1.0 EM point improves across five span-prediction reading-comprehension tasks, while multiple-choice accuracy improves by 0.89 points on average.
- Natural Language Generation: 48.46% Rouge-L is achieved on LCSTS, outperforming CPM-2 with a comparable parameter count and ProphetNet-zh.
- Natural Language Generation: 75% accuracy on Math23K exceeds CPM-2’s 69.37% accuracy after fine-tuning to generate postfix expressions.
4.3 Experiments on Zero-shot Learning
Zero-shot experiments compare ERNIE 3.0 with large-scale language models across classification, completion, reading comprehension, question answering, and generation tasks. Results are generally strong, though zero-shot semantic similarity and natural language inference remain limited.
- Overall zero-shot performance: ERNIE 3.0 achieves strong zero-shot performance against large-scale models on most downstream tasks.The comparison includes CPM-1, PanGu-α-2.6B, and PanGu-α-13B.
- Classification: ERNIE 3.0 reaches competitiveness with fine-tuning approaches on TNEWS and performs slightly well on IFLYTEK.The evaluation samples three negative labels and compares per-token perplexity among four choices.
- Remaining limitations: Zero-shot semantic similarity exceeds baselines by a large margin but remains only slightly above random guessing, while zero-shot NLI is comparable to baselines.The authors attribute the semantic-similarity limitation partly to suboptimal prompting and identify substantial room for improvement in zero-shot NLI.
- Coreference resolution: ERNIE 3.0 improves WSC2020 performance by 3.38 percentage points over PanGu-α-13B.WSC2020 is formalized as a multi-choice completion task using per-token perplexity.
- Cloze and completion: ERNIE 3.0 achieves the best CHID score, much lower Chinese WPLC perplexity than PanGu-α, and large-margin gains on other cloze tasks.The evaluated tasks include CHID, Chinese WPLC, CMRC2019, PD, CFT, and CMRC2017.
- Machine reading comprehension: ERNIE 3.0 outperforms baselines by a large margin on CMRC2018, DRCD, and DuReader machine reading comprehension datasets.The evaluation uses both multiple-choice and generation-based prompting across reading-comprehension formats.
- Closed-book question answering: ERNIE 3.0 achieves better closed-book question-answering performance than baselines without additional evidence.The tasks require answers generated from knowledge learned during pre-training.
- Generation quality: Human evaluation scores zero-shot generations on coherence, fluency, and accuracy across 450 cases from 13 tasks.Annotators use a 0–2 scale, with average scores reported for the three dimensions.
4.4 Experiments on SuperGLUE
ERNIE 3.0 is evaluated on the diverse SuperGLUE benchmark, covering question answering, inference, reasoning, comprehension, word-sense disambiguation, and coreference. Its English version surpasses T5 and DeBERTa with a score of 90.6.
- Benchmark scope: SuperGLUE evaluates ERNIE 3.0 across a diverse set of natural language understanding datasets.The benchmark includes BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC.
- Task examples: The benchmark spans textual entailment and word-sense disambiguation tasks evaluated with accuracy.RTE measures textual entailment, while WiC is binary classification of sentence pairs.
- Task examples: SuperGLUE also includes coreference resolution through WSC, where the model selects the correct pronoun referent from candidate noun phrases.WSC is evaluated with accuracy.
- Results: 90.6 is ERNIE 3.0’s SuperGLUE score, surpassing T5 and DeBERTa and taking first place on the benchmark.The reported leaderboard results were recorded on July 3, 2021.
5 Analysis
The analysis examines task-specific representation modules, universal knowledge-text prediction, and progressive learning. Results support separate task-specific networks, knowledge enhancement, and faster convergence under the tested settings.
- Task-specific representation modules: Task-specific networks converge faster and develop a larger performance gap than shared task-specific networks as training progresses.The ablation compares understanding and generation paradigms using a unified base model.
- Universal knowledge-text prediction: Knowledge enhancement achieves impressive empirical performance on relation extraction tasks using universal knowledge-text prediction.The experiments use SanWen and FinRE and mark head and tail entities with special tokens.
- Universal knowledge-text prediction: Zero-shot CKBQA experiments further evaluate whether universal knowledge-text prediction supports knowledge learning from a knowledge graph.The task requires searching and reasoning for answers based on a knowledge graph.
- Progressive learning: ERNIEBase convergence time is reduced by 65.21%, from 11 hours to 4 hours, under progressive learning.The ERNIE1.5B setting reports a 48% reduction, with both experiments using 8 NVIDIA Tesla V100 GPUs.
6 Conclusion
The conclusion presents ERNIE 3.0 as a knowledge-enhanced 10-billion-parameter framework trained on plain text and knowledge-graph data. Its unified architecture targets understanding and generation across zero-shot, few-shot, and fine-tuning settings.
- Conclusion: ERNIE 3.0 pre-trains a knowledge-enhanced 10-billion-parameter model on a 4TB corpus containing plain texts and a knowledge graph.The framework integrates auto-encoder and auto-regressive networks.
- Conclusion: The unified framework handles language understanding and generation through zero-shot learning, few-shot learning, and fine-tuning.The conclusion bases this claim on experiments across varied datasets and task paradigms.
- Conclusion: Experiments demonstrate ERNIE 3.0’s effectiveness compared with previous state-of-the-art pre-trained models.