Source-linked AI summary
ERNIE: Enhanced Representation through Knowledge Integration
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, Hua Wu
TL;DR
ERNIE addresses the limited incorporation of prior knowledge into pretrained language representations. It uses phrase- and entity-level masking during pretraining, and reports state-of-the-art results across five Chinese NLP tasks together with stronger cloze-test knowledge inference.
Problem
Prior language representation methods predict missing words from context without explicitly considering prior knowledge about entities and their relationships.
Method
ERNIE integrates phrase- and entity-level knowledge through multi-stage masking, masking complete conceptual units so their information is learned during language-model training.
Results
ERNIE advances state-of-the-art results on five Chinese NLP tasks and shows better knowledge inference capacity than strong baselines on a cloze test.
Takeaways & Limitations
Knowledge integration and heterogeneous-data pretraining enable ERNIE to obtain better language representations than BERT across the evaluated Chinese tasks.
Abstract
from arXiv · showhide
We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.
1 Introduction
ERNIE addresses the limited use of prior knowledge in context-based language representation by masking phrases and entities as complete units. It reports improvements across Chinese NLP tasks and stronger knowledge inference.
- Prior representation methods predict missing words from context without explicitly considering prior knowledge about entities and their relationships.The Harry Potter example illustrates why local word cooccurrence may be insufficient for relationship-based prediction.
- ERNIE uses phrase-level and entity-level masking to implicitly learn knowledge and longer semantic dependencies during pretraining.All words in a selected phrase or entity are masked together rather than masking only one word or character.
- ERNIE is pretrained on heterogeneous Chinese data and applied to five Chinese NLP tasks, where it advances state-of-the-art results.A cloze test additionally reports better knowledge inference capacity than strong baseline methods.
- ERNIE introduces unit-level masking to learn syntactic and semantic information from phrases and entities.
- ERNIE significantly outperforms previous state-of-the-art methods on various Chinese natural language processing tasks.
- The authors report released ERNIE code and pretrained models for public use.
2 Related Work
Related work develops contextual and transferable language representations, while BERT adds masked language modeling and sentence-pair prediction. Other approaches incorporate tasks, languages, or heterogeneous data into pretraining.
- Early representation methods learn context-independent word vectors from large unlabeled corpora, assigning one embedding to each vocabulary word.
- Context-sensitive approaches improve representations by modeling sentence context, transfer learning, or Transformer-based language-model features.The cited examples include Skip-thought, Cove, ULMFit, ELMo, and GPT.
- BERT randomly masks words and predicts whether two sentences are adjacent, targeting both masked-word prediction and sentence relationships.
- Other models add multitask information, task information, or language embeddings to pretrained representations for broader task transfer.MT-DNN, GPT-2, and XLM represent these directions.
- Heterogeneous unsupervised data can support transfer learning, with prior work combining sources such as Wikipedia, news, web QA, forums, or parallel corpora.
3 Methods
ERNIE uses a Transformer encoder and a multi-stage masking procedure that progresses from basic units to phrases and named entities. The procedure integrates higher-level semantic information into language representations.
- Transformer Encoder: ERNIE uses a multilayer Transformer encoder to capture contextual information for each token and generate contextual embeddings.
- Transformer Encoder: Input representations combine token, segment, and position embeddings after Chinese text is tokenized with WordPiece.
- Knowledge Masking: ERNIE integrates phrase- and entity-level knowledge through multi-stage masking rather than directly adding knowledge embeddings.
- Basic-Level Masking: Basic-level masking randomly masks 15 percent of basic language units and trains the Transformer to predict them from the remaining units.Basic units are words for English and Chinese characters for Chinese; this stage alone makes high-level semantic knowledge difficult to model fully.
- Phrase-Level Masking: Phrase-level masking selects phrases and masks all basic units within each phrase so phrase information is encoded into word embeddings.Phrase boundaries are obtained using language-analysis and segmentation tools.
- Entity-Level Masking: Entity-level masking analyzes named entities and masks and predicts all slots within each entity before producing representations with richer semantic information.
4 Experiments
The experiments use a BERT-base-sized ERNIE pretrained on heterogeneous Chinese corpora. The supplied passages describe the corpus sources and preprocessing, but not task-result values.
- ERNIE matches BERT-base in model size, using 12 encoder layers, 768 hidden units, and 12 attention heads.
- ERNIE is pretrained on a heterogeneous corpus combining Chinese Wikipedia, Baidu Baike, Baidu news, and Baidu Tieba.The sources contain 21M, 51M, 47M, and 54M sentences, respectively.
- The preprocessing converts traditional Chinese characters to simplified characters and English letters to lowercase.
- The model uses a shared vocabulary of 17,964 Unicode characters.
4.2 DLM
ERNIE models query–response structure with dialogue embeddings and masked prediction, extending beyond single-turn role representations to multi-turn conversations.
- Dialogue Language Model: Dialogue embeddings identify query and response roles in the Dialogue Language Model task.They serve a role analogous to BERT token-type embeddings.
- Dialogue Language Model: The task masks words and predicts them using both query and response context.This parallels masked language modeling while conditioning on dialogue structure.
- Dialogue Language Model: ERNIE represents multi-turn conversation patterns including QRQ, QRR, and QQR.Here, Q and R denote query and response, respectively.
- Dialogue Language Model: Randomly replacing a query or response creates fake samples for judging whether a multi-turn conversation is coherent.The replacement objective adds dialogue-level relationship learning to masked prediction.
- Dialogue Language Model: The Dialogue Language Model task is pre-trained alternately with masked language modeling because their architectures are compatible.The task is intended to learn implicit relationships in dialogues and enhance semantic representation.
4.3 Experiments on Chinese NLP Tasks
ERNIE is evaluated on five Chinese NLP tasks spanning inference, similarity, entity recognition, sentiment classification, and question answering.
- Task Coverage: ERNIE is applied to natural language inference, semantic similarity, named entity recognition, sentiment analysis, and question answering.These tasks cover several common Chinese language-understanding settings.
- Natural Language Inference: XNLI evaluates textual entailment using contradiction, neutral, and entailment labels in Chinese-translated sentence pairs.The corpus is a crowdsourced extension of MultiNLI translated into 14 languages, including Chinese.
- Semantic Similarity: LCQMC tests whether two Chinese sentences express the same intention through binary classification.Each sentence pair carries a binary label indicating whether the intentions match.
- Named Entity Recognition: MSRA-NER frames recognition of person, place, organization, and other entity types as sequence labeling.The dataset was published by Microsoft Research Asia.
- Sentiment Analysis: ChnSentiCorp classifies sentences as positive or negative across comments about hotels, books, and electronic computers.The dataset targets sentence-level sentiment judgment.
- Question Answering: NLPCC-DBQA selects answers for corresponding questions and evaluates performance with MRR and F1 score.The task is answer selection rather than free-form answer generation.
4.4 Experiment results
ERNIE outperforms BERT across all five Chinese NLP tasks and establishes new state-of-the-art results.
- Experiment Results: ERNIE outperforms BERT on all five Chinese NLP tasks, creating new state-of-the-art results.The tasks span inference, semantic similarity, named entity recognition, sentiment analysis, and question answering.
- Experiment Results: More than 1% absolute accuracy improvement over BERT is reported for XNLI, MSRA-NER, ChnSentiCorp, and nlpcc-dbqa.The passage identifies four tasks with this improvement threshold.
- Experiment Results: The reported gains are attributed to ERNIE’s knowledge integration strategy.The comparison is based on test results presented in Table 1.
4.5 Ablation Studies
Ablation studies examine knowledge masking and dialogue pre-training, showing incremental gains from phrase and entity masking and benefits from the Dialogue Language Model task.
- Knowledge Masking: Adding phrase-level masking to word-level masking improves model performance on 10% of the training data.The experiment samples 10% of the full corpus to test the knowledge masking strategy.
- Knowledge Masking: Adding entity-level masking after phrase-level masking further improves performance.The ablation isolates the incremental contribution of the two knowledge masking strategies.
- Knowledge Masking: 0.8% performance gain is achieved with a pre-training dataset ten times larger on XNLI.This result is reported alongside the masking ablation.
- Dialogue Language Model: 0.7%/1.0% improvement in develop/test accuracy is achieved by the Dialogue Language Model task on XNLI.The result averages fine-tuning outcomes from five random restarts.
4.6 Cloze Test
The cloze test examines whether ERNIE can infer removed named entities, comparing its predictions with BERT. ERNIE generally performs better in context-based knowledge reasoning, including correct entity completion and semantic-type prediction.
- The experiment removes named entities from paragraphs and compares BERT’s and ERNIE’s predictions.The task tests whether the models can infer the missing entity from context.
- ERNIE uses relationship knowledge in the article, whereas BERT tends to copy names appearing in the context in one case.
- In cases 2 and 5, ERNIE fills slots with the correct entities while BERT identifies the entity type but misses the exact entity.
- In cases 3, 4, and 6, ERNIE predicts the correct entities except in case 4 and still identifies an Australian city and the semantic type there.
- Overall, the cases show that ERNIE performs better than BERT in context-based knowledge reasoning.
5 Conclusion
The paper concludes that ERNIE integrates knowledge into a pre-training language model and outperforms BERT across five Chinese language-processing tasks. Future work will extend the approach to other knowledge types and languages.
- ERNIE integrates knowledge into a pre-training language model and outperforms BERT across five Chinese language-processing tasks.
- The authors plan to integrate syntactic parsing and weakly supervised signals from other tasks, and to validate the idea in other languages.