Source-linked AI summary
ERNIE: Enhanced Language Representation with Informative Entities
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu
TL;DR
Pretrained language models capture rich textual patterns but rarely incorporate structured knowledge from knowledge graphs, creating a gap for knowledge-driven language understanding. ERNIE combines text and knowledge-graph entities through entity-aware representation learning and pretraining, significantly improving knowledge-driven tasks while remaining comparable to BERT on common NLP tasks.
Problem
Pretrained language models rarely incorporate knowledge-graph information, despite the structured facts it provides for language understanding.
Method
ERNIE aligns text mentions with knowledge-graph entities, integrates entity embeddings into language representations, and trains with a denoising entity auto-encoder objective.
Results
ERNIE significantly outperforms BERT on entity typing and relation classification and achieves comparable results on other common NLP tasks.
Takeaways & Limitations
ERNIE demonstrates that combining lexical, syntactic, and knowledge information supports stronger knowledge-driven language understanding while preserving broad NLP performance.
Takeaways & Limitations
The model represents tokens at the subword level.
Abstract
from arXiv · showhide
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code of this paper can be obtained from https://github.com/thunlp/ERNIE.
1 Introduction
ERNIE addresses the limited incorporation of structured knowledge in pretrained language models by integrating textual corpora, knowledge graphs, and entity-aware pretraining. It is designed to improve knowledge-driven understanding while retaining strong performance on common NLP tasks.
- Motivation: Pretrained language models capture textual information but rarely incorporate knowledge graphs, limiting access to structured facts for language understanding.The paper illustrates that missing entity knowledge can hinder entity typing and fine-grained relation classification.
- Challenges: External-knowledge integration poses challenges in encoding relevant graph facts and fusing lexical, syntactic, and knowledge information across vector spaces.The paper identifies structured knowledge encoding and heterogeneous information fusion as separate challenges.
- Approach: ERNIE pretrains a language representation model on large-scale textual corpora and knowledge graphs.It aligns named entity mentions with knowledge-graph entities and feeds informative entity embeddings into the model.
- Approach: ERNIE combines masked language modeling and next sentence prediction with a denoising objective that masks entity alignments and predicts appropriate knowledge-graph entities.The objective requires aggregating contextual and knowledge information to predict both tokens and entities.
- Results: ERNIE significantly outperforms BERT on entity typing and relation classification while achieving comparable results on other common NLP tasks.These experiments evaluate both knowledge-driven tasks and broader NLP tasks.
2 Related Work
Prior language representation methods learn from text through feature-based or fine-tuning approaches, but related work also motivates incorporating external knowledge and jointly representing words and entities.
- Language Representation Pretraining: Pretraining approaches are broadly divided into feature-based methods and fine-tuning methods.Both classes aim to capture language information from text for downstream NLP applications.
- Feature-Based Approaches: Feature-based approaches transform words into distributed representations used as input embeddings or initialization parameters for later NLP models.These representations capture syntactic and semantic information from textual corpora.
- Fine-Tuning Approaches: Fine-tuning approaches pretrain model architectures and parameters on unlabeled text before adapting them to specific NLP models.This line of work builds on pretrained models as starting points for downstream tasks.
- Knowledge Integration: Despite their success, feature-based and fine-tuning models generally ignore knowledge information.The paper contrasts this limitation with prior evidence that extra knowledge benefits several NLP applications.
- Knowledge Integration: Prior work has explored injecting external knowledge and jointly learning word and entity representations to leverage knowledge graphs.These efforts span tasks including reading comprehension, machine translation, inference, knowledge acquisition, and dialogue.
3 Methodology
ERNIE combines a textual encoder with a knowledgeable encoder and pre-trains their representations using token, entity, and alignment information. It then adapts these representations to common and knowledge-driven NLP tasks through task-specific fine-tuning.
- Entity representation: Entity alignment maps a named entity to the first token in its phrase, while unaligned tokens remain without corresponding KG entities.The token and entity sequences can have different lengths because not every token aligns to an entity.
- Model architecture: ERNIE represents tokens and aligned KG entities jointly by combining a BERT-style textual encoder with a knowledgeable encoder.The textual encoder captures lexical and syntactic features, while the knowledgeable encoder fuses token and entity representations.
- Knowledge fusion: The knowledgeable encoder uses stacked aggregators with multi-head self-attention and information fusion to integrate token and entity features.Its top aggregator produces the final output embeddings for both tokens and entities.
- Pre-training objectives: The denoising entity auto-encoder masks token-entity alignments and predicts the corresponding entities using the given entity sequence.ERNIE also randomly replaces 5% of alignments and masks 15% to address incorrect or incomplete entity linking.
- Pre-training and fine-tuning: ERNIE combines the dEA, masked language model, and next sentence prediction losses to learn knowledge, lexical, and syntactic information.For downstream tasks, it uses BERT-like fine-tuning, with marked entity mentions for relation classification and entity typing.
4 Experiments
ERNIE is evaluated on knowledge-driven and common NLP tasks using pre-training and fine-tuning experiments. It improves over BERT on entity typing and relation classification while remaining comparable on GLUE.
- Experimental Setup: ERNIE is evaluated on entity typing, relation classification, and common NLP tasks, including GLUE.The experiments use five NLP datasets spanning knowledge-driven and common-language benchmarks.
- Parameter Settings and Training Details: ERNIE combines BERT initialization with multi-task pre-training on annotated text aligned to Wikidata.The corpus contains nearly 4,500M subwords and 140M entities; entity embeddings are trained from Wikidata facts using TransE.
- Entity Typing: ERNIE significantly improves FIGER strict accuracy over BERT, indicating that external knowledge helps avoid fitting noisy distant-supervision labels.FIGER is distantly supervised for training but human-annotated for testing.
- Entity Typing: On Open Entity, ERNIE improves precision by 2% and recall by 2% over BERT, and outperforms baselines on gold annotations.Open Entity is completely manually annotated.
- Relation Classification: ERNIE increases FewRel F1 by 3.4% over BERT and raises TACRED F1 by nearly 2.0%, achieving the best recall and F1 scores on TACRED.Pre-training models also improve FewRel F1 by at least 15% over the CNN baseline, while extra knowledge is especially useful with smaller training data.
- Common NLP Tasks: ERNIE achieves results comparable to BERTBASE on GLUE, though performance is less stable on smaller datasets.It is better on CoLA and RTE but worse on STS-B and MRPC.
5 Conclusion
ERNIE incorporates knowledge information into language representation through a knowledgeable aggregator and the dEA pre-training task. Experiments show stronger denoising of distantly supervised data and fine-tuning on limited data than BERT.
- ERNIE incorporates knowledge information into language representation models using a knowledgeable aggregator and the dEA pre-training task.
- ERNIE demonstrates better abilities than BERT for denoising distantly supervised data and fine-tuning on limited data.
- Future work includes injecting knowledge into feature-based models, adding diverse structured knowledge, and building larger heuristically annotated pre-training corpora.