Source-linked AI summary
Knowledge Enhanced Contextual Word Representations
Matthew E. Peters, Mark Neumann, Robert L. Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
TL;DR
Contextual representations lack explicit grounding to real-world entities and often fail to retain factual knowledge. KnowBert integrates multiple knowledge bases into BERT with entity linking, word-to-entity attention, and joint multitask training. With WordNet and Wikipedia, it improves intrinsic and downstream evaluations while maintaining comparable runtime and scaling to large knowledge bases.
Problem
Contextual word representations trained on raw text lack explicit grounding to real-world entities and often have difficulty recovering factual knowledge.
Method
KnowBert inserts multiple knowledge bases into BERT using entity linking, word-to-entity attention, and jointly trained self-supervised language modeling.
Results
KnowBert improves masked-language-model quality, factual recall, relationship extraction, entity typing, and word sense disambiguation across intrinsic and downstream evaluations.
Takeaways & Limitations
Knowledge-enhanced representations can be applied across downstream tasks while adding minimal runtime and parameter overhead and supporting large knowledge bases.
Takeaways & Limitations
Entity-linking performance lags the current state of the art on AIDA, partly because candidate generation includes common nouns beyond AIDA’s named-entity annotations.
Abstract
from arXiv · showhide
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update contextual word representations via a form of word-to-entity attention. In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation. KnowBert's runtime is comparable to BERT's and it scales to large KBs.
1 Introduction
KnowBert inserts multiple knowledge bases into pretrained models through entity-aware attention and jointly trained entity linkers. The resulting representations are general-purpose, efficient, and improve intrinsic and downstream evaluations.
- Method: KnowBert uses Knowledge Attention and Recontextualization to retrieve entity embeddings and update contextual word representations through word-to-entity attention.KAR models entity spans, retrieves relevant entities, and is inserted between two middle layers of BERT.
- Training and transfer: The approach jointly trains entity linkers with self-supervised language modeling on unlabeled data, producing representations usable across downstream tasks.Training combines limited entity-linking supervision with large amounts of raw text.
- Efficiency and extensibility: KAR leaves BERT’s top layers unchanged, adds minimal parameters and runtime, and supports incorporating additional knowledge bases at other locations.These design choices allow KnowBert to replace BERT in downstream applications while retaining output loss layers.
- Knowledge bases: KnowBert supports varied knowledge-base forms and experiments with WordNet and Wikipedia for word senses and named-entity facts.The framework can also be extended to commonsense or domain-specific knowledge bases.
- Results: KnowBert improves masked-language-model perplexity, factual recall, relationship extraction, entity typing, and word sense disambiguation, including comparisons against BERTLARGE.The evaluations combine intrinsic and extrinsic tasks, with improvements reported across the listed datasets.
2 Related Work
Prior work developed contextual embeddings, entity embeddings, entity-aware language models, and task-specific knowledge-base architectures. KnowBert builds on these lines by learning transferable knowledge-enhanced representations from unannotated text.
- Pretrained word representations: KnowBert extends context-sensitive word representations by incorporating structured knowledge into pretrained language models.This work follows the field’s shift from static word vectors toward context-sensitive embeddings.
- Entity embeddings: Entity embedding methods represent external knowledge sources continuously, using translational-distance or similarity-based linear models.KnowBert is agnostic to the details of the entity embedding method.
- Entity-aware language models: Earlier entity-aware language models required full annotation, latent-variable marginalization, or entity-centric modeling, whereas KnowBert trains with large unannotated text collections.The contrast concerns training requirements for integrating knowledge into language models.
- Task-specific KB architectures: Task-specific architectures integrate knowledge bases for individual downstream tasks, while KnowBert learns transferable representations for multiple tasks.The distinction is between task-supervised integration and general-purpose representation learning.
3 KnowBert
KnowBert extends BERT with KAR, which identifies candidate mention spans, retrieves and integrates entity embeddings, and recontextualizes word representations with entity-span attention. The component supports entity linking, knowledge injection, and end-to-end multitask training while preserving much of BERT’s architecture.
- KAR overview: KAR inserts knowledge into BERT by projecting word-piece representations, pooling candidate mention spans, linking entities, and recontextualizing words with entity-span attention.The component produces enhanced representations that are passed into the next pretrained BERT layer.
- Training procedure: KnowBert jointly trains entity linkers and language modeling in a multitask regime, while supporting multiple KBs at different layers.The training procedure incrementally pretrains larger portions of the model before fine-tuning trainable parameters with available entity-linking supervision.
- Mention-span representations: Mention-span representations are formed by linearly projecting BERT states to the entity dimension and self-attentively pooling word pieces within each candidate span.The resulting span vectors are stacked into a C × E matrix for subsequent linking.
- Entity linker: The entity linker uses mention-span self-attention and candidate entity embeddings to make linking decisions using global span information and entity-entity cooccurrence.Candidate scores combine span–entity dot products with candidate priors, and a small transformer layer contextualizes the spans.
- Knowledge injection: KnowBert filters candidates below a score threshold, softmax-normalizes the remaining scores, and computes a weighted entity embedding for each span.If every candidate falls below the threshold, the model substitutes a special NULL embedding.
- Recontextualization: Word-to-entity-span attention uses projected word-piece states as queries and knowledge-enhanced span representations as keys and values.This replaces the corresponding self-attention interaction before the remaining BERT layers run normally.
4 Experiments
KnowBert integrates Wikipedia and WordNet into BERT and improves intrinsic and downstream evaluations while retaining efficient runtime. Experiments examine knowledge-base construction, training data, perplexity, factual recall, and multiple downstream tasks.
- Training and setup: The experiments train KnowBert-Wiki, KnowBert-WordNet, and KnowBert-W+W from English uncased BERTBASE using Wikipedia and Books Corpus text.The models insert knowledge components near the top BERT layers, with KnowBert-W+W placing WordNet after Wikipedia.
- Knowledge bases: KnowBert-Wiki uses approximately 470K pretrained Wikipedia entity embeddings, while KnowBert-WordNet combines synset and lemma metadata with a relational graph.Wikipedia embeddings are 300-dimensional; WordNet embeddings combine graph-based 200-dimensional vectors with 2048-dimensional gloss vectors.
- Intrinsic evaluation: KnowBert improves masked language model perplexity over BERTLARGE despite being derived from BERTBASE and significantly improves factual recall from Wikidata tuples.The factual-recall evaluation masks either the subject or object in natural-language relation instances and measures masked word-piece mean reciprocal rank.
- Efficiency: KnowBert is 8% slower than BERTBASE for KnowBert-Wiki and 32% slower for KnowBert-W+W, while remaining much faster than BERTLARGE.Runtime is independent of KB size because only a fixed small candidate set is considered for each input; scaling mainly increases embedding memory.
- Relation extraction: KnowBert-W+W significantly outperforms comparable BERTBASE systems on TACRED by 3.5% and improves over BERTLARGE by 1.4%.On SemEval 2010 Task 8, its F1 falls between the cited entity-aware BERTBASE and BERTLARGE systems.
- Downstream tasks: KnowBert-W+W sets a new state of the art on WiC, improving over BERTLARGE by 1.4%, and improves entity typing F1 by 0.6% over ERNIE and 2.5% over BERTBASE.The WiC result also reduces the relative gap to 80% human performance by 13.3%.
5 Conclusion
The paper presents an efficient, general method for inserting prior knowledge into deep neural models. Adding WordNet and Wikipedia to BERT improves masked language modeling, factual recall, and several downstream tasks.
- KnowBert efficiently inserts prior knowledge into a deep neural model.
- Adding WordNet and Wikipedia to BERT improves masked language model quality and significantly improves factual recall.
- Downstream evaluations show improvements on relationship extraction, entity typing, and word sense disambiguation datasets.
- Future work will incorporate diverse domain-specific knowledge bases for specialized NLP applications.
A.1 KnowBert training details
KnowBert training combines standard fine-tuning practices with layer-specific learning rates, limited pretraining hyperparameter tuning, and mixed unlabeled/labeled batches.
- KAR layers surrounding each knowledge base receive the largest learning rate, while lower BERT layers use a smaller rate.
- Pretraining hyperparameter tuning was limited by computational expense and used partial runs monitored through early masked language model perplexity.
- KnowBert-WordNet and KnowBert-Wiki used batches sampled at an 80% unlabeled and 20% labeled ratio.
- KnowBert-W+W used an 85% unlabeled, 7.5% labeled, and 7.5% labeled sampling ratio.
A.2 Task fine-tuning details
Task fine-tuning used Adam, warmup and decay schedules, gradient clipping, weight decay, and task-specific hyperparameter searches across several datasets.
- All end-task optimization used Adam with 10% linear learning-rate warmup, linear decay, gradient clipping above norm 1.0, and 0.01 weight decay.
- TACRED: TACRED used 106K annotated sentences across 41 relationship classes, with maximum development micro F1 of 71.7%.
- SemEval 2010 Task 8: SemEval 2010 Task 8 used 10K sentences across 18 relationship classes and achieved maximum development F1 of 89.1 ± 0.77.
- WiC: WiC used 7.5K annotated sentence pairs, with model averaging helping reduce development-accuracy variance across random restarts.
- WiC: WiC’s maximum development accuracy was 72.6.
- Entity typing: Entity typing covered nine general entity classes and achieved maximum development F1 of 75.5 ± 0.38 across five seeds.
A.3 Wikidata probing results
The Wikidata probing evaluation measures whether KnowBert can recover masked entities from factual tuples across multiple relations.
- The Wikidata probing task reports results for all relationships.