Source-linked AI summary
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Wenhan Xiong, Jingfei Du, William Yang Wang, Veselin Stoyanov
TL;DR
The paper examines whether pretrained language models encode real-world entity knowledge and addresses the lack of entity-centric modeling in standard token-level objectives. It introduces weakly supervised entity replacement pretraining, which improves fact completion and entity-related downstream tasks, including QA and fine-grained entity typing.
Problem
Existing pretraining objectives usually operate at the token level and do not explicitly model entity-centric knowledge, motivating evaluation of pretrained models’ real-world knowledge representations.
Method
The method replaces entity mentions with randomly selected same-type entities and trains the model to distinguish original factual statements from negative replacements.
Results
The model establishes new state of the art on three entity-related QA datasets and FIGER, with larger entity-typing improvements than BERT-based comparisons and stronger fact completion performance.
Takeaways & Limitations
Directly learning entity-level knowledge from unstructured text can improve fact completion and downstream entity-related NLP performance without additional downstream fine-tuning computation, memory, or architectural overhead.
Takeaways & Limitations
Pretraining requires entity links, and the QA evaluation treats answers as entities only when they correspond to Wikidata entities or Wikipedia titles.
Abstract
from arXiv · showhide
Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.
1 INTRODUCTION
The paper examines whether pretrained language models capture encyclopedic entity knowledge and introduces weakly supervised training to make that knowledge more explicit. The resulting model improves entity-related QA and fine-grained entity typing.
- 1 INTRODUCTION: Existing pretraining objectives capture syntax and semantics but do not explicitly model entity-centric knowledge.The paper investigates whether entity-focused pretraining can improve knowledge-intensive NLP tasks.
- 1 INTRODUCTION: The model is evaluated on fact completion, open-domain question answering, and fine-grained entity typing.The QA evaluation uses four previously published datasets, while FIGER is used for entity typing.
- 1 INTRODUCTION: The method replaces entity mentions with randomly selected entities of the same type and trains the model to distinguish the correct mention from false alternatives.This derives knowledge directly from unstructured text rather than requiring an external knowledge base.
- 1 INTRODUCTION: The approach requires no additional data processing, memory, or model modifications during downstream fine-tuning.The contribution is designed to incorporate entity knowledge during pretraining while preserving the downstream fine-tuning setup.
- 1 INTRODUCTION: On three QA datasets and FIGER, the model establishes new state-of-the-art results among the reported comparison settings.Ablation analysis indicates that the entity-centric training objective is instrumental for these results.
2 ENTITY REPLACEMENT TRAINING
The entity replacement training objective creates factual discrimination signals from Wikipedia by replacing linked entities with type-matched alternatives. It combines this objective with masked language modeling while retaining the BERT architecture for downstream use.
- ENTITY REPLACEMENT TRAINING: The method identifies entity mentions, links them to Wikipedia entities, and treats original documents as positive knowledge statements.Negative statements are created by replacing mentioned entities with random entities of the same type.
- ENTITY REPLACEMENT TRAINING: Type-constrained replacement preserves sentence-level linguistic correctness while requiring factual judgment.The objective is defined at the entity level and supplies stronger negative signals than token-level language modeling.
- Data Preparation: The training data consists of English Wikipedia, with entity mentions recognized from Wikipedia anchor links and Wikidata aliases.Documents are split into text chunks of 512 tokens for training.
- Replacement Strategy: Replacement samples entities of the same Wikidata type and avoids replacing adjacent entities.Each text chunk is replicated 10 times with different negative entities.
- Model Architecture: The model uses the BERT-base Transformer architecture and predicts entity replacement from the representations of the mention’s boundary words.The architecture has 12 Transformer layers with hidden dimension 768.
- Training Objectives: Entity replacement training is combined with masked language modeling, using masks outside entity spans at a 5% ratio.The model is trained for approximately 1 million updates with batch size 128.
3 EXPERIMENTS
The experiments evaluate WKLM on zero-shot fact completion and entity-related downstream tasks, comparing it with pretrained language-model baselines and training-objective variants. WKLM achieves strong relation-level fact completion and consistent gains on question answering and fine-grained entity typing, while ablations examine the roles of masked language modeling and masking ratios.
- Experimental Setup: The evaluation covers zero-shot fact completion, open-domain question answering, and fine-grained entity typing.Fact completion tests recovery of missing entities, while the downstream tasks require entity-level knowledge.
- Zero-Shot Fact Completion: Fact completion converts Wikidata triples into natural-language templates, removes an entity, and ranks candidate entities to recover the missing object.The entity-ranking setup supports automatic evaluation of multi-token entities.
- Zero-Shot Fact Completion: WKLM achieves the best fact-completion results on 8 of 10 relations and is less sensitive than BERT to candidate-set size and answer cardinality.WKLM also outperforms BERT and GPT-2 while remaining robust across relations with different properties.
- Question Answering: Across entity-related QA datasets, WKLM improves by an average of 2.7 F1 over BERT without ranking scores and reaches state of the art on three of four datasets with a BERT-based ranker.The gains are larger on WebQuestions, TriviaQA, and Quasar-T than on SearchQA.
- Entity Typing: On FIGER, WKLM achieves a 5.68-point absolute improvement and sets a new state of the art for fine-grained entity typing.The comparison includes BERT, ERNIE, and neural baselines using sparse hand-crafted features.
- Ablation Study: Ablations show that masked language modeling is essential, excessive masking harms results, and continued MLM training is less effective than WKLM on entity-centric tasks.The authors describe WKLM as a complementary recipe to masked language modeling for entity-related NLP tasks.
4 RELATED WORK
Prior work developed contextual language representations and explored adding knowledge to NLP models, often through external resources or entity-level objectives. This paper’s approach instead uses minimal external entity information without extra memory or architectural overhead.
- Contextual language representations improve on static word embeddings by modeling word meaning in context.
- Knowledge-enhanced NLP models address the limitation that standard language encoders do not explicitly model background knowledge.
- Recent approaches incorporate entity-level knowledge into pretrained models through specialized training objectives or external knowledge bases.
- Unlike these methods, the proposed method uses minimal external entity information and requires no additional memory or architectural overhead.
5 CONCLUSION
The paper introduces weakly supervised pretraining to encourage entity-level knowledge learning with minimal entity information and no downstream fine-tuning overhead. Results show strong performance on fact completion, question answering, and entity typing.
- The method uses minimal entity information during pretraining without adding computation, memory, or architectural overhead to downstream fine-tuning.
- The trained model performs strongly on fact completion and two entity-related NLP tasks.
- The results support directly learning entity-level knowledge from unstructured natural language through knowledge-aware pretraining.
A APPENDIX
The appendix describes implementation, QA context collection, and relation-property analyses. BERT’s fact-completion performance varies with candidate-set and answer-set properties, whereas WKLM is often less sensitive.
- 32 V100 GPUs were used for three days to pretrain the models.
- QA evidence contexts were collected using dataset-specific retrieval systems, including TF-IDF retrieval, Lucene-ranked paragraphs, and search-engine rankings.
- BERT’s fact-completion performance is strongly correlated with candidate-set size and the number of groundtruth answers.
- WKLM is often less sensitive than BERT to candidate-set size and groundtruth-answer count.
- Figure 2 plots hits@10 against candidate-set size and number of groundtruth answers.