PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Entity Linking by Reading Entity Descriptions

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, Honglak Lee

arXiv:1906.07348v1cs.CLcs.LG

TL;DR

Zero-shot entity linking asks models to link mentions to unseen specialized entities without in-domain labels, alias tables, or metadata, using textual descriptions alone. The paper combines reading-comprehension models with domain-adaptive pre-training and reports improved performance on a newly constructed benchmark, while retaining scope limitations around NIL recognition and candidate generation.

  • Problem

    Entity linking must generalize to unseen specialized entity dictionaries when labeled mentions and powerful linking resources are unavailable.

  • Method

    The paper constructs a multi-world zero-shot entity-linking dataset and combines neural reading comprehension with domain-adaptive pre-training.

  • Results

    77.05% normalized accuracy and 56.58% unnormalized accuracy are achieved by the best model chaining all pretraining strategies.

  • Takeaways & Limitations

    The dataset provides a benchmark for specialized-domain entity linking when entities are defined through descriptions alone and labeled mentions are unavailable.

  • Takeaways & Limitations

    The task assumes the target entity exists in the dictionary and leaves NIL recognition or clustering to future editions.

Abstract

from arXiv · show

We present the zero-shot entity linking task, where mentions must be linked to unseen entities without in-domain labeled data. The goal is to enable robust transfer to highly specialized domains, and so no metadata or alias tables are assumed. In this setting, entities are only identified by text descriptions, and models must rely strictly on language understanding to resolve the new entities. First, we show that strong reading comprehension models pre-trained on large unlabeled data can be used to generalize to unseen entities. Second, we propose a simple and effective adaptive pre-training strategy, which we term domain-adaptive pre-training (DAP), to address the domain shift problem associated with linking unseen entities in a new domain. We present experiments on a new dataset that we construct for this task and show that DAP improves over strong pre-training baselines, including BERT. The data and code are available at https://github.com/lajanugen/zeshel.

1 Introduction

The paper introduces zero-shot entity linking for unseen specialized entities identified only by textual descriptions, and presents reading-comprehension models plus domain-adaptive pre-training as solutions.

  • Motivation: Zero-shot entity linking targets specialized entity dictionaries where labeled data, frequency statistics, and metadata are unavailable.The task is designed to generalize to unseen entities and domains using minimal assumptions.
  • Task and dataset: The new task evaluates models on disjoint test entity dictionaries whose entities are represented by text descriptions.The dataset uses multiple Wikia sub-domains and automatically extracts labeled mentions from hyperlinks.
  • Challenges: Models must read entity descriptions and reason about their correspondence with mentions in context, making strong reading comprehension important.The task also requires adaptation to new mention contexts and entity descriptions without labeled test-entity mentions.
  • Baseline: Attention between the mention context and entity descriptions is critical for this task.This attention mechanism had not been used in prior entity linking work, according to the paper.
  • Adaptation: Domain-adaptive pre-training (DAP) further improves entity linking performance.The paper presents DAP as a simple adaptation strategy for the task.

2 Zero-shot Entity Linking

The paper defines zero-shot entity linking by removing common entity-linking resources and requiring transfer from labeled source worlds to disjoint target worlds using textual descriptions.

  • 2.1 Review: Entity linking: Standard entity linking grounds a mention in context to an entity from a potentially very large entity set.Entity sets may contain tens of thousands or millions of entities.
  • 2.1 Review: Entity linking: Conventional systems often assume a shared entity set, alias tables, frequency statistics, or structured data.These resources reduce ambiguity or provide candidate and popularity information.
  • 2.2 Task Definition: Zero-shot entity linking drops these assumptions and represents each entity through a text description in an entity dictionary.The remaining weak assumption is access to an entity dictionary of entity-description pairs.
  • 2.2 Task Definition: A world consists of mention and document distributions plus an associated entity dictionary, with training on labeled source worlds and testing on a disjoint target world.Target-world documents and entity descriptions may support unsupervised adaptation, while target mention boundaries are unavailable during training but provided at test time.
  • Related Work: The task differs from related linking-to-any-database settings by avoiding alias-table or token-overlap shortcuts and structured-data assistance.This leaves textual information as the basis for linking in a larger candidate space.

3 Dataset Construction

The dataset uses Wikia worlds to evaluate zero-shot entity linking across specialized, disjoint entity dictionaries, with mentions and descriptions drawn from richly contextual domains. It excludes NIL recognition and emphasizes difficult low-overlap cases.

  • Wikia provides specialized, community-written encyclopedias whose hyperlinks yield labeled mentions and whose entities have rich thematic context.
  • The dataset uses 16 Wikias: 8 for training, 4 for validation, and 4 for testing.
  • String-matched mentions comprise approximately 5% of the final dataset, limiting their representativeness of natural mention distributions.
  • The task assumes every target entity exists in the dictionary and postpones NIL recognition and clustering to future dataset editions.
  • Mentions are categorized as High Overlap, Multiple Categories, Ambiguous Substring, or Low Overlap according to token overlap with the entity title.
  • Each domain contains 10,000–100,000 entities, with 49,275 labeled training mentions and 10,000-mention validation and test sets composed entirely of unseen entities.

4 Models for Entity Linking

The model pipeline generates candidates with BM25 and ranks them using a Transformer that jointly reads mention context and entity descriptions. Its deep cross-attention distinguishes the main architecture from pooling-based alternatives.

  • The system uses a fast candidate-generation stage followed by a more expensive candidate-ranking stage.
  • Candidate generation: BM25 retrieves the top 64 candidate entities by measuring similarity between the mention string and candidate documents.Top-64 candidate coverage is less than 77% on average.
  • Candidate ranking: A deep Transformer jointly encodes the mention context and candidate entity description as a sequence pair for ranking.Each component is represented by 128 word-piece tokens, and mention words receive a special embedding.
  • Candidate ranking: The main architecture uses deep cross-attention, allowing representations of the mention and entity description to interact directly.
  • Candidate ranking: Pool-Transformer and Cand-Pool-Transformer provide comparison variants using separate encoders with pooled representations and optional token-level attention.

5 Adapting to the Target World

The paper combines open-corpus, task-adaptive, and target-domain pre-training to address domain shift in zero-shot entity linking. Domain-adaptive pre-training inserts target-only training before source-labeled fine-tuning, and chained stages provide additive gains.

  • The approach combines task-adaptive and open-corpus pre-training with a new domain-adaptive pre-training strategy.
  • Task-adaptive pre-training: Task-adaptive pre-training uses unlabeled source and target data jointly before fine-tuning on source-domain labeled data.
  • Open-corpus pre-training: Open-corpus pre-training applies unsupervised training to large corpora before source-domain supervised fine-tuning.Examples include ELMo, OpenAI GPT, and BERT.
  • Domain-adaptive pre-training: Domain-adaptive pre-training inserts a penultimate stage trained only on target-domain data, followed by source-domain labeled fine-tuning.Its rationale is to prioritize target-domain representations when representational capacity is limited.
  • Chained pre-training: Pre-training stages can be chained, such as open-corpus, combined source-target, and target-only training before source-domain fine-tuning.
  • Chained pre-training: Chaining different pre-training strategies provides additive gains.

6 Experiments

Experiments show that zero-shot entity linking depends on strong reading-comprehension architectures and carefully chosen pre-training data. Domain-adaptive pre-training improves performance across strategies, while candidate generation remains a key bottleneck.

  • Test results and performance analysis: Candidate generation limits overall performance: top-64 recall is 76% on validation and 68% on test, with especially poor recall for low-overlap mentions.Strengthening candidate generation would improve unnormalized performance, whereas the ranking model handles low-overlap mentions comparably to other hard categories.
  • Baselines: Pretraining is necessary for reasonable Full-Transformer performance, and the choice of source, target, or external data affects results.The evaluated strategies include Usrc, Utgt, Usrc+tgt, and UWB.
  • Baselines: Fine-grained cross-attention in the Full-Transformer substantially outperforms simpler Transformer variants, establishing reading comprehension as central to zero-shot linking.The model compares mention context and entity descriptions through cross attention.
  • Generalization to unseen entities and new worlds: Seen entities are easiest; unseen training-world entities incur a 5-point drop, while entities from new worlds incur an 11-point drop.The larger gap reflects simultaneous shifts in language distribution and entity sets.
  • Impact of domain-adaptive pre-training: DAP improves all evaluated pre-training strategies by adding target-domain pretraining, with UWB →Usrc+tgt →Utgt outperforming a strong BERT model by 2%.This setting chains open-corpus, source-plus-target, and target-only pretraining.
  • Test results and performance analysis: 77.05% normalized accuracy and 56.58% unnormalized accuracy are achieved by the model chaining all pretraining strategies on test worlds.Unnormalized accuracy requires selecting the correct entity from tens of thousands of candidates.

7 Related Work

Related work spans entity-linking systems that use mention similarity, structured information, collective inference, and unsupervised domain adaptation. Zero-shot entity linking differs because source and target domains have disjoint entity labels rather than a shared label space.

  • Entity linking models: Entity-linking systems commonly separate candidate generation from candidate ranking, using mention similarity, contextual representations, and structured or frequency information.Prior methods exploit alias tables, metadata, or target-database relations when available.
  • Entity linking models: Prior work most similar to this task compares mentions with entity descriptions and structured data, while distant supervision and collective inference offer complementary approaches.Collective inference may be limited when metadata is unavailable.
  • Unsupervised domain adaptation: Standard unsupervised domain adaptation assumes labeled source and unlabeled target examples share a fixed label set, unlike zero-shot entity linking.Zero-shot linking has disjoint source and target entity labels.
  • Unsupervised domain adaptation: Denoising objectives, adversarial training, and multi-source adaptation are related strategies that can complement domain-adaptive pre-training.The paper positions its improved pretraining method within this broader adaptation literature.

8 Conclusion

The paper introduces zero-shot entity linking and a multi-world benchmark for specialized domains where labeled mentions are unavailable and entities are represented by descriptions. It combines neural reading comprehension with domain-adaptive pre-training as a strong baseline, while identifying several directions for future task extensions.

  • Conclusion: The multi-world dataset benchmarks entity linking to specialized dictionaries using entity descriptions alone and without labeled target mentions.It is intended as a shared benchmark for this setting.
  • Conclusion: Combining powerful neural reading comprehension with domain-adaptive pre-training provides the paper’s strong baseline.This combination addresses the task’s description-matching and domain-adaptation demands.
  • Conclusion: Future task variants could add NIL recognition and mention detection, while document-level joint resolution and better candidate generation remain open directions.The current task provides mention boundaries, and candidate generation leaves significant room for improvement.

A Examining model errors and predictions

The appendix presents examples of model predictions across four specialized worlds. Each example pairs a mention context with a gold entity and the model’s top-ranked candidate predictions.

  • Prediction examples: Examples report the gold entity and top-five model predictions for each mention.This format supports direct inspection of ranking errors.
  • Prediction examples: Mention contexts are centered on the target mention and paired with the beginning of candidate entity documents.Both contexts use the first 32 tokens available in the examples.
  • Prediction examples: The examples cover Coronation Street, Muppets, Ice Hockey, and Elder Scrolls worlds.The tables identify the source world for each set of mention and candidate examples.
Loading 1906.07348v1…