Source-linked AI summary
Improving Entity Linking by Modeling Latent Relations between Mentions
Phong Le, Ivan Titov
TL;DR
Entity linking must align mentions with knowledge-base entities while maintaining compatibility across related mentions, yet prior systems often rely on supervised or heuristic relation prediction. This paper induces mention relations as latent variables in an end-to-end neural model, achieving the best reported AIDA-CoNLL result and faster training. The findings support modeling multiple latent relations rather than ignoring relations altogether.
Problem
NEL systems need relations between mentions to enforce compatible linking decisions, but prior approaches commonly rely on supervised systems, heuristics, or relation-agnostic modeling.
Method
The model represents mention relations as latent variables and learns relation, mention, and context embeddings while optimizing entity linking end to end.
Results
0.85% F1 improvement over the best previous results on AIDA-CoNLL is reported, with substantially better performance than the relation-agnostic version and ten times shorter training time.
Takeaways & Limitations
Latent multi-relational modeling provides a relation-aware NEL system that reduces reliance on human expertise and may be more portable across languages and domains.
Takeaways & Limitations
A small fixed number of relations is used because directly predicting context-specific relation embeddings caused overfitting and poor performance in preliminary experiments.
Abstract
from arXiv · showhide
Entity linking involves aligning textual mentions of named entities to their corresponding entries in a knowledge base. Entity linking systems often exploit relations between textual mentions in a document (e.g., coreference) to decide if the linking decisions are compatible. Unlike previous approaches, which relied on supervised systems or heuristics to predict these relations, we treat relations as latent variables in our neural entity-linking model. We induce the relations without any supervision while optimizing the entity-linking system in an end-to-end fashion. Our multi-relational model achieves the best reported scores on the standard benchmark (AIDA-CoNLL) and substantially outperforms its relation-agnostic version. Its training also converges much faster, suggesting that the injected structural bias helps to explain regularities in the training data.
1 Introduction
NEL assigns textual entity mentions to knowledge-base entries, while document-level relations constrain compatible assignments. This paper induces such relations as latent variables in an end-to-end neural linker and reports improved benchmark performance and faster training.
- Motivation: Named entity linking assigns textual mentions to corresponding entries in a knowledge base and supports downstream natural-language understanding tasks.The paper cites information extraction and question answering as common uses.
- Motivation: Coreference and semantic relations constrain entity assignments, such as requiring coreferent mentions to resolve identically and participants to match the linked event.For example, linking World Cup to FIFA WORLD CUP favors interpreting England as a football team rather than a basketball team.
- Limitations of prior work: Prior NEL systems typically use coreference from external tools or heuristics, while relation-agnostic models treat predicted entities as an unordered bag.Only a few earlier systems exploit additional syntactic-semantic relations.
- Approach: The proposed model treats relations between mentions as latent variables and induces them without supervision while optimizing entity linking end to end.It learns embeddings of mentions, contexts, and relations through representation learning.
- Results: 0.85% F1 improvement over the best reported scores on AIDA-CoNLL is achieved by the multi-relational model.The model also requires ten times shorter wall-clock training time than the simpler relation-agnostic version.
2 Background and Related work
The background frames entity linking as local scoring plus document-level coherence, whose exact global decoding is difficult. Prior work largely relies on coreference, hand-crafted relations, or relation-agnostic representation learning; this paper argues for relation-aware pairwise modeling.
- Entity linking formulation: Entity linking assigns each mention a candidate knowledge-base entity or NILL, commonly after heuristic candidate selection reduces the search space.Statistical models then choose among the remaining candidate lists.
- Local and global models: Local models score each candidate from a mention’s local context, whereas global models additionally score coherence among entity assignments.Global coherence is represented by a function over the document’s entity assignments.
- Local and global models: Global models commonly sum pairwise coherence scores, but exact decoding is NP-hard and therefore requires approximate inference such as loopy belief propagation.Other work approximates decoding by decomposing it into one problem per entity.
- Relations between mentions: NEL systems widely use coreference, but selecting additional semantic relations is challenging and earlier approaches often depend on labor-intensive rules, relation knowledge bases, or supervised external tools.These dependencies are difficult to generalize when suitable knowledge bases or domain experts are unavailable.
- Representation learning: Representation-learning approaches replace most feature engineering with pretrained word and entity embeddings and can achieve state-of-the-art accuracy on AIDA-CoNLL.The cited formulation uses embeddings, diagonal parameter matrices, and an attention-based context representation.
- Representation learning: The paper extends relation-agnostic pairwise scoring by modeling relations between mentions with relation embeddings.The prior global component treats entities as a bag and ignores relation identities and ordering.
3 Multi-relational models
The model represents mention pairs with latent relations, assigning relation-specific confidence weights and combining their pairwise scores. It compares relation-wise and mention-wise normalization, using padding and diversity regularization to address structural limitations.
- 3.1 General form: Each mention pair is assigned K latent relations with non-negative confidence weights, and its pairwise score is a weighted sum of relation-specific scores.Each relation uses a diagonal embedding matrix Rk; the weights are normalized scores derived from mention and context representations.
- 3.1 General form: The model uses a single-layer neural network for f(mi, ci), with context formed from averaged embeddings of words surrounding each mention.The network uses tanh activation and dropout.
- 3.2 Rel-norm: Relation-wise normalization: Rel-norm normalizes relation weights over k, while ment-norm normalizes them over other mentions j.With K = 1, rel-norm reduces to the bag-of-entities model, while mono-relational ment-norm with uniform weights matches Ganea and Hofmann (2017).
- 3.3 Ment-norm: Mention-wise normalization: Ment-norm can focus on related mentions for each relation, but its normalization forces every relation to distribute weight even when that relation is inapplicable.A padding mention and entity let the model damp the probability assigned to ordinary mentions in such cases.
- 3.4 Implementation: The model defines a CRF with local and relation-aware pairwise scores, estimates max-marginals using max-product loopy belief propagation, and trains with a ranking loss optimized by Adam.The local entity probability combines mention-entity hyperlink statistics from Wikipedia, a large Web corpus, and YAGO.
- 3.4 Implementation: A diversity regularizer encourages different relation embeddings because otherwise the relation representations tend to collapse toward similar matrices.The regularization term is added to the loss to encourage exploration of different relations.
4 Experiments
The experiments compare four multi-relational and ablated models across in-domain and out-of-domain entity-linking settings. Models are run repeatedly and evaluated with micro F1, with relation counts selected using development scores.
- Models: The evaluation compares rel-norm, ment-norm, ment-norm with K = 1, and ment-norm without padding.Mono-relational rel-norm is equivalent to the relation-agnostic baseline of Ganea and Hofmann (2017).
- Evaluation: Each model is run 5 times, reporting average micro F1 and its 95% confidence interval.Micro F1 aggregates performance over all mentions.
- Datasets: The study evaluates in-domain performance on AIDA-CoNLL and out-of-domain transfer to MSNBC, AQUAINT, ACE2004, WNED-CWEB, and WNED-WIKI.AIDA-train is used for training, with AIDA-A for development and AIDA-B for testing.
Candidate selection
The experiments use fixed candidate-selection and optimization settings, with K selected on development data under a GPU-memory constraint.
- Each mention is initially assigned 30 candidates, after which candidates are retained using mention-entity probabilities and model scores.
- The models use 300-dimensional representations, pretrained GloVe word embeddings, entity embeddings, dropout, and local context windows.GloVe embeddings were trained on 840B tokens; the local context window for pairwise scores is six.
- The selected numbers of relations were 6 for rel-norm and 3 for ment-norm, with K restricted to at most 7 by GPU memory.
- Training used early stopping, reducing the learning rate at 91% dev F1 for rel-norm and 91.5% for ment-norm.Training stopped after 20 epochs without F1 improvement; the learning-rate schedule differed between models.
- All hyperparameters except K and the early-stopping turning point matched Ganea and Hofmann (2017), leaving systematic tuning as a possible source of improvement.
4.2 Results
On AIDA-B and five out-domain test sets, ment-norm provides the strongest reported performance among the paper’s models, with gains that vary by evaluation setting.
- All four proposed models outperform previous methods on AIDA-B, with ment-norm achieving 0.85% higher F1 than Ganea and Hofmann (2017).The comparison uses micro F1 and the same Wikipedia and YAGO mention-entity index across methods.
- Ment-norm achieves the highest F1 on MSNBC and ACE2004 among the five out-domain test sets.
- On average, ment-norm’s F1 is 0.3% higher than Ganea and Hofmann (2017) but 0.2% lower than Guo and Barbosa (2016).Guo and Barbosa performs exceptionally well on WIKI but substantially worse than ment-norm on the other datasets.
- The other three proposed models have lower average F1 scores than the best previous model on the out-domain evaluation.
Mono-relational v.s. multi-relational
Multi-relational modeling generally improves ment-norm, while rel-norm shows a domain-dependent trade-off; oracle results further highlight the value of multiple relations.
- Mono-relational v.s. multi-relational: Multi-relational rel-norm outperforms its mono-relational version on AIDA-CoNLL but performs significantly better on all five out-domain datasets.The authors interpret this pattern as evidence that multi-relational rel-norm does not generalize well across domains.
- Mono-relational v.s. multi-relational: Multi-relational ment-norm performs better than its mono-relational version on every test set except AQUAINT.The authors speculate that multiple factors make ment-norm less sensitive to individual prediction errors.
- Oracle: Oracle rel-norm performs close to oracle ment-norm, unlike the substantial difference without the oracle.This suggests rel-norm is more sensitive to prediction errors than ment-norm.
- Oracle: Ganea and Hofmann (2017), even with the oracle, performs only slightly better than non-oracle ment-norm using LBP.The authors connect this result to an overly simplistic global coherence scoring component.
- Oracle: Both multi-relational oracle models substantially outperform the two mono-relational oracle models.The oracle removes imperfections in LBP and mistakes in predicting other entities when evaluating a mention.
Relations
The learned relations differ by model: rel-norm is difficult to interpret, whereas ment-norm yields relations with distinct, interpretable mention preferences.
- Rel-norm’s relation meanings are difficult to interpret because its first relation dominates the other two with high weights for most mentions.Its improvement over the baseline nevertheless suggests that the learned relations encode useful information.
- Ment-norm’s first relation resembles coreference by preferring mentions that may refer to the same entity or have semantically similar mentions.
- Ment-norm’s second and third relations prefer more distant meanings and complement the first relation.In the example, they assign high weights to “Belgium” and “Brussels” but low weights to “Liege” mentions.
Complexity
Although rel-norm and ment-norm have complexity linear in K, they converge much faster than the relation-agnostic model, reducing training time substantially.
- Complexity of rel-norm and ment-norm is linear in K, making them theoretically more expensive than the relation-agnostic model.
- 120 epochs versus 1250 epochs are needed on average for the proposed and relation-agnostic models, respectively.
- Just under 1.5 hours of wall-clock training time is required, ten times faster than the relation-agnostic model.
- The difference in testing time is negligible when using a GPU.
5 Conclusion and Future work
The paper models relations as latent variables in NEL without extra supervision and reports improved benchmark performance. It distinguishes multiple relations from generic coherence and proposes richer relational modeling as future work.
- 5 Conclusion and Future work: The models treat relations as latent variables, avoiding extra supervision, and use representation learning for relation embeddings instead of extensive feature engineering.
- 5 Conclusion and Future work: The best model achieves the best reported F1 on AIDA-CoNLL, improving over previous results by 0.85% F1.
- 5 Conclusion and Future work: Modeling multiple relations differs substantially from modeling coherence and may support follow-up work that injects prior knowledge into individual relations.
- 5 Conclusion and Future work: Future work includes using syntactic and discourse structures to encourage discovery of a richer set of relations and combining ment-norm with rel-norm.
- 5 Conclusion and Future work: The authors also propose examining whether induced latent relations could help relation extraction.