Source-linked AI summary
Improving Coreference Resolution by Learning Entity-Level Distributed Representations
Kevin Clark, Christopher D. Manning
TL;DR
Coreference systems often rely on mention-pair links and therefore miss entity-level information. This paper learns distributed representations for cluster pairs and trains cluster merging with learning-to-search, substantially outperforming state-of-the-art systems on English and Chinese CoNLL 2012.
Problem
Mention-ranking systems consider only local mention pairs, limiting their ability to consolidate entity-level information in coreference decisions.
Method
A neural cluster-ranking model learns distributed representations of coreference-cluster pairs and uses learning-to-search to optimize local merges toward high-scoring final partitions.
Results
65.29 CoNLL F1 for English and 63.66 for Chinese, substantially outperforming other state-of-the-art systems on the CoNLL 2012 Shared Task.
Takeaways & Limitations
Incorporating learned entity-level information improves coreference results across both languages and all evaluation metrics, though gains are fairly modest in practice.
Takeaways & Limitations
The model depends on search-space pruning, which removes many candidate actions and leaves its unpruned performance uncertain.
Abstract
from arXiv · showhide
A long-standing challenge in coreference resolution has been the incorporation of entity-level information - features defined over clusters of mentions instead of mention pairs. We present a neural network based coreference system that produces high-dimensional vector representations for pairs of coreference clusters. Using these representations, our system learns when combining clusters is desirable. We train the system with a learning-to-search algorithm that teaches it which local decisions (cluster merges) will lead to a high-scoring final coreference partition. The system substantially outperforms the current state-of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task dataset despite using few hand-engineered features.
1 Introduction
Coreference resolution is a clustering problem, but many systems rely only on mention pairs and therefore lack entity-level information. This work learns cluster representations and merge decisions, achieving strong results across English and Chinese.
- Coreference resolution identifies mentions referring to the same real-world entity and is fundamentally a clustering problem.
- Entity-level features describe relationships between clusters rather than only between two mentions.
- The system learns continuous distributed representations for cluster pairs instead of relying mainly on hand-crafted categorical constraints.
- Learning-to-search trains the model to choose cluster merges that lead toward a high-scoring final coreference partition.
- 65.29 CoNLL F1 for English and 63.66 CoNLL F1 for Chinese substantially outperform other state-of-the-art systems.
2 System Architecture
The architecture uses neural subnetworks to encode mention pairs and cluster pairs, while a mention-ranking model initializes and prunes the cluster-ranking model. Figure 1 summarizes these component relationships.
- A single cluster-ranking neural network learns which coreference cluster merges are desirable.
- The mention-pair encoder creates representations for mention pairs, and the cluster-pair encoder pools relevant mention-pair representations into cluster-pair vectors.
- A mention-ranking model initializes the cluster-ranking model and prunes candidate merges so the system runs faster.
- Figure 1 uses solid arrows for neural-network component relationships and dashed arrows for other dependencies.
3 Building Representations
The system builds mention-pair representations from lexical, structural, distance, speaker, and document features, then aggregates them into cluster-pair representations. These representations are produced by feedforward networks and pooling operations.
- Representation Overview: The architecture and trained models are documented through Figures 2 and 3, with code and trained models made available online.
- Mention-Pair Encoder: The mention-pair encoder maps a mention and candidate antecedent, including NA, to a distributed vector representation.
- Mention-Pair Encoder: Input features combine word embeddings with mention, distance, speaker, string-matching, and document-genre information.
- Mention-Pair Encoder: For NA candidates, a separate network with the same architecture except for its input layer produces anaphoricity scores.
- Mention-Pair Encoder: Three fully connected ReLU hidden layers transform the input into the mention-pair representation rm(a, m) = h3(a, m).
- Cluster-Pair Encoder: The cluster-pair encoder pools the matrix of relevant mention-pair representations using concatenated max-pooling and average-pooling.
4 Mention-Ranking Model
The mention-ranking model scores candidate antecedents for each mention and uses these scores for linking, pretraining, and candidate selection. Its training objective penalizes different antecedent errors differently and is tuned through search.
- Pretraining: Mention-ranking pretraining initializes the cluster-ranking model and supplies scores for easy-first ordering and candidate pruning.The network uses all-pairs classification for 150 epochs followed by top-pairs classification for 50 epochs.
- Model: The mention-ranking model scores each mention–antecedent pair and links each mention to its highest-scoring candidate at test time.The candidate set includes preceding mentions and NA for mentions without an antecedents.
- Training: Its slack-rescaled max-margin objective separates the highest-scoring true antecedent from false antecedents.The objective considers candidate antecedents preceding the mention and NA when the mention has no antecedent.
- Training: Different penalties represent false-new, false-anaphoric, wrong-link, and correct-coreference decisions.These penalties allow tuning toward coreference evaluation metrics by biasing the system toward more or fewer links.
- Training: The system searches error-penalty settings with αWL fixed at 1.0, finding (αFN, αFA, αWL) = (0.8, 0.4, 1.0) for English and (0.7, 0.4, 1.0) for Chinese.The search varies αFA and αFN over {0.1, 0.2, ..., 1.5}.
5 Cluster-Ranking Model
The cluster-ranking model represents and scores pairs of clusters, then incrementally merges them through a sequential decision process. Easy-first ordering, pruning, and learning-to-search connect local merge actions to final clustering quality.
- 5 Cluster-Ranking Model: The cluster-ranking model addresses mention-ranking’s limitation by scoring pairs of clusters rather than only pairs of mentions.Its cluster-pair encoder produces a representation used to estimate cluster compatibility.
- 5 Cluster-Ranking Model: At each step, the system starts from singleton clusters and can merge the current mention’s cluster with a preceding cluster or PASS.The resulting clustering and next mention define the next state.
- 5 Cluster-Ranking Model: The policy assigns probabilities to available actions and executes the highest-scoring action during inference.Actions are evaluated from the current state, which contains existing clusters and the current mention.
- 5.2 Easy-First Cluster Ranking: Easy-first ordering processes mentions by descending highest-scoring candidate-link scores, delaying harder decisions until more information is available.This ordering is described as reducing errors compounding during incremental inference.
- 5.2 Easy-First Cluster Ranking: Pruning high-scoring antecedents removes over 95% of candidate actions without decreasing performance, enabling faster learning and inference.The candidate set is reduced instead of including all previously occurring mentions.
- 5.3 Deep Learning to Search: Learning-to-search rolls out candidate actions with a reference policy to estimate their eventual loss and train the merge policy.The algorithm visits states from the current policy, evaluates actions through end states, and updates the policy using the resulting costs.
6 Experiments and Results
Experiments evaluate mention- and cluster-ranking models on CoNLL 2012 English and Chinese data, testing preprocessing, pretraining, ordering, learning-to-search, and semantic similarity. The cluster-ranking system improves performance across languages and metrics, while its gains over mention ranking are fairly modest in practice.
- Experimental Setup: Experiments use system-produced predicted mentions on English and Chinese CoNLL 2012 data, reporting MUC, B3, CEAFφ4, and their average CoNLL F1.Mention detection uses a rule-based procedure that extracts pronouns and maximal noun phrases before filtering spurious mentions.
- Mention-Ranking Model Experiments: Pretraining substantially improves mention-ranking accuracy, especially when both pretraining stages transition toward the max-margin ranking objective.The study reports these comparisons on development sets in Table 2.
- Cluster-Ranking Model Experiments: The easy-first strategy slightly outperforms left-to-right ordering for cluster ranking.Left-to-right ordering processes mentions by document position, whereas easy-first ordering uses the highest-scoring coreference link.
- Cluster-Ranking Model Experiments: Using a fixed trajectory of gold decisions significantly decreases performance compared with learning to search.The authors attribute this to training that exposes the model only to correct decisions, leaving it unprepared for mistakes.
- Capturing Semantic Similarity: 18.9 F1 vs. 10.7 F1: the neural model substantially improves nominal coreference with no head match over the earlier statistical system.These links represent about 1.2% of positive test-set coreference links, so the improvement does not significantly affect the final score.
- Final System Performance: The cluster-ranking model improves results across both languages and all evaluation metrics, with the largest improvement in CEAFφ4.The authors associate entity-level information with preventing bad merges between large clusters, but characterize the practical gains over the more complex model as fairly modest.
7 Related Work
The paper builds on mention-ranking, incremental clustering, entity-level modeling, and learning-to-search approaches. It distinguishes its cluster ranker as a fundamentally clustering-based model and compares the work through CoNLL 2012 state-of-the-art results.
- Mention-Ranking Approaches: Mention-ranking models are a popular machine-learning approach to coreference resolution, but they primarily link individual mention pairs.The paper positions its work against this established line of research.
- Benchmark Comparison: The paper’s cluster ranker is evaluated against current state-of-the-art approaches on CoNLL 2012 English and Chinese test sets.Table 5 identifies the neural mention ranker and neural cluster ranker as contributions of this work.
- Entity-Level Modeling: The paper differs from recurrent entity-level augmentation by making cluster ranking fundamentally a clustering model rather than an enhanced mention ranker.Both approaches incorporate entity-level information, but the paper emphasizes the distinction in modeling level.
- Incremental Clustering: Its approach follows prior incremental clustering work and combines cluster-ranking with easy-first clustering strategies.Related systems have also incorporated entity-level information through joint inference and incremental cluster construction.
- Learning to Search: The system uses a learning-to-search algorithm similar to SEARN, following prior work that trains coreference resolvers on test-like decision trajectories.Other related approaches use structured perceptron models for similar training purposes.
8 Conclusion
The paper presents a coreference system that represents cluster pairs with distributed vectors and learns merge decisions through learning to search. On CoNLL 2012 English and Chinese, it reports substantial improvement over the current state of the art.
- Conclusion: Distributed, dense, high-dimensional vectors represent coreference cluster pairs and help distinguish beneficial from harmful merges.The cluster-ranking model uses these representations to capture entity-level information.
- Conclusion: Learning to search trains the model to learn how local cluster-merge decisions affect the final coreference score.The training procedure addresses the dependence of current decisions on earlier clustering actions.
- Conclusion: The system achieves substantial improvement over the current state of the art on the English and Chinese CoNLL 2012 Shared Task portions.The conclusion reports this evaluation outcome without specifying additional metric values.