Source-linked AI summary
End-to-end Neural Coreference Resolution
Kenton Lee, Luheng He, Mike Lewis, Luke Zettlemoyer
TL;DR
Coreference systems previously depended on syntactic parsers and hand-engineered mention detectors. This paper instead considers all spans as potential mentions, learns antecedent distributions end-to-end, and achieves state-of-the-art OntoNotes performance, including a 3.1 F1 gain with a 5-model ensemble.
Problem
Previous coreference models relied on syntactic parsers for head features and hand-engineered mention proposal algorithms.
Method
The model considers all document spans as potential mentions, learns span representations with head-finding attention, and optimizes antecedent decisions end-to-end with pruning.
Results
3.1 F1 improvement is achieved on the OntoNotes benchmark with a 5-model ensemble, while the single model improves the state-of-the-art average F1 by 1.5.
Takeaways & Limitations
The approach achieves strong coreference performance without external preprocessing tools and learns useful mention candidates from all possible spans.
Takeaways & Limitations
The model remains prone to false positive links from conflating paraphrasing or relatedness with coreference and struggles with decisions requiring world knowledge.
Abstract
from arXiv · showhide
We introduce the first end-to-end coreference resolution model and show that it significantly outperforms all previous work without using a syntactic parser or hand-engineered mention detector. The key idea is to directly consider all spans in a document as potential mentions and learn distributions over possible antecedents for each. The model computes span embeddings that combine context-dependent boundary representations with a head-finding attention mechanism. It is trained to maximize the marginal likelihood of gold antecedent spans from coreference clusters and is factored to enable aggressive pruning of potential mentions. Experiments demonstrate state-of-the-art performance, with a gain of 1.5 F1 on the OntoNotes benchmark and by 3.1 F1 using a 5-model ensemble, despite the fact that this is the first approach to be successfully trained with no external resources.
1 Introduction
The paper presents an end-to-end neural coreference model that learns mention detection and antecedent selection directly from all document spans, without parsers or hand-engineered mention proposals. Factoring and interpretability support efficient inference and analysis.
- The model jointly learns which spans are entity mentions and how to select their antecedents without syntactic parsers or hand-engineered mention proposals.
- The model represents spans with context-dependent boundary embeddings and head-finding attention over span-internal words.
- Unary mention scores and pairwise antecedent scores factor the model, allowing aggressive pruning of spans and antecedents before pairwise computation.
- 1.5 F1 improvement over existing models is achieved on OntoNotes, increasing to 3.1 F1 with a 5-model ensemble.
- The model's factors and attention weights provide interpretable signals for diagnosing absent links and identifying influential mention-internal words.
2 Related Work
Prior coreference systems relied heavily on parsing, hand-engineered structure, or more complex global modeling. The paper positions its approach as a non-pipelined alternative that performs well with stronger independence assumptions.
- Earlier hand-engineered coreference systems built on automatically produced parse trees outperformed learning approaches until highly lexical learning methods reversed that trend.
- Recent neural coreference models achieved substantial gains but still used parsers for head features and mention proposal algorithms.
- The paper differs from a prior non-pipelined transition system by using stronger independence assumptions that enable straightforward inference.
- Coreference learning approaches include mention-pair classifiers, entity-level models, latent-tree models, and mention-ranking models.
- Some earlier systems used lexicalized rules over English parse trees to remove pleonastic mentions.
3 Task
The task assigns every possible document span an antecedent decision. A selected preceding span creates a coreference link, while a dummy antecedent represents either a non-mention or a first mention.
- The task is defined over every possible span in a document containing T words and metadata such as speaker and genre information.
- Spans are ordered by their start indices, with spans sharing a start ordered by their end indices.
- Each span i receives an antecedent yi chosen from a dummy antecedent ǫ or all preceding spans.
- The dummy antecedent ǫ represents either a non-entity mention or an entity mention without a previous coreferent.
- The predicted antecedent links define the final coreference clustering by connecting spans into groups.
4 Model
The model represents spans with context and learned internal head information, then scores antecedent links by combining mention and pairwise compatibility factors.
- Model: The model learns a conditional antecedent distribution for each span using a product of per-span multinomials.Each span chooses among a dummy antecedent and preceding spans.
- Scoring: Each pairwise coreference score combines whether both spans are mentions with whether the earlier span is an antecedent.The final score adds unary mention scores and a pairwise antecedent score.
- Model: The model considers spans up to a maximum width, scores potential mentions, and prunes low-scoring spans before coreference decisions.The factorized design supports aggressive pruning of spans unlikely to belong to a coreference cluster.
- Scoring Architecture: The architecture computes span and antecedent scores with feed-forward neural networks using learned span representations and pairwise features.Pairwise features include element-wise span similarity, speaker and genre metadata, and span distance.
- Span Representations: Span embeddings combine context-dependent boundary representations, a learned attention-based head vector, and span features.The attention mechanism replaces reliance on syntactic parses and automatically learns weights over words inside each span.
5 Inference
Inference avoids the full quartic computation by pruning candidate spans and antecedents, then computes the remaining document-level antecedent distribution in one forward pass.
- Inference: O(T^4) full-model complexity motivates greedy pruning during both training and evaluation.The system limits span width and candidate counts to maintain computational efficiency.
- Pruning: The system retains at most λT high-scoring spans and up to K antecedents per span, while enforcing non-crossing bracketing.Candidates are selected by unary mention scores, with spans accepted in decreasing score order subject to suppression.
- Pruning: Over 92% gold-mention recall is maintained when λ = 0.4 despite aggressive pruning.This result measures recall of gold mentions after candidate reduction.
- Inference: For the remaining mentions, the model computes a joint antecedent distribution in one forward pass and outputs the most likely clustering.The final clustering is recovered from the most likely configuration.
6 Learning
Training uses gold clustering information to maximize the marginal likelihood of correct antecedents, while the factorized objective also supports accurate pruning.
- Learning Objective: The model optimizes the marginal log-likelihood of all correct antecedents implied by each gold coreference cluster.Antecedents are latent because the training data provides clustering information rather than explicit antecedent assignments.
- Learning Objective: The learning objective helps the model learn pruning because only gold mentions receive positive updates during training.This signal supports credit assignment to mention scores used for pruning.
- Dummy Antecedent: Fixing the dummy antecedent score at zero removes a spurious degree of freedom related to mention detection and limits pruning noise.When a gold antecedent is pruned, the objective pushes non-gold antecedents down without incorrectly raising the dummy score.
- Evaluation Constraint: The model enforces non-crossing mention structures because the official CoNLL-2012 evaluation considers crossing predictions invalid.The paper notes that this consistency constraint is not inherently necessary for the model.
- Learning Objective: A simple maximum-likelihood objective was more effective than the cost-sensitive and margin-based alternatives tested.The objective is described as a span-level, cost-insensitive analogue of earlier work.
7 Experiments
Experiments use CoNLL-2012 data and a neural architecture with pretrained word representations, character CNNs, LSTMs, feature embeddings, pruning, and five-model ensembles.
- Data: The experiments use 2802 training, 343 development, and 348 test documents from the CoNLL-2012 English coreference shared task.Training documents average 454 words and reach a maximum of 4009 words.
- Architecture: Word representations concatenate fixed 300-dimensional GloVe and 50-dimensional Turian embeddings with character-CNN features.The character CNN uses learned 8-dimensional character embeddings and filters over windows of 3, 4, and 5 characters.
- Architecture: The model uses 200-dimensional LSTM states and two-layer feed-forward networks with 150-dimensional hidden layers and ReLU activations.These dimensions specify the principal neural components used in the experiments.
- Features: Speaker, genre, span-distance, and mention-width features are represented as learned 20-dimensional embeddings.Distance values are grouped into predefined buckets, and speaker information records whether two spans share a speaker.
- Pruning: Pruning uses a maximum span width L = 10, λ = 0.4 spans per word, and at most K = 250 antecedents.Training documents are randomly truncated to at most 50 sentences.
- Ensembling: Five-model ensembles average mention scores before pruning and antecedent scores afterward.The models use different random initializations, with ensemble decisions applied to both pruning and antecedent scoring.
8 Results
The model achieves state-of-the-art OntoNotes performance, with gains linked especially to recall and end-to-end mention handling. Ablations identify span features, word representations, metadata, attention, and mention proposals as important factors.
- Coreference Results: Recall provides the most significant gains, likely because end-to-end training avoids discarding mentions missed by pipelined detectors.Pipelined systems discard more than 9% of labeled mentions in one comparison, whereas the model discards less than 2% due to its maximum width of 10.
- Coreference Results: 1.5 F1 improves the single model’s state-of-the-art average on OntoNotes, while the 5-model ensemble improves it by 3.1 F1.Average F1 is computed from MUC, B3, and CEAFφ4 F1.
- Ablations: 3.8 F1 comes from span-distance and span-width features, while character-level modeling contributes 0.9 F1 through morphological and rare-word information.Both fixed pretrained word-embedding sets also contribute to development F1.
- Ablations: Removing speaker and genre metadata reduces performance by 1.4 F1, yet the resulting system remains on par with prior state-of-the-art systems using that metadata.These indicators may be unavailable in downstream applications.
- Ablations: Removing head-finding attention causes a 1.3 F1 drop, showing that task-specific attention to mention-internal words contributes to performance.The attention can favor multiple words useful for coreference rather than only traditional syntactic heads.
- Ablations: Rule-based mention proposals reduce performance by 1 F1, whereas oracle mentions improve the model by over 17.5 F1.These comparisons isolate the contribution of mention proposal quality on development data.
9 Analysis
The analysis shows that unary mention scores learn useful span rankings, achieving high mention recall and syntactic plausibility while revealing both effective attention behavior and neural-model weaknesses.
- The model automatically learns a useful ranking of spans despite singleton clusters lacking explicit labels.
- Mention Recall: 92.7% recall of gold mentions is achieved by retaining 0.4 spans per word in development data.Recall increases with more retained spans but shows diminishing returns.
- Mention Precision: 75–90% of predictions for 2–5-word spans are syntactic constituents, but precision falls to 46% for 10-word spans.Longer spans are relatively rare and more difficult for the model.
- Head Agreement: The learned attention-selected words match syntactic heads in 68–93% of evaluated spans without explicit syntactic-head supervision.The model learns these preferences from coreference clustering because head words help make coreference decisions.
- Qualitative Analysis: Attention supports coreference links by selecting informative words such as fire, factory, and region within mention spans.The analysis also shows attention to coordinators such as and when plurality provides a useful cue.
- Qualitative Analysis: Word-embedding similarity can produce false-positive links by conflating paraphrasing or relatedness with coreference.The authors identify a need for representations that distinguish equivalence, entailment, and alternation.
- Qualitative Analysis: The model performs poorly on coreference decisions requiring world knowledge, such as resolving them to a person rather than some ships.The paper points to external knowledge or much larger training data as possible ways to address this limitation.
10 Conclusion
The paper presents a first end-to-end, state-of-the-art coreference model that learns mention candidates and task-specific head preferences without external preprocessing tools. Its gains remain potentially complementary to entity-level inference and world-knowledge methods.
- The final model ensemble improves OntoNotes performance by over 3 F1 without external preprocessing tools used by previous systems.
- The model implicitly learns useful mention candidates from all possible spans and task-specific head-word preferences that correlate strongly with traditional definitions.
- The improvements are potentially complementary to entity-level inference and incorporating world knowledge.The paper identifies both as important avenues for future work.