Source-linked AI summary
Higher-order Coreference Resolution with Coarse-to-fine Inference
Kenton Lee, Luheng He, Luke Zettlemoyer
TL;DR
Coreference systems based on independent pairwise decisions can produce globally inconsistent clusters, while higher-order inference is computationally costly. The paper uses iterative attention-based span refinement with coarse-to-fine antecedent pruning, achieving 73.0 F1 versus 72.3 F1 for the tuned span-ranking baseline on English OntoNotes. Its conclusion is that higher-order interactions and efficient pruning improve coreference resolution performance within this benchmark setting.
Problem
First-order models can produce globally inconsistent coreference clusters, while explicit document-level higher-order inference is needed beyond what span encoders capture.
Method
The model iteratively refines span representations using antecedent distributions as attention and uses a cheaper bilinear coarse factor to prune candidates before fine scoring.
Results
73.0 F1 versus 72.3 F1 for the ELMo-augmented, tuned span-ranking baseline on the English OntoNotes test set.
Takeaways & Limitations
The approach models higher-order interactions while enabling efficient antecedent pruning, and the full system sets a new state of the art for coreference resolution.
Takeaways & Limitations
The prior distance-based pruning strategy considers only the K = 250 nearest mentions, although coreference links can reach much farther in natural language discourse.
Abstract
from arXiv · showhide
We introduce a fully differentiable approximation to higher-order inference for coreference resolution. Our approach uses the antecedent distribution from a span-ranking architecture as an attention mechanism to iteratively refine span representations. This enables the model to softly consider multiple hops in the predicted clusters. To alleviate the computational cost of this iterative process, we introduce a coarse-to-fine approach that incorporates a less accurate but more efficient bilinear factor, enabling more aggressive pruning without hurting accuracy. Compared to the existing state-of-the-art span-ranking approach, our model significantly improves accuracy on the English OntoNotes benchmark, while being far more computationally efficient.
1 Introduction
First-order span-ranking models efficiently score pairwise links but can produce globally inconsistent clusters. The paper introduces differentiable higher-order inference and coarse-to-fine pruning to improve accuracy while controlling computation.
- Motivation: First-order models score entity-mention pairs independently, making them vulnerable to clusters that are locally consistent but globally inconsistent.In the example, underspecified plurality makes each pair compatible while the complete cluster mixes plurality.
- Higher-order inference: The proposed higher-order inference iteratively uses antecedent distributions as attention to update span representations.Later coreference decisions can therefore softly condition on earlier decisions.
- Coarse-to-fine inference: The coarse-to-fine approach adds a cheaper, less accurate factor that prunes antecedents before expensive fine scoring.It is learned jointly with a single end-to-end objective and computes a rough sketch of likely antecedents first.
2 Background
The baseline formulates coreference resolution as antecedent assignment over spans and learns antecedent distributions from pairwise scores. Span representations and mention-pruning stages make this formulation tractable for documents.
- Task definition: Each span selects a dummy antecedent or one of its preceding spans, and these assignments implicitly define the final coreference clusters.The dummy antecedent indicates either a non-mention or an entity mention without a prior coreferent span.
- Baseline model: The baseline learns a distribution P(y_i) over possible antecedents for every span.The antecedent set includes the dummy option and all preceding spans.
- Baseline model: Pairwise coreference scores combine mention scores for both spans with an antecedent score s_a(i, j).The three factors assess whether each span is a mention and whether j is an antecedent of i.
- Representations and features: Span representations g_i are computed with bidirectional LSTMs and supplied to the mention and antecedent scoring functions.The antecedent scorer also uses element-wise span similarity and speaker, genre, and distance features.
- Inference and learning: A two-stage beam search retains up to M high-scoring mentions before computing pairwise coreference scores among survivors.This pruning is applied during both training and inference.
- Inference and learning: Training maximizes the marginal log-likelihood of potentially correct antecedents because each span’s best antecedent is latent.The supervision consists of gold coreference clusters.
3 Higher-order Coreference Resolution
The paper extends first-order coreference models with fully differentiable higher-order inference. Iteratively refined span representations let antecedent decisions softly condition on multiple hops in predicted clusters.
- First-order models score span pairs independently, making them susceptible to globally inconsistent clusters despite locally consistent links.Document-level higher-order decisions require explicit inference because mentions can be separated by large surface distances.
- The model performs N iterations of fully differentiable inference to condition coreference decisions on higher-order structures.Each iteration refines span representations and antecedent distributions using the baseline scoring function.
- At each iteration, antecedent probabilities weight previous span representations in an attention-based update.The baseline initializes the representations, which are then updated and used to refine antecedent distributions.
- A learned gate interpolates each span representation with its expected antecedent representation, dimension by dimension.The gate chooses whether to retain current span information or integrate information from the expected antecedent.
- After n iterations, a span representation can softly condition on up to n other spans in its predicted cluster.This interpretation assumes the antecedent distribution is peaked and treats the update as an element-wise weighted average.
- The iterative model can be interpreted as implicitly modeling directed paths of up to length N + 1 in latent antecedent trees.Span-ranking predicts antecedent trees whose trees correspond to predicted clusters.
4 Coarse-to-fine Inference
The coarse-to-fine approach reduces the cost of higher-order inference by pruning antecedents with an efficient bilinear factor before applying the more accurate scorer. It avoids a fixed maximum coreference distance and remains robust under aggressive pruning.
- The fine antecedent scorer is a computational bottleneck because it builds an M × M × (3|g| + |φ|) tensor at every inference iteration.Heuristic pruning reduces this to M × K × (3|g| + |φ|), but limits each span to its K nearest antecedents.
- Nearest-antecedent pruning imposes an a priori maximum link distance, even though coreference links can reach much farther in discourse.Previous work considered up to K = 250 nearest mentions.
- Reducing K from 250 to 50 drops distance-based pruning performance by almost 5 F1, versus less than 0.2 F1 for coarse-to-fine pruning.Figure 2 reports these development-set differences across the two antecedent-pruning strategies.
- The coarse-to-fine method uses a learned bilinear factor as an efficient rough sketch of likely antecedents without imposing an a priori maximum distance.The bilinear factor is less accurate than the fine scorer but can be learned end-to-end.
- Inference uses three beam-search stages: mention pruning, coarse antecedent pruning, then fine scoring with higher-order inference.The second stage ranks antecedents with sm(i) + sm(j) + sc(i, j), while the third computes the overall score on surviving pairs.
- The coarse-to-fine model achieves better performance with a much smaller K because its extra factor expands the set of learnable coreference links.The maximum-likelihood objective is still computed over span pairs retained by the final pruning stage.
5 Experimental Setup
The experiments use English CoNLL-2012 coreference data and largely reuse the baseline's settings, with selected embedding and hyperparameter changes. The final model uses more aggressive antecedent pruning, while development results favor second-order inference over third-order inference.
- The experiments use English coreference data from the CoNLL-2012 shared task, with replication code publicly available.
- The models reuse Lee et al.'s hyperparameters, while reporting ELMo and hyperparameter tuning as orthogonal improvements.
- The hyperparameter changes increase maximum span width from 10 to 30 words, add highway LSTMs, and use specified GloVe windows.
- The final model considers only 50 antecedents per span because coarse-to-fine pruning is insensitive to more aggressive pruning.The baseline considers up to 250 antecedents per span.
- 0.8 F1 separates the second-order model from the first-order model on development data, while third order adds only 0.1 F1.The authors therefore compute test results only for the second-order model.
6 Results
Evaluation uses the official CoNLL-2012 scripts and average F1 across MUC, B3, and CEAFφ4. The full coarse-to-fine, higher-order system reaches 73.0 F1, while using fewer expensive antecedent scores and improving recall and precision.
- The main evaluation is average F1 across MUC, B3, and CEAFφ4 precision, recall, and F1 scores.The official CoNLL-2012 evaluation scripts are used.
- 73.0 F1 makes the full approach a new state of the art, compared with 72.3 F1 for the augmented span-ranking baseline.The baseline includes ELMo and hyperparameter tuning.
- The coarse-to-fine model computes expensive sa(i, j) scores for 50 antecedents while coarse scores can cover all antecedents.This enables potential links between any two document spans and yields much higher recall.
- Second-order inference further improves performance, largely through increased precision as higher-order inference rules out inconsistent clusters.
7 Related Work
The paper builds higher-order coreference modeling on an existing pairwise span-ranking architecture. It contrasts a lightweight gating mechanism with more expressive recurrent cluster models.
- The proposed model uses the end-to-end span-ranking architecture as a building block for modeling more complex coreference structures.
- A relatively lightweight gating mechanism is reported as sufficient to effectively model higher-order structures.This is contrasted with Wiseman et al.'s highly expressive recurrent neural networks for clusters.
8 Conclusion
The paper presents a state-of-the-art coreference system that models higher-order interactions between spans. Its coarse-to-fine approach reduces the cost of higher-order inference while preserving end-to-end learnability.
- The system models higher-order interactions between spans in predicted clusters and achieves state-of-the-art coreference resolution.
- The coarse-to-fine approach alleviates the computational cost of higher-order inference while maintaining end-to-end learnability.