Source-linked AI summary
Deep Reinforcement Learning for Mention-Ranking Coreference Models
Kevin Clark, Christopher D. Manning
TL;DR
Coreference systems rely on heuristic losses with carefully tuned hyperparameters because evaluation metrics do not decompose over local decisions. This paper instead applies reinforcement learning to a neural mention-ranking model, comparing REINFORCE with reward-rescaled max-margin training. Reward rescaling significantly outperforms both REINFORCE and the heuristic loss on English and Chinese CoNLL 2012 data, producing significant gains over the current state of the art.
Problem
Coreference evaluation metrics do not decompose over local linking decisions, leading systems to use heuristic losses with hyperparameters that require careful tuning across languages and datasets.
Method
The paper applies reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics, using REINFORCE and reward-rescaled max-margin training.
Results
Reward rescaling significantly outperforms REINFORCE and the heuristic loss on both English and Chinese portions of the CoNLL 2012 Shared Task.
Takeaways & Limitations
The reward-rescaling approach yields significant gains over the current state of the art while directly optimizing coreference evaluation metrics.
Abstract
from arXiv · showhide
Coreference resolution systems are typically trained with heuristic loss functions that require careful tuning. In this paper we instead apply reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics. We experiment with two approaches: the REINFORCE policy gradient algorithm and a reward-rescaled max-margin objective. We find the latter to be more effective, resulting in significant improvements over the current state-of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task.
1 Introduction
Coreference systems make local linking decisions, but evaluation metrics do not decompose over those decisions, complicating training with heuristic losses and tuned hyperparameters. The paper applies reinforcement learning to directly optimize evaluation metrics and finds reward rescaling more effective than REINFORCE and heuristic loss functions.
- Coreference systems make sequences of local mention-linking decisions, while evaluation metrics assess global outcomes whose utility is unavailable for individual decisions.
- Heuristic training losses define the goodness of individual decisions and require hyperparameters tuned across languages, datasets, and evaluation settings.
- The paper explores reinforcement-learning variants that directly optimize coreference systems for coreference evaluation metrics.
- The model is a neural mention-ranking system whose independent actions make each action’s effect on final reward efficient to compute.
- REINFORCE is competitive with the heuristic loss, whereas reward rescaling significantly outperforms both.
2 Neural Mention-Ranking Model
The neural mention-ranking model scores candidate antecedent–mention pairs using feedforward networks and links each mention to its highest-scoring candidate at test time.
- For each mention m and candidate antecedent c, the model uses a feedforward neural network to produce a compatibility score s(c, m).
- Candidate antecedents include preceding mentions and NA, which indicates that the mention has no antecedent.
- Input Layer: The input combines word and word-group representations with distance, string-matching, and speaker-identification features.
- Hidden Layers: The input passes through three fully connected hidden layers of rectified linear units.
- Scoring Layer: At test time, each mention links to the candidate antecedent with the highest score.
3 Learning Algorithms
The paper replaces heuristically tuned coreference losses with reinforcement-learning objectives that directly optimize evaluation rewards in a neural mention-ranking model. It compares REINFORCE with reward-rescaled max-margin training, using independent mention-linking actions to compute reward-sensitive losses.
- Heuristic Max-Margin Objective: Heuristic mention-ranking losses assign error costs and require hyperparameter tuning to align training with coreference metrics.The paper searches over false-anaphor and false-new penalties, finding different best settings for English and Chinese.
- Heuristic Max-Margin Objective: The heuristic objective is a slack-rescaled max-margin loss based on error types for false-new, false-anaphor, wrong-link, and correct decisions.For each mention, candidate antecedents include preceding mentions and NA; the model compares the highest-scoring true antecedent with candidate links.
- Reinforcement Learning: Mention-ranking treats each mention-to-antecedent link as an independent action, with the model selecting the highest-scoring candidate at each step.This independence enables evaluating alternative actions while holding the rest of the action sequence fixed.
- Reward Rescaling: The reward-rescaled objective sets each action’s slack according to how much replacing that action changes the final reward, rather than its heuristic error type.The reward-sensitive comparison is computed relative to the model’s highest-scoring action sequence.
- REINFORCE: REINFORCE maximizes expected reward by sampling action sequences, while using action independence and a baseline to reduce gradient-estimate variance.The full expectation is exponential in sequence length, so the method uses an unbiased sampled gradient estimate and a lower-variance independence-based estimate.
4 Experiments and Results
Experiments on English and Chinese CoNLL 2012 compare heuristic loss, REINFORCE, and reward-rescaled max-margin training. Reward rescaling significantly outperforms both alternatives, apparently by prioritizing costly decisions and producing less severe errors.
- Results: Reward rescaling significantly outperforms heuristic loss and REINFORCE on both English and Chinese CoNLL 2012 data.REINFORCE is only slightly better than the heuristic loss.
- The Benefits of Reinforcement Learning: 0.79 for false-negative errors and 0.38 for false-anaphoric errors are the English average reward-based costs when wrong-link cost is scaled to 1.0.These averages are close to the grid-searched heuristic settings, but costs vary substantially within each error type.
- The Benefits of Reinforcement Learning: Reward rescaling incorporates reward-based costs into the max-margin loss, directly prioritizing decisions with larger effects on final coreference scores.The action cost also depends on existing cluster errors and the utilities of alternative actions.
- The Benefits of Reinforcement Learning: The reward-rescaling model makes slightly more total errors, so its performance improvement comes from errors being less severe.Table 2 reports false-new, false-anaphoric, and wrong-link error counts on the English test set.
- The Benefits of Reinforcement Learning: Proper nouns have average false-new cost 0.90 versus 0.77 for other mention types; reward rescaling reduces these errors while increasing mostly low-cost false-anaphoric errors.The figures come from English CoNLL 2012 test-set examples of mention classes improved by reward-based costs.
- The Benefits of Reinforcement Learning: Wrong-link errors in telephone-conversation pronoun clusters have a high average cost of 1.21, and reward rescaling produces significantly fewer such errors than the heuristic loss.The large clusters make incorrect links especially damaging to the score.
5 Related Work
Prior coreference work largely trains mention-ranking models with heuristic error-cost losses, while related imitation-learning methods directly optimize evaluation metrics using expert policies. Reinforcement learning differs by relying on rewards rather than an expert policy.
- Mention-ranking coreference models have commonly been trained with heuristic loss functions that assign costs to different error types.
- Earlier imitation-learning approaches, including SEARN, directly optimize coreference evaluation metrics for training coreference resolvers.
- Unlike reinforcement learning, imitation learning requires an expert policy rather than relying on rewards alone.
6 Conclusion
The paper applies reinforcement learning to directly optimize mention-ranking models for coreference evaluation metrics, avoiding carefully selected hyperparameters for each language, dataset, and metric. The reward-rescaling approach also improves accuracy and achieves significant gains over the current state-of-the-art.
- Reinforcement learning directly optimizes mention-ranking models for coreference evaluation metrics.
- The approach obviates hyperparameters that must be carefully selected for each language, dataset, and evaluation metric.
- The reward-rescaling approach increases model accuracy and produces significant gains over the current state-of-the-art.