PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing the Myth of Higher-Order Inference in Coreference Resolution

Liyan Xu, Jinho D. Choi

arXiv:2009.12013v2cs.CLcs.LG

TL;DR

The paper examines how effective higher-order inference is for coreference resolution, where its contribution has not been studied extensively relative to representation learning. It compares four HOI methods with BERT and SpanBERT, finding HOI’s impact negative to marginal with a high-performing encoder; cluster merging reaches 80.2 Avg-F1 on CoNLL 2012.

  • Problem

    Recent coreference models commonly use HOI, but its effectiveness relative to representation learning has received limited investigation.

    Source: §1 · p. 1

  • Method

    The authors implement an end-to-end coreference system and evaluate four HOI approaches, including two proposed methods, with BERT and SpanBERT encoders.

    Source: §1 · p. 1

  • Results

    80.2 Avg-F1: cluster merging surpasses the previous best 79.6, while other HOI models show no clear advantage over SpanBERT without HOI.

    Source: §4.1 · p. 3

  • Takeaways & Limitations

    The authors conclude that current HOI does not meet its original motivation in this setting and that coreference resolution needs a new perspective on HOI in deep learning-based NLP.

    Source: §5 · p. 5

    Because HOI and local ranking are mutually dependent during training, HOI’s indirect influence on local decisions makes its true impact difficult to assess.

    Source: §4.2 · p. 4

Abstract

from arXiv · show

This paper analyzes the impact of higher-order inference (HOI) on the task of coreference resolution. HOI has been adapted by almost all recent coreference resolution models without taking much investigation on its true effectiveness over representation learning. To make a comprehensive analysis, we implement an end-to-end coreference system as well as four HOI approaches, attended antecedent, entity equalization, span clustering, and cluster merging, where the latter two are our original methods. We find that given a high-performing encoder such as SpanBERT, the impact of HOI is negative to marginal, providing a new perspective of HOI to this task. Our best model using cluster merging shows the Avg-F1 of 80.2 on the CoNLL 2012 shared task dataset in English.

1 Introduction

Recent coreference gains have come largely from contextualized representations, while the contribution of higher-order inference (HOI) has received less direct examination and reported gains have been marginal. The paper compares four HOI approaches alongside an end-to-end system to assess their effectiveness with BERT and SpanBERT encoders.

1 Introduction

  • Recent state-of-the-art coreference performance improved 12.4% over a model from 2.5 years earlier, with most improvement attributed to representation learning.The introduction points to contextualized encoders such as ELMo and BERT as contributors to this progress.Source: §1 · p. 1
  • Although most recent models use HOI for global coreference-link optimization, reported gains have been marginal and its effectiveness merits direct analysis.The authors frame the analysis as informing future research directions for HOI in coreference resolution.Source: §1 · p. 1
  • The study implements an end-to-end system and compares four HOI approaches with BERT and SpanBERT encoders.The approaches are attended antecedent and entity equalization from prior work, plus the authors’ span clustering and cluster merging methods.Source: §1 · p. 1

2 Related Work

Prior coreference systems typically rank antecedents locally, while some work adds entity-level information or refines mention representations. Progress has also tracked stronger contextualized encoders, and a question-answering framework recently reported a new CoNLL’12 result.

2 Related Work

  • Antecedent-ranking models make local decisions between each mention and its antecedents.The cited systems include Wiseman et al. (2015), Clark and Manning (2015), Lee et al. (2017, 2018), and Joshi et al. (2019, 2020).Source: §2 · p. 1
  • Some HOI approaches build entity representations during ranking, while others refine mention representations by aggregating antecedent information.The former are associated with Wiseman et al. (2016), Clark and Manning (2016), and Yu et al. (2020); the latter with Lee et al. (2018) and Kantor and Globerson (2019).Source: §2 · p. 2
  • ELMo, BERT, and SpanBERT each brought reported gains over earlier encoders, including 3.3% for ELMo, another 3.3% for BERT, and 2.7% for SpanBERT.These comparisons are reported against the preceding models in the same end-to-end coreference line.Source: §2 · p. 2
  • A question-answering framework for coreference resolution reported a state-of-the-art CoNLL’12 result of 83.1.The paper attributes this result to Wu et al. (2020).Source: §2 · p. 2

3 Approach

The approach pairs an end-to-end antecedent-ranking system with span-refinement and clustering-based higher-order inference methods, including two new methods: span clustering and cluster merging.

3.1 End-to-End Coreference System

  • Candidate spans are scored for mention validity and coreference, then assigned a distribution over their antecedents.The system uses Transformer-based encoders and an independent setup for long documents.Source: §3.1 · p. 2

3.2 Span Refinement

  • Span refinement interpolates each span with nonlocal context before another ranking round; attended antecedent aggregates antecedents, while entity equalization models entity distributions through clustering relaxation.Source: §3.2 · p. 2

3.3 HOI with Clustering

  • Span clustering constructs actual clusters from antecedent probabilities and uses each span’s cluster representation as refined context, trading differentiability for the same empirical inference time as entity equalization.Source: §3.3 · p. 3
  • Cluster merging sequentially ranks antecedents using both antecedent and cluster scores, then merges the chosen clusters; training searches and merges clusters without oracle clusters.The ranking can be left-to-right or easy-first, with element-wise mean or max reduction when merging.Source: §3.3 · p. 3

4 Experiments

On the CoNLL 2012 English dataset, the experiments compare BERT and SpanBERT baselines with four HOI methods. SpanBERT's gains exceed those of most HOI methods, while cluster merging achieves the best HOI result.

4.1 Results

  • On CoNLL’12 English shared-task test data, SpanBERT improved 2.4% over BERT on average; no HOI model clearly surpassed SpanBERT, and CM achieved Avg-F1 80.2 versus c2f-coref’s previous-best 79.6 (Joshi et al., 2020).Source: §4.1 · p. 3, Table 1 · p. 4

4.2 Impact Analysis of HOI

  • Less than 0.2 Avg-F1 was lost across all methods when HOI was disabled at test time, indicating no significant direct impact on final performance.The test-time comparison uses trained HOI models with HOI turned off.Source: §4.2 · p. 4
  • About 1% of links became correct after HOI, with roughly the same share becoming wrong, so the opposing changes limited overall improvement.Table 2 reports link changes before and after HOI.Source: Table 2 · p. 4, §4.2 · p. 4
  • Because HOI and local ranking are mutually dependent during training, their interaction makes HOI’s true impact difficult to assess.The paper identifies this indirect influence as an issue for future investigation.Source: §4.2 · p. 4

4.3 Analysis of Pronoun Resolution

  • In direct pronoun links, HOI had slightly more impact than switching encoders; AA lowered singular-to-plural errors while raising reverse errors, whereas BERT–SpanBERT differences were trivial.The analysis excludes ambiguous pronouns such as “you.”Source: Table 3 · p. 4, §4.3 · p. 4
  • EE and CM reduced false-link errors by 4+%, while SpanBERT reduced wrong-link errors almost 10% compared with BERT, indicating greater encoder impact on semantic matching.False links connect a non-anaphoric pronoun to another pronoun; wrong links select an incorrect pronoun antecedent.Source: Table 3 · p. 4, §4.3 · p. 5
  • Very few erroneous mixed-number clusters contained ambiguous pronouns; SpanBERT reduced these clusters 10%, while the four HOI methods showed no significant difference compared with SpanBERT.The analysis concerns clusters containing both singular and plural pronouns.Source: Table 3 · p. 4, §4.3 · p. 5

5 Conclusion

The paper evaluates four higher-order inference (HOI) methods in an end-to-end coreference system and finds that, with SpanBERT, HOI’s impact is negative to marginal. Cluster merging achieves Avg-F1 of 80.2 on CoNLL 2012.

5 Conclusion

  • The study implements an end-to-end coreference system and investigates four HOI methods, including two original methods.The conclusion identifies span clustering and cluster merging as the two original approaches.Source: §5 · p. 5
  • 80.2 Avg-F1 on CoNLL 2012 is the best model’s result, achieved with cluster merging.The conclusion presents this as a new result on the dataset.Source: §5 · p. 5
  • With SpanBERT, HOI does not meet its original motivation and has negative-to-marginal impact compared with encoder improvements.The paper argues that HOI needs a new perspective for coreference resolution in deep learning-based NLP.Source: §5 · p. 5

A.1 Experimental Settings

Experiments use BERTLarge and SpanBERTLarge encoders, selecting the best development-set model for test evaluation. The appendix also reports training settings, runtime, and an EE-specific candidate cap imposed by memory requirements.

A.1 Experimental Settings

  • The best-performing model on the development set is selected for test evaluation in each experiment.The encoders are BERTLarge and SpanBERTLarge.Source: §A.1 · p. 7
  • Documents are segmented to at most 384 word pieces for BERTLarge and 512 for SpanBERTLarge.The final setting uses separate learning rates of 1×10−5 for BERT parameters and 3×10−4 for task parameters, with 24 training epochs and dropout 0.3.Source: §A.1 · p. 7
  • EE is limited to 300 span candidates to keep GPU memory within 32GB, which may negatively affect its performance.The EE implementation requires O(k^2) memory, compared with O(k) for the other HOI approaches.Source: §A.1 · p. 7
  • Average training takes around 7 hours without HOI and 9–15 hours with HOI on Nvidia Tesla V100 GPUs with 32GB memory.Source: §A.1 · p. 7

A.2 Results

Across five repeated developments, cluster merging has the best macro-average F1, while attended antecedent, entity equalization, and span clustering remain below local-only SpanBERT. The final cluster-merging configuration uses sequential order and max reduction.

A.2 Results

  • 79.9 averaged F1 is the best performance, achieved by cluster merging across five repeated developments.The results are reported as macro-average F1 scores.Source: §A.2 · p. 7
  • The maximum number of antecedents is set to 50 for all models.Source: §A.2 · p. 7
  • Attended antecedent, entity equalization, and span clustering have lower F1 scores than local-only SpanBERT.Source: §A.2 · p. 7
  • The final cluster-merging configuration uses sequential order and max reduction, as other tested configurations had no large performance impact.Source: §A.2 · p. 7

A.3 Analysis

The analysis describes a test-set evaluation that turns off HOI in trained models to assess its direct impact on local decisions. It also identifies the pronouns treated as ambiguous in the pronoun-resolution analysis.

A.3 Analysis

  • Table 5 reports performance drops and standard deviations for Avg-F1 after the corresponding HOI is turned off in trained models.This evaluation is intended to examine HOI’s direct impact over local decisions.Source: §A.3 · p. 7, Table 5 · p. 7
  • The pronoun-resolution analysis treats “you”, “your”, and “yours” as ambiguous personal pronouns.Source: §A.3 · p. 7
  • Table 4 reports macro-average F1 scores from five repeated developments for each approach on the CoNLL 2012 English test set.Source: Table 4 · p. 7
Loading 2009.12013v2…