Source-linked AI summary
BERT for Coreference Resolution: Baselines and Analysis
Mandar Joshi, Omer Levy, Daniel S. Weld, Luke Zettlemoyer
TL;DR
Coreference resolution requires models to identify which spans refer to the same entity across benchmarks and contexts. The paper fine-tunes BERT within c2f-coref using span representations and independent or overlapping document segments. BERT-large improves substantially on OntoNotes and GAP, but errors remain in document context, conversations, and mention paraphrasing.
Problem
Coreference resolution remains challenging across paragraph- and document-level benchmarks, motivating evaluation of BERT for this task.
Method
The paper replaces c2f-coref’s LSTM encoder with BERT and uses span representations with independent or overlapping document segments.
Results
BERT-large improves over ELMo-based c2f-coref by 3.9% on OntoNotes and 11.5% on GAP, both absolute.
Takeaways & Limitations
BERT-large is particularly better at distinguishing related but distinct entities, while effective document context and difficult conversational and paraphrasing cases remain unresolved.
Takeaways & Limitations
Errors indicate remaining difficulty with document-level context, conversational pronouns, and mention paraphrasing.
Abstract
from arXiv · showhide
We apply BERT to coreference resolution, achieving strong improvements on the OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. A qualitative analysis of model predictions indicates that, compared to ELMo and BERT-base, BERT-large is particularly better at distinguishing between related but distinct entities (e.g., President and CEO). However, there is still room for improvement in modeling document-level context, conversations, and mention paraphrasing. Our code and models are publicly available.
1 Introduction
The paper applies BERT to coreference resolution and reports strong gains on GAP and OntoNotes, while analysis identifies persistent challenges in context, conversations, and paraphrased mentions.
- BERT-large improves over ELMo-based c2f-coref by 3.9% on OntoNotes and 11.5% on GAP, both absolute.
- BERT-large is better than BERT-base and ELMo at distinguishing related but distinct entities or concepts.Examples include Repulse Bay versus Victoria Harbor.
- Both BERT variants often struggle with coreference cases requiring world knowledge, such as “the developing story” and “the scandal.”
- Pronoun modeling remains difficult, especially in conversations.
- Longer context windows and overlapping segments do not consistently improve performance, indicating limitations in effective document-level context modeling.BERT-large benefits from 384 word pieces, BERT-base from 128, but both perform much worse at 512 tokens; overlap provides no improvement.
2 Method
The method replaces c2f-coref’s LSTM encoder with BERT and adapts span representations and document segmentation to handle coreference inputs within limited context windows.
- The model uses c2f-coref, a higher-order coreference model, as the experimental base.
- For each mention span, the model learns a distribution over possible antecedent spans.
- Span-pair scores combine mention scores for each span with their joint compatibility as references to the same entity.
- The BERT adaptation replaces the LSTM encoder and represents spans using endpoint word-pieces plus attention over span tokens.
- Documents are processed with either independent non-overlapping segments or overlapping segments whose token representations are interpolated.The overlap variant creates a segment every T/2 tokens and combines representations from both segments.
3 Experiments
The experiments evaluate BERT-extended c2f-coref on paragraph-level GAP and document-level OntoNotes, comparing variants, baselines, and context-window settings. BERT improves coreference performance, but longer document context remains difficult to exploit.
- Experimental setup: The experiments compare BERT-extended c2f-coref with ELMo-based c2f-coref and e2e-coref on GAP and OntoNotes.OntoNotes is document-level, whereas GAP examples fit within a single BERT segment.
- OntoNotes evaluation: BERT improves c2f-coref by 0.9% and 3.9% on OntoNotes for the base and large variants, respectively.The main evaluation is average F1 across MUC, B3, and CEAFφ4.
- GAP evaluation: BERT improves c2f-coref by 11.5% on GAP, while the base model improves by 9%.GAP evaluates F1 on masculine, feminine, and overall examples, plus a bias factor.
- Document-level analysis: Performance generally drops as OntoNotes document length and cluster spread increase.Spread is measured by the average number of tokens between the first and last mentions in a cluster.
4 Analysis
The analysis finds that BERT-large more reliably separates related entities, while coreference remains difficult for long documents, conversations, and paraphrased mentions.
- Qualitative comparison: 93 errors were found for BERT-base and 74 for BERT-large across the same 15 OntoNotes development documents.Incorrect clusters could belong to multiple error categories.
- Strengths: BERT-large improves over BERT-base in pronoun resolution and lexical matching, and more often avoids merging distinct entities.Examples include distinguishing Ocean Theater from Marine Life Center.
- Weaknesses: Better modeling of document-level context, conversations, and entity paraphrasing could further improve coreference resolution.These weaknesses are identified from error analysis rather than benchmark scores alone.
- Document context: Models perform distinctly worse on longer OntoNotes documents, whose clusters are generally larger and more spread out.The analysis links document length with increasingly difficult coreference structure.
- Document context: Using larger segments or overlapping segments does not improve results, despite the intended extension of available context.The reported segment lengths include 450 and 512, while overlapping segments provide no improvement.
- Model scale: Larger BERT models may encode longer contexts better, but they also intensify the memory costs of span representations.The paper suggests sparse representations as a direction for encoding document-level context more effectively.
5 Related Work
Coreference resolution has traditionally relied on scoring span or mention pairs, while recent progress increasingly uses unsupervised contextualized representations such as BERT.
- Span-based models: Span or mention-pair scoring has been one of the dominant paradigms in coreference resolution.The paper's Lee et al. (2018) base model belongs to this family.
- Contextualized representations: Recent advances in coreference resolution and other NLP tasks have been driven by unsupervised contextualized representations.The cited examples include ELMo, BERT, and other contextualized representation methods.
- BERT: BERT uses passage-level pretraining with bidirectional masked language modeling to model long-range dependencies more effectively.This positions BERT's pretraining strategy as distinct from sentence-level contextualization.