Source-linked AI summary
Coreferential Reasoning Learning for Language Representation
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, Zhiyuan Liu
TL;DR
Existing language representation models capture contextual semantics but do not explicitly handle coreference, which is important for coherent discourse understanding. CorefBERT introduces Mention Reference Prediction with mention-reference masking and copy-based training, and improves performance on coreferential-reasoning tasks while maintaining comparable results on common NLP tasks.
Problem
Existing language representation models mainly learn local information and do not explicitly model long-distance coreference, which is essential for coherent discourse understanding.
Method
CorefBERT pre-trains on unlabeled text with Mention Reference Prediction, masking repeated mentions and using a copy-based objective to select contextual referents.
Results
CorefBERT outperforms vanilla BERT on almost all coreferential-reasoning benchmarks, strengthens RoBERTa, and remains comparable to BERT on common NLP tasks.
Takeaways & Limitations
Coreferential reasoning can be incorporated into language-model pre-training from unlabeled text without impairing common language understanding.
Abstract
from arXiv · showhide
Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning. However, most existing language representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse. To address this issue, we present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks that require coreferential reasoning, while maintaining comparable performance to previous models on other common NLP tasks. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/CorefBERT.
1 Introduction
CorefBERT addresses the limited coreferential reasoning of existing language representation models by introducing a pre-training task that learns from repeated mentions in unlabeled text. It improves performance on tasks requiring coreferential reasoning while preserving comparable performance on common NLP tasks.
- Motivation: Existing masked language modeling mainly captures local semantics and syntax, limiting modeling of long-distance coreference across sentence boundaries.Coreference is essential for coherent discourse understanding and higher-level NLP tasks requiring full-text understanding.
- Motivation: Fine-tuning on small supervised coreference datasets did not improve downstream performance, while large-scale supervised data is impractical to obtain.
- Method: CorefBERT introduces Mention Reference Prediction, which masks repeated mentions and predicts their contextual referents from large-scale unlabeled text.The task uses repeated noun or noun-phrase mentions to acquire co-referring relations.
- Method: MRP adds a copy-based objective that encourages selecting referents from context rather than from the whole vocabulary.This is intended to produce more context-sensitive representations for coreferential reasoning.
- Results: CorefBERT outperforms vanilla BERT on almost all evaluated coreferential-reasoning benchmarks and strengthens RoBERTa, while remaining comparable to BERT on common NLP tasks.The evaluated tasks include extractive question answering, relation extraction, fact extraction and verification, and coreference resolution.
2 Related Work
Prior work developed contextual language representation models, alternative pre-training objectives, external-knowledge integration, multilingual learning, and unsupervised approaches to coreference resolution.
- Language Representation Models: Contextual language representation models extend earlier static embeddings by learning representations from large-scale unlabeled corpora.
- Pre-training Directions: Related pre-training directions include sequence-to-sequence learning, replaced-token detection, contrastive learning, external knowledge integration, and multilingual learning.
- Unsupervised Coreference Resolution: Unsupervised coreference research has used feature-based methods, language-model probabilities for pronoun resolution, and WikiCREM.
3 Methodology
CorefBERT combines masked language modeling with Mention Reference Prediction to learn coreferential reasoning from repeated mentions and contextual copying.
- Training tasks: CorefBERT trains with Mention Reference Prediction (MRP) alongside vanilla BERT’s Masked Language Modeling (MLM).MLM learns general language understanding, while MRP enhances coreferential reasoning.
- Copy-based Training Objective: MRP uses a copy-based objective that predicts masked mention tokens by copying corresponding unmasked context tokens.The copy mechanism is intended to help process low-frequency tokens such as proper nouns.
- Model architecture: The model encodes token and position embeddings with a deep bidirectional Transformer to obtain contextual representations for pre-training losses.The overall loss combines mention reference prediction loss L_MRP and masked language modeling loss L_MLM.
- Mention Reference Masking: Mention reference masking masks repeated mentions rather than random tokens, relying on unmasked references and context to recover them.The method assumes repeated mentions in a sequence refer to each other.
- Mention Reference Masking: Nouns are extracted, grouped by identical noun identity, and sampled by group to select masked mentions.For example, repeated occurrences of Jane or Claire are grouped before one mention is sampled.
- Training procedure: CorefBERT jointly samples MLM and MRP masks, using a 4:1 ratio while sampling 15% of tokens for each masking strategy.For both strategies, 80% are replaced by [MASK], 10% by random tokens, and 10% remain unchanged.
4 Experiment
The experiments evaluate CorefBERT across coreferential reasoning tasks and common language-understanding benchmarks. CorefBERT improves over BERT-based baselines on several reasoning-focused tasks while retaining comparable performance on GLUE.
- Experiments cover extractive question answering, relation extraction, fact verification, coreference resolution, and eight GLUE tasks.
- Extractive Question Answering: On QUOREF, CorefBERTBASE and CorefBERTLARGE exceed adapted BERT baselines by 4.4% and 2.9% F1, respectively.CorefRoBERTa achieves about 1% EM improvement over RoBERTa on QUOREF.
- Extractive Question Answering: Over 1% average F1 improvement occurs across all six MRQA datasets, including NewsQA and HotpotQA.The reported datasets are SQuAD, NewsQA, SearchQA, TriviaQA, HotpotQA, and Natural Questions.
- Relation Extraction: CorefBERTBASE and CorefBERTLARGE outperform BERT counterparts on DocRED by 0.7% and 0.5% F1, respectively.DocRED requires document-level relation extraction by synthesizing information from entity mentions across the document.
- Fact Extraction and Verification: KGAT with CorefBERTBASE improves FEVER score by 0.4% over KGAT with BERTBASE, while CorefRoBERTaLARGE improves it by 1.9% over RoBERTaLARGE.The latter reaches a new state-of-the-art on the FEVER benchmark.
- Coreference Resolution and GLUE: CorefBERT significantly outperforms BERT-LM on coreference resolution and achieves comparable performance to WikiCREM, while matching BERT on GLUE.WikiCREM is specialized for sentence-level coreference resolution; GLUE is not strongly dependent on coreference resolution.
5 Ablation Study
The ablation study compares masking and training-objective choices, finding that removing NSP and using mention-reference masking with copy-based training generally improves downstream performance.
- Deleting NSP improves performance on almost all evaluated tasks.
- MRM usually matches WWM, while both outperform original subword masking on NewsQA and TriviaQA.The gains are +1.7% F1 on NewsQA and +1.5% F1 on TriviaQA.
- The copy-based objective explicitly requires the model to find mention referents in context, incorporating sequence-level coreference information.
- +2.3% F1 on QUOREF results from adding the copy-based objective to MRM.
6 Conclusion and Future Work
The paper concludes that CorefBERT strengthens BERT's coreferential reasoning through Mention Reference Prediction and reports improvements over BERT and RoBERTa, while identifying noisy labels and omitted pronouns as future challenges.
- CorefBERT is trained with Mention Reference Prediction to strengthen BERT's coreferential reasoning ability.
- Experiments show CorefBERT significantly outperforms BERT and improves the performance of the strong RoBERTa model.
- The distant-supervision assumption can produce incorrect labels, leaving mitigation of labeling noise as an open problem.
- Because distant supervision does not consider pronouns, future work should develop strategies that incorporate them into coreferential reasoning.
Appendices
The appendix describes masked language modeling as predicting original tokens from corrupted input, using a specified mixture of mask, random-token, and unchanged replacements.
- Masked language modeling predicts missing tokens from their contextual representations.
- Among sampled tokens, 80% are replaced with [MASK], 10% with random tokens, and 10% remain unchanged.
- The task samples 15% of input-sequence tokens as missing tokens and predicts their originals from the corrupted sequence.
B Leaderboard Results on QUOREF
The QUOREF leaderboard section reports results for TASE with CorefRoBERTa and identifies the corresponding test-set leaderboard table.
- TASE with a CorefRoBERTa encoder gains about 1% EM over TASE with a RoBERTa encoder.
- The comparison indicates that CorefBERT improves performance across different question-answering frameworks.
- Table 8 presents leaderboard results on the QUOREF test set.
C Case Study on QUOREF
The QUOREF case studies show CorefBERT solving questions by linking mentions across sentences and aggregating the information they convey. Its predictions depend on resolving references such as pronouns and repeated names.
- C Case Study on QUOREF: CorefBERT links “the asthmatic boy” to Barry before combining two Mr. Lee mentions to infer that Noreen’s uncle trains him.The reasoning requires synthesizing relations stated in different mentions and positions.
- C Case Study on QUOREF: The case-study table contains examples correctly predicted by CorefBERTBASE but wrongly predicted by BERTBASE, with answers and clues distinguished.
- C Case Study on QUOREF: CorefBERT infers that Tippett is a composer from the second sentence to answer a question based on the first.
- C Case Study on QUOREF: CorefBERT resolves “she” to Elena and “he” to Ector, enabling accurate answers in the corresponding examples.The passage attributes these results to distant-supervised coreference resolution training data.
D Case Study on DocRED
The DocRED case study illustrates CorefBERT recovering relational facts by linking referring expressions across sentence boundaries. The FEVER example shows its encoder helping evidence aggregation for claim verification.
- D Case Study on DocRED: CorefBERT connects Eclipse across the first and third sentences to recover that New Moon and Breaking Dawn are novels in the Twilight Saga.
- D Case Study on DocRED: The FEVER table caption describes five evidence pieces retrieved from articles about Bob Ross and The Joy of Painting.
- D Case Study on DocRED: Linking “it,” “the novel,” and “the book” to Eclipse helps CorefBERT identify the novel’s characters and publication date.
- D Case Study on DocRED: With a CorefBERT encoder, KGAT propagates and aggregates entity information from evidence to refute the fabricated Joy of Painting claim more accurately.
F Task-Specific Model Details
The task-specific details cover optimization, data splits, model representations, evaluation settings, and efficiency across CorefBERT benchmarks. CorefBERT adds few parameters relative to BERT and has similar inference behavior for comparable architectures.
- Optimization and input settings: CorefBERT models use Huggingface Transformers and Adam optimization, with task-specific batch sizes and sequence-length limits.For QA, the batch size is 32 and the maximum sequence length is 512.
- Question answering: QUOREF uses 19399 / 2418 / 2537 train, development, and test instances, while MRQA fixes lr = 3 × 10^-5 and 2 epochs across experiments.
- Document-level relation extraction: DocRED represents each mention by averaging its word representations, each entity by averaging its mentions, and entity pairs through a bi-linear relation layer.
- Document-level relation extraction: DocRED uses 3053 / 1000 / 1000 train, development, and test documents, with the best reported configuration using learning rate 4 × 10^-5 and 200 epochs.
- Coreference resolution: The WikiCREM loss combines correct-candidate negative log-likelihood with a max-margin term, using lr = 3 × 10^-5, α = 10, and β = 0.2 in the best validation configuration.
- Model size and efficiency: CorefBERT adds a few parameters for its copy-based objective while keeping a similar parameter count to same-size BERT models.