PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, Jie Zhou

arXiv:2010.01923v2cs.CL

TL;DR

The paper asks which information neural relation-extraction models use and how to improve them without relying on shallow entity cues. It studies textual context and entity mentions, then introduces entity-masked contrastive pre-training. The analyses find that context is primary but entity types matter, while the framework improves effectiveness and robustness across relation-extraction settings.

  • Problem

    The paper addresses limited understanding of how textual context and entity mentions affect neural relation-extraction decisions and how models can be improved without superficial mention cues.

  • Method

    The paper proposes entity-masked contrastive pre-training using Wikidata sentences with similar or different relational semantics while randomly masking entity mentions.

  • Results

    Experiments find that context is the main predictive source, entity mentions provide critical information mainly through types, and datasets may leak shallow mention cues; the framework improves effectiveness and robustness across settings.

  • Takeaways & Limitations

    Understanding relational context and entity types while limiting entity memorization is supported as a direction for more effective and robust relation-extraction models.

  • Takeaways & Limitations

    The pre-training examples can contain noisy or incorrect relations because generated sentences may express no relation or a different relation than expected.

Abstract

from arXiv · show

Neural models have achieved remarkable success on relation extraction (RE) benchmarks. However, there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models. To this end, we empirically study the effect of two main information sources in text: textual context and entity mentions (names). We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks. Based on the analyses, we propose an entity-masked contrastive pre-training framework for RE to gain a deeper understanding on both textual context and type information while avoiding rote memorization of entities or use of superficial cues in mentions. We carry out extensive experiments to support our views, and show that our framework can improve the effectiveness and robustness of neural models in different RE scenarios. All the code and datasets are released at https://github.com/thunlp/RE-Context-or-Names.

1 Introduction

Relation extraction models use both textual context and entity mentions, but their respective influence and the possibility of dataset shortcuts remain unclear. The paper studies these information sources and proposes entity-masked contrastive pre-training to improve contextual and type understanding while limiting memorization and biased mention cues.

  • Motivation: Relation extraction identifies relational facts between entities in text and supports knowledge-graph construction, completion, and downstream applications.Examples include extracting (SpaceX, founded by, Elon Musk) and supporting question answering, dialog systems, and search.
  • Motivation: Neural relation extraction models achieve strong benchmark performance, motivating analysis of what information they actually use.The paper frames this analysis as a way to understand model behavior and identify problems for further study.
  • Information sources: Textual context and entity mentions are the two main information sources available for relation classification.Context expresses relational patterns such as “be founded ... by,” while mentions can provide entity types, linked knowledge, and pretrained entity knowledge.
  • Findings: Context is the main source supporting predictions, but entity mentions also provide critical information, most of it related to entity types.The experiments examine how much existing models rely on each source.
  • Findings: Existing relation-extraction benchmarks may leak shallow statistical cues through entity mentions, contributing to models’ high performance.Models can still achieve high performance when given only entity mentions, indicating biased cues in the datasets.
  • Approach: The proposed entity-masked contrastive pre-training framework learns relational semantics from context while using entity types and avoiding rote mention memorization.It uses Wikidata sentences expressing similar relations, contrasts relationally similar and dissimilar sentences, and randomly masks entity mentions.

2 Pilot Experiment and Analysis

The pilot study tests how neural relation extraction models use textual context and entity mentions under controlled input formats. Results show that both sources matter, entity types explain much of mention information, and datasets contain shallow mention-based cues.

  • Main findings: OnlyC, OnlyM, and OnlyT perform substantially worse than C+M and C+T, showing that context and entity mentions both support relation classification.The comparison indicates that relying on only one information source is insufficient for correct prediction.
  • Main findings: C+T achieves results comparable to C+M for BERT and MTB, indicating that most useful mention information is entity type information.Case analysis finds 95.7% overlap in correct predictions between C+M and C+T, while 68.1% of C+M errors are shared with C+T.
  • Dataset biases: Models achieve high performance with entity mentions alone, suggesting that existing RE datasets leak superficial statistical cues through mentions.The same phenomenon appears beyond TACRED, and mention distributions can bias predictions toward relations seen with particular entities.
  • Model differences: CNN behaves differently from BERT and MTB: its OnlyC score is nearly equal to OnlyM, while C+M is 5% lower than C+T.The authors attribute this pattern to CNN’s limited encoding capacity and greater tendency to overfit mention-based cues.
  • Case study: Among OnlyC errors, 42% involve clear relational patterns that the model fails to extract, suggesting that C+M can exploit mention heuristics when context is misunderstood.Other errors arise because masking makes relations difficult for humans to identify or leaves sentences ambiguous.

3 Contrastive Pre-training for RE

The framework uses Wikidata-linked sentence pairs for relational contrastive pre-training, while masking entity mentions to learn relational context and entity types without simple memorization.

  • 3.1 Relational Contrastive Example Generation: Compared with MTB, the framework samples more diverse entity pairs and contexts, increasing type coverage while reducing entity-name memorization.MTB requires sampled sentences to share the same entity pair, which reduces noise but limits diversity and type learning.
  • 3.1 Relational Contrastive Example Generation: The method gathers sentences linked to the same Wikidata relation as positive pairs and samples other sentences as negative pairs.Sentences sharing a relation should receive similar representations, while sentences with different relations should be separated.
  • 3.1 Relational Contrastive Example Generation: Randomly masking entity mentions reduces memorization of entity names while preserving opportunities to learn entity-type information.The masking ratio is set to PBLANK = 0.7; masking every mention would prevent models from using entity information during fine-tuning.
  • 3.1 Relational Contrastive Example Generation: The generated examples may contain sentences with no relation or a relation different from the expected one, so pre-training tolerates noisy supervision.The framework seeks representations relatively better suited to RE than raw BERT, rather than directly training a downstream RE model.
  • 3.2 Training Objectives: Training uses a BERT-style Transformer with entity markers and combines contrastive learning with masked language modeling.The contrastive objective brings representations of same-relation sentences closer, while MLM preserves contextual language knowledge and helps avoid catastrophic forgetting.
  • 3.2 Training Objectives: The combined objective optimizes relational representations while retaining general language understanding inherited from BERT.The contrastive loss encourages same-relation sentences to have similar representations, and MLM supplies contextual semantic and syntactic knowledge.

4 Experiment

Experiments span supervised, few-shot, cross-domain, and low-resource RE settings. Contrastive pre-training improves both context and type-information use, with larger gains when annotations are scarce.

  • RE Tasks: The evaluation covers supervised RE on TACRED, SemEval, Wiki80, and ChemProt, plus few-shot learning on FewRel.Supervised experiments include 1% and 10% training-data settings; FewRel evaluates N-way K-shot classification.
  • Strength of Contrastive Pre-training: CP improves model performance across C+M, OnlyC, and OnlyM settings, indicating gains in context understanding and type-information extraction.The comparison includes BERT, MTB, and the proposed contrastive pre-trained model.
  • Strength of Contrastive Pre-training: Performance gains on C+M and OnlyC are universal, including biomedical ChemProt and FewRel 2.0, suggesting relational patterns transfer across domains.The models are pre-trained on Wikipedia but evaluated on biomedical datasets.
  • Strength of Contrastive Pre-training: CP prominently improves OnlyM on TACRED, Wiki80, and FewRel 1.0, indicating stronger extraction of type information from mentions.These datasets are closely related to Wikipedia.
  • Strength of Contrastive Pre-training: 7% improvement occurs on 10-way 1-shot FewRel 1.0, 18% on 1% TACRED, and 24% on 1% Wiki80 for C+M.Similar gains occur for OnlyC and OnlyM, with larger improvements in low-resource and few-shot settings.

5 Related Work

Prior RE research spans multiple modeling paradigms and increasingly uses pre-trained language models. This paper addresses a less systematically analyzed question: how models use context versus entity mentions and their shallow cues.

  • Development of RE: RE research has progressed through pattern-based, feature-based, kernel-based, graphical, and neural methods.Neural RE became prominent after recursive neural networks were introduced for relation extraction.
  • Pre-training for RE: BERT-based RE pre-training learns relational patterns or injects entity knowledge, whereas this paper focuses on information available in text itself.The paper distinguishes its focus from approaches whose gains come from external relational knowledge.
  • Analysis of RE: Earlier work suggested studying context and mentions and identified possible shallow mention cues, but the authors describe their analysis as the first systematic treatment.The paper combines experiments and case studies to examine both information sources.

6 Conclusion

The paper finds that both textual context and entity mentions—mainly type information—matter for neural RE, while datasets may leak superficial mention cues. It proposes entity-masked contrastive pre-training and reports effective results.

  • Conclusion: Experiments and case studies show that context and entity mentions, mainly type information, both provide critical information for relation extraction.The conclusion treats these as separate information sources affecting RE models.
  • Conclusion: Existing RE datasets may leak superficial cues through entity mentions, and models may not understand context as strongly as expected.The conclusion frames this as a finding from the paper’s analyses.
  • Conclusion: The proposed entity-masked contrastive pre-training framework targets better understanding of textual context and entity types.Experimental results are reported as supporting the method’s effectiveness.
  • Conclusion: Future work focuses on open relation extraction and relation discovery, where relational representations are needed with limited or zero annotations.These are identified as continuing directions rather than evaluated settings in this paper.

A Pre-training Details

The pre-training dataset is built from Wikipedia linked to Wikidata, and contrastive training groups same-relation sentences as positives. The implementation uses BERTBASE-scale models and reports distinct training costs for MTB and CP.

  • Pre-training Dataset: The pre-training corpus contains 744 relations and 867,278 sentences constructed by linking Wikipedia mentions to Wikidata entities.Anchors, spaCy mention discovery, and name matching are used for entity linking.
  • Pre-training Dataset: Entity pairs without Wikidata relations are filtered from the MTB pre-training data because they are expected to contribute little relational information.The filtering also improves training efficiency.
  • Data Sampling Strategy: For CP, sentences with the same relation form positive pairs, while sentences from different relation bags form negative pairs.Positive and negative samples are dynamically constructed from relation-labeled sentence bags.
  • Hyperparameters: PBLANK is the probability of replacing entities with [BLANK] in the pre-training models’ hyperparameter specification.Hyperparameters are selected using TACRED micro F1, with implementation details reported for MTB and CP.
  • Pre-training Efficiency: The models use BERTBASE architecture with approximately 110M parameters, while CP pre-training takes 3,500 steps and approximately 12 hours.The reported setup uses four Nvidia 2080Ti GPUs.

B RE Fine-tuning

The experiments use several supervised and few-shot relation extraction datasets, with controlled training-data proportions, task-specific hyperparameters, and repeated or high-volume evaluation procedures.

  • RE Datasets: The study uses TACRED, Wiki80, SemEval, ChemProt, and FewRel for relation extraction experiments.TACRED is downloaded from LDC, while the other datasets are obtained from the sources specified in the passage.
  • RE Datasets: Supervised RE training data are evaluated under 1%, 10%, and 100% proportion settings, sampled separately for each relation.The resulting totals are not necessarily exact proportions of the original datasets.
  • RE Datasets: The 1% SemEval and ChemProt settings contain extremely few training instances, producing abnormal performance.
  • Hyperparameters: BERT, MTB, and CP use task-specific fine-tuning hyperparameters, while CNN is trained with SGD at learning rate 0.5, batch size 160, and hidden size 230.Few-shot RE uses the recommended FewRel hyperparameters.
  • Multiple Trial Settings: Supervised results are medians across five seeds, whereas few-shot RE uses one trial with 10,000 evaluation episodes.The supervised seeds are 42, 43, 44, 45, and 46; the reported Wiki80 and FewRel metric details continue beyond the supplied passage.
Loading 2010.01923v2…