Source-linked AI summary
Matching the Blanks: Distributional Similarity for Relation Learning
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski
TL;DR
General-purpose relation extractors have limited ability to generalize, motivating task-agnostic relation representations learned directly from entity-linked text. The paper introduces matching the blanks with BERT-based relation encoders, achieving state-of-the-art results across three relation extraction tasks and outperforming prior work on few-shot relation matching.
Problem
Existing general-purpose relation extractors have limited ability to generalize beyond surface forms or relations jointly embedded with an existing knowledge graph.
Method
The paper learns BERT-based relation representations from entity-linked text by matching blank-containing relation statements, without relation labels or knowledge-graph supervision.
Results
The models achieve state-of-the-art results on three relation extraction tasks and outperform prior work on FewRel without FewRel training data, including gains of 8.8% and 12.7% for BERTEM+MTB.
Takeaways & Limitations
Matching the blanks is particularly effective in low-resource settings and may reduce the human effort required to create relation extractors.
Takeaways & Limitations
The approach assumes access to text whose entities are linked to unique identifiers.
Abstract
from arXiv · showhide
General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris' distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task's training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.
1 Introduction
General-purpose relation extraction seeks representations that generalize beyond fixed schemas and surface forms. The paper learns relation representations directly from entity-linked text by matching blanked relation statements, without knowledge-graph or human-annotator supervision.
- Prior approaches: Relation extraction approaches include fixed-schema supervised systems, surface-form open extraction, and universal schema methods combining textual and schematic relations.Universal schema extends joint representations to arbitrary textual input and entity pairs.
- Prior approaches: Surface-form representations lack generality because many surface forms can express the same relation, while universal schema methods rely on large aligned knowledge graphs.The paper identifies these as limitations of two general-purpose extraction strategies.
- Approach: The paper proposes learning relation representations directly from text, building on distributional similarity for relations and contextual representation learning with Transformer architectures.It first studies Transformer-based relation encoders, then trains representations without knowledge-graph or human-annotator supervision.
- Approach: Matching the blanks replaces linked entities with a special symbol and trains paired statements to have similar representations when they range over the same entity pair.This objective uses redundancy in entity-linked text to learn relation representations.
- Evaluation: The method is evaluated on FewRel using task-agnostic representations, including comparisons with supervised relation-matching systems.The supplied introduction states that the method can outperform FewRel results without seeing its training data and that tuned models outperform humans on evaluation.
2 Overview
The paper defines relation representations as fixed-length vectors for marked entity pairs and evaluates Transformer-based encoders on supervised relation extraction benchmarks. Its larger contribution is learning the encoder from entity-linked text through distant supervision without relation labels.
- Task definition: A relation statement consists of tokens plus two entity spans, and the encoder maps it to a fixed-length vector representing the relation between those entities.The formal input includes token sequence x and span indices s1 and s2.
- Architecture study: The paper investigates Transformer-based architectures for relation encoding and evaluates them on supervised relation extraction benchmarks.The overview describes a suite of benchmark evaluations for the candidate architectures.
- Matching the blanks: The paper’s second contribution is learning the relation encoder from widely available entity-linked text as distant supervision.This training signal does not require relation labels at training time.
3 Architectures for Relation Learning
The paper evaluates BERT-based architectures for representing relations in supervised extraction and few-shot matching. It varies how entity spans are marked in the input and how relation representations are extracted from Transformer outputs, finding that entity markers paired with entity-start outputs perform best across all four tasks.
- Relation Classification and Extraction Tasks: The experiments cover fully supervised relation extraction and few-shot relation matching, using SemEval 2010 Task 8, KBP-37, TACRED, and FewRel.Supervised tasks predict a relation from a fixed dictionary, whereas few-shot matching ranks candidate statements whose test relations may be unseen during training.
- Architecture Variants: BERT architectures differ in both entity-span input encoding and fixed-length relation output representation, yielding six combinations.The input options are STANDARD, POSITIONAL EMBEDDINGS, and ENTITY MARKERS; the output options are [CLS], MENTION POOLING, and ENTITY START where applicable.
- Entity Span Identification: STANDARD input gives BERT no explicit indication of which two entity mentions are the relation’s focus.This is used as a reference condition because a sentence can contain more than two entity mentions.
- Entity Span Identification: POSITIONAL EMBEDDINGS identify each focus span by adding separate segmentation embeddings to the tokens of the two entity mentions.The approach adapts segmentation embeddings to encode entity-span information in the input.
- Entity Span Identification: ENTITY MARKERS insert reserved start and end tokens around each entity mention, with updated span indices accounting for the inserted tokens.The markers are [E1start], [E1end], [E2start], and [E2end].
- Results: For all four tasks, ENTITY MARKERS input combined with ENTITY START output achieves the best scores and outperforms previously published models on every task.The ENTITY START representation concatenates the final hidden states associated with the two entity-start markers and is used for the paper’s subsequent experiments.
4 Learning by Matching the Blanks
The paper introduces Matching the Blanks (MTB), which learns relation representations from entity-linked text without relation labels or a predefined ontology. It uses shared entity pairs as positive evidence, carefully chosen negatives, and blanked entity mentions to encourage semantic relation encoding.
- MTB learns relation representations without a predefined ontology or relation-labeled training data.
- The training objective assigns high inner products to statements expressing semantically similar relations and low inner products to semantically different relations.
- Positive statement pairs share both resolved entities, while strong negatives share one entity but differ in the other.This construction exploits repeated mentions of relations between the same entity pairs while distinguishing superficially related statements.
- Entity mentions are replaced with [BLANK] independently with probability α = 0.7, limiting reliance on entity identification.Only α^2 of relation statements explicitly name both participating entities, so minimizing the objective requires more than recognizing named entities.
- Entity-linked Wikipedia paragraphs provide the corpus, with entity mentions annotated by unique knowledge-base identifiers.The corpus excludes lists and tables, and includes proper names, common nouns, and pronouns as referential entities.
- Noise-contrastive estimation replaces exhaustive negative-pair comparison with uniformly sampled negatives and hard negatives sharing one entity.Hard negatives are included to expose the model to similar but distinct relations rather than mostly unrelated random pairs.
5 Experimental Evaluation
The experiments evaluate matching-the-blanks representations on FewRel and supervised relation extraction, including task-agnostic, fully supervised, and low-resource settings. Across these settings, MTB improves relation matching and reduces the amount of task-specific annotation needed.
- Few-shot Relation Matching: MTB-trained BERT representations outperform prior FewRel state of the art without seeing FewRel training data, with gains of 8.8% on 5-way-1-shot and 12.7% on 10-way-1-shot.BERTEM+MTB also significantly outperforms BERTEM in this unsupervised setting.
- Few-shot Relation Matching: FewRel performance increases as training examples per relation type or the number of tuned relation types increases.Figure 4 varies examples per relation while holding 64 relation types fixed, and varies relation types while holding 700 examples per type fixed.
- Few-shot Relation Matching: 6% of FewRel training data lets BERTEM+MTB match BERTEM trained on all training data when all relation types remain represented.Maintaining diverse relation types while reducing examples per type is reported as the most effective way to reduce annotation effort.
- Few-shot Relation Matching: BERTEM+MTB outperforms the reported human upper bound and all other published or unpublished FewRel leaderboard submissions on fully supervised tasks.The comparison covers all of FewRel’s fully supervised tasks.
- Supervised Relation Extraction: MTB-based classifiers improve F1 scores over BERTEM on SemEval 2010 Task 8, KBP37, and TACRED, whose BERTEM classifiers already outperform prior published results.The additional MTB training increases F1 on all three supervised relation extraction tasks.
- Supervised Relation Extraction: MTB training is more effective in low-resource supervised settings, supporting reduced human input for creating relation extractors and populating a knowledge base.Table 5 varies the amount of task-specific tuning data and reports a larger BERTEM versus BERTEM+MTB gap when less data is available.
6 Conclusion and Future Work
The paper introduces matching the blanks for learning relation representations directly from text using entity-resolution annotations. It reports state-of-the-art relation extraction results, strong few-shot relation matching, and particular effectiveness in low-resource regimes, while identifying relation discovery and distributed knowledge-base representations as future directions.
- Conclusion: Matching the blanks learns relation representations directly from text using only entity-resolution annotations.The method is coupled with a BERT architecture for fine-tuning relation representations.
- Conclusion: The models achieve state-of-the-art results on three relation extraction tasks and outperform reported human accuracy on few-shot relation matching.The conclusion also reports particular effectiveness in low-resource regimes.
- Future Work: Future work will cluster similarly represented relation statements for relation discovery and study distributed representations for storing relation triples in knowledge bases.These directions are presented as steps toward general-purpose relation identification and extraction.