Source-linked AI summary
Inducing Relational Knowledge from BERT
Zied Bouraoui, Jose Camacho-Collados, Steven Schockaert
TL;DR
The paper investigates whether BERT captures relational knowledge beyond standard word embeddings. It proposes automatically finding relational templates in a large corpus and fine-tuning BERT to classify candidate pairs in instantiated templates, obtaining strong results overall but not for every relation type.
Problem
It is unclear whether pre-trained language models capture relational knowledge beyond standard word embeddings and improve relation induction.
Method
Starting from seed relation pairs, the method extracts and filters corpus sentences into templates, then fine-tunes BERT to classify candidate pairs in instantiated templates.
Results
The method obtains strong results overall, but its performance is not consistent across relation types and is poor for morphological relations.
Takeaways & Limitations
Relational knowledge can be obtained from BERT in a fully automated way without hand-coded templates, within the method’s supported relation types.
Abstract
from arXiv · showhide
One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.
1 Introduction
The paper asks whether pre-trained language models capture relational knowledge beyond word embeddings and proposes extracting that knowledge through corpus-derived trigger sentences. The method uses BERT to assess candidate word pairs without requiring corpus sentences containing those candidates.
- Motivation: Relation induction predicts likely instances of a relation from example word pairs, commonly by extracting their co-occurring sentences and training a relation-extraction model.Word-vector methods can predict pairs that never co-occur, whereas extraction methods rely on corpus sentences mentioning the examples.
- Motivation: Pre-trained language models raise the question of whether they capture more relational knowledge than standard word embeddings.Because BERT and XLNet complete masked sentences, relationally expressive sentences may provide inputs for extracting that knowledge.
- Contribution: The proposed methodology finds corpus sentences containing seed pairs, filters them for relational expressiveness, and uses the survivors as trigger templates.The templates are selected automatically rather than hand-coded.
- Contribution: Candidate-pair predictions rely only on the pre-trained language model’s relational knowledge, not on corpus sentences mentioning the candidate words.The corpus is used to find predictive trigger sentences, while BERT evaluates instantiated templates.
2 Related Work
Related work studies relational and attributive knowledge in word embeddings and the importance of input sentences for language-model probing. It also documents limitations in neural language models’ factual knowledge.
- Word embeddings: Word-embedding research has examined analogies, taxonomic and attributive properties, ordinal attributes, and relational transformations.These studies use pre-trained vectors to predict properties or relations between words.
- Language-model probing: Prior language-model work shows that choosing the surrounding input sentence can improve word-sense characterization.Amrami and Goldberg compared alternative sentence contexts for probing the sense of “sound.”
- Language-model probing: Neural language models have also been described as severely limited in capturing factual knowledge, motivating knowledge-graph-enhanced models.This frames factual knowledge as a distinct challenge from the relational probing studied here.
3 Methodology
The methodology extracts candidate templates from sentences containing seed relation pairs, ranks them using BERT predictions, and fine-tunes BERT to classify instantiated candidate pairs. It aggregates predictions across selected templates while avoiding misleading single-token completions.
- Pipeline: The pipeline extracts sentences containing seed pairs, filters them into relation-expressive templates, and fine-tunes a language model on instantiated templates.The three stages are extraction, template filtering, and classification of candidate pairs.
- Filtering templates: Templates are useful when their instantiated sentences remain natural across many known relation pairs rather than being specific to one pair.Indirectly informative templates can also provide evidence when several assertions jointly support the relation.
- Filtering templates: BERT ranks templates by checking whether masked-word top-k predictions contain the known source and target words.The fast score uses two top-k prediction sets per sentence, whereas the full score requires 2n predictions.
- Fine-tuning BERT: For relation classification, the model feeds the complete instantiated sentence to BERT and classifies its [CLS] representation as a relation assertion.This avoids treating a plausible predicted tail word as sufficient when the full sentence is nonsensical.
- Relation classification: Each candidate pair receives one probability per selected template, which the method combines using either a maximum-based or an average-based strategy.The BERTmax variant accepts a pair when its strongest positive template prediction exceeds the strongest negative alternative.
4 Experiments
Experiments evaluate whether the proposed BERT-based relation-induction method models relations better than pretrained word vectors across Google, BATS, and DiffVec. Results show usefulness of automatic template scoring, strong performance on several non-morphological relations, and weaknesses on morphological and some lexical relations.
- Experimental setting: The evaluation splits relation examples into 90% training and 10% test sets, using Wikipedia sentences to filter templates and fine-tune BERT.Negative test examples include reversed, randomly paired, cross-relation, and unrelated word pairs, producing five times as many negatives as positives.
- Overall results: K = 100 outperforms K = 50 and K = 1000, showing that the proposed scoring function improves performance when selecting templates.K = 1000 omits the scoring function, while BERTmax and BERT* show no consistent difference.
- Results by relation type: The model does not consistently outperform the Trans baseline; Trans performs notably better on BATS and on morphological relations.For DiffVec Verb 3rd, Trans reaches F1 97.0 versus 20.3 for the proposed model, while hypernymy reaches 73.6 versus 54.3.
- Results by relation type: On DiffVec, the proposed model outperforms the baseline for most non-morphological, non-Expression relations involving attributive, causal, and commonsense knowledge.Its weaknesses include morphological relations and lexical relations such as hypernymy.
- Qualitative analysis: Automatically extracted templates provide richer linguistic context than short manual patterns and can encode implicit evidence, such as embassies being located in capitals.Manual templates perform poorly in this setting, reinforcing the value of automatic extraction, although the comparison is not strictly equivalent because the proposed model uses multiple templates.
5 Conclusions
The paper shows that BERT can yield high-quality relational knowledge automatically by mining predictive templates from text, but its suitability depends on relation type.
- The method identifies and filters corpus sentences mentioning known relation instances to produce predictive templates without hand-coded patterns.These templates are then used to evaluate candidate word pairs with BERT.
- The approach is not suitable for morphological relations and performs broadly on par with word-vector methods for lexical relations such as meronymy and hypernymy.
- For encyclopedic and commonsense relations, BERT consistently and often substantially outperforms methods based on pre-trained word vectors.