Source-linked AI summary
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction
Christoph Alt, Marc Hübner, Leonhard Hennig
TL;DR
Distantly supervised relation extraction relies on noisy labels, and existing methods may overlook long-tail relations. DISTRE fine-tunes a pre-trained GPT with selective attention for bag-level relation prediction. It reports state-of-the-art AUC on NYT10 and broader relation recognition, especially at higher recall.
Problem
Distant supervision supplies noisy labels, and current relation extraction methods can overlook long-tail relations despite high precision on a limited set.
Source: §1 · p. 1Method
DISTRE fine-tunes GPT for distantly supervised relation extraction, using selective attention to aggregate sentence representations into bag-level predictions.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The approach uses features captured implicitly in pre-trained representations, which the authors say allows increased domain and language independence and avoids preprocessing.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Distantly supervised relation extraction is widely used to extract relational facts from text, but suffers from noisy labels. Current relation extraction methods try to alleviate the noise by multi-instance learning and by providing supporting linguistic and contextual information to more efficiently guide the relation classification. While achieving state-of-the-art results, we observed these models to be biased towards recognizing a limited set of relations with high precision, while ignoring those in the long tail. To address this gap, we utilize a pre-trained language model, the OpenAI Generative Pre-trained Transformer (GPT) [Radford et al., 2018]. The GPT and similar models have been shown to capture semantic and syntactic features, and also a notable amount of "common-sense" knowledge, which we hypothesize are important features for recognizing a more diverse set of relations. By extending the GPT to the distantly supervised setting, and fine-tuning it on the NYT10 dataset, we show that it predicts a larger set of distinct relation types with high confidence. Manual and automated evaluation of our model shows that it achieves a state-of-the-art AUC score of 0.422 on the NYT10 dataset, and performs especially well at higher recall levels.
1 Introduction
Distant supervision provides training labels for relation extraction but introduces noise, while existing methods may overlook long-tail relations. DISTRE fine-tunes a pre-trained Transformer for this setting and reports state-of-the-art AUC alongside broader relation coverage and stronger performance at higher recall.
1 Introduction
- Distant supervision aligns entity tuples in text with known knowledge-base relations, but its labels can be noisy and the knowledge base incomplete.The resulting labels can mark mentions as positive even when they do not express the relation.Source: §1 · p. 1
- Existing relation extraction methods can recognize a limited set of relations with high precision while ignoring long-tail relations.These methods use multi-instance learning and may incorporate linguistic or contextual side information.Source: §1 · p. 1
- Pre-trained language models are hypothesized to supply semantic, syntactic, and limited common-sense information that could support more diverse relation recognition.The paper proposes that implicit features may reduce reliance on explicit linguistic and side information.Source: §1 · pp. 1–2
- AUC 0.422 on NYT10 is reported as state of the art, with DISTRE performing especially well at higher recall levels.The model is compared with RESIDE and PCNN+ATT.Source: §1 · p. 2
- DISTRE extends GPT with selective attention to aggregate sentence representations into bag-level predictions for distantly supervised relation extraction.This supports fine-tuning the pre-trained language model directly on the task.Source: §1 · p. 2
2 Transformer Language Model
This section reviews the Transformer language model, its decoder architecture, and the language-modeling objective used to learn contextualized representations.
2.1 Transformer-Decoder
- The decoder-only Transformer repeatedly applies masked multi-head self-attention and a position-wise feedforward operation across layers.Masking prevents attention to positions ahead of the current token; the architecture can model long-range dependencies.Source: §2.1 · p. 2
- Each Transformer block produces layer states, while the relation-extraction architecture encodes bag sentences and uses selective attention to aggregate their representations for classification.The Transformer adds learned positional embeddings to token embeddings because it has no implicit notion of token positions.Source: §2.1 · p. 2, Figure 2 · p. 3
2.2 Unsupervised Pre-training of Language Representations
- The language-modeling objective predicts each token from a context window, using final-layer states to produce a probability distribution over tokens.The model parameters are optimized by stochastic gradient descent.Source: §2.2 · p. 3
3 Multi-Instance Learning with the Transformer
This section extends the Transformer to bag-level multi-instance learning for distantly supervised relation extraction and specifies the task-specific input.
3.1 Distantly Supervised Fine-tuning on Relation Extraction
- Noisy sentence labels are handled by classifying each entity-pair bag, with selective attention weighting sentence representations by relation relevance before classification.The bag representation is a weighted sum of sentence representations.Source: §3.1 · p. 3
- Fine-tuning combines the relation-extraction and language-modeling objectives, with λ weighting the language-model objective.Source: §3.1 · p. 4
3.2 Input Representation
- Inputs place head and tail entities before the sentence and end with a classification token, using delimiters and summed token and positional embeddings.The argument-first ordering biases left-to-right attention toward the entity representations while the sentence is processed.Source: §3.2 · p. 4, Figure 3 · p. 4
4 Experiment Setup
The experiments evaluate the model on NYT10 against two state-of-the-art baselines and describe the dataset, reused language model, and fine-tuning settings.
4 Experiment Setup
- Experiments use NYT10 and compare DISTRE with PCNN+ATTN and RESIDE as state-of-the-art baselines.PCNN+ATTN uses piecewise convolution and selective attention; RESIDE combines a GRU, dependency-tree GCN, and entity types.Source: §4 · p. 4
4.1 NYT10 Dataset
- NYT10 contains 53 relation types and uses 2005–2006 for training and 2007 for testing; its distant-supervision test labels are approximate.The training and test sets contain 522,611 and 172,448 sentences, respectively; manual evaluation also reports P@100, P@200, and P@300.Source: §4.1 · pp. 4–5
4.2 Pre-training
- The experiments reuse Radford et al.’s GPT, pretrained on BooksCorpus, with 12 decoder blocks, 12 attention heads, and 768-dimensional states.Its byte-pair vocabulary is extended with task-specific tokens.Source: §4.2 · p. 5
4.3 Hyperparameters
- Fine-tuning runs for 3 epochs with batch size 8, learning rate 6.25e-5, and residual and attention dropout of 0.1.Classifier dropout is 0.2, with linear learning-rate decay and warm-up over 0.2% of updates.Source: §4.3 · p. 5
5 Results
On NYT10, DISTRE achieves state-of-the-art held-out AUC and stronger precision at higher recall, while manual analysis finds a broader range of high-confidence relation types.
5 Results
- DISTRE's more balanced performance, especially in the long tail, accompanies its broader relation coverage without explicit side information or linguistic features.Figure 4 presents the NYT precision-recall curves; its markers distinguish results reported by Vashishth et al. (2018) from OpenNRE4 results obtained by the authors.Source: Figure 4 · p. 5, §5 · p. 5
5.1 Held-out Evaluation
- 60.2% precision for the top 1,000 predictions is state of the art, and DISTRE continues to outperform for the remaining, larger part of predictions.DISTRE has lower precision for the top 500 predicted instances.Source: §5.1 · p. 5
- 0.422 AUC gives DISTRE a new state of the art on held-out evaluation, and it outperforms RESIDE and PCNN+ATT at higher recall.Precision is lower for the top predicted instances; PCNN+ATT performs better only at the curve's very beginning.Source: §5.1 · p. 5, Table 1 · p. 6
5.2 Manual Evaluation and Analysis
- Manual ratings give PCNN+ATT the highest average precision at 94.3%, driven by P@100 and P@200; models are similar at P@300, and DISTRE errors often involve mislabeled capital relations.Three annotators rated each model's top 300 predictions; a correct rating required a true fact and at least one sentence clearly expressing it.Source: §5.2 · p. 6, Table 2 · p. 7
- DISTRE's top 300 predictions span 10 relation types, versus RESIDE's 7 and PCNN+ATT's 4, with the latter concentrated on two types.The two most frequent types account for 67% of DISTRE's predictions, 82% of RESIDE's, and 91% of PCNN+ATT's.Source: §5.2 · p. 6, Table 3 · p. 7
6 Related Work
Prior work in relation extraction moved from discrete-feature classifiers to neural encoders, while distant supervision developed multi-instance methods and increasingly rich architectures. This paper contrasts those approaches with fine-tuning pre-trained language representations using distant-supervision labels.
Relation Extraction
- Relation extraction progressed from statistical and kernel methods with discrete syntactic features to recurrent and convolutional neural networks with distributed representations.Dependency-path methods used the shortest dependency path to focus on actions and agents (Bunescu and Mooney, 2005; Socher et al., 2014).Source: §6 · p. 7
Distantly Supervised Relation Extraction
- Unlike methods relying on explicit linguistic features or task- and domain-specific pre-training, this work fine-tunes pre-trained language representations using only distant-supervision labels.Verga et al. (2018) extended Transformers for supervised biomedical entity and relation extraction.Source: §6 · p. 7
- Distantly supervised RE evolved from multi-instance learning to PCNN-based systems, selective attention, noise handling, graph convolutions, and capsule networks.The classic multi-instance assumption is that at least one sentence per relation instance expresses the relation.Source: §6 · p. 7
Relation Extraction
- Table 4 presents challenging relation mentions as examples that benefit from capturing more complex features, with relation arguments marked in bold.The caption does not identify the specific examples or their outcomes.Source: Table 4 · p. 8
Language Representations and Transfer Learning
- ELMo contextualized representations significantly outperformed static pre-trained word vectors on NLP tasks; general-domain pre-training with task-specific fine-tuning achieved state-of-the-art results on question answering, text classification, textual entailment, and semantic similarity.Devlin et al. (2018) introduced a slot-filling objective to jointly train a bidirectional language model.Source: §6 · p. 8
7 Conclusion
DISTRE combines a Transformer with attentive selection for multi-instance distant supervision and achieves a more balanced, diverse set of predictions despite lower precision among its top 300. The authors propose further study of captured syntax and integration of entity and relation knowledge.
7 Conclusion
- DISTRE extends a Transformer with attentive selection for multi-instance learning in distantly supervised relation extraction.Compared with RESIDE, it uses features captured in pre-trained representations rather than explicitly provided side information and linguistic features.Source: §7 · p. 8
- DISTRE achieves state-of-the-art AUC and more balanced performance, especially at higher recall, while its top 300 predictions have lower precision.It also predicts a larger set of distinct relation types with high confidence.Source: §7 · p. 8
- Future work will investigate how much syntactic structure deep language representations capture and whether entity or relation background knowledge could improve performance.The authors also note that DISTRE's generic architecture permits integration of such contextual information.Source: §7 · p. 8