Improving Relation Extraction by Pre-trained Language Representations

Christoph Alt; Marc Hübner; Leonhard Hennig

Source-linked AI summary

Improving Relation Extraction by Pre-trained Language Representations

Christoph Alt, Marc Hübner, Leonhard Hennig

arXiv:1906.03088v1Jun 7, 2019cs.CL

TL;DR

Relation extraction traditionally depends on explicit linguistic features that require language-specific resources and introduce preprocessing errors. TRE replaces those features with pre-trained language representations in a self-attentive Transformer, learning from plain text before task fine-tuning. It outperforms state-of-the-art methods on TACRED and SemEval 2010 Task 8 while substantially improving sample efficiency.

Problem
Explicit linguistic features restrict relation extraction’s portability to novel languages, require annotated resources, and introduce additional preprocessing errors.
Method
TRE uses pre-trained deep language representations and a self-attentive Transformer, pre-training on plain text before fine-tuning for relation extraction.
Results
TRE outperforms state-of-the-art methods on TACRED and SemEval 2010 Task 8 and substantially improves sample efficiency.
Takeaways & Limitations
Pre-trained language representations capture features informative for relation extraction and reduce reliance on explicit linguistic features and large labeled datasets.
Takeaways & Limitations
The paper leaves the extent of syntactic structure captured by language representations, compared with dependency parsing, for future investigation.

Abstract

from arXiv · show

Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

1. Introduction

Relation extraction supports natural language understanding and applications but traditionally depends on explicit linguistic features that limit portability and add preprocessing risks. TRE replaces those features with pre-trained language representations in a Transformer architecture, achieving state-of-the-art results while improving sample efficiency.

1. Introduction: Explicit feature extraction requires annotated language-specific resources, restricts portability to novel languages, and can introduce cascading errors.Earlier systems use lexical, syntactic, and semantic features such as morphology, part-of-speech tags, dependency parses, named entities, and WordNet hypernyms.
1. Introduction: TRE replaces explicit linguistic features with deep language representations learned through unsupervised pre-training on plain text.Fine-tuning then adapts the representations directly to relation extraction, reducing reliance on annotated, language-specific resources.
1. Introduction: TRE uses the Transformer’s self-attentive architecture to model long-range dependencies between entity mentions.The Transformer is presented as an architecture that captures long-range dependencies efficiently.
1. Introduction: Pre-trained language representations improve sample efficiency and match baseline performance with substantially fewer labeled examples.The paper attributes this benefit to reduced overfitting and better generalization for complex entity mentions.
1. Introduction: The authors release trained models, experiments, and source code to support wider adoption and further research.

2. TRE

TRE is a decoder-only Transformer relation extraction model that replaces explicit linguistic features with representations learned through language-model pre-training and fine-tuned for relation classification.

2.1 Model Architecture: TRE uses a multi-layer Transformer-Decoder with masked multi-headed self-attention and position-wise feedforward operations.The model repeatedly encodes input representations across L Transformer blocks.
2.1 Model Architecture: Input representations sum byte-pair token embeddings with learned positional embeddings, while the self-attentive states model long-range dependencies.The architecture adds positional information because the Transformer has no implicit notion of token positions.
2.2 Input Representation: TRE converts relation-extraction inputs into an ordered sequence containing both arguments, delimiters, the sentence, and a final classification token.Arguments are placed first to bias left-to-right attention toward their representations while processing the sentence.
2.3 Unsupervised Pre-training of Language Representations: Unsupervised pre-training maximizes next-token likelihood over plain-text corpora, learning deep lexical, syntactic, and semantic representations before supervised fine-tuning.The conditional token distribution is produced from the final Transformer states, embedding matrix, and optimized model parameters.
2.4 Supervised Fine-tuning on Relation Extraction: During fine-tuning, the pre-trained model maps labeled token sequences to a final-state representation and predicts relation labels with a linear layer and softmax.The objective combines relation-classification loss with a weighted language-model objective, where λ controls the auxiliary contribution.

3. Experiment Setup

The experiments evaluate TRE on TACRED and SemEval 2010 Task 8, using established dataset splits, a PCNN baseline, entity masking strategies, and specified optimization settings.

3. Experiment Setup: Experiments use TACRED and SemEval 2010 Task 8, with the PCNN implementation of Zeng et al. [2015] as a state-of-the-art baseline.TACRED is substantially larger and more negative-example-heavy than SemEval, making its classification setting more challenging.
3.1 Datasets: TACRED contains 106k sentences with typed entity mentions, whereas SemEval contains 8,000 training and 2,717 test sentences with untyped nominals and nine directed relation types.TACRED results use micro-averaged F1 and model selection based on median validation F1 across five runs.
3.2 Pre-trained Language Model: The experiments reuse Radford et al. [2018]’s language model, trained on BooksCorpus, because pre-training is computationally expensive.The reused model has 12 layers, 12 attention heads, 768-dimensional states, and a 40,000-token byte-pair encoding vocabulary extended with task-specific tokens.
3.3 Entity Masking: Four entity-masking strategies vary how much entity information is exposed, supporting analysis of entity types, roles, overfitting, and generalization to unseen entities.Masking ranges from replacing all entity mentions with an unknown token to more informative entity representations.
3.4 Hyperparameter Settings and Optimization: Fine-tuning follows Adam with β1 = 0.9, β2 = 0.999, batch size 8, warm-up over 0.2% of updates, and residual and classifier dropout of 0.1.The best dataset-specific configurations are reported in Table 3.

4. Results

TRE achieves state-of-the-art relation extraction performance on TACRED and SemEval 2010 Task 8, including when entity mentions are masked. The results support using pre-trained representations instead of explicit linguistic features.

4.1 TACRED: Masking was important for the TACRED result: unmasked entity mentions reduced performance to 62.8, indicating overfitting to specific entity types.The reported state-of-the-art result used entity masking, while the masked strategy supplied named entity information.
4.1 TACRED: 67.4 F1 on TACRED lets TRE outperform state-of-the-art single-model systems, including methods designed for complex syntactic and long-range dependencies.The result indicates that TRE implicitly captures patterns similar to complex syntactic features while modeling long-range dependencies.
4.2 SemEval: 87.1 F1 on SemEval 2010 Task 8 establishes TRE as the new state of the art over methods using explicit lexical or syntactic features.The comparison includes SVM, RNN, BCRNN, DRNN, and CGCN approaches.
4.2 SemEval: 79.1 F1 with all SemEval entity mentions replaced by UNK improves 2.6 points over the previous state of the art.This setting tests generalization beyond entity mentions and simulates unseen entities.

5. Analysis & Ablation Studies

Ablations show that pre-training improves performance and generalization, while entity masking controls reliance on mention-specific information. TRE also maintains strong performance with substantially less TACRED training data.

5.1 Effect of Pre-training: Pre-training raised SemEval validation F1 from 75.6 to 85.6 and increased TACRED validation F1 by 20 points to 63.3 without entity masking.The larger gains without masking suggest a regularizing effect that helps prevent overfitting and adapt to complex entities.
5.1 Effect of Pre-training: Pre-trained byte-pair embeddings benefited both datasets, with a larger benefit on the smaller SemEval corpus where embeddings were harder to learn reliably.The supplied passage reports that the smaller corpus increased the risk of overfitting to entity mentions.
5.1 Effect of Pre-training: Providing entity and grammatical-role information increased TACRED validation F1 from 63.3 to 68.0, while pre-trained representations remained informative without those features.The comparison supports the claim that contextualized representations can capture features as informative as entity type and grammatical role information.
5.3 Effect of Entity Masking: UNK masking reduced TACRED validation F1 to 51.0, whereas grammatical-role masking raised it to 56.1 and NE masking substantially improved recall.Combining NE and grammatical-role masking increased recall from 65.3% to 67.2% while precision stayed at 68.8%.
5.4 Sample Efficiency: More than 60 F1 with only 20% of TACRED training data was achieved by the pre-trained model with NE+GR masking, which consistently outperformed alternatives.Performance increased steeply with the first portion of the training data and continued improving as more data was added.
5.4 Sample Efficiency: Without pre-training, TRE overfit early and declined after 70% of training data, while several models dropped or stagnated after about 80%.The authors suggest that later examples may not increase models’ regularization capabilities.

6. Related Work

Relation extraction has traditionally used explicit lexical, syntactic, and semantic features, while newer work increasingly uses neural models and contextualized language representations.

Relation Extraction: Earlier relation extraction systems combined statistical or kernel methods with discrete syntactic features such as part-of-speech, named-entity, morphological, and WordNet features.More recent neural networks superseded many of these approaches.
Language Representations and Transfer Learning: ELMo introduced contextualized word representations learned by bidirectional language-model training, replacing static pre-trained word vectors with deep language representations.This work is presented as part of the broader development of unsupervised pre-training and transfer learning.

7. Conclusion

TRE replaces explicit linguistic features with implicit features learned through pre-trained language representations, outperforming state-of-the-art methods on two relation extraction datasets. The paper identifies further investigation of TRE’s captured linguistic structure as important future work.

TRE outperforms the state of the art on the TACRED and SemEval 2010 Task 8 relation extraction datasets.
Pre-trained language representations drastically improve TRE’s sample efficiency.The experiments also found these representations capture features highly informative for relation extraction.
TRE replaces previous methods’ explicit linguistic features with implicit features captured in pre-trained language representations.
Future work should investigate which linguistic features TRE captures, including how much syntactic structure its representations contain relative to dependency parsing.The generic architecture could also integrate additional contextual information and entity background knowledge.