DARE: Data Augmented Relation Extraction with GPT-2

1 Introduction

• DARE fine-tunes GPT-2 for each relation type, then combines generated samples with gold data to train a BERT-based relation extractor.The introduction presents this as an application of GPT-2 generation to limited-data and class-imbalance settings.Source: §1 · p. 1
• Experiments compare DARE with two strong baselines and state-of-the-art methods across three established biomedical relation-extraction benchmarks.The paper reports significant improvements, but this passage does not give the numerical results.Source: §1 · p. 1
• Table 1 presents generated examples from GPT-2 models fine-tuned on examples for specific relation types.The introduction directs readers to Section 4 for the experimental datasets.Source: §1 · p. 1, Table 1 · p. 2

2 Related Work

• Class-imbalance methods include class-weighted errors or adjusted class priors, as well as majority-class undersampling and balanced bagging.Balanced bagging trains an ensemble using random majority undersampling.Source: §2 · p. 2
• Text augmentation approaches include synonym replacement, topic-model generation, and learned sequence generation, but text semantics depend on word order and structure.The paper describes textual oversampling as more limited than image augmentation.Source: §2 · p. 2
• Sun et al. (2019a) fine-tuned GPT-2 to generate training samples for earlier tasks in lifelong learning, then combined those samples with data for a new task.That approach was intended to avoid catastrophic forgetting.Source: §2 · p. 2
• DARE places relation extraction within text-oversampling work and does not require domain expertise, templates, a synonym thesaurus, or training a model from scratch.The authors characterize these properties as making the approach adaptable across domains with relatively low resource requirements.Source: §2 · p. 2

3.1 GPT-2

• GPT-2 is pretrained to predict the next word token from the preceding k tokens, and the experiments use its 774M-parameter version as a compromise between accuracy and hardware requirements.The language model is pretrained on large amounts of text using a standard language-modeling objective.Source: §3.1 · p. 3

3.2 Data Augmented Relation Extraction

• The BERT-based classifier uses a classification token and masks entity mentions with generic entity types; the method is not restricted to this classifier.The paper describes the classifier as pretrained BERT fine-tuned for relation extraction.Source: §3.2 · p. 3
• DARE splits training data by relation type, fine-tunes one GPT-2 per type, generates sentences, filters out samples missing entity masks or containing fewer than eight tokens, and combines the remaining samples.Each GPT-2 is fine-tuned for five epochs.Source: §3.2 · p. 3
• Each ensemble classifier trains on all gold data plus relation-specific synthetic subsets sized to the gold count for that relation multiplied by ratio r.The experiments set r = 1.0 and use majority voting for prediction.Source: §3.2 · p. 3
• An early shared GPT-2 model trained on all relations with relation-prefix tokens performed worse than using gold data alone, largely because frequent classes led to incorrectly labeled generations.The authors contrast this experiment with fine-tuning separate models by relation type.Source: §3.2 · p. 3

4.1 Setup

• A BERT-large RE classifier is trained on gold data or gold plus generated data; DARE ensembles 20 classifiers trained on the full gold set and generated-data subsets.Source: §4.1 · p. 4

4.2 Datasets

• The evaluation covers CDR, DDI2013, and ChemProt; the rarest training classes have 153 instances in DDI2013 and 173 in ChemProt.Source: §4.2 · p. 4, §4.3 · p. 4

4.3 Baselines

• Balanced bagging and class weighting serve as baselines for datasets with class imbalance and few positive examples.Source: §4.3 · p. 4

4.4 Fine-tuning GPT-2 on In-domain Data

• On CDR, first fine-tuning GPT-2 on 500k PubMed abstracts proved significantly favorable before relation-specific fine-tuning and generation.Source: Table 3 · p. 5, §4.4 · p. 5

4.5 DARE on Imbalanced Datasets

• DARE’s advantage over balanced bagging reaches 11 F1 points when CDR has 250 or fewer positive samples.Source: Figure 1 · p. 5, §4.5 · p. 6

4.6 Effect of Generated Data Size

• More generated data does not necessarily improve performance; the authors choose ratio=1, adding generated instances equal to gold instances for each relation type.Source: Figure 2 · p. 6, §4.6 · p. 6

4.7 DARE against the SOTA and Baselines

• DARE beats the SOTA by 3 to 8 F1 points across datasets and the baselines by 2 to 5 F1 points, significantly in all baseline comparisons.Source: §4.7 · pp. 6–7, Table 4 · p. 7

• DARE fine-tunes a pretrained GPT-2 model per relation type, generates training data, and combines it with gold data to train BERT-based RE classifiers.Source: §5 · p. 7
• Up to 11 F1 score points over two strong baselines and new state-of-the-art performance on three biomedical RE benchmarks are reported.Source: §5 · p. 7
• The authors identify extending DARE to other Natural Language Understanding tasks as future work.Source: §5 · p. 7

• The supplemental material gives per-class results for DARE against the state of the art and class weighting baseline on ChemProt and DDI2013.Source: A Supplemental Material · p. 9
• Table 5 presents ChemProt F1 results by relation type for DARE, the state of the art, and the best baseline.Source: Table 5 · p. 9
• Table 6 presents DDI2013 F1 results by relation type for DARE, the state of the art, and the best baseline.Source: Table 6 · p. 9