PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Relation Extraction in the era of Large Language Models

Somin Wadhwa, Silvio Amir, Byron C. Wallace

arXiv:2305.05003v2cs.CL

TL;DR

Relation extraction requires identifying entities and their semantic relationships, but generative approaches are difficult to evaluate with exact matching because valid outputs can vary in wording. The paper evaluates GPT-3 and Flan-T5, finding that few-shot GPT-3 is comparable to supervised SOTA models while GPT-3-generated CoT supervision gives Flan-T5 SOTA performance across the considered datasets.

  • Problem

    Generative RE outputs can vary in wording, making exact-match evaluation against reference targets unreliable.

  • Method

    The paper evaluates GPT-3 with few-shot prompting and fine-tunes Flan-T5 using relation labels augmented with GPT-3-generated CoT explanations.

  • Results

    GPT-3 performs comparably to fully supervised SOTA models with only tens of examples, while CoT-augmented Flan-T5 achieves SOTA performance across all considered datasets, often by 5-10 F1 points.

  • Takeaways & Limitations

    LLMs should be considered a standard baseline for RE, including the smaller open-source Flan-T5 model when trained with GPT-3-generated CoT data.

  • Takeaways & Limitations

    The study excludes complex n-ary relation datasets and lengthy, relation-dense datasets such as DocRED because of prompt-length constraints.

Abstract

from arXiv · show

Relation extraction (RE) is the core NLP task of inferring semantic relationships between entities from text. Standard supervised RE techniques entail training modules to tag tokens comprising entity spans and then predict the relationship between them. Recent work has instead treated the problem as a \emph{sequence-to-sequence} task, linearizing relations between entities as target strings to be generated conditioned on the input. Here we push the limits of this approach, using larger language models (GPT-3 and Flan-T5 large) than considered in prior work and evaluating their performance on standard RE tasks under varying levels of supervision. We address issues inherent to evaluating generative approaches to RE by doing human evaluations, in lieu of relying on exact matching. Under this refined evaluation, we find that: (1) Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly equivalent to existing fully supervised models; (2) Flan-T5 is not as capable in the few-shot setting, but supervising and fine-tuning it with Chain-of-Thought (CoT) style explanations (generated via GPT-3) yields SOTA results. We release this model as a new baseline for RE tasks.

1 Introduction

The paper extends generative relation extraction to larger language models and finds strong results from GPT-3 few-shot prompting and Flan-T5 fine-tuned with GPT-3-generated Chain-of-Thought explanations. It also addresses the difficulty of evaluating flexible generated outputs.

  • Generative RE work had previously considered only moderately sized pre-trained models such as BART.
  • The paper investigates end-to-end RE via generation with very large language models, including GPT-3.
  • Few-shot GPT-3 achieves near-SOTA performance on standard RE datasets and outperforms fully supervised models.
  • Flan-T5 is less capable even when fine-tuned, but GPT-3-generated CoT explanations enable it to achieve SOTA results.
  • Human annotations are used to evaluate generated outputs because exact matching is inadequate for generative RE.
  • LLMs are presented as a potential default approach to RE, including a smaller publicly available Flan-T5 model trained with CoT supervision.

2 RE via Text Generation

The paper treats relation extraction as conditional text generation, representing relation tuples as linearized target strings. It highlights both the flexibility of generated answers and the resulting challenges for exact-match evaluation.

  • RE is treated as a conditional text generation task that produces linearized relation-triplet strings from input text.
  • The generated relation probability is conditioned on a context containing n linearized examples, with n much smaller than the dataset size N.
  • Experiments cover ADE, CoNLL, NYT, and DocRED, which vary in numbers of entities and relation types.
  • Targets for single-relation datasets are lists of tuples, while multiple-relation datasets use subject–relation–object triplets with entity types.
  • Flexible language-model outputs can express the same relation in many forms, making structured parsing and exact comparison non-trivial.
  • Because accurate outputs may differ from reference targets, human annotators judge whether generations convey the same information.

3 In-Context Few-Shot Learning with GPT-3 for RE

GPT-3 is evaluated for relation extraction with instructional few-shot prompts and human review of outputs, revealing strong performance alongside dataset- and prompt-specific limitations.

  • Prompting strategy: GPT-3 uses instructional in-context prompts containing 12–20 training examples, with dataset-specific prompt construction.ADE uses 12 randomly selected examples; CoNLL includes all entity and relation types; NYT uses 20 examples because of its 24 relations.
  • Manual evaluation: Human annotations show that strict evaluation mislabels many GPT-3 outputs: 51.67% of ADE false positives and 50.27% of CoNLL false positives were valid.Annotators also judged 32.61% of ADE false negatives and 36.6% of CoNLL false negatives accurate.
  • Manual evaluation: Strict exact-matching evaluation is inaccurate and pessimistic for generative relation extraction, so reported LLM results incorporate manual assessments.The evaluation process uses human judgments against reference labels to quantify errors and identify erroneous gold references.
  • Results: GPT-3 performs comparably to fully supervised state-of-the-art models without fine-tuning, using only 12–20 training examples in most cases.The analysis also identifies relation pairs that GPT-3 predicts correctly but that are incorrectly marked in the references.
  • Results: GPT-3 produces 18 CoNLL validation instances containing at least one generated triplet outside the valid relation set.These out-of-domain relations were often closely related to correct relations or otherwise semantically correct.
  • Results: NYT performance declines by approximately 30 Micro-F1 points when relation descriptions are omitted, with approximately 10.6% invalid or empty outputs.The authors attribute this setting’s limitation to the difficulty of fitting detailed instructions for datasets with long texts or many targets, leaving traditional fine-tuning as the practical option.

4 SOTA RE Performance with Flan-T5

Flan-T5 performs poorly in few-shot prompting but reaches state-of-the-art relation-extraction performance when fine-tuned with relation labels and GPT-3-generated Chain-of-Thought explanations. This approach achieves substantial gains over existing supervised generative baselines.

  • Few-shot prompting: Few-shot Flan-T5 generates many malformed or out-of-domain relations and is not competitive with GPT-3 or state-of-the-art RE models.Non-conforming outputs occur in 13.9% of ADE generations and 12.5% of CoNLL generations; CoNLL also produces over 120 unique out-of-domain relations.
  • Standard fine-tuning: Standard fine-tuning makes Flan-T5 equivalent to, but not better than, existing fully supervised models such as REBEL.
  • CoT prompting: CoT improves GPT-3 output standardization more than its overall performance in few-shot RE.Non-conforming relations fall from 13.9% to 0.8% on ADE and from 12.5% to 1.1% on CoNLL, while performance gains remain moderate.
  • CoT supervision: GPT-3-generated CoT explanations are added to Flan-T5 training targets alongside reference relation labels.GPT-3 produces explanations conditioned on input texts and relation reference labels, which are then used with the labels to fine-tune Flan-T5.
  • Results: 9.97 points: Flan-T5 gains this much in micro F-1 over REBEL on ADE when fine-tuned with labels and GPT-3-generated explanations.The gain is averaged over ADE’s ten train/test folds.
  • Results: 5-10 points: fine-tuning Flan-T5 with labels and GPT-3-produced CoT explanations yields this approximate micro-F1 margin across RE datasets.The strategy achieves state-of-the-art performance across the considered datasets and is faster to train than existing fully supervised models.

5 Related work

Earlier relation-extraction research used supervised structured models and later conditional sequence-to-sequence generation. This work extends that line by evaluating large language models in few-shot, in-context RE.

  • Prior RE methods: Traditional RE methods include Conditional Random Fields, structured SVMs, and joint deep-learning models for identifying entities and relations.
  • Generative RE: Conditional generative models formulate RE as sequence-to-sequence generation of linearized relation strings.This framework was applied to relation extraction and other structured prediction tasks.
  • Supervised generation: Prior approaches focused mainly on fine-tuning sequence-to-sequence models with standard supervised relation-generation objectives.REBEL added RE-specific pretraining but still used a fine-tuning step.
  • Few-shot LLMs: This work evaluates few-shot in-context relation extraction with large language models, described as the first such evaluation specifically for RE.Few-shot prompting incorporates training examples into inference prompts without updating model weights.

6 Conclusions and Future Directions

The paper finds that GPT-3 can match fully supervised RE models with only tens of examples, while GPT-3-generated CoT supervision enables Flan-T5 to achieve strong performance across datasets. Future work targets cheaper evaluation and unresolved generalization concerns.

  • Conclusions: GPT-3 performs comparably to fully supervised state-of-the-art RE models when given only tens of examples.
  • Conclusions: GPT-3-generated CoT explanations distilled into Flan-T5 training produce state-of-the-art performance across all considered datasets, often by 5-10 F1 points.
  • Conclusions: The authors suggest using LLMs as a standard RE baseline where feasible.
  • Future directions: Manual annotations were required to identify apparently incorrect GPT-3 outputs that were actually accurate, creating evaluation costs.The paper leaves automated evaluation as a future direction and reports preliminary BERT-style classifier work.
  • Future directions: Precision fell ∼16 points for GPT-3 and ∼7 points for Flan-T5 on post-September-2021 news examples, but the cause could not be determined.The authors could not distinguish domain shift from data leakage and leave further analysis for future work.

Limitations

The paper identifies scope, explanation-quality, model-access, and language limitations. Its experiments cover only three standard binary-relation RE datasets, while several broader settings remain untested.

  • Scope: Experiments covered three standard RE datasets with binary relations, excluding n-ary and other complex relation settings.The authors also could not test lengthy-text, many-relation datasets such as DocRED because of prompt-length requirements.
  • Supervision: The quality of GPT-3-generated Chain-of-Thought explanations was not evaluated, although it may affect model performance.
  • Model access: GPT-3 was not fine-tuned on RE datasets, primarily because of its cost and the resulting weights’ lack of local accessibility.The authors instead fine-tuned the smaller, open-source Flan-T5 model.
  • Language: All experiments used English-language datasets, so replication of the highlighted issues in other languages is unknown.

Ethics Statement

The ethics statement describes compensation procedures for the manual annotation and evaluation work conducted through Amazon Mechanical Turk.

  • Mechanical Turk workers were paid at an estimated average rate of $15 per hour per annotation task.The authors estimated task duration using a small set of annotations they completed themselves and applied the rate regardless of worker location.
  • The authors used their own pilot annotations to estimate task time and set fair approximate compensation.
  • The annotation process required payment to workers for each manual evaluation.

A Datasets

The paper evaluates relation extraction on four datasets spanning sentence-level and document-level settings, different entity and relation types, and varying structural complexity.

  • ADE: ADE contains binary drug–adverse-event relations, uses two entity types, and provides a 10-fold split.
  • CONLL04: CoNLL04 contains news sentences annotated with four entity types and five relation types.Its relation labels are KILL, WORK_FOR, LIVE_IN, LOCATED_IN, and ORG_BASED_IN.
  • NYT: NYT contains distantly annotated FreeBase relation triplets from New York Times articles, with three overlapping entity types and 24 relation types.
  • DocRED: DocRED uses long documents with document-level relations, six entity types, 96 relation types, and averages 19.9 entities and 19.5 relation instances per document.

B Models and Reproducibility

The reproducibility details report averaging micro metrics over five random seeds and describe Flan-T5 fine-tuning choices and the experimental hardware.

  • Evaluation: Average micro metrics were reported over five seeds for each dataset.
  • Fine-tuning: Flan-T5-Large experiments used manual tuning for some hyperparameters while leaving most at default values.The manually tuned final values were documented in Table 4.
  • Hardware: All experiments ran on a single NVIDIA Quadro RTX 8000 with 64GB of RAM and an Intel Xeon E502680v4 processor.

B.1 Costs ($$$)

The prompts combine few-shot exemplars, instructional prefixes, and explanatory relation outputs, while the supplementary tables document fine-tuning settings and GPT-3 costs.

  • Prompt construction: Few-shot prompting uses dataset-specific exemplars whose inputs and target relations come from original training sets, with explanations added for experiments.The exemplars include biomedical adverse-effect relations and general-domain entity relations.
  • Prompt construction: Instructional prefixes specify the relation types or output format expected for each input text.Examples ask for drug–adverse-effect pairs or specified relation types among typed entities.
  • Prompt construction: The examples pair extracted relations with concise explanations connecting entities to their stated relationships.Biomedical examples describe drug effects, while general-domain examples describe locations, employment, and organizational bases.
  • Compute and settings: Table 4 reports hyperparameters and compute time for fully fine-tuned Flan models, corresponding to the main results table.The supplementary text notes that most hyperparameters remained at default values, while selected values were manually tuned.
  • Compute and settings: Table 5 summarizes costs incurred when prompting GPT-3 and using it as a relation-extraction labeler.The surrounding text directs readers to the table for experiment-cost details.

D Learning to Identify False False Positives and Negatives

Generative RE evaluation is complicated because language models can express valid entities and relations differently from exact references. The paper therefore combines human reassessment with learned classifiers to identify erroneous false positives and negatives.

  • Evaluation challenge: Exact-match metrics can label generated relations as errors even when manual review finds them semantically correct.Generative models may produce relation pairs or triplets that differ from reference surface forms while remaining inferable from the text.
  • Human evaluation: Human annotators reassessed ostensible false positives and false negatives using the input text and relevant generated or reference relations.The study recruited qualified Mechanical Turk workers after pilot experiments and reported high annotator agreement.
  • Learned evaluation: A fine-tuned BERT classifier predicts whether potential false positives are accurate designations from the input text and generated relation.For false negatives, the classifier also receives the potential missing relation and the full set of generated labels.
  • Results: AUC 0.88 was achieved for identifying potential false positives on CoNLL, compared with AUC 0.73 for identifying false negatives on CoNLL.The authors describe false-positive identification as the strongest and false-negative identification as the weakest setting examined.
  • Implication: The authors characterize learned identification of erroneous false positives and false negatives as a promising route toward automated evaluation of generative RE.This conclusion follows the reported AUC results for the BERT-based classifiers.
Loading 2305.05003v2…