PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-RE: In-context Learning for Relation Extraction using Large Language Models

Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, Sadao Kurohashi

arXiv:2305.02105v3cs.CL

TL;DR

Relation extraction remains challenging for LLM in-context learning because retrieved demonstrations may lack entity-relation relevance and NULL examples are often overpredicted as labeled relations. GPT-RE addresses these issues with task-aware retrieval and gold label-induced reasoning, achieving state-of-the-art results on Semeval and SciERC and competitive results on TACRED and ACE05.

  • Problem

    In-context learning for relation extraction is limited by low entity-relation relevance in retrieved demonstrations and overprediction of NULL examples.

  • Method

    GPT-RE combines task-aware retrieval using entity- and relation-focused representations with gold label-induced reasoning in demonstrations.

  • Results

    GPT-RE improves over GPT-3 and fully supervised baselines, reaches state-of-the-art performance on Semeval and SciERC, and is competitive on TACRED and ACE05.

  • Takeaways & Limitations

    Task-specific retrieval and reasoning evidence enable GPT-3 in-context learning to outperform fine-tuning baselines on three datasets and reach state-of-the-art results on two.

  • Takeaways & Limitations

    NULL recall still lags behind fully supervised baselines, especially on datasets with many NULL examples such as ACE05.

Abstract

from arXiv · show

In spite of the potential for ground-breaking achievements offered by large language models (LLMs) (e.g., GPT-3), they still lag significantly behind fully-supervised baselines (e.g., fine-tuned BERT) in relation extraction (RE). This is due to the two major shortcomings of LLMs in RE: (1) low relevance regarding entity and relation in retrieved demonstrations for in-context learning; and (2) the strong inclination to wrongly classify NULL examples into other pre-defined labels. In this paper, we propose GPT-RE to bridge the gap between LLMs and fully-supervised baselines. GPT-RE successfully addresses the aforementioned issues by (1) incorporating task-specific entity representations in demonstration retrieval; and (2) enriching the demonstrations with gold label-induced reasoning logic. We evaluate GPT-RE on four widely-used RE datasets, and observe that GPT-RE achieves improvements over not only existing GPT-3 baselines, but also fully-supervised baselines. Specifically, GPT-RE achieves SOTA performances on the Semeval and SciERC datasets, and competitive performances on the TACRED and ACE05 datasets.

1 Introduction

GPT-RE addresses two weaknesses limiting in-context learning for relation extraction: irrelevant demonstrations and overprediction of NULL examples. It combines task-aware retrieval with gold label-induced reasoning, improving performance across four RE datasets.

  • Motivation: LLM-based in-context learning for relation extraction can underperform fine-tuned models because retrieved demonstrations may poorly match entities and relations.Sentence-level retrieval emphasizes overall semantics rather than the specific entity pair and relation required by RE.
  • Approach: GPT-RE uses task-aware retrieval representations that emphasize entity and relation information rather than general sentence semantics.The proposed approaches include entity-prompted sentence embeddings and fine-tuned relation representations.
  • Approach: GPT-RE enriches demonstrations with gold label-induced reasoning explaining why an input matches a relation and why NULL examples match none.The reasoning provides additional evidence for aligning diverse relational expressions with labels.
  • Motivation: GPT-3 also tends to classify NULL examples as predefined relations instead of recognizing that no relation exists.The paper identifies this overprediction as a separate challenge in RE.
  • Results: GPT-RE improves over existing GPT-3 and fully supervised baselines, achieving state-of-the-art performance on Semeval and SciERC and competitive performance on TACRED and ACE05.The evaluation covers three general-domain datasets and one scientific-domain dataset.

2 Methodology: GPT-RE

GPT-RE formalizes relation extraction as language generation and improves in-context learning through task-aware demonstration retrieval and gold label-induced reasoning. It emphasizes entity-pair information when selecting demonstrations and adds label-conditioned clues to make demonstrations more informative.

  • Task Definition: Relation extraction predicts a predefined relation between a subject–object entity pair or NULL when no predefined relation holds.
  • Prompt Construction: GPT-RE constructs prompts containing task instructions, retrieved demonstrations, and a test input for GPT-3 to generate the corresponding relation.
  • Task-aware Demonstration Retrieval: Sentence-embedding retrieval can mismatch relation extraction because whole-sentence semantics underrepresent the specific entity pair and relation.
  • Task-aware Demonstration Retrieval: GPT-RE uses entity-prompted sentence embeddings or fine-tuned relation representations to emphasize entity and relation information during demonstration retrieval.Entity prompting preserves sentence meaning while adding entity-pair-centered information; fine-tuned representations encode entity information through marked subject and object tokens.
  • Task-aware Demonstration Retrieval: Fine-tuned relation representations can also help recognize similar NULL demonstrations, addressing the tendency to overpredict predefined labels.
  • Gold Label-induced Reasoning: Gold label-induced reasoning asks GPT-3 to generate clues explaining why a selected demonstration expresses its labeled relation, then appends those clues to the demonstration.The reasoning prompt explicitly conditions the generated explanation on the entity pair, relation label, and context.

3 Experiment Setup

The experiments evaluate GPT-RE across three general-domain and one scientific-domain relation extraction datasets, comparing GPT-3 retrieval baselines with fine-tuned baselines. The setup includes random and sentence-embedding GPT-3 baselines, while some analyses use sampled test subsets because of GPT-3 API cost.

  • Datasets: GPT-RE is evaluated on Semeval 2010 Task 8, TACRED, ACE05, and SciERC.Semeval, TACRED, and ACE05 are general-domain datasets, while SciERC represents the scientific domain.
  • Evaluation Constraints: Main results sample subsets from the ACE05 and TACRED test sets because running GPT-3 through the API is costly.
  • GPT-3 Baselines: The GPT-3 experiments use text-davinci-003 with a maximum of 4,097 input tokens and identical prompt construction.
  • GPT-3 Baselines: GPT-Random selects few-shot demonstrations with additional constraints designed to make their label distribution more uniform.The authors report that this is a stronger baseline than vanilla random selection in preliminary experiments.
  • Baselines: GPT-Sent retrieves demonstrations using SimCSE sentence embeddings, while PURE supplies a directly comparable entity-marker-based fine-tuned baseline.The study also compares against corresponding state-of-the-art fine-tuned baselines on Semeval and TACRED.

4 Experimental Results

GPT-RE improves RE performance through task-aware retrieval and reasoning-enhanced demonstrations. Across ablations, demonstration quality is more influential than quantity, while reasoning helps especially with fewer demonstrations and low-shot settings.

  • Main Results: GPT-RE_FT outperforms PURE by +2.00, +2.42, and +0.55 Micro-F1 on three datasets and achieves SOTA results on Semeval and SciERC.
  • Task-aware Retrieval: Task-aware retrieval improves performance, with GPT-RE_FT outperforming all retrieval methods and beating fine-tuning when k > 15.
  • Task-aware Retrieval: 80.30 −→83.43(+3.13) shows that five GPT-RE_FT demonstrations outperform thirty GPT-RE_SimCSE demonstrations, highlighting demonstration quality over quantity.
  • Reasoning Enhancement: Reasoning-enhanced demonstrations improve both retrieval variants across all k-shot settings, with smaller GPT-RE_FT gains when more high-quality demonstrations are available.
  • Reasoning Enhancement: Reasoning enhancement is expected to help low-shot relation extraction, although reasoning tokens create a trade-off with adding more demonstrations under GPT-3's input limit.
  • Low-resource Scenario: In Semeval, GPT-3-based methods outperform fine-tuning below 650 training examples, while GPT-RE_FT maintains the upper-bound performance across resource settings.

5 Analysis

The analysis examines NULL overprediction and demonstration quality. Task-aware retrieval improves entity-pair relevance, while GPT-RE remains challenged by NULL cases involving relations outside predefined classes.

  • The Issue of Overpredicting: Semeval contains 17.4% NULL examples, whereas SciERC contains 90.16%, and the larger NULL proportion produces a more significant performance drop.
  • The Issue of Overpredicting: GPT-RE_FT is most robust to NULL examples, but all GPT-3 methods still underperform fine-tuning on NULL cases.
  • The Issue of Overpredicting: GPT-3 tends to overpredict predefined relations when contextual relation information is present but falls outside the defined classes, suggesting relevance to open RE.
  • Case Study of Demonstration Quality: GPT-Sent retrieves demonstrations matching the shared relation phrase but not the target entity pair, whereas GPT-RE_FT retrieves a structurally similar entity-pair example.
  • Case Study of Demonstration Quality: GPT-RE_SimCSE improves relation matching but retains a mismatch between the demonstrated and target entity pairs.

6 Related Work

Prior work studies in-context learning, demonstration selection, and GPT-3 for information extraction. GPT-RE addresses the reported weaknesses of GPT-3 ICL in relation extraction across general and scientific domains.

  • In-context Learning: ICL enables GPT-3 to perform diverse tasks from a few natural-language demonstrations, with prior work examining prompt design and output coherence.
  • In-context Learning: Earlier biomedical RE results suggested that GPT-3 ICL significantly underperformed fine-tuned models, while GPT-RE reports stronger performance across general and scientific RE.
  • Retrieval-based Demonstrations: Dynamic retrieval of nearest demonstrations generally improves GPT-3 ICL over fixed or distant examples, motivating better retrieval modules.

7 Conclusions

GPT-RE combines task-aware demonstration retrieval with gold label-induced reasoning to improve GPT-3 in-context learning for relation extraction. It significantly outperforms fine-tuning on three datasets and reaches SOTA on Semeval and SciERC.

  • Task-aware retrieval emphasizes entity and relation information, while gold label-induced reasoning enriches the evidence available for each demonstration.
  • GPT-RE is reported as the first GPT-3 ICL method to significantly outperform fine-tuning on three datasets and achieve SOTA on Semeval and SciERC.

Limitations

GPT-RE retains two limitations despite positive results: NULL overprediction is reduced but not eliminated, and task-aware retrieval depends on representations whose robustness may be limited.

  • Limitations: Task-aware retrieval optimizes representations from PLMs such as SimCSE and BERT, but the passage notes that LLMs may generate more robust representations.
  • Experimental setting: The experiments use the GPT-3 API with temperature 0.0, so results are reported from a single run.
  • Experimental setting: Different demonstration-search ranges are used across datasets because GPT-3 has input-length limits and datasets have different average context lengths.
  • Experimental setting: The study follows a single-sentence setup across datasets and trains the PLM baselines with dataset-specific encoders using two NVIDIA RTX3090 GPUs.

A.3 Sentence Embedding Methods

The sentence-embedding demonstrations illustrate relation labels through entity-pair contexts, with COMPONENT AND WHOLE explicitly identified as a gold label. The section also includes an example figure entry and a fragmentary case label.

  • Sentence embedding methods: SimCSE uses the supervised sup-simcse-bert-base-uncased sentence-embedding model for demonstration retrieval.
  • Demonstrations: The demonstrations assign MEMBER AND COLLECTION to principles–set and survivors–sisterhood entity pairs.
  • Demonstrations: The demonstrations assign COMPONENT AND WHOLE to ink–screen, mouse–computer, and screen–computer entity pairs.
  • Demonstrations: One annotation states that COMPONENT AND WHOLE denotes the gold label.

B Case Study

The case study compares retrieval strategies and shows that task-aware methods select demonstrations with more relevant entity pairs and relations than sentence-level retrieval.

  • Case study: The authors provide additional cases to verify the effectiveness of task-aware demonstration retrieval.
  • Case study: GPT-Sent retrieves a semantically similar “design” example whose entity pair is irrelevant, producing a noisy demonstration.
  • Case study: GPT-RE_SimCSE retrieves a more relevant demonstration whose entity pair is closer and shares the same relation label.
  • Case study: GPT-RE_FT retrieves a demonstration with both the closest entity pair and the same linguistic structure between entities.

C Subset

Subset construction accounts for NULL prevalence and long-tail labels, preserving label distributions while showing a stronger GPT-RE_FT result on an enlarged TACRED subset.

  • C Subset: Subset sampling preserves each relation label’s proportion from the original test set, with NULL prevalence determining the required sample size.
  • C Subset: The sampled-example count depends not only on training-set size but also on the proportion of NULL examples.
  • C Subset: Table 6 presents the TACRED label distribution used for subset construction.
  • C Subset: 73.16 GPT-RE_FT performance at k = 15 on the enlarged TACRED subset exceeds PURE’s 70.48.
  • C Subset: The enlarged subset addresses TACRED’s long-tail labels, some of which have only one example in the current subset.
Loading 2305.02105v3…