GPT-RE: In-context Learning for Relation Extraction using Large Language Models

• GPT-RE improves over existing GPT-3 and fully supervised baselines across four datasets, achieving SOTA on Semeval and SciERC and competitive results on TACRED and ACE05.Figure 1 compares micro F1 on two RE datasets and shows GPT-RE substantially outperforming the pictured baselines.Source: Figure 1 · p. 1, §1 · p. 2
• In RE, in-context learning prompts an LLM with task instructions, retrieved training demonstrations, and a test sentence to identify a predefined relation or NULL.The introduction describes ICL as making predictions from a few demonstrations rather than fine-tuning on a task-specific dataset.Source: §1 · pp. 1–2
• GPT-3 ICL for RE faces low entity-and-relation relevance in retrieved demonstrations and weak input-label explanations, while also tending to misclassify NULL examples as predefined relations.Sentence-embedding retrieval favors overall sentence semantics over specific entities and relations; the NULL overprediction issue can be alleviated by supervising retrieval representations with all training NULL examples.Source: §1 · p. 2, Figure 3 · p. 2
• GPT-RE retrieves demonstrations using entity-prompted sentence embeddings or fine-tuned relation representations that emphasize RE-specific entity and relation information.These representations are intended to address the low relevance of demonstrations retrieved using sentence semantics alone.Source: §1 · p. 2
• GPT-RE adds gold label-induced reasoning to demonstrations, explaining why an input fits a relation and why a NULL example fits none of the predefined categories.The paper says this reasoning improves alignment between relations and their diverse expression forms.Source: §1 · p. 2

2.1 Task Definition

• GPT-RE predicts a predefined relation or NULL for a subject–object entity pair in a context.The task is to identify the relation between the entities, or return NULL when no predefined relation applies.Source: §2.1 · p. 3

2.2 Overview

• GPT-RE improves in-context learning through task-aware demonstration retrieval and gold label-induced reasoning.Its workflow combines retrieval of higher-quality examples with explanations added to each demonstration.Source: §2.2 · p. 3

2.3 Prompt Construction

• Each GPT-3 prompt combines task instructions, retrieved and reasoning-enriched demonstrations, and a test input for relation generation.Instructions specify the predefined relation classes and require NULL when none applies; demonstrations are formed as (x_i, y_i, r_i), and GPT-3 generates y_test for the test input.Source: §2.3 · p. 3

2.4 Task-aware Demonstration Retrieval

• Sentence-embedding kNN retrieval can be ambiguous for relation extraction because sentence semantics may diverge from the entity-pair relation being classified.GPT-RE therefore proposes entity-prompted sentence embeddings and fine-tuned relation representations for retrieval.Source: §2.4 · pp. 3–4

2.4.1 Entity-Prompted Sentence Embedding

• Entity-prompted retrieval reconstructs each context to explicitly state the relation between the subject and object while retaining the original sentence meaning.The paper uses SimCSE to compute sentence-embedding similarity for these reconstructed contexts.Source: §2.4.1 · p. 4

2.4.2 Fine-tuned Relation Representation

• The fine-tuned relation representation concatenates BERT hidden states at subject and object markers to predict relation probabilities.Entity markers encode the pair and their types when available; training on the full training set can improve retrieval quality and help retrieve NULL examples accurately.Source: §2.4.2 · p. 4

2.5 Gold Label-induced Reasoning

• GPT-RE asks GPT-3 to generate clues for each selected demonstration conditioned on its gold relation label, then adds those clues to the demonstration.Conditioning the reasoning prompt on the label focuses the clues on the relation that actually holds between the entity pair in context.Source: §2.5 · p. 4

3.1 Datasets

• The four datasets span general-domain RE—Semeval 2010 Task 8, TACRED, and ACE05—and scientific-domain RE through SciERC.Semeval covers semantic relations between nominals; TACRED has 106,264 newswire and web-text examples; SciERC annotates AI-paper abstracts; ACE05 includes entity, relation, and event annotations across multiple domains. (Hendrickx et al., 2010; Zhang et al., 2017; Luan et al., 2018.)Source: §3.1 · p. 5
• In the main results, because running GPT-3 through the API is costly, the authors sample a subset of the original test set for ACE05 and TACRED.Source: §3.1 · p. 5

3.2 Baseline Methods

• GPT-3 baselines use text-davinci-003, a 4,097-token input limit, and the same prompt construction; GPT-Random balances demonstration-label distributions, while GPT-Sent uses SimCSE retrieval.The authors report GPT-Random was stronger than vanilla random selection in preliminary experiments. SimCSE is cited as state of the art for sentence similarity (Gao et al., 2021).Source: §3.2 · p. 5
• PURE supplies retrieval representations and serves as a directly comparable fine-tuned baseline, alongside SOTA fine-tuned baselines for Semeval and TACRED.The Semeval baseline reformulates RE as question answering, while TACRED uses extra pre-training to capture RE structure (Wang et al., 2022a).Source: §3.2 · p. 5

4.1 Main Results

• GPT-RE_FT beats PURE by +2.00, +2.42, and +0.55 Micro-F1 on three datasets and reaches SOTA on Semeval and SciERC.Both GPT-RE_SimCSE and GPT-RE_FT also outperform GPT-Sent, supporting the value of task-specific information for selecting demonstrations.Source: §4.1 · p. 6
• Reasoning improves GPT-RE_SimCSE by around 2% Micro-F1, but high-quality GPT-RE_FT demonstrations leave relatively little room for further gains.The paper attributes the limited gains to GPT-RE_FT's high-quality demonstrations and notes that reasoning may help more with fewer demonstrations.Source: §4.1 · p. 6

4.4 Low-resource Scenario

• With fewer than 650 training examples (10%), all GPT-3-based results outperform fine-tuning on Semeval, and GPT-RE_FT leads across all tested settings.GPT-RE_FT remains strong even when fine-tuning performs poorly with 100–400 examples; GPT-RE_SimCSE shows a substantial difference from GPT-Sent after training size surpasses 30%.Source: Figure 7 · p. 6, §4.4 · p. 7

4.2 Ablation Study on Task-aware Retrieval

• At k = 5, GPT-RE_FT scores 83.43 versus GPT-RE_SimCSE’s 80.30 at k = 30 (+3.13), showing demonstration quality can outweigh quantity.All retrieval-based models outperform GPT-Random; entity-aware SimCSE performs better across k-shot settings, while GPT-RE_FT significantly outperforms the retrieval methods and beats the fine-tuning baseline when k > 15.Source: §4.2 · p. 6

4.3 Ablation Study on Reasoning Enhancing

• With k ≤ 15, reasoning-enhanced demonstrations improve GPT-RE_SimCSE and GPT-RE_FT across all tested settings.For GPT-RE_FT, the improvement becomes less significant as more demonstrations are added; the authors suggest reasoning enhancement may help low-shot relation extraction for novel relations with few or no examples (Han et al., 2018; Geng et al., 2020; Liu et al., 2022a).Source: §4.3 · pp. 6–7

5.1 The Issue of “Overpredicting”

• As NULL examples rise from 17.4% in Semeval to 90.16% in SciERC, performance drops more; GPT-RE_FT is most robust but GPT-3 methods still trail fine-tuning on NULL.The authors attribute residual overprediction to cases where entities have a contextual relation outside the predefined classes, and suggest this tendency may be useful for open RE.Source: §5.1 · pp. 7–8

5.2 Case Study of Demonstration Quality

• The case study shows retrieval improving level by level, from a sentence-level mismatch to demonstrations that better reflect the target entity pair.GPT-Sent matches context semantics rather than the entity pair; GPT-RE_SimCSE is more relevant but mismatches the gold pair, while GPT-RE_FT shares a similar entity-pair structure that clues the relation.Source: Figure 9 · p. 8, §5.2 · p. 8

In-context Learning

• Prior GPT-3 ICL research examined prompt design, output coherence calibration, ordered prompts, and retrieval-based demonstrations (Brown et al., 2020; Perez et al., 2021; Malkin et al., 2022; Lu et al., 2022; Rubin et al., 2022; Liu et al., 2022b; Shin et al., 2021).Source: §6 · p. 8
• The authors report no prior GPT-3 work on general-domain RE and say their approach partly overcomes ICL issues in information extraction, confirming potential in general and scientific RE.The cited biomedical work applied GPT-3 to biomedical NER and RE and identified issues that may also affect information extraction more broadly.Source: §6 · p. 8

Retrieval-based Demonstrations

• Dynamic, test-specific demonstrations and nearest-neighbor examples improve GPT-3 ICL, while existing retrieval methods use sentence embeddings; GPT-RE instead fine-tunes PLMs on target RE tasks for task-specific, robust representations.Prior methods include BERT, RoBERTa, KATE, SimCSE, and Sentence-BERT sentence encoders (Devlin et al., 2019; Zhuang et al., 2021; Liu et al., 2022b; Gao et al., 2021; Reimers and Gurevych, 2019; Wolf et al., 2020).Source: §6 · p. 8

7 Conclusions

• Task-aware retrieval emphasizes entity and relation information, while gold label-induced reasoning enriches each demonstration’s reasoning evidence.These two strategies aim to improve demonstration search accuracy and help bridge the performance gap to fine-tuning baselines.Source: §7 · pp. 8–9
• GPT-RE significantly outperforms fine-tuning baselines on three datasets and achieves state of the art on Semeval and SciERC.The authors describe it, to their knowledge, as the first GPT-3 in-context learning research to significantly outperform fine-tuning on three datasets.Source: §7 · p. 9
• The paper reports detailed studies exploring how GPT-3 overcomes difficulties such as the influence of NULL examples.Source: §7 · p. 9

Limitations

• GPT-RE’s overprediction has been significantly alleviated but not eliminated; its NULL recall still trails fully supervised baselines, particularly on NULL-heavy datasets such as ACE05, where it is 95.60%.Source: Limitations · p. 9

A.1 GPT-3 Hyperparameters

• GPT-3 experiments used the API with Temperature set to 0.0 for stable output, and the authors report a single run for all experiments.The passage says the hyperparameters are listed in Table 3 but does not provide their values here.Source: §A.1 · p. 13
• Because of GPT-3’s input-length limit and dataset-specific context lengths, the authors searched different demonstration-count ranges for each dataset.Those ranges are referenced as being shown in Table 4, whose values are not included in the supplied passages.Source: §A.1 · p. 13

A.2 Fine-tuning Baseline PURE

• For PURE, the authors followed a single-sentence setup to maintain consistency across datasets, including sentence-level Semeval and TACRED.Source: §A.2 · p. 13
• PURE’s base encoder was scibert-scivocabuncased for SciERC and bert-base-uncased for the other three general-domain datasets.These encoder choices follow PURE and the cited work’s domain-specific setup.Source: §A.2 · p. 13
• The PURE fine-tuning baseline followed the hyperparameters in its paper and used two NVIDIA RTX3090 GPUs for training.Source: §A.2 · p. 13

A.3 Sentence Embedding Methods

• Gutiérrez et al. (2022) uses RoBERTa-large [CLS] for retrieval; Liu et al. (2022b) fine-tunes RoBERTa-large on SNLI and MultiNLI, while experiments use sup-simcse-bert-base-uncased.Liu et al. fine-tunes on the two NLI datasets to enhance sentence-embedding quality.Source: §A.3 · p. 13

B Case Study

• In Figure 10a, GPT-Sent retrieves a semantically similar but entity-irrelevant demonstration, GPT-RE_SimCSE retrieves a closer entity pair with the same relation, and GPT-RE_FT also matches entity-pair structure; Figure 10b shows a similar pattern.The example illustrates level-by-level improvement from the proposed methods.Source: B Case Study · p. 14

C Subset

• The subset sampling rule keeps each relation label’s proportion consistent with the original test set, particularly when NULL is prevalent.The paper notes that the sample size depends importantly on the proportion of NULL.Source: C Subset · p. 14
• At k = 15, GPT-RE_FT scores 73.16 versus PURE’s 70.48 on an enlarged #3200 TACRED subset.The enlarged set is twice the size of the current version.Source: Table 6 · p. 14, C Subset · p. 14