PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models

Andrea Papaluca, Daniel Krefl, Sergio Mendez Rodriguez, Artem Lensky, Hanna Suominen

arXiv:2312.01954v1cs.CL

TL;DR

Supervised TE models rely on labeled data that cover only part of the possible entities and relations, while earlier zero- and few-shot LLM performance was often underwhelming. This paper dynamically retrieves KB context—triplets or sentence–triplet examples—for LLM prompts. The added context improves TE performance, though LLMs remain below classical state-of-the-art models on the two tested benchmarks.

  • Problem

    Supervised TE models are limited by labeled datasets that cover only part of the possible entities and relations, and tailored training data can be expensive to produce.

    Source: §1 · p. 1

  • Method

    The pipeline dynamically retrieves relevant KB triplets or sentence–triplet examples and provides them in prompts to LLMs for TE.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    KB context substantially improved LLM TE performance, with sentence–triplet examples bringing larger models closer to classical state-of-the-art models and outperforming most older baselines, although LLMs remained below state of the art on both tested benchmarks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    TE performance was associated with KB context quality and improved approximately logarithmically with model size, suggesting that improving the KB and retriever might be more effective than increasing model size.

    Source: §5 · p. 10

    Even the largest tested LLMs did not match classical end-to-end state-of-the-art models on the two standard benchmark datasets.

    Source: §5 · p. 9

Abstract

from arXiv · show

In this work, we tested the Triplet Extraction (TE) capabilities of a variety of Large Language Models (LLMs) of different sizes in the Zero- and Few-Shots settings. In detail, we proposed a pipeline that dynamically gathers contextual information from a Knowledge Base (KB), both in the form of context triplets and of (sentence, triplets) pairs as examples, and provides it to the LLM through a prompt. The additional context allowed the LLMs to be competitive with all the older fully trained baselines based on the Bidirectional Long Short-Term Memory (BiLSTM) Network architecture. We further conducted a detailed analysis of the quality of the gathered KB context, finding it to be strongly correlated with the final TE performance of the model. In contrast, the size of the model appeared to only logarithmically improve the TE capabilities of the LLMs.

1 Introduction

Supervised TE models depend on labeled datasets that cover only part of the possible entities and relations, limiting their generality. This work adds dynamically retrieved KB context to LLM prompts, improving their TE performance and bringing example-augmented results closer to fully trained models.

1 Introduction

  • TE extracts subject–predicate–object triplets that carry sentence meaning, making triplet identification a key ingredient in sentence understanding.The passage describes the task as important for NLP.Source: §1 · p. 1
  • Labeled TE models are restricted by dataset coverage and the closed sets of entities and relations seen in training, while tailored training data can be expensive to produce.The paper investigates LLM prompting as a lower-data alternative, but notes that prior zero- and few-shot results were often underwhelming against fully trained models.Source: §1 · pp. 1–2
  • The paper proposes dynamically retrieving KB context for LLM prompts, reporting substantial TE improvement and closer performance to fully trained state-of-the-art models when context includes relevant sentence–triplet examples.The introduction frames the approach as a way to make LLMs more competitive with classic NLP baselines.Source: §1 · p. 2

2 Related Work

Earlier TE work includes supervised end-to-end models and several LLM approaches for low-data settings. This study tests dynamic KB retrieval on two standard datasets and analyzes performance across models of varying sizes and context quality.

2 Related Work

  • Classical end-to-end supervised models hold the strongest TE performance, with Transformer-based UniRel described as the current state of the art.Earlier architectures used bidirectional RNNs before Transformer-based models improved performance.Source: §2 · p. 2
  • Prior LLM work explored generating training examples or filling relation templates, while later studies tested prompting, sequential extraction, and zero- or one-shot settings.Classical fine-tuned models remained superior in most cases reported by the cited benchmark evaluations.Source: §2 · p. 2
  • This study tests zero- and few-shot LLMs on two standard TE datasets not covered by the cited prior work, using dynamically retrieved rather than manually crafted static context.The authors also vary model size and analyze both model scaling and the quality of contextual KB information.Source: §2 · p. 2

3 The Pipeline

The pipeline prompts LLMs to extract subject–predicate–object triplets and augments each input with relevant information retrieved from a KB. It retrieves either triplets or sentence–triplet examples by comparing their embeddings with the input sentence.

3.1 Task Formulation

  • TE identifies relations and extracts them as subject–predicate–object triplets, with subjects and objects corresponding to entities and predicates mapping to relation types.The paper describes the task over token spans in an input sentence.Source: §3.1 · pp. 2–3

3.3 KB-aided Triplet Extraction

  • For each test sentence, the pipeline retrieves relevant information from a KB built from training and validation data and adds it to the LLM prompt.Retrieval compares the input sentence embedding with embeddings of context triplets or sentence–triplet pairs using cosine similarity.Source: Figure 1 · p. 3

3.2 LLMs as Triplet Generators

  • LLMs are prompted to generate triplets without task-specific training; after preliminary testing found no significant F1 differences among three prompt variants, the experiments use the base prompt.The base prompt specifies the text to process and the dataset’s maximum number of triplets per sentence.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 KB-aided Triplet Extraction

  • The KB augments the prompt either with retrieved context triplets or with relevant sentence–triplet examples replacing the static examples.A MiniLM sentence encoder represents triplets, while example retrieval can encode the sentence alone or the sentence with its triplets; the top similar pairs are used as few-shot examples. Triplet retrieval selects up to two items per relation type to diversify context, and more sophisticated representations were not investigated because simple embeddings performed well.Source: §3.3 · pp. 3–4

4 Experiments

The experiments test LLM triplet extraction on WebNLG and NYT across zero- and few-shot prompts, then examine how retrieved KB context and model size relate to performance.

4.1 Datasets and Models

  • The experiments used WebNLG and NYT, building each KB from its training and validation splits; tested models included local 8-bit LLMs and API-served GPT models.Source: §4.1 · p. 5

4.2 Zero- and 2-Shots without the KB

  • Two fixed examples substantially improved performance over zero-shot, but LLMs generally remained below classical baselines and struggled to reach 1% F1 on NYT.Source: §4.2 · pp. 5–6, Table 4 · p. 5

4.3 Zero-shot with KB Triplets (0.5-Shots)

  • KB triplets improved NYT results without reaching classical baselines, while GPT-2 base was competitive with context-free LLaMA-65B on both datasets.On WebNLG, larger LLMs performed better than or on par with some classical baselines.Source: §4.3 · p. 6

4.4 Few Shots with KB Sentence-Triplets Pairs

4.5 Quality of the KB Context

  • Correct triplets appeared in retrieved context in more than 50% of most cases and approached 70–80% on WebNLG; increasing examples from five to eight only marginally improved final TE performance.For triplets alone, five worked best in the reported tests, while retrieval probability gains became marginal beyond roughly 10–20 triplets.Source: §4.5 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

4.6 Ablation Study

  • TE performance was strongly correlated with retrieved-data quality and appeared to scale linearly with KB size, but only logarithmically with model size.The KB-size ablation used LLaMA-65b on WebNLG; a preliminary NYT test yielded similar results.Source: §4.6 · pp. 8–9, Figure 7 · p. 9

5 Conclusion

The pipeline’s KB context improves few-shot triplet extraction, with sentence–triplet examples bringing larger LLMs closer to classical SOTA models. The authors also find that retrieved-context quality strongly tracks performance, while benchmark overlap and persistent gaps to SOTA limit the conclusions.

5 Conclusion

  • Sentence–triplet KB examples brought larger LLMs closer to classical SOTA models and above most older baselines.Even the largest tested models remained below classical end-to-end SOTA on WebNLG and NYT.Source: §5 · p. 9
  • The solution was often already present in retrieved context, and random sampling sometimes matched LLaMA-65b; the LLM’s main advantage was robustness to diluted context.Random-model performance decayed as context grew.Source: §5 · pp. 9–10
  • Competitive random-model results raise questions about WebNLG and NYT as general TE benchmarks, with reduced train–validation–test overlap proposed as a more realistic alternative.The random model also matched some older BiLSTM baselines (Zheng et al., 2017).Source: §5 · p. 10
  • TE performance improved approximately logarithmically with model size, while LLaMA-65b performance scaled linearly with the probability that retrieved context contained the solution.The authors suggest improving the KB and retriever might be more effective than increasing model power for TE.Source: §5 · p. 10, Figure 8 · p. 10

A Appendix

The appendix reports prompt variants and their comparison, alongside the prompts used for Chain-of-Thought, task-definition, and KB-context settings.

A Appendix

  • The Chain-of-Thought prompt asks the LLM to proceed step by step through predicate, subject, and object identification before extracting triplets.It limits extraction to {max_triplets}.Source: A Appendix · p. 12, Figure 9 · p. 12
  • The base, Chain-of-Thought, and expanded task-definition prompts produced similar micro-averaged F1 scores, with standard deviation never exceeding 0.003.The comparison covers WebNLG and NYT with GPT-2 xl and LLaMA 65B.Source: A Appendix · p. 12, Table 6 · p. 13
  • The expanded task-definition prompt describes subjects, objects, predicates, and knowledge triplets before requesting extraction.It also caps output at {max_triplets}.Source: A Appendix · p. 12, Figure 10 · p. 12
  • The 0.5-Shots prompt adds retrieved context triplets and asks the LLM to select relevant ones before extracting triplets from the input text.The prompt is an adaptation of the base prompt for KB triplets.Source: A Appendix · pp. 12–13, Figure 11 · p. 13
Loading 2312.01954v1…