PaperlayerView exact revision on arXiv

Source-linked AI summary

EntGPT: Entity Linking with Generative Large Language Models

Yifan Ding, Amrit Poudel, Qingkai Zeng, Tim Weninger, Balaji Veeramani, Sanmitra Bhattacharya

arXiv:2402.06738v3cs.CL

TL;DR

LLMs’ structured-output abilities remain underdeveloped, while entity disambiguation links mentions to knowledge-base entries. EntGPT uses multi-step prompting and instruction tuning to ground LLMs in entities, with reported gains across ED benchmarks and QA tasks.

  • Problem

    LLMs’ ability to produce structured output remains underdeveloped, and entity disambiguation requires linking text mentions to knowledge-base entries.

    Source: §1 · pp. 1–2

  • Method

    EntGPT-P generates and filters entity candidates, augments mentions with LLM-generated information, and prompts the model to select a candidate.

    Source: §2.1 · p. 4

  • Results

    EntGPT-P outperforms other models across six zero-shot ED benchmarks, with a +7.0% average micro-F1 improvement over the second-best model.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    The authors report that enhancing entity-based knowledge helps entity disambiguation and provides better performance in question answering.

    Source: §5 · p. 10

    The reported framework addresses entity disambiguation; extending it to entity linking, which also requires mention extraction, is future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Entity Linking in natural language processing seeks to match text entities to their corresponding entries in a dictionary or knowledge base. Traditional approaches rely on contextual models, which can be complex, hard to train, and have limited transferability across different domains. Generative large language models like GPT offer a promising alternative but often underperform with naive prompts. In this study, we introduce EntGPT, employing advanced prompt engineering to enhance EL tasks. Our three-step hard-prompting method (EntGPT-P) significantly boosts the micro-F_1 score by up to 36% over vanilla prompts, achieving competitive performance across 10 datasets without supervised fine-tuning. Additionally, our instruction tuning method (EntGPT-I) improves micro-F_1 scores by 2.1% on average in supervised EL tasks and outperforms several baseline models in six Question Answering tasks. Our methods are compatible with both open-source and proprietary LLMs. All data and code are available on GitHub at https://github.com/yifding/In_Context_EL.

1. Introduction

The introduction frames structured output from LLMs as underdeveloped and describes linking LLMs to knowledge-base entities as important for connecting neural models with symbolic reasoning.

1. Introduction

  • LLMs’ ability to produce structured output remains underdeveloped and relatively unexplored.The paper situates this gap alongside the impressive text-generation and unstructured-reasoning capabilities of LLMs.Source: §1 · p. 1
  • Entity disambiguation links text fragments representing real-world entities to entries in structured knowledge bases.The introduction presents this integration of deep neural networks and knowledge-base reasoning as essential to advancing AI problem-solving and reasoning.Source: §1 · p. 2

2. EntGPT

EntGPT grounds generative LLMs in knowledge-base entities for entity disambiguation. It offers a three-step prompting approach and an instruction-tuned variant.

2. EntGPT

  • Entity disambiguation maps each document mention to its corresponding entity in a knowledge base.The task is defined over a document’s token sequence and its contained mention subsequences.Source: §2 · p. 4

2.1. EntGPT-P: Entity Disambiguation with Multi-Step Prompting

  • EntGPT-P generates and filters candidates, augments mentions with LLM-generated information, then prompts the LLM to select an entity.Candidate generation combines hyperlink-based Prior and BLINK retrieval, retaining the top 10; the final choice includes candidate descriptions and a None option.Source: §2.1 · pp. 4–5

2.2. EntGPT-I: Entity Disambiguation with Instruction Tuning

  • EntGPT-I instruction-tunes GPT3.5 and Llama2-chat on an instruction dataset created by transforming AIDA examples.The models receive supervised fine-tuning before evaluation on entity disambiguation and question answering.Source: §2.2 · p. 5

3. Methodology

The methodology evaluates entity disambiguation on public benchmarks and question answering on commonsense reasoning datasets, using task-specific metrics and comparisons against supervised and zero-shot baselines.

3. Methodology

  • The evaluation covers ten public entity-disambiguation datasets and six question-answering datasets.The ED datasets use Wikipedia as the knowledge base, with spurious, duplicate, and empty items removed during preprocessing.Source: §3 · pp. 5–6
  • ED comparisons group models by fine-tuning data, while QA comparisons use supervised and zero-shot baselines.EntGPT-I is also evaluated on six commonsense reasoning benchmarks: ARC-C, OBQA, aNLI, CSQA, PIQA, and SIQA.Source: §3 · p. 6

3.1. Evaluation Metrics

  • ED is measured with in-KB micro-F1, excluding empty or out-of-KB entities, and QA is measured with accuracy.The ED metric averages micro-F1 per mention.Source: §3.1 · p. 6

4. Results

EntGPT performs competitively across entity-disambiguation benchmarks, question answering, and ablations. Results also include a case where EntGPT-I links a mention correctly, alongside reported errors.

4.1. Entity Disambiguation

  • +7.0% average micro-F1 improvement over the second-best model, with EntGPT-P outperforming other models across six zero-shot ED benchmarks.EntGPT-I, fine-tuned only on AIDA, achieves SOTA micro-F1 on AQUAINT, ACE2004, CWEB, and WIKI and a +2.1% mean micro-F1 improvement across six datasets, with performance comparable to CoherentED.Source: §4.1 · p. 7, Table 1 · p. 7

4.2. Question Answering

  • EntGPT-I attains SOTA performance on six QA datasets in a zero-shot setting and comparable, sometimes superior, performance to supervised methods.The passage reports accuracy in Table 3 but gives no specific scores.Source: §4.2 · p. 8, Table 3 · p. 8

4.3. Ablation Study

  • Removing EntGPT-P’s auxiliary content severely decreases performance, while removing BLINK candidate generation does not substantively affect it.The ablation covers eight of ten ED benchmarks; WIKI and CWEB are omitted because of budget constraints.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Removing entity candidates from GPT3.5 fine-tuning on AIDA produces a sharp decline in mean micro-F1.The authors attribute the performance increase to entity grounding.Source: §4.3 · p. 9

4.4. Case Study

  • In an OKE15 case, EntGPT-I correctly links “Farnborough” to Farnborough, Hampshire, but the passage also reports cases where it fails.The study compares GPT3.5, EntGPT-P, and EntGPT-I.Source: §4.4 · p. 9, Table 5 · p. 10

5. Discussion

EntGPT’s prompt-based and instruction-tuned variants show entity disambiguation ability, and experiments indicate that entity-based knowledge can also improve question answering. The discussion identifies extending the framework to full entity linking and investigating entity correlation in QA as future work.

5. Discussion

  • EntGPT includes prompt-based and instruction-tuned variants, and the authors report ability in entity disambiguation.They also conclude that entity-based knowledge helps disambiguation and improves question answering.Source: §5 · p. 10
  • Extending EntGPT from disambiguation to full entity linking remains future work, because linking also requires mention extraction.The authors note that prior generative-model studies have not consistently shown advantages over supervised methods.Source: §5 · p. 10
  • The authors propose investigating how entity correlation relates to the observed QA improvement.They identify entity correlation as a potential factor in the better QA performance and suggest incorporating it more deeply into QA.Source: §5 · p. 10

Appendix A. A Case Study on Entity Disambiguation

The disambiguation case study attributes some apparent errors to incorrect ground truth and finds that many EntGPT-P mistakes are reasonable but imperfect predictions. Examples also show differing model outputs across questions.

Appendix A. A Case Study on Entity Disambiguation

  • About one-third of the reported errors are estimated to stem from incorrect ground truth labels.These cases are marked “GT is incorrect” in Table A.6.Source: Appendix A · p. 21
  • Many EntGPT-P mistakes still partially match the ground truth, while most mistakes arise during Step 3 candidate selection.Step 2 augmentation occasionally fails to provide useful auxiliary content.Source: Appendix A · pp. 21–22
  • Table A.7 presents examples from ARC-C and OBQA, with model performance shown for particular questions.The caption identifies ARC-C on the left and OBQA on the right.Source: Table A.7 · p. 23
  • For the summer-solstice question, GPT3.5 answered B, EntGPT-I C, and LLAMA2 A.Source: Appendix A · p. 23
  • For the soil question, GPT3.5 and EntGPT-I answered B, while LLAMA2 answered C.Source: Appendix A · p. 23
  • For the shark question, EntGPT-I answered C, while GPT3.5 and LLAMA2 answered D.Source: Appendix A · p. 23

Appendix B. Case Study on Question Answering

A brief analysis of ARC-C and OBQA examples finds that entity-based knowledge can help with commonsense-plus-logical reasoning but does not necessarily help mathematical reasoning. The examples also include cases where entity linking leads a model astray.

Appendix B. Case Study on Question Answering

  • The analysis of ARC-C and OBQA examples is explicitly brief rather than a comprehensive examination of error types.Source: Appendix B · p. 23
  • All models fail on the cited mathematical-reasoning example, and entity-based knowledge does not necessarily help mathematical reasoning.The example requires understanding kinetic energy and logical deduction.Source: Appendix B · pp. 23–24
  • Entity-based knowledge can improve context understanding and mitigate hallucination in questions requiring both commonsense and logical reasoning, with exceptions.Source: Appendix B · p. 24
  • The soil example shows an exception: entity linking led the model to choose “roots” instead of the more-direct answer “microscope.”Source: Appendix B · p. 24

Related papers

Loading 2402.06738v3…