Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguation

1 Introduction

• LLMs may hallucinate or lack knowledge about concepts outside their training corpus, while retraining to address these gaps is time-consuming and expensive.The proposed approach uses KG information during inference rather than retraining the LLM.Source: §1 · p. 2
• KG class taxonomies prune candidate entities, while descriptions enrich prompts for difficult or unseen entities.When remaining candidates fall under the same class, their descriptions are retrieved and appended to the prompt.Source: §1 · pp. 2–3
• Across ten ED datasets, the approach improves LLM disambiguation and is more adaptable across domains than a task-specific model.The evaluation also compares against non-enhanced and description-only enhanced LLMs.Source: §1 · p. 3
• The paper studies how YAGO and DBpedia affect pruning, ED results, and algorithm performance, and classifies failure cases to inform possible improvements.The supplied contribution passages identify these as evaluation and error-analysis goals.Source: §1 · p. 3

2 Related Work

• The related-work section surveys ED methods that use external information and introduces KG-enhanced LLM methods for NLP tasks.Source: §2 · p. 3

2.1 Knowledge-augmented ED

• Prior knowledge-augmented ED approaches add descriptions, types, or KG-derived information, often using models trained or fine-tuned on large datasets.Examples include incorporating entity types or graph embeddings, predicting facts between mentions, and verbalizing KG triples.Source: §2.1 · p. 4

2.2 KG-enhanced LLMs

• KG-enhanced LLM work includes prompting with retrieved subgraphs or triples, while this approach uses a KG ontology's class hierarchy to guide ED.The related approaches described use KG information for question answering.Source: §2.2 · pp. 4–5

3.1 Problem Formulation

• The task is to assign a mention in document d to its referring entity from candidate set C.Source: §3.1 · p. 5

3.2 Method

• The LLM repeatedly prunes candidates until one entity remains, using class-level choices when available and descriptions when selecting among entities.The procedure can prompt the LLM up to k times in the worst case.Source: §3.2 · p. 5, Pruning Algorithm · pp. 7–8

Subgraph Generation

• The candidate subgraph is a DAG rooted at Thing, with candidate entities as leaves; nodes may have multiple predecessors.Construction links candidates to their classes and simplifies the graph by removing irrelevant classes, inferred path relations, self-links, and certain intermediate nodes.Source: Subgraph Generation · pp. 5–6

Pruning Algorithm

• At each step, the algorithm finds the candidates’ deepest shared ancestor and considers its direct successors.It handles class-only, entity-only, and mixed successors with different LLM choice prompts and removes alternatives that are not selected.Source: Pruning Algorithm · pp. 7–8
• A None choice or a negative check of the final entity triggers an entity-selection prompt with the remaining candidates.Source: Pruning Algorithm · p. 8

4 Experiments

• The experiments compare the proposal with other methods, examine the KG used, and analyze cases where disambiguation fails.Source: §4 · p. 8

Datasets

• The evaluation uses ten ED datasets spanning news and online articles, Wikipedia, and hand-crafted brief ambiguous sentences.Table 1 summarizes their statistics.Source: Datasets · p. 8, Table 1 · p. 9

Candidate Sets

• Candidate sets contain ten entities, combining Wikipedia hyperlink-count candidates with BLINK retrieval when needed.Source: Candidate Sets · p. 8

Knowledge Graphs

• YAGO supplies the class hierarchy, while Wikipedia descriptions are truncated to 250 characters before being appended to prompts.DBpedia provides a shallower, manually created ontology for comparison with YAGO’s more granular taxonomy.Source: Knowledge Graphs · p. 9

Evaluation Metric

• Results use inKB micro-F1 and the percentage of Gold F1 achieved to compare KGs with different entity coverage.Gold F1 is the maximum inKB micro-F1 attainable when candidate sets may omit the ground-truth entity.Source: Evaluation Metric · p. 9
• Dataset scores are combined as a weighted average using each dataset’s number of instances as its weight.Source: Evaluation Metric · p. 9

Large Language Models

• The experiments use GPT-3.5-turbo-1106 at temperature 0, chosen as a trade-off among reasoning capability, operating cost, and API availability.Temperature 0 was selected to decrease response randomness and creativity for factual answers.Source: Large Language Models · p. 9

4.2 Results

• ChatEL can return an empty prediction when its response contains no candidate, unlike this approach, which always returns an answer.The authors note that this difference can affect precision and F1 scores, and use achieved gold percentage to make comparisons more comparable.Source: §4.2 · p. 10
• The KG-enhanced approach outperforms the non-enhanced baseline on all ten datasets.The comparison is reported in Table 2.Source: §4.2 · p. 10, Table 2 · p. 11
• Against ChatEL, the approach scores higher on 6 of 10 datasets, with a weighted average 1.8 percentage points higher.ChatEL’s complete evaluation uses GPT-4, whose cost per token is more than 20 times higher; graph manipulation runs two orders of magnitude faster than LLM calls.Source: §4.2 · p. 12
• The task-specific model scores higher on 6 datasets and averages 1.5 percentage points higher, while the proposed model leads by more than 25 points on KORE.The paper attributes the task-specific model’s training to a large Wikipedia dataset and fine-tuning on news ED data, and describes the LLM methods as more adaptable.Source: §4.2 · p. 12

4.3 KG Expressivity Impact

• YAGO has over a thousand times as many classes as DBpedia and nearly twice the average path depth; Barcelona is classified as both Place and Organization in YAGO, but only Place in DBpedia.The comparison illustrates the taxonomies’ differing granularity and semantic expressivity.Source: §4.3 · p. 12, Figure 4 · p. 13, Table 3 · p. 13
• YAGO outperforms DBpedia on 7 of 10 datasets, with a weighted average score 2.4 percentage points higher.The paper reports that YAGO’s advantage is absent primarily on the OKE datasets.Source: §4.3 · p. 12
• On OKE datasets, generic occupations can lead to a disambiguation between Head of Government and Other, with the LLM choosing the contextual fit over the ground-truth occupation entity.The paper gives the example of Reagan’s mention as governor, where the ground truth is the entity “Governor.”Source: §4.3 · pp. 12–13
• Both taxonomies average close to 2.2 iterations; despite YAGO’s deeper hierarchy, the paper reports that graph depth does not significantly affect performance.Graph manipulation takes two orders of magnitude less time than LLM calls.Source: §4.3 · p. 13, Table 4 · p. 14

4.4 Error Analysis

• The authors examine cases in which the method produces an incorrect disambiguation.They frame this error analysis as important for assessing the method’s capabilities and limitations.Source: §4.4 · p. 14
• The error analysis classifies the scenarios that lead to incorrect predictions.The passage describes examining and categorizing these cases.Source: §4.4 · p. 14
• The stated purpose is to assess the capabilities and limitations of LLMs on this task.Source: §4.4 · p. 14

Ground truth errors

• Incorrect dataset annotations can make the gold entity wrong, as when “English” is labeled “England” instead of “English Language.”Source: Ground truth errors · p. 14

KG errors

• KG class errors can cause pruning to fail, and inconsistent class assignments can place similar entities under different categories.The passage describes Bounty (chocolate bar) being labeled an Architectural Structure and the Supreme Courts of Florida and California assigned different classes.Source: KG errors · p. 14

Ambiguous errors

• Highly ambiguous context can leave multiple candidates plausible, as with Justin Timberlake and Justin Bieber in a sentence about MTV and Twitter.The passage also notes that some ground-truth labels could be argued to be incorrect.Source: Ambiguous errors · p. 14

• The error analysis covers ACE2004 and KORE only; extending it to all datasets was unfeasible because of their sizes.Source: LLM errors · pp. 14–15, Table 5 · p. 15
• Among the examined errors, 5 LLM errors involved missed context and 7 involved interpreting the mention as the wrong class.The passage gives “Tiger lost the US Open” as an example of missed context and a Fox Sports mention as an example of wrong class interpretation.Source: LLM errors · p. 15
• In a small experiment on 12 of these LLM errors, GPT-4 correctly disambiguated 8 and Mistral Large correctly disambiguated 7.Source: LLM errors · p. 15

• The proposed approach outperforms non-enhanced and description-enhanced LLMs and has higher adaptability across domains than task-specific methods.Source: §5 · p. 15
• The method uses KG class taxonomies to gradually prune candidates and adds entity descriptions to prompts for entity-level disambiguation.Source: §5 · p. 15
• Using more semantically expressive KGs improves ED results without degrading pruning performance.Source: §5 · p. 15
• The authors propose studying more powerful LLMs as future work, since they could help disambiguate difficult mentions.Source: §5 · p. 16