PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

Derong Xu, Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, Enhong Chen

arXiv:2412.18537v2cs.CL

TL;DR

LLMs can produce incorrect answers when complex reasoning requires specialized or up-to-date knowledge, and KG retrieval can add irrelevant context. AMAR aligns and gates retrieved entities, relations, and subgraphs as prompt embeddings; it reports state-of-the-art performance against 22 baselines.

  • Problem

    KG retrieval can supply irrelevant or partially relevant context, and prior approaches did not account for cross-aspect commonalities or adaptively learn relevance to the question.

    Source: Introduction · p. 2

  • Method

    AMAR converts retrieved entities, relations, and subgraphs into prompt embeddings, aligns their commonalities, and applies a soft gate to select relevant information for LLM reasoning.

    Source: Introduction · p. 2

  • Results

    AMAR outperforms 22 baseline models and achieves state-of-the-art performance, with validation across two datasets and five metrics.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The paper reports that AMAR is more effective than simply appending retrieved text as context and minimizes noise interference.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) demonstrate remarkable capabilities, yet struggle with hallucination and outdated knowledge when tasked with complex knowledge reasoning, resulting in factually incorrect outputs. Previous studies have attempted to mitigate it by retrieving factual knowledge from large-scale knowledge graphs (KGs) to assist LLMs in logical reasoning and prediction of answers. However, this kind of approach often introduces noise and irrelevant data, especially in situations with extensive context from multiple knowledge aspects. In this way, LLM attention can be potentially mislead from question and relevant information. In our study, we introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. The Amar framework comprises two key sub-components: 1) a self-alignment module that aligns commonalities among entities, relations, and subgraphs to enhance retrieved text, thereby reducing noise interference; 2) a relevance gating module that employs a soft gate to learn the relevance score between question and multi-aspect retrieved data, to determine which information should be used to enhance LLMs' output, or even filtered altogether. Our method has achieved state-of-the-art performance on two common datasets, WebQSP and CWQ, showing a 1.9\% improvement in accuracy over its best competitor and a 6.6\% improvement in logical form generation over a method that directly uses retrieved text as context prompts. These results demonstrate the effectiveness of Amar in improving the reasoning of LLMs.

Introduction

AMAR addresses the challenge of using KG retrieval to support LLM reasoning when retrieved context can contain irrelevant or partially relevant information. It combines multi-aspect prompting with self-alignment and relevance gating, and reports state-of-the-art results against 22 strong baselines.

Introduction

  • LLMs can produce factually incorrect outputs when specialized knowledge is absent from training data or reasoning involves outdated knowledge and hallucinations.Source: Introduction · p. 1
  • KG retrieval can return large amounts of irrelevant or partially relevant information, while prior work overlooked cross-aspect commonalities and adaptive question–retrieval relevance learning.Source: Introduction · p. 2
  • AMAR maps retrieved entities, relations, and subgraphs into prompt embeddings, aligns their commonalities, and uses a soft relevance gate to select useful information for LLM reasoning.Source: Introduction · p. 2
  • The work presents multi-aspect KG information as prompt embeddings and pairs self-alignment with relevance gating to select knowledge while reducing noise interference.Source: Introduction · p. 2
  • AMAR achieved state-of-the-art results compared with 22 strong baselines across two datasets and five metrics.Source: Introduction · p. 2

Preliminaries

The preliminaries define knowledge graphs, logical forms, and the KGQA task. They describe a pipeline that retrieves KG knowledge, generates an S-expression, and executes its converted query to obtain answers.

Preliminaries

  • A knowledge graph stores factual knowledge as triples containing a head entity, relation, and tail entity or literal.Source: Preliminaries · p. 2
  • A logical form is a structured representation of a question; this work uses S-expressions with set-based functions such as JOIN and AND.Source: Preliminaries · p. 2
  • KGQA retrieves question-related knowledge, generates an S-expression, converts it to SPARQL, and executes the query against the graph to obtain answers.Source: Preliminaries · p. 2

Methodology

AMAR retrieves multiple forms of KG knowledge, aligns and gates them as prompt embeddings, then generates and refines logical forms for KG querying.

Retrieval of Multi-Aspect Knowledge

  • AMAR retrieves entities, relations, and subgraphs using different methods suited to their distinct representations.Entity retrieval uses ELQ and FACC1; relation retrieval masks linked entity mentions and searches a shared dense space, while 1-hop subgraphs group linearized triples by head entity and use sparse BM25 retrieval.Source: Retrieval of Multi-Aspect Knowledge · pp. 2–3

Methodology

  • AMAR combines question and retrieved-knowledge embeddings as LLM input, then refines the generated logical expression and queries the KG for answers.The framework overview depicts retrieved knowledge and question tokens entering the LLM before the logical expression is refined for KG querying.Source: Figure 2 · p. 3

Self-Alignment

  • Self-alignment maps retrieved texts to prompt embeddings and uses attention across retrieval texts to align commonalities among entities, relations, and subgraphs.Entity and relation self-attention produces consistency tokens; cross-attention between entities, relations, and subgraphs weights subgraph information through shared content.Source: Self-Alignment · pp. 3–4

Relevance Gating

  • Relevance gating estimates question–retrieval relevance and applies soft gates to consistency tokens before combining them with the question for LLM input.The model uses a shared MLP, similarity scores, averaging, and a sigmoid to produce relevance scores; the LLM parameters remain frozen while model and LoRA parameters are optimized.Source: Relevance Gating · p. 4

Query Execution

  • Generated logical forms may contain nonexistent entities or relations, so AMAR matches their terms against KG entities and nearby relations before SPARQL execution.The refinement step uses entity-label similarity, identifies relations within two hops of matched entities, and converts the resulting logical forms to SPARQL for KG querying.Source: Query Execution · p. 4

Experiments

Experiments use WebQSP and CWQ, datasets whose questions have corresponding SPARQL queries executable on Freebase.

Experiments

22 baselines,

AMAR outperforms baselines across both datasets and all reported metrics; ablations and analyses also report benefits from its modules and multi-aspect retrieval, particularly as retrieval volume grows.

22 baselines,

  • The evaluation spans EM-, IR-, SP-, and LLM-based methods, using Hits@1, F1, and Acc; fine-tuning uses LLaMA2-7B/13B with LoRA.Inference uses beam search to generate logical forms and selects the executable form with the highest score.Source: §22 · pp. 4–5

Main Results

  • AMAR outperforms all baselines across every reported metric on both datasets, with WebQSP accuracy 1.6% above second-best ChatKBQA.The comparison covers KGQA methods on WebQSP and CWQ.Source: Table 1 · p. 5, Main Results · p. 5

Ablation Study

  • Removing SN, SA, or RG lowers most metrics, while removing the full framework causes a significant performance drop versus AMAR.The full-framework variant directly appends retrieved knowledge as context rather than converting it to prompt embeddings.Source: Table 2 · p. 5, Ablation Study · p. 6
  • Removing any retrieval aspect reduces performance; relation removal causes the largest drop, while subgraphs are less critical for logical forms than relations or entities.Source: Table 3 · p. 6, Ablation Study · p. 6

Number of Retrieval Analysis

Efficiency of Fine-Tuning

  • AMAR outperforms direct-context baselines with either PT or LoRA, PT+LoRA performs best, and LoRA alone outperforms PT.The authors attribute LoRA's advantage to more tunable parameters and its ability to act on all projection layers.Source: Efficiency of Fine-Tuning · p. 6

Case Study

  • Table 4 assigns high scores to correct entity Harper Lee (0.9272) and relation people.person.education (0.9844), while direct context can yield an erroneous school-type relation.Source: Case Study · p. 6, Table 4 · p. 7

Analysis of LLM Backbones

  • Fine-tuned AMAR with LLaMA2-7B/13B beats same-backbone ChatKBQA and surpasses GPT-4 ToG-R on both datasets; 13B trails 7B on WebQSP, possibly from overfitting.The authors connect the WebQSP result to its smaller scale and maximum two-hop question complexity.Source: Table 5 · p. 7, Analysis of LLM Backbones · p. 7

Related Work

KGQA research includes methods that retrieve or embed KG information, generate executable logical forms, or use LLMs for reasoning. Prior work also identifies irrelevant multi-aspect retrieved knowledge and lengthy context as challenges.

  • KGQA methods are grouped as embedding-based, information retrieval-based, semantic parsing-based, and LLM-based approaches.Source: Related Work · p. 7
  • Semantic parsing methods translate questions into executable queries such as SPARQL or S-expressions, while retrieval-based methods find or rerank KG answers.The cited examples include executable-query approaches and answer retrieval/reranking.Source: Related Work · p. 7
  • LLM-based KGQA includes planning-retrieval-reasoning pipelines and interactive path exploration, while multi-aspect retrieved input can introduce irrelevant knowledge.The passage also notes that lengthy texts challenge LLM reasoning.Source: Related Work · p. 7

Conclusion

AMAR retrieves multi-aspect KG knowledge and uses self-alignment and relevance gating to enhance and select relevant information. It outperforms 22 baselines and achieves state-of-the-art performance.

  • AMAR retrieves multi-aspect knowledge from KGs to enhance LLM reasoning and factual output.Source: Conclusion · p. 7
  • AMAR outperforms 22 baseline models and achieves new state-of-the-art performance.Source: Conclusion · p. 8
  • Self-alignment and relevance gating adaptively enhance and select relevant information, minimizing noise interference compared with simply appending retrieved text.Source: Conclusion · p. 8

Appendix Comparison in Oracle Entities setting

In the oracle entity-linking setting, AMAR reports state-of-the-art improvements and also exceeds several oracle-dependent baselines without using the oracle setting itself.

Logical Form

The paper represents questions with S-expressions that use set-based operators, balancing readability and compactness for KGQA.

Logical Form

  • S-expression F is the chosen logical form, using set-based functions such as JOIN and AND for KGQA.Source: Logical Form · p. 11
  • JOIN queries a triple through either its head or tail, with (JOIN r t) and (JOIN (R r) h) expressing the respective patterns.Source: Logical Form · p. 11
  • The operator set includes intersection, counting, extrema, and numeric comparisons such as GT, GE, LT, and LE.Source: Logical Form · p. 11

Retrieval of Multi-Aspect Knowledge

AMAR retrieves candidate entities, relations, and subgraphs using distinct retrieval procedures; the accompanying tables report recall on WebQSP and CWQ.

Entity Retrieval

Relation Retrieval

Retrieval Evaluation

Subgraph Retrieval

Query Execution

Query execution refines generated S-expressions by identifying entities and relations while preserving logical structure, then converts the refined form to SPARQL and executes it against the KG.

Query Execution

  • Refinement targets entities and relations in the generated S-expression while leaving its logical structure unchanged.The example identifies Oceania and Country as entities and two relations as targets.Source: Query Execution · p. 12
  • Entity candidates are selected by similarity threshold, while relation candidates come from connected neighbors and are ranked by similarity.The refinement follows Luo et al. (2023) and uses SimCSE for similarity scoring.Source: Query Execution · pp. 12–13
  • The refined S-expression is converted to a SPARQL query and executed against the KG to obtain final results.The passage illustrates the conversion with the refined example.Source: Query Execution · p. 13

Experiment Settings

The experiments use WebQSP and CWQ, compare AMAR with a broad set of KGQA baselines, and report training-time and space comparisons against direct text-context use.

Datasets

  • WebQSP and CWQ are Freebase-based datasets, with WebQSP providing full SPARQL parses for 4,737 questions and partial annotations for 1,073 others.CWQ is described as a natural-language dataset of complex questions requiring reasoning over multiple web snippets.Source: Datasets · p. 13
  • The comparison includes 22 baselines across embedding-, information-retrieval-, semantic-parsing-, and LLM-based KGQA, plus six additional baselines in the Oracle entities setting.The baseline set includes methods from all four categories.Source: Datasets · p. 13
  • The baselines cover methods such as graph-based retrieval and reasoning, logical-form generation, interactive KG exploration, and retrieval-augmented approaches.Examples include GraftNet and PullNet, GMT-KBQA, ToG-R, G-Retriever, and GNN-RAG.Source: Datasets · pp. 13–14

Implementation Details

  • Training uses approximately 48GB of VRAM on A6000 GPUs, while querying the KG requires 3GB; the reported hyperparameter search varied the number of retrieved data.The tested retrieval counts were 4, 8, 16, 32, 64, and 100 for both datasets.Source: Implementation Details · p. 14

Experiments on training time and space consumption

  • Training and testing time and space are reported as within acceptable range, with AMAR significantly reducing both costs versus directly using text as context.The results are reported for WebQSP using LLaMA2-7b on an A6000 GPU; Table 11 presents the time and space comparison.Source: Table 11 · p. 14, Experiments on training time and space… · p. 14

Related papers

Loading 2412.18537v2…