PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities

Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, Haofen Wang

arXiv:2505.20099v2cs.CLcs.AIcs.IR

TL;DR

LLM-based QA struggles with limited reasoning, outdated knowledge, and hallucinations in complex tasks. This survey structures and reviews methods that synthesize LLMs with KGs, aligns them with QA challenges, and identifies open issues including retrieval, dynamic integration, scalability, explainability, and fairness. Its analysis indicates that KGs can augment LLMs for multi-hop and iterative reasoning over factual knowledge graphs.

  • Problem

    LLM-based QA struggles with limited reasoning, outdated knowledge, and hallucinated content in complex QA tasks.

  • Method

    The survey proposes a structured taxonomy of LLM–KG methods by QA category and KG role, reviews state-of-the-art approaches, and aligns them with complex QA challenges.

  • Results

    The analysis highlights that LLM–KG approaches can support multi-hop and iterative reasoning over factual knowledge graphs for complex QA.

  • Takeaways & Limitations

    Future work should address efficient retrieval, dynamic knowledge integration, reasoning at scale, and explainable and fairness-aware QA.

  • Takeaways & Limitations

    The survey may miss newly released works and underemphasizes quantitative comparisons because of rapid field expansion, implementation differences, and diverse benchmarks.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable performance on question-answering (QA) tasks because of their superior capabilities in natural language understanding and generation. However, LLM-based QA struggles with complex QA tasks due to poor reasoning capacity, outdated knowledge, and hallucinations. Several recent works synthesize LLMs and knowledge graphs (KGs) for QA to address the above challenges. In this survey, we propose a new structured taxonomy that categorizes the methodology of synthesizing LLMs and KGs for QA according to the categories of QA and the KG's role when integrating with LLMs. We systematically survey state-of-the-art methods in synthesizing LLMs and KGs for QA and compare and analyze these approaches in terms of strength, limitations, and KG requirements. We then align the approaches with QA and discuss how these approaches address the main challenges of different complex QA. Finally, we summarize the advancements, evaluation metrics, and benchmark datasets and highlight open challenges and opportunities.

1 Introduction

LLM-based QA faces reasoning, knowledge freshness, and hallucination challenges in complex settings. This survey organizes LLM–KG methods through a structured taxonomy and aligns them with complex QA challenges.

  • 1 Introduction: LLM-based QA struggles with complex scenarios because of limited reasoning, outdated domain knowledge, and hallucinated content.These limitations arise alongside the broad use of QA in applications including search, chatbots, and fact-checking.
  • 1 Introduction: GraphRAG and KG-RAG combine LLMs with KGs through knowledge integration, reasoning guidelines, and knowledge validation or refinement.These modules target the limitations of LLMs for knowledge-intensive complex QA.
  • 1 Introduction: The survey reviews recent LLM–KG integration methods for complex QA under open-domain and long-context settings.It emphasizes the roles KGs play when synthesized with LLMs.
  • 1 Introduction: Its structured taxonomy categorizes methods from multiple perspectives and aligns LLM–KG approaches with different complex QA types.The categories are non-exclusive, so one work may belong to multiple categories.
  • 1 Introduction: The survey contributes a review of state-of-the-art methods, alignments between approaches and QA challenges, and discussion of current opportunities.It also highlights how methods address specific challenges of complex QA.

2 Complex QA

Complex QA requires decomposition, knowledge fusion, and reasoning across documents, modalities, and sources. The surveyed methods use graph retrieval, multimodal integration, graph reasoning, and specialized pipelines to address these demands.

  • Complex QA: Complex QA commonly requires question decomposition, cross-source or cross-modal knowledge fusion, and complex knowledge reasoning.These requirements motivate methods that synthesize LLMs and KGs.
  • Multi-document QA: Multi-document QA methods use graph traversal, KG prompting, reasoning agents, and multimodal RAG to retrieve and fuse relevant evidence.KGP and CuriousLLM target retrieval across multiple documents, while VisDom integrates multimodal knowledge with CoT reasoning.
  • Multi-modal QA: Multi-modal QA methods select and retrieve knowledge across visual and textual sources, including adaptive selection and learning-to-rank.MMJG, RAMQA, and KVQA integrate multimodal knowledge with LLM-based QA.
  • Multi-hop QA: Multi-hop QA decomposes questions into single-hop subquestions, retrieves relevant subgraphs, generates intermediate answers, and integrates them.GraphLLM exemplifies this process using LLMs and GNNs for graph reasoning.
  • Conversational QA: Conversational QA uses question reformulation, reinforcement learning, soft rewards, knowledge aggregation, and graph reasoning for multi-turn interactions.These methods support repeated rounds of interaction to refine answers.
  • Explainable and Temporal QA: Explainable and temporal QA use structured reasoning and specialized retrieval pipelines to support explanations and temporal reasoning.Examples include HQDT-based probabilistic reasoning, heterogeneous evidence graphs, and retrieve–reretrieve–rerank pipelines.

3 Approaches and Alignments

The survey organizes LLM–KG QA methods by the KG’s role, covering background knowledge, reasoning guidelines, refiners and validators, and hybrid approaches. These methods support complex QA through retrieval, fusion, joint reasoning, and answer verification, while facing scalability and KG-quality challenges.

  • The survey compares approaches for synthesizing LLMs and KGs and aligns them with complex QA tasks.It summarizes their strengths, limitations, and KG requirements.
  • KG as Background Knowledge: Knowledge integration fuses aligned text and KG information to jointly train language models for complex QA.Local subgraph extraction and entity linking connect text and KGs before cross-model encoding.
  • KG as Background Knowledge: RAG retrieves relevant knowledge before augmenting LLMs, while Graph RAG and KG-RAG retrieve structured graph knowledge instead of only textual chunks.Graph-based retrieval preserves structured information and interrelations among knowledge items.
  • KGs as Reasoning Guidelines: KG-guided reasoning supports offline path selection, process-integrated guidance, and agent-based reasoning over structured knowledge.Agent systems can use KG executors, memory, observation, action, reflection, and self-correction.
  • KGs as Reasoning Guidelines: Joint KG–LLM reasoning enables multi-hop and iterative reasoning and can provide logical inference chains and factual evidence for explainable answers.The survey links these capabilities to multi-document, multi-modal, and multi-hop QA.
  • KGs as Refiners and Validators: KG refiners and validators filter, rerank, refine, and verify LLM outputs using structured factual knowledge.Examples include type-based candidate filtering, relevance ranking, medical-KG reranking, and factual-statement validation.
  • Hybrid Approach: Hybrid methods combine multiple KG roles to support question decomposition, conversational context understanding, multimodal interaction, and answer explainability.A remaining challenge is efficient vector indexing and search over large-scale KGs while balancing cost and performance.

4 Evaluation and Application

The survey summarizes evaluation metrics and benchmark datasets for LLM–KG QA, emphasizing answer, retrieval, and reasoning quality. It also showcases industrial and scientific applications and demonstrations.

  • The survey summarizes and compares metrics and benchmark datasets for evaluating LLM–KG QA.The comparison focuses on answer quality, retrieval quality, and reasoning quality.
  • The evaluation framework includes Answer Quality, Retrieval Quality, and Reasoning Quality.These are abbreviated as AnsQ, RetQ, and ReaQ.
  • The survey showcases industrial and scientific applications and demonstrations of synthesizing LLMs with KGs for QA.

5 Open Challenges and Opportunities

The survey identifies unresolved challenges for LLM+KG question answering in efficient retrieval, dynamic knowledge integration, large-scale reasoning, explainability, and fairness. It outlines opportunities for structure-aware retrieval, conflict-aware updates, and more interpretable and fair QA systems.

  • Open challenges: Future LLM+KG QA research must address efficient knowledge retrieval, dynamic integration, scalable reasoning, explainability, and fairness.These challenges are presented as remaining limitations of current synthesis approaches rather than as a completed solution.
  • Scaling to both Effectiveness and Efficiency: LLM+KG systems must retrieve factual knowledge and perform multi-hop reasoning under tight latency and memory budgets.Structure-aware retrieval is needed because vanilla dense or sparse retrieval discards topological cues useful for pruning the search space.
  • Knowledge Alignment and Dynamic Integration: Dynamic knowledge integration remains difficult because KG snapshots become outdated as entities and relations change and contradictions emerge.Open directions include quantifying structural alignment, facilitating real-time updates, and detecting and resolving conflicts across modalities and sources.
  • Explainable and Fairness-Aware QA: Explainable QA requires retrieving comprehensible subgraphs and guiding LLMs to generate explicit reasoning steps grounded in them.The survey also highlights fairness-aware retrieval and counterfactual prompting because both LLMs and KGs can contain biased or incomplete knowledge.

6 Conclusion

The survey synthesizes research on combining LLMs and KGs for QA through a taxonomy organized by QA types and KG roles. It highlights both the potential of KGs to address complex-QA weaknesses and unresolved challenges requiring adaptive, scalable, explainable, and fair systems.

  • Conclusion: The survey presents a taxonomy of LLM+KG QA methodologies based on QA types and the roles played by KGs.It uses this synthesis to examine current strengths, limitations, and remaining research directions.
  • Conclusion: LLM+KG approaches can help address hallucinations, limited reasoning capabilities, and knowledge conflicts in complex QA scenarios.The conclusion frames these benefits as potential opportunities rather than universally established outcomes.
  • Conclusion: Remaining challenges include efficient retrieval, dynamic knowledge integration, reasoning over knowledge at scale, and explainable and fairness-aware QA.The survey calls for adaptive frameworks that dynamically integrate up-to-date knowledge with LLMs while scaling reasoning, explainability, and fairness.

Limitations

The survey’s scope and coverage are constrained by the rapidly expanding literature, overlapping taxonomies, and heterogeneous evaluation settings. Its review emphasizes alignment between methodologies and complex QA challenges more than standardized quantitative comparison.

  • Scope: The survey may miss newly released works because research on synthesizing LLMs and KGs for QA is expanding rapidly.This limits the completeness of the surveyed literature.
  • Taxonomy: The two taxonomies can overlap because their categories are non-exclusive and describe methodologies from different perspectives.A single work may therefore fit multiple categories rather than one mutually exclusive class.
  • Evaluation: The survey underemphasizes quantitative and experimental comparison because implementations, benchmarks, and evaluation metrics vary substantially.The authors specifically attribute this limitation to diverse implementation details, benchmark datasets, and non-standardized metrics.
  • Relation to prior surveys: Earlier surveys covered broader knowledge-intensive tasks, narrower closed-domain KGQA, or user-centric integration of LLMs, KGs, and search engines.The present survey positions its scope around synthesizing LLMs and KGs for complex QA.
  • Search procedure: The review retrieves papers primarily from 2021 onward using Google Scholar and PaSa2, while extending benchmark-dataset searches to 2016.Papers are screened according to relevance and publication-venue quality.
  • Complex QA categories: Complex QA is organized into multi-document, multi-modal, multi-hop, conversational, explainable, and temporal categories according to technical challenges.These categories cover retrieval and synthesis across sources, multiple modalities, iterative reasoning, multi-turn context, explanations, and temporal conditions.

B.2 Approaches of Synthesizing LLMs and KGs for QA

The survey classifies LLM+KG QA methods by the role assigned to the KG and compares these approaches across techniques, models, graphs, datasets, QA types, and evaluation metrics. It also aligns the approaches with diverse complex-QA settings.

  • Role-based taxonomy: The methodology taxonomy categorizes LLM+KG QA approaches according to the role of KGs.The section introduces this role-based organization as a basis for surveying synthesis methods.
  • Survey comparison: Existing surveys are compared across coverage of LLMs, KGs, LLMs+KGs, GraphRAG, and QA, including multilingual and temporal KGQA.The comparison table marks whether topics are covered or discussed.
  • Approach categories: The taxonomy includes KGs as background knowledge, reasoning guidelines, refiners and validators, and hybrid approaches.These categories distinguish whether KGs supply information, guide reasoning, refine or validate outputs, or combine roles.
  • Method comparison: The survey summarizes strengths and limitations for each synthesis method using the role-based taxonomy.Detailed comparisons cover main techniques, language models, knowledge graphs, datasets, QA types, and evaluation metrics.
  • QA alignment: The approaches are aligned with diverse complex-QA tasks in a dedicated comparison table.This alignment connects synthesis methodologies with the QA settings they address.

D Extended Optimization

The survey reviews optimization techniques for reducing the cost of graph retrieval, graph reasoning, vector indexing, and LLM context processing, while noting that computational costs remain substantial.

  • Index-based Optimization: Index-based optimization accelerates embedding learning and vector retrieval through dynamic indexes, hierarchical index graphs, and optimized graph-structure indexing.Examples include PG-RAG, GoR, KG-Retriever, and NodeRAG.
  • Prompt-based Optimization: Prompt-based optimization improves fusion between KG-retrieved knowledge and LLM knowledge through historical-prompt guidance, relevance preflight checks, and multi-source evidence filtering.StraGo, BriefContext, and QUASAR represent these strategies.
  • Cost-based Optimization: Cost-based optimization reduces LLM calls and accelerates knowledge probing through sampling, constrained inference procedures, and clue-guided path exploration.GLens, Coke, and CGPE target efficiency while preserving knowledge access for unfamiliar questions.
  • Remaining Limitations: Despite optimization and ranking strategies, extracting relevant subgraphs, performing graph reasoning, and conducting vector retrieval remain computationally costly.The survey identifies these operations as continuing efficiency bottlenecks.

E Evaluations and Applications

The survey summarizes evaluation metrics, benchmark datasets, industrial and scientific applications, and demonstrations for synthesizing LLMs and KGs for QA.

  • Scope: The evaluation-and-application overview covers metrics, benchmark datasets, industrial and scientific applications, and demonstrations of LLM–KG synthesis for QA.These materials are summarized and compared in the following sections and tables.

E.1 Evaluations

The survey organizes evaluation around answer quality, retrieval quality, and reasoning quality, with metrics and benchmark evaluations tailored to these dimensions.

  • Metrics: Answer-quality metrics assess generated responses using BERTScore, answer relevance, hallucination, accuracy matching, and human-verified completeness.These metrics target the quality and factual properties of generated answers.
  • Metrics: Retrieval-quality metrics include context relevance, faithfulness score, precision, context recall, mean reciprocal rank, and normalized discounted cumulative gain.These measures evaluate the relevance and ranking of retrieved context in retrieval-augmented QA.
  • Metrics: Reasoning-quality metrics for multi-hop QA include Hop-Acc and reasoning accuracy.They evaluate the correctness of reasoning processes or chains in multi-hop tasks.
  • Benchmark Dataset: Benchmark datasets evaluate answer correctness with evidence, retrieved-context relevance, and the correctness of generated reasoning chains.The survey labels these dimensions Answer Quality, Retrieval Quality, and Reasoning Quality.

E.2 Applications

The survey presents applications and demonstrations that use LLM–KG synthesis for knowledge-intensive QA, explanation, retrieval, reasoning, and domain-specific assistance.

  • Applications: KAG uses KGs and vector retrieval to bidirectionally enhance LLMs for knowledge-intensive tasks such as QA.It is presented as a domain-knowledge augmented generation framework.
  • Applications: PIKE-RAG guides LLMs toward accurate responses by extracting, understanding, and applying domain-specific knowledge and rationales.The system is described as a specialized knowledge and rationale augmented generation system.
  • Applications: GraphRAG-QA combines NLP2Cypher, vector RAG, and graph-vector RAG query engines to augment QA.It is described as an industrial GraphRAG demonstration by NebulaGraph.
  • Applications: MedRAG uses KG-elicited reasoning within a healthcare copilot to generate diagnoses and treatment recommendations from patient manifestations.The application targets healthcare QA and assistance.
  • Applications: Fact Finder retrieves from medical KGs to improve the completeness and correctness of LLM answers.It is presented as a query-based medical-KG retrieval application.
  • Demonstrations: AprèsCoT visualizes LLM-generated answers through chain-of-thought path exploration and inference over KGs.The tool focuses on explaining generated answers.
Loading 2505.20099v2…