Source-linked AI summary
A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, Ji-Rong Wen
TL;DR
Complex KBQA must handle questions with difficult compositional and linguistic structures, while existing methods face challenges in parsing, retrieval, supervision, and generalization. This survey organizes prior work into semantic parsing-based and information retrieval-based approaches, analyzes their challenges and solutions, and identifies future directions. It emphasizes the need for systems that are more robust, interpretable, evolutionary after deployment, and able to use broader knowledge sources.
Problem
Complex KBQA involves challenging query structures, while existing approaches face difficulties with parsing coverage, retrieval scope, computational cost, weak supervision, and out-of-distribution questions.
Method
The survey categorizes complex KBQA methods into semantic parsing-based and information retrieval-based approaches, then analyzes their modules, challenges, solutions, and future directions.
Results
The survey synthesizes existing complex KBQA datasets, methods, challenges, solutions, and promising research directions across the two mainstream approaches.
Takeaways & Limitations
Future complex KBQA research should pursue continuous improvement after deployment, robust and interpretable models, and more general knowledge bases.
Takeaways & Limitations
Existing methods often perform well under i.i.d. benchmark assumptions but may fail on out-of-distribution cases, including few-shot and compositional-generalization settings.
Abstract
from arXiv · showhide
Knowledge base question answering (KBQA) aims to answer a question over a knowledge base (KB). Recently, a large number of studies focus on semantically or syntactically complicated questions. In this paper, we elaborately summarize the typical challenges and solutions for complex KBQA. We begin with introducing the background about the KBQA task. Next, we present the two mainstream categories of methods for complex KBQA, namely semantic parsing-based (SP-based) methods and information retrieval-based (IR-based) methods. We then review the advanced methods comprehensively from the perspective of the two categories. Specifically, we explicate their solutions to the typical challenges. Finally, we conclude and discuss some promising directions for future research.
1 Introduction
Complex KBQA extends fact-based question answering to questions requiring multi-hop reasoning, constrained relations, numerical operations, or combinations of these. This survey organizes existing solutions into semantic parsing-based and information retrieval-based approaches, identifies their challenges, and discusses future directions.
- Task context: A KB stores facts as subject–relation–object triples, and KBQA answers natural-language questions using those facts as its knowledge source.Early KBQA focused on simple questions answerable from a single fact.
- Motivation: Complex KBQA questions can involve multiple subjects, compound relations, numerical operations, or combinations of multi-hop reasoning and constraints.The example composes nominee and spouse relations, applies a TV-producer type constraint, and selects candidates by earliest marriage date.
- Approaches: SP-based methods execute parsed symbolic logic forms, whereas IR-based methods construct question-specific graphs and rank entities by relevance.Both approaches first identify and link the topic entity before deriving answers in its neighborhood.
- Challenges: Complex questions challenge parsers through diverse query structures and challenge retrieval methods through limited entity scope and untraceable reasoning.Additional relations and subjects enlarge the logic-form search space and can prevent retrieval of all relevant entities.
- Challenges: Question understanding and weak supervision remain shared difficulties because complex language requires strong understanding and ground-truth paths are expensive to annotate.Datasets generally provide question–answer pairs rather than correct logic forms or reasoning paths.
- Survey scope: The survey differs from earlier work by analyzing challenges and solutions module by module across the two approaches and outlining promising research directions.Earlier surveys addressed simple KBQA or provided broader technique-focused views of complex KBQA.
2 Background
The background section defines complex KBQA over a fact-based knowledge base, describes how benchmark datasets are constructed, and introduces standard evaluation metrics. Complex datasets require reasoning over multiple facts and may add numerical operations or constraints.
- Task: Complex KBQA takes a KB of connected subject–relation–object facts and seeks answers from the KB entity set.Unlike simple KBQA, answers are not necessarily directly connected to the topic entity.
- Datasets: Complex KBQA datasets typically generate simple questions first, then compose them into complex questions using KB neighborhoods and predefined templates.Some datasets additionally use crowd workers to rewrite questions in natural language, as indicated by the LF and NL table fields.
- Datasets: These datasets require multiple KB facts and may include counting, comparative or superlative ranking, entity constraints, and temporal constraints.Such operations and constraints increase the difficulty of reasoning from KBs.
- Evaluation: Evaluation commonly uses precision, recall, F1, or Hits@1 to assess predicted answer entities.Precision and recall measure complementary answer-set properties, F1 combines them, and Hits@1 measures whether correct answers rank first.
3 Two Mainstream Approaches
Complex KBQA methods follow either a parse-then-execute paradigm or a retrieval-and-rank paradigm. SP-based systems offer interpretable symbolic reasoning but depend on expressive parsing, while IR-based systems are easier to train yet less interpretable.
- Overall paradigms: SP-based and IR-based systems correspond to parse-then-execute and retrieval-and-rank paradigms, respectively.These paradigms are presented as the two mainstream approaches for complex KBQA.
- Semantic Parsing-based Methods: SP-based methods encode questions, parse them into uninstantiated logic forms, ground and validate them against the KB, then execute them to produce answers.Some systems combine logical parsing and KB grounding while the logic form is partially parsed.
- Information Retrieval-based Methods: IR-based methods extract a question-specific graph from the KB and rank its entities according to their relevance to the question.The top-ranked entities are predicted as answers.
- Pros and Cons: SP-based methods provide more interpretable reasoning through expressive logic forms but rely heavily on logic-form and parsing design.This reliance is described as a bottleneck for performance improvement.
- Pros and Cons: IR-based methods fit end-to-end training and are easier to train, but their black-box reasoning makes intermediate steps less interpretable.They conduct reasoning over graph structure while performing semantic matching.
4 Challenges and Solutions
Complex KBQA challenges affect both semantic parsing and information retrieval pipelines, from understanding and representing compositional queries to grounding, retrieval, reasoning, and supervision. Proposed solutions expand query representations, reduce search and retrieval failures, supplement incomplete KBs, and improve interpretability and learning signals.
- Semantic Parsing-based Methods: SP-based methods follow question understanding, logical parsing, KB grounding, and execution, each facing distinct difficulties on complex questions.Complexity affects both semantic and syntactic understanding and the coverage of query types.
- Semantic Parsing-based Methods: Complex compositional semantics and long-distance dependencies make syntactic analysis unreliable, motivating skeleton-based parsing and structural ranking of candidate logic forms.Skeleton-based parsing extracts a simple-question trunk with branches for expansion, while other methods exploit tree or graph structure.
- Semantic Parsing-based Methods: Expressive query graphs better match KB schemas and support complex queries, but manually predefined generation rules limit scalability and coverage of long-tail types.Earlier templates covered 1-hop, 2-hop, and single-constraint relations but suffered from limited coverage; later work uses frequent substructures and syntactic annotation.
- Semantic Parsing-based Methods: Grounding becomes computationally expensive because KB entities may connect to hundreds or thousands of relations, making exhaustive logic-form exploration unaffordable.The grounding module must instantiate executable logic forms while managing both computational resources and time complexity.
- Semantic Parsing-based Methods: Weak supervision leaves SP-based methods with sparse end-of-execution rewards, so reward shaping supplies additional feedback during parsing evaluation.Strategies compare answer types or generated forms with high-reward logic forms and can provide intermediate rewards.
- Information Retrieval-based Methods: IR-based methods construct a question-specific graph, represent the question, reason over the graph, and rank candidate answers, but complex queries strain every module.The retrieval graph is intended to cover relevant entities and relations, while later modules address semantic composition, reasoning, and ranking.
- Information Retrieval-based Methods: Incomplete KBs can omit correct multi-hop paths, so systems supplement question-specific subgraphs with textual evidence or fuse external information into the graph.This incompleteness is more severe for complex questions than for simple 1-hop queries.
- Information Retrieval-based Methods: Static vector instructions inadequately capture compositional semantics, encouraging dynamic updates during graph reasoning.Dynamic updates adapt reasoning instructions to the current state of the reasoning process.
5 Conclusion and Future Directions
The survey organizes complex KBQA challenges and solutions across SP-based and IR-based methods, then identifies open problems and future directions. It highlights evolutionary systems, robust and interpretable models, and broader knowledge bases as promising research avenues.
- Conclusion: The survey synthesizes datasets, SP-based methods, IR-based methods, and their challenges and solutions, while noting that many issues remain open and under-explored.It also acknowledges approaches outside the two main categories, such as rule-based decomposition into simple questions.
- Evolutionary KBQA: Future evolutionary KBQA systems should use deployment-time user feedback for continuous improvement and better handling of newly emerging knowledge and questions.Prior work used feedback to rectify answers and directly involved users in question parsing.
- Robust and Interpretable Models: Robust and interpretable KBQA remains challenging because models performing well under i.i.d. benchmarks may fail on out-of-distribution, few-shot, and compositional-generalization settings.GrailQA and CFQ are cited as datasets supporting research on out-of-distribution questions and explainable reasoning.
- More General Knowledge Base: A broader KB notion could extend complex KBQA by combining structured KBs with text, images, human interactions, and implicit relational knowledge in pretrained language models.The survey connects this direction to incomplete-KB compensation and related tasks such as visual question answering and commonsense reasoning.