PaperlayerView exact revision on arXiv

Source-linked AI summary

Complex Knowledge Base Question Answering: A Survey

Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2108.06688v5cs.CL

TL;DR

Complex KBQA addresses the gap between successful simple-question answering and unsatisfactory performance on questions requiring multiple subjects, compound relations, or numerical operations. The survey organizes recent work into semantic parsing-based and information retrieval-based methods, reviews their challenges and solutions, and identifies low-resource robustness and interaction evaluation as open directions.

  • Problem

    Simple-question KBQA methods have not adequately solved complex questions involving multi-hop reasoning, constrained relations, or numerical operations.

  • Method

    The survey synthesizes complex-KBQA research through SP-based and IR-based method categories, analyzing their procedures, challenges, and advanced solutions.

  • Results

    The survey finds no absolute agreement that either SP-based or IR-based methods are better, while outstanding methods in both categories typically use advanced techniques.

  • Takeaways & Limitations

    Complex KBQA research is organized around complementary symbolic parsing and retrieval-based approaches, with future work needed on robust low-resource systems and richer evaluation.

  • Takeaways & Limitations

    Current KBQA datasets are mostly template-generated, lack diversity, and are limited in scale by expensive labeling.

Abstract

from arXiv · show

Knowledge base question answering (KBQA) aims to answer a question over a knowledge base (KB). Early studies mainly focused on answering simple questions over KBs and achieved great success. However, their performance on complex questions is still far from satisfactory. Therefore, in recent years, researchers propose a large number of novel methods, which looked into the challenges of answering complex questions. In this survey, we review recent advances on KBQA with the focus on solving complex questions, which usually contain multiple subjects, express compound relations, or involve numerical operations. In detail, we begin with introducing the complex KBQA task and relevant background. Then, we describe benchmark datasets for complex KBQA task and introduce the construction process of these datasets. Next, we present two mainstream categories of methods for complex KBQA, namely semantic parsing-based (SP-based) methods and information retrieval-based (IR-based) methods. Specifically, we illustrate their procedures with flow designs and discuss their major differences and similarities. After that, we summarize the challenges that these two categories of methods encounter when answering complex questions, and explicate advanced solutions and techniques used in existing work. Finally, we conclude and discuss several promising directions related to complex KBQA for future research.

1 INTRODUCTION

Complex KBQA extends simple fact retrieval to questions requiring multi-hop reasoning, constrained relations, or numerical operations. This survey organizes the field around SP-based and IR-based methods, their challenges, and corresponding solutions.

  • Complex KBQA questions may involve multiple subjects, compound relations, or numerical operations requiring multi-hop, constrained, or aggregate reasoning.The example combines nominee and spouse relations, a type constraint, and selection by earliest marriage date.
  • SP-based methods parse questions into symbolic logic forms and execute them against the KB, whereas IR-based methods construct question-specific graphs and generate answers from retrieved information.Both approaches first identify and link the question’s topic entity.
  • Existing SP-based parsers struggle to cover diverse complex queries, while IR-based answers may lack traceable reasoning.The two approaches therefore face different limitations in representing and explaining complex reasoning.
  • Additional relations and subjects enlarge the parsing search space and make IR retrieval less complete, intensifying computational cost and incomplete-KB problems.These pressures complicate both logic-form search and retrieval of all facts needed for reasoning.
  • Complex questions require stronger language understanding because small syntactic changes can alter which phrase an ordinal constraint modifies.The survey illustrates this with the ordinal term “first” constraining either “wife” or “TV producer.”
  • The survey reviews recent complex-KBQA advances, compares the two mainstream method categories, and organizes their challenges and solutions.It also adds recent papers and refines the discussion of challenges and solutions relative to earlier surveys.

2 PRELIMINARY

This section defines knowledge bases and complex KBQA, then outlines the traditional pipeline and its semantic parsing-based and information retrieval-based approaches.

  • 2.1 Knowledge Base: Freebase uses compound value types to represent n-ary facts involving entities and values such as marriage dates.The example encodes Jeff Probst’s marriage to Shelley Wright in 1996 as one relational statement.
  • 2.1 Knowledge Base: Large-scale open KBs commonly use RDF and SPARQL to support structured access, retrieval, and data manipulation.The section gives an executable SPARQL example for retrieving Jeff Probst’s spouses.
  • 2.1 Knowledge Base: A knowledge base is modeled as triples linking entities through relations, with KBs differing in purpose and schema design.The formal representation is G = {⟨e, r, e′⟩ | e, e′ ∈ E, r ∈ R}.
  • 2.2 Task Formulation: Complex KBQA maps a tokenized natural-language question to predicted answers derived from KB entities or their surface names.Unlike simple KBQA, complex answers may be multiple hops from topic entities or involve more complex answer structures.
  • 2.3 Traditional Approaches: Traditional KBQA first identifies the question’s topic entity through recognition, disambiguation, and linking before predicting answers.Entity linking tools such as S-MART, DBpedia Spotlight, and AIDA are cited for this preliminary step.
  • 2.3 Traditional Approaches: Semantic parsing-based systems construct executable logic forms, while information retrieval-based systems retrieve and rank candidates in a question-specific KB subgraph.Early semantic parsing methods often relied on rule-based mappings, whereas IR-based methods represent questions and candidates as dense vectors for ranking.

3 TWO MAINSTREAM APPROACHES

Complex KBQA is organized around two mainstream paradigms: SP-based methods parse questions into executable logic forms, while IR-based methods retrieve question-specific graphs and reason over them. The survey compares their modules, shared components, limitations, and their convergence toward neural-symbolic reasoning.

  • 3.1 Semantic Parsing-based Methods: SP-based methods parse questions into logic forms, ground them to KB entities and relations, and execute the resulting forms to produce answers.Their pipeline includes question understanding, logical parsing, KB grounding, and KB execution.
  • 3.2 Information Retrieval-based Methods: IR-based methods retrieve a question-specific graph, encode reasoning instructions, propagate information through graph reasoning, and generate entity or text answers.Answer generation either ranks graph entities or produces free text from a vocabulary.
  • 3.3 A Comparison of Core Modules: SP-based systems emphasize interpretable logic forms but face weak supervision and costly expert annotation, whereas IR-based systems depend partly on retrieval recall and graph reasoning quality.The approaches differ in their core mechanisms while sharing dependencies on topic entities and parameter-free KB-related modules.
  • 3.4 A Unified Paradigm - Neural Symbolic Reasoning: Neural-symbolic reasoning combines symbolic search and inference over structured data with neural modeling of diverse language and heterogeneous or incomplete KB information.The survey describes symbolic systems as supporting interpretability and compositional generalization, while neural systems support flexible representation and latent reasoning.

4 SEMANTIC PARSING-BASED METHODS

SP-based methods parse complex questions into executable logic forms, but must address diverse query structures, large search spaces, and costly KB grounding. Existing solutions use structural question analysis, expressive parsing targets, decomposition, and iterative query expansion.

  • SP-based systems use question understanding, logical parsing, KB grounding, and KB execution, with complex questions increasing difficulty across these modules.
  • 4.1 Overview: Structural analysis of syntax and semantics improves alignment between question constituents and logic-form elements, including entities, relations, types, and attributes.
  • 4.3 Parsing Complex Queries: Query graphs extend fixed templates by representing entities, variables, and functions while attaching constraints and aggregations to core inference chains.
  • Complex-question grounding is computationally expensive because entities may connect to hundreds or thousands of relations, making exhaustive logic-form exploration impractical.
  • Decompose-execute-join methods split complex questions into simple sub-questions, whereas expand-and-rank methods iteratively extend and rank candidate query graphs.

5 INFORMATION RETRIEVAL-BASED METHODS

IR-based methods retrieve a question-specific graph and reason over it, but complex questions expose missing reasoning paths and noisy graph context. Research therefore supplements retrieval sources and improves graph construction and reasoning relevance.

  • IR-based methods typically construct a question-specific graph, represent the question, perform graph-based reasoning, and generate an answer.
  • Complex questions make incomplete knowledge bases more problematic because correct multi-hop reasoning paths may be absent from the retrieved graph.
  • Researchers supplement incomplete sources with retrieved sentences, textual information fused into entity representations, or pre-trained knowledge-base embeddings.

Supplementing incomplete graph with pre-trained KB

Pre-trained KB embeddings address incomplete knowledge sources by adding learned entity and relation information to IR-based question answering. This approach scores candidate triples after mapping question representations into the embedding space.

  • Pre-trained knowledge-base embeddings are used to supplement incomplete graphs and alleviate KB sparsity through missing-link prediction.
  • Apoorv et al. pre-trained entity and relation embeddings with ComplEX and predicted answers using a triple-scoring function over the topic entity, question, and answer entity.
  • Question representations are mapped to the embedding space so that natural-language questions can fit the original ComplEX scoring function.

5.3 Understanding Complex Semantics

IR-based systems must represent compositional semantics, knowledge-intensive expressions, named entities, and numerical constraints. Solutions dynamically update reasoning instructions, incorporate graph and textual knowledge, and encode numerical properties.

  • Static question vectors poorly represent compositional semantics, so IR-based methods use step-aware and dynamically attended instructions during graph reasoning.
  • Reasoning instructions can be updated with predicted intermediate relations or retrieved KB facts through explicit paths and key-value memory networks.
  • Graph neural networks jointly update entity representations and reasoning instructions through iterative reasoning over heterogeneous information sources.
  • Named entities and other knowledge-intensive phrases motivate injecting KB-derived representations and external textual knowledge into question understanding.
  • Copying, retrieval, and structured-memory mechanisms help generate named entities, while numerical reasoning methods encode magnitude and ordinal properties in entity representations.

5.4 Uninterpretable Reasoning

Complex KBQA requires interpretable multi-step reasoning because black-box reasoning obscures how answers are derived. Existing approaches expose this process through relational paths or intermediate entities.

  • 5.4 Uninterpretable Reasoning: Multi-hop KBQA reasoning is difficult to interpret when neural modules operate as black boxes, motivating traceable intermediate predictions.Traceability is intended to make reasoning more interpretable and support user interaction for improvement.
  • 5.4 Uninterpretable Reasoning: Relational-path methods represent reasoning as sequences of relations and generate distributions over candidate relations at each step.Reasoning status can be represented with vectors matched against relation candidates during each hop.
  • 5.4 Uninterpretable Reasoning: Intermediate-entity methods predict question-relevant entities at successive steps to provide a traceable account of multi-hop reasoning.A key-value memory network uses instruction matching to derive reasoning status and predict intermediate entities.

5.5 Training under Weak Supervision Signals

Weak supervision makes complex KBQA training difficult because systems receive answers but lack annotations for intermediate reasoning steps. Proposed solutions add intermediate feedback, inferred supervision, or joint learning for entity linking.

  • 5.5 Training under Weak Supervision Signals: Answer-only supervision provides no guidance for intermediate reasoning, making both SP-based and IR-based approaches vulnerable to training difficulties.For IR-based methods, the absence of intermediate feedback can produce spurious reasoning and misleading rewards.
  • 5.5 Training under Weak Supervision Signals: Reward shaping supplies intermediate feedback by evaluating the semantic similarity between a question and the relation path selected during reasoning.The approach frames multi-hop reasoning as graph-path expansion and uses attention over question parts during triple selection.
  • 5.5 Training under Weak Supervision Signals: Pseudo intermediate supervision can be learned through bidirectional reasoning, using inferred entity distributions to augment training signals.The method is motivated by bidirectional search over the knowledge-base graph.
  • 5.5 Training under Weak Supervision Signals: Variational multi-task learning can train entity linking jointly without annotations, addressing topic-entity errors that otherwise propagate through the pipeline.Most existing systems rely on off-the-shelf tools to locate topic entities, leaving this step weakly supervised.

6 PLM APPLICATIONS ON COMPLEX KBQA

Pre-trained language models are applied across complex KBQA to generate executable logic forms, improve low-resource training, construct cleaner question-specific sources, and support unified graph reasoning.

  • 6 PLM APPLICATIONS ON COMPLEX KBQA: PLMs support complex KBQA by jointly improving question understanding, logical parsing, KB grounding, source construction, and reasoning.Their transferable knowledge is especially relevant when labeled data are sparse.
  • 6 PLM APPLICATIONS ON COMPLEX KBQA: Seq2seq PLM frameworks generate executable programs such as SPARQL from questions and related KB information.Knowledge-enhanced text generation formalizes the mapping from user requests to executable programs.
  • 6 PLM APPLICATIONS ON COMPLEX KBQA: A BART-based generator on KQA Pro achieved near-human performance and robustness to sparse training data without incorporating external knowledge.The result illustrates PLM usefulness for low-resource logic-form and program generation.
  • 6 PLM APPLICATIONS ON COMPLEX KBQA: PLMs can retrieve hop-wise question-related relations more precisely than noisy heuristic graph construction by ranking candidates conditioned on the question and prior selections.The retriever constructs a question-specific source incrementally across reasoning hops.
  • 6 PLM APPLICATIONS ON COMPLEX KBQA: PLM relational knowledge may complement incomplete KBs, while PLM similarity scores can help filter noisy graph context during reasoning.These uses extend PLMs beyond generation to knowledge sourcing and graph-context selection.

7 EVALUATION AND RESOURCE

The survey organizes KBQA evaluation around reliability, robustness, and system-user interaction, and reviews complex-question datasets, construction procedures, and leaderboard practices. Its analysis finds no universally superior method category and highlights technique choices associated with strong performance.

  • 7.1 Evaluation Protocol: KBQA evaluation should cover reliability, robustness, and system-user interaction rather than relying on a single measurement aspect.The protocol distinguishes correctness, out-of-distribution generalization, and user-facing behavior.
  • 7.1 Evaluation Protocol: Precision, Recall, F1, and Hits@1 measure answer-set correctness, with Hits@1 assessing whether the correct answer ranks first.The survey presents these as classical reliability metrics for KBQA systems.
  • 7.1 Evaluation Protocol: Current datasets have limited robustness coverage because template generation, costly labeling, and combinatorial query diversity restrict training distributions.The survey distinguishes i.i.d., compositional, and zero-shot generalization as robustness levels.
  • 7.1 Evaluation Protocol: Quantitative measures for system-user interaction remain lacking, so human evaluation is presented as an efficient and comprehensive alternative.Relevant behaviors include clarification questions and responses to user-reported errors.
  • 7.2 Datasets And Leaderboard: Complex KBQA datasets are typically generated from topic entities, simple-question templates, neighborhood facts, composition templates, and rule-based answer extraction.Some construction pipelines also use crowd workers to rewrite questions in natural language.
  • 7.2 Datasets And Leaderboard: Leaderboard comparisons standardize source selection and experimental setups, while excluding or narrowing certain datasets for comparability and presentation.The survey uses official leaderboards when available and published results otherwise, with stated exceptions for QALD, MetaQA, and LC-QuAD 2.0.
  • 7.2 Datasets And Leaderboard: Neither SP-based nor IR-based methods is consistently superior; SP-based methods cover broader question types, whereas IR-based methods emphasize multi-hop benchmarks.Strong systems in both categories use advanced encoders and structured multi-step reasoning techniques.

8 RECENT TRENDS

The survey identifies evolutionary, robust, conversational, neural-symbolic, and more general knowledge-base systems as promising directions for complex KBQA. These directions address deployment feedback, limited or shifted data, interaction, reasoning capabilities, and KB incompleteness.

  • Evolutionary KBQA systems: Evolutionary KBQA systems should learn from deployment failures, unseen question templates, user feedback, and the changing world knowledge base.The survey frames such systems as potentially practical for industrial needs, while noting the need for further user-interaction designs.
  • Robust KBQA systems: Robust KBQA research must address low-resource training and out-of-distribution generalization because existing methods often assume sufficient, identically distributed data.The survey highlights meta-learning, knowledge transfer, pretrained language models, and evaluation across multiple levels of generalization as relevant techniques.
  • Conversational KBQA systems: Conversational KBQA is needed because users commonly ask follow-up questions, creating ambiguity and coreference challenges for deployed systems.Initial work has begun addressing these conversational difficulties in factoid question answering applications.
  • Neural symbolic KBQA systems: Neural-symbolic reasoning could combine SP-based interpretability with IR-based neural reasoning power while adding explicit numerical and logical operations.The survey presents symbolic modules coupled with neural reasoning modules as a way to unify the two mainstream paradigms.
  • More general knowledge bases: More general knowledge bases can supplement missing KB information with text, images, and human interactions to improve complex question answering.The survey emphasizes text as especially attractive because it is semantically rich and comparatively easy to collect.

9 CONCLUSION

The survey organizes complex KBQA research around challenges and solutions for semantic parsing-based and information retrieval-based methods. It concludes that complex KBQA remains an active area with open challenges and broad applications.

  • 9 CONCLUSION: The survey synthesizes typical challenges, solutions, and advanced techniques for the two dominant complex KBQA categories: SP-based and IR-based methods.It also notes that some approaches instead focus on question decomposition rather than KB reasoning or logic-form generation.
  • 9 CONCLUSION: Complex KBQA remains promising for natural language understanding, compositional generalization, and multi-hop reasoning, while many research challenges remain under-explored.The conclusion presents these applications and open problems as directions for continued research.
Loading 2108.06688v5…