PaperlayerView exact revision on arXiv

Source-linked AI summary

KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base

Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, Hanwang Zhang

arXiv:2007.03875v4cs.CL

TL;DR

Complex KBQA benchmarks lack explicit reasoning processes and often have limited diversity or scale. KQA Pro addresses these gaps with roughly 120K diverse questions annotated with compositional KoPL programs and SPARQL queries, and evaluation shows current KBQA models struggle substantially on the dataset. The paper also identifies omitted Wikidata types as a scope limitation.

  • Problem

    Existing Complex KBQA benchmarks often provide QA pairs without explicit reasoning processes and have insufficient question diversity or scale.

  • Method

    KQA Pro combines approximately 120K diverse natural-language questions with compositional KoPL programs and corresponding SPARQL queries.

  • Results

    Existing KBQA models show significant performance drops on KQA Pro compared with established benchmarks, revealing weaknesses in comparison reasoning and unseen-answer questions.

  • Takeaways & Limitations

    KQA Pro supports both KBQA and semantic parsing evaluation while serving as a diagnostic dataset for multiple reasoning skills.

  • Takeaways & Limitations

    The KB design omits geographical and time types that Wikidata also contains, leaving them for future work.

Abstract

from arXiv · show

Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are poor in diversity or scale. To this end, we introduce KQA Pro, a dataset for Complex KBQA including ~120K diverse natural language questions. We introduce a compositional and interpretable programming language KoPL to represent the reasoning process of complex questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro serves for both KBQA and semantic parsing tasks. Experimental results show that SOTA KBQA methods cannot achieve promising results on KQA Pro as on current datasets, which suggests that KQA Pro is challenging and Complex KBQA requires further research efforts. We also treat KQA Pro as a diagnostic dataset for testing multiple reasoning skills, conduct a thorough evaluation of existing models and discuss further directions for Complex KBQA. Our codes and datasets can be obtained from https://github.com/shijx12/KQAPro_Baselines.

1 Introduction

KQA Pro addresses limitations in Complex KBQA benchmarks by providing diverse questions with explicit compositional reasoning programs and paired SPARQL queries. Evaluation shows existing KBQA models perform substantially worse on KQA Pro than on established benchmarks.

  • Motivation: Existing Complex KBQA benchmarks often omit explicit reasoning processes, forcing models to learn compositional reasoning through large search spaces or weak supervision.One cited result reports 96.52% F1 on simple CSQA questions but only 0.33% on complex comparative-count questions.
  • Motivation: Prior datasets also have limited diversity or scale, including template-generated questions, omitted literal attributes, or fewer than 40K questions.MetaQA uses 36 templates, while LC-QuAD2.0 and ComplexWebQuestions contain fewer than 40K questions.
  • KQA Pro: KQA Pro introduces KoPL to explicitly represent complex-question reasoning and provides a corresponding SPARQL query for every question.KoPL composes symbolic functions that expose intermediate operations, while SPARQL represents the query graph.
  • KQA Pro: The dataset is generated through canonical question-program-query triples, structural and linguistic diversification, and crowdsourced paraphrasing, yielding 117,970 questions.Structural variety comes from varied templates, random sampling, and recursive composition; linguistic variety is filtered using edit distance.
  • Evaluation: Existing state-of-the-art KBQA models show significant performance drops on KQA Pro compared with existing benchmarks, confirming the challenge of Complex KBQA.The dataset also exposes weaknesses in comparison reasoning and questions whose answers are absent from training data.

2 Related Work

Prior Complex KBQA datasets and methods trade off scale, knowledge coverage, natural-language diversity, and explicit reasoning annotations. KQA Pro combines KoPL and SPARQL annotations in a large, diverse semantic-parsing benchmark.

  • Existing Methods: Complex KBQA methods either parse questions into symbolic logic forms or retrieve and rank entities in question-specific KB graphs.Semantic parsing offers explicit intermediate reasoning, whereas retrieval-based methods construct question-specific graphs from the KB.
  • Existing Methods: Semantic parsing is interpretable but difficult to train because manually annotated logic forms are expensive and question-answer supervision is weak.The lack of logic-form annotations is identified as a major bottleneck for semantic parsing.
  • Existing Datasets: Existing datasets variously omit literal knowledge, lack logic-form annotations, rely on templates, or provide only small-scale question-to-SPARQL resources.SPARQL annotations help, but SPARQL describes query graphs less explicitly than intermediate solution procedures.
  • KQA Pro: KQA Pro introduces KoPL as a multi-step logic form and provides approximately 120K diverse natural-language questions with both KoPL and SPARQL annotations.The paper presents it as the largest NLQ-to-SPARQL dataset known to the authors.

3 Background

The paper models knowledge bases as collections of entities, concepts, relations, attributes, and qualifiers, then represents Complex KBQA reasoning with compositional KoPL programs. KoPL functions operate over textual and intermediate functional inputs, including relational and literal knowledge.

  • KB Definition: The KB representation distinguishes entities, concepts, relations, literal attributes, relational knowledge, literal knowledge, and qualifier knowledge.Attributes contain typed values such as strings, numbers, dates, or years; qualifiers restrict or contextualize relational or literal triples.
  • KoPL Design: KoPL represents complex-question procedures as compositional programs whose intermediate steps form a binary tree or post-order function sequence.Each function receives textual arguments and functional arguments produced by earlier operations.
  • KoPL Design: The Relate function takes a relation, direction, and entity, returning entities connected by that relation in the specified direction.For example, applying Relate to LeBron James Jr. with the forward father relation returns LeBron James.
  • KoPL Design: KoPL defines 27 functions covering KB-item manipulation, reasoning skills such as intersection and union, and multiple question types.Composing these finite functions yields programs for complex questions.
  • KoPL Design: The dataset explicitly models qualifiers because they can disambiguate or restrict a fact's validity, a capability the paper describes as absent from prior Complex KBQA models and datasets.The paper presents this as a first explicit treatment of qualifiers in Complex KBQA.

4 KQA Pro Construction

KQA Pro is constructed from a focused knowledge base, compositional question-generation strategies, executable annotations, and crowdsourced paraphrases, yielding a large and diverse benchmark.

  • Question generation strategies: Canonical questions, KoPL programs, and SPARQL queries are generated together, then paraphrased by workers and filtered for semantic consistency, fluency, and linguistic diversity.Paraphrases are rejected when workers identify meaning changes, average fluency is below 3, or edit distance is too small.
  • Knowledge base extraction: The knowledge base uses FB15k-237 entities aligned with Wikidata and adds 3,000 same-name entities to increase disambiguation difficulty.The aligned subset provides rich literal and qualifier knowledge while avoiding the scale of a full knowledge base.
  • Question generation strategies: Questions are generated by composing locating and asking stages, using sampled restrictions, quantitative comparisons, qualifiers, recursive multi-hop descriptions, and selection operations.The strategy design aims to cover possible queries through random sampling and recursive composition.
  • Dataset construction: 117,970 instances combine questions, SPARQL queries, KoPL programs, ten answer choices, and golden answers, with only uniquely answerable questions retained.Choices are generated by dropping one clause from the complete SPARQL query.
  • Dataset analysis: Multi-hop questions comprise 73.7% of KQA Pro, including 4.7% requiring at least five hops, while average question, program, and SPARQL lengths are 14.95, 4.79, and 35.52.The dataset contains 24,724 unique answers and longer questions on average than other Complex KBQA datasets.

5 Experiments

The experiments evaluate representative KBQA models and supervised KoPL/SPARQL parsers on KQA Pro, finding substantial difficulty and weaknesses in reasoning and generalization.

  • 5.2 Difficulty of KQA Pro: Models perform well on MetaQA and WebQSP but substantially worse on KQA Pro, whose relational, attribute, qualifier, linguistic, and reasoning diversity makes it more challenging.KQA Pro additionally covers verification, counting, intersection, and union, beyond the primarily relational focus of the comparison benchmarks.
  • 5.3 Analyses on Reasoning Skills: KoPL and SPARQL annotations provide intermediate supervision: the RNN and BART parsers outperform current models across the evaluated reasoning skills.The authors attribute this improvement to supervision from explicit intermediate programs and queries.
  • 5.3 Analyses on Reasoning Skills: Current models perform poorly on comparison and zero-shot questions, reflecting limited handling of literal and qualifier knowledge and weak generalization to unseen answers.The analysis links comparison weakness to models’ focus on relational knowledge and reports near-zero performance for zero-shot questions.
  • 5.4 Compositional Generalization: BART KoPL accuracy drops from 90.55% to 77.86% when longer programs are held out for testing, exposing a compositional-generalization gap.The productivity experiment trains on short programs and evaluates on longer programs or greater compositional depths.
  • 5.5 Case Study: A case study shows SPARQL making a semantic error while KoPL produces a semantically correct alternative, making KoPL predictions easier to inspect and modify.The authors describe KoPL as capable of learning multiple solutions and exposing human-readable intermediate reasoning.

6 Conclusion and Future Work

The paper concludes that KQA Pro supports both Complex KBQA and semantic parsing through explicit KoPL and SPARQL annotations, while KoPL offers interpretable reasoning traces.

  • 6 Conclusion and Future Work: KQA Pro provides each question with a KoPL program and SPARQL query, supporting both KBQA and semantic parsing research.The dataset is presented as a large-scale resource with explicit compositional programs.
  • 6 Conclusion and Future Work: KoPL improves interpretability by exposing reasoning steps and intermediate-function outputs that help humans locate parser errors.The authors also identify incremental training from human corrections as future work.

A Function Library of KoPL

KoPL defines a function-based program space for compositional KB reasoning, with typed arguments, relation directions, comparison operators, and grammar-based generation strategies.

  • A Function Library of KoPL: KoPL uses functions with fixed arguments to represent complex question answering as multi-step programs that can be represented as binary trees.Its supported program space is defined by a synchronous context-free grammar paired with canonical questions.
  • A Function Library of KoPL: The function library distinguishes entities, names, keys, values, comparison operators, relation directions, facts, and qualifier fields as program elements.Comparison operators vary by operation, including equality or inequality, greater or less, and largest or smallest.
  • A Function Library of KoPL: The underlying knowledge base combines FB15k-237 entities with Wikidata literal and qualifier knowledge, adds same-name entities, merges 363 relations, and filters unused content.These construction choices increase knowledge coverage and entity-disambiguation difficulty.
  • A Function Library of KoPL: KoPL’s generation strategy separates locating entity sets from asking for information about target entities or sets, using seven locating and nine asking strategies.The locating stage applies restrictions, while the asking stage queries specific information.

E SPARQL Implementation Details

The SPARQL implementation executes generated queries in Virtuoso and represents qualifiers and unit-bearing numeric attributes with virtual graph nodes.

  • E SPARQL Implementation Details: Generated SPARQL queries are executed using a Virtuoso 8 engine.
  • E SPARQL Implementation Details: Qualifier facts are represented by virtual nodes connected to subject, relation, and object through special edges, with qualifier values added as linked facts.The same virtual-node approach represents numeric attribute values that carry an extra unit.

F Generation Examples

The generation example illustrates how KQA Pro represents reasoning with textual descriptions, SPARQL, and KoPL, while accompanying figures summarize answer and program-length distributions.

  • The Cleveland Cavaliers example maps each generation stage to a textual description, SPARQL fragment, and KoPL function.The example proceeds from asking for an entity to locating it through concept and relational information.
  • Figure 5 reports the 20 most frequent answers in KQA Pro, with “yes” as the most frequent answer.About half of Verify questions have “yes” as their answer.
  • Figure 6 presents the distribution of program lengths in KQA Pro.

G Data Analysis

KQA Pro contains varied answer types, program complexities, and question forms, including common verification and counting questions alongside long multi-step programs.

  • “yes” and “no” dominate answers because they cover Verify questions, while Count questions account for 11.5% of the dataset.KQA Pro contains 24,724 unique answers; quantity answers include “0”, “1”, “2”, and “3”.
  • 28.42% of problems can be solved in 4 functional steps, while 1.24% require more than 10 steps.
  • Questions commonly begin with “what”, “which”, “how many”, “when”, “is”, or “does”, and frequently concern people, movies, countries, and universities.

H Baseline Implementation Details

The baseline implementations adapt memory, recurrent, embedding, and graph models to KQA Pro, while KoPL and SPARQL parsers generate structured reasoning representations.

  • KVMemNet retrieves 1,000 question-linked key-value pairs and iteratively updates a query vector through attentive memory reading.Keys concatenate subjects with attributes, relations, or qualifier keys, while values store corresponding objects or qualifier values.
  • SRN is restricted to relational questions beginning with Find and ending with QueryName, excluding literal and qualifier knowledge.The filtered data retain 5,004 training questions and 649 test questions.
  • EmbedKGQA adds virtual nodes for qualifier knowledge and applies its flexible classification layer to the entire KQA Pro dataset.
  • RGCN represents entities as graph nodes and relations as labeled edges, concatenating literal attributes into node descriptions while ignoring qualifiers.The implementation uses one graph layer and 32-dimensional node and edge hidden states.
  • The recurrent parsers encode questions with GRUs and decode either KoPL function sequences with textual inputs or SPARQL token sequences.Attention focuses on relevant question words during function and input prediction.
  • KQA Pro examples contain a textual question, SPARQL query, KoPL program, 10 candidate choices, and a golden answer.Verify choices contain “yes”, “no”, and eight “unknown” padding tokens.
Loading 2007.03875v4…