PaperlayerView exact revision on arXiv

Source-linked AI summary

FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering

Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, Jianyong Wang

arXiv:2308.12060v3cs.CLcs.AI

TL;DR

KBQA needs effective systems despite scarce annotations, large knowledge bases, and heterogeneous query languages. FlexKBQA uses LLMs to translate sampled programs into synthetic questions, trains lightweight models, and applies execution-guided self-training. It outperforms prior few-shot baselines across three datasets, approaches supervised performance, and pioneers zero-shot KBQA.

  • Problem

    KBQA annotation is difficult because knowledge bases are large and evolving, while heterogeneous schemas and query languages require separate training processes; existing LLM methods also face context, efficiency, and domain-specific generation limits.

  • Method

    FlexKBQA samples diverse knowledge-base programs, translates them into natural-language questions with LLMs, trains lightweight models, and uses execution-guided self-training on real queries.

  • Results

    Across GrailQA, WebQSP, and KQA Pro, FlexKBQA outperforms prior methods in few-shot settings, approaches supervised models, and extends evaluation to zero-shot KBQA.

  • Takeaways & Limitations

    FlexKBQA offers flexibility in data annotation, deployment, and domain coverage by integrating LLM capabilities with lightweight KBQA models.

  • Takeaways & Limitations

    On KQA Pro, unseen test-set relations or entities can yield semantically correct but unexecutable programs because of the entity-linking stage.

Abstract

from arXiv · show

Knowledge base question answering (KBQA) is a critical yet challenging task due to the vast number of entities within knowledge bases and the diversity of natural language questions posed by users. Unfortunately, the performance of most KBQA models tends to decline significantly in real-world scenarios where high-quality annotated data is insufficient. To mitigate the burden associated with manual annotation, we introduce FlexKBQA by utilizing Large Language Models (LLMs) as program translators for addressing the challenges inherent in the few-shot KBQA task. Specifically, FlexKBQA leverages automated algorithms to sample diverse programs, such as SPARQL queries, from the knowledge base, which are subsequently converted into natural language questions via LLMs. This synthetic dataset facilitates training a specialized lightweight model for the KB. Additionally, to reduce the barriers of distribution shift between synthetic data and real user questions, FlexKBQA introduces an executionguided self-training method to iterative leverage unlabeled user questions. Furthermore, we explore harnessing the inherent reasoning capability of LLMs to enhance the entire framework. Consequently, FlexKBQA delivers substantial flexibility, encompassing data annotation, deployment, and being domain agnostic. Through extensive experiments on GrailQA, WebQSP, and KQA Pro, we observe that under the few-shot even the more challenging zero-shot scenarios, FlexKBQA achieves impressive results with a few annotations, surpassing all previous baselines and even approaching the performance of supervised models, achieving a remarkable 93% performance relative to the fully-supervised models. We posit that FlexKBQA represents a significant advancement towards exploring better integration of large and lightweight models. The code is open-sourced.

Introduction

FlexKBQA addresses few-shot KBQA by using LLMs to generate synthetic program-question data for lightweight models and execution-guided self-training for real queries. Across GrailQA, WebQSP, and KQA Pro, it outperforms prior methods with few labels and approaches supervised performance.

  • Introduction: Manual KBQA annotation is difficult because knowledge bases are large and evolving, while heterogeneous schemas and query languages require KB-specific training processes.These challenges expand the sample space and increase the labor needed to develop and adapt models.
  • Introduction: Existing in-context LLM approaches face limited context windows, high inference overhead, and difficulty generating domain-specific entities and relations without fine-tuning.Their benefits can also diminish as more training data is introduced.
  • Introduction: LLMs generate synthetic program-question pairs from sampled knowledge-base programs, which train lightweight KBQA models while reducing dependence on manual annotation.The framework samples S-expressions from the KB and translates them into fluent natural-language questions.
  • Introduction: Execution-guided self-training uses a lightweight model to annotate real user queries, mitigating distribution shift between synthetic training data and real questions.The resulting annotations are added as training data for iterative self-improvement.
  • Introduction: FlexKBQA significantly outperforms existing baselines across GrailQA, WebQSP, and KQA Pro in few-shot settings, approaching several fully supervised models.With 25 labeled examples, it surpasses previous methods using 100 shots on GrailQA.

Related Work

Prior KBQA work includes semantic parsing, step-wise search, and few-shot in-context learning, while broader model-combination research transfers LLM capabilities to smaller models. FlexKBQA extends teaching via data by using LLMs as program translators for KBQA.

  • Related Work: Semantic-parsing KBQA maps natural-language questions to formal programs, while step-wise discriminative methods constrain search and reduce generation errors.These approaches have produced state-of-the-art results but rely on substantial annotated data.
  • Related Work: Few-shot KBQA studies have explored in-context learning with LLMs, including Pangu and KB-BINDER, using minimal demonstrations to generate programs.These methods represent early efforts to apply LLM few-shot capabilities to KBQA.
  • Related Work: Research on combining large and lightweight models includes knowledge distillation and teaching via data, where LLM-generated synthetic examples train smaller student models.This line of work addresses the computational cost and deployment difficulty of large models.
  • Related Work: FlexKBQA adopts teaching via data but uniquely uses an LLM as a program translator to address KBQA training-data annotation.The approach generates synthetic data for fine-tuning a smaller task-specific model.

Methodology

FlexKBQA samples executable programs, translates them into natural-language questions to create synthetic training data, and uses execution-guided self-training plus inherent reasoning to address distribution shift and improve flexibility.

  • Automatic Program Sampling: FlexKBQA samples valid executable programs from knowledge-base templates, using template collection and step-wise grounding to cover diverse question types.The sampling objective is to generate programs executable on the knowledge base while avoiding costly manual annotation of many programs.
  • Low-Resource Program Translation: LLMs translate sampled programs into natural-language questions, producing synthetic pairs for training lightweight knowledge-base models with few-shot or zero-shot demonstrations.The prompt uses an instruction and diverse seed program-question pairs; N = 0 denotes zero-shot translation.
  • Execution-Guided Self-Training: Execution-guided self-training reduces synthetic-to-real distribution discrepancy by filtering pseudo-labeled user questions before combining them with synthetic and labeled data.Filtering removes execution failures, low semantic similarity, and samples whose pseudo-answers do not meet the inherent-reasoning criterion.
  • Inherent Reasoning Augmentation: Inherent reasoning augments the pipeline by selecting answer-aligned samples and supplying final answers when semantic parsing cannot retrieve one, while retaining interpretability as a motivation for program execution.The method uses inherent reasoning as data augmentation rather than relying on it exclusively for domain-specific knowledge-base questions.
  • Flexibility: FlexKBQA is designed to be data-efficient, domain-agnostic, and deployable through lightweight models that can integrate domain-specific knowledge.The framework requires only a small number of question-program pairs as prompts and is intended to reduce deployment costs relative to closed-source LLMs.

Experimental Setup

The evaluation spans three KBQA datasets with different knowledge bases, program types, and generalization demands, using established lightweight models and few-shot baselines for comparison.

  • Datasets: Experiments cover GrailQA, WebQSP, and KQA Pro, representing diverse datasets, knowledge bases, program types, and reasoning or generalization settings.GrailQA includes i.i.d., compositional, and zero-shot generalization; WebQSP reflects real-world query logs; KQA Pro requires multi-hop and set-based reasoning.
  • Underlying Models: FlexKBQA uses RnG-KBQA for GrailQA and WebQSP and BART-SPARQL for KQA Pro, making the framework model-agnostic in the evaluation.The selected underlying models are described as well-performing and reproducible choices.
  • Baselines: Baselines include Pangu and KB-BINDER, with LLM-ICL additionally reimplemented for KQA Pro and supervised models provided as comparison references.The baseline set is limited because few-shot KBQA is a relatively new task and prior KQA Pro results for Pangu and KB-BINDER were unavailable.
  • Data Construction: The setup treats original training data as unlabeled user questions and constructs 6,184 Freebase and 5,017 Wikidata synthetic pairs after entity-based sampling and deduplication.The evaluation uses the same off-the-shelf entity linkers as Pangu for fair comparison.

Results

FlexKBQA substantially outperforms few-shot KBQA baselines across GrailQA, WebQSP, and KQA Pro, approaching supervised performance in some settings. Its gains extend beyond few-shot learning, while EGST and inherent reasoning improve performance and address distribution shift.

  • Main Results: 62.8 EM and 69.4 F1 on GrailQA with 25 annotations outperform Pangu by 6.7 F1 points and reach 93% of fully supervised RnG-KBQA performance.FlexKBQA also surpasses several supervised models trained on tens of thousands of samples.
  • Main Results: 6.1 F1 points over Pangu on WebQSP under 100 shots, while KQA Pro shows a gap from the best model because it lacks entity linking.Without entity linking, unseen test relations or entities can yield semantically correct but unexecutable programs.
  • EGST and IR: Inherent reasoning improves F1 by 1.4 on GrailQA and 2.4 on WebQSP, and accuracy by 13.5 on KQA Pro.The GrailQA EM score remains unchanged because the metric focuses on structural-query consistency; on KQA Pro, reasoning helps when entity-linking absence causes execution errors.
  • Beyond Few-Shot KBQA: With 1000 real samples, synthetic-data pretraining retains an 8-point advantage over models trained solely on real samples, supporting use beyond few-shot settings.Performance steadily improves as annotated data increases, making the approach applicable as data augmentation.
  • Ablation and Case Studies: Performance improves with each EGST iteration and converges around the sixth epoch as pseudo-program error rates diminish.Case studies show synthetic coverage helps FlexKBQA handle non-i.i.d. relations, while EGST helps resolve similar ambiguous or complex questions.

Conclusion and Future Work

FlexKBQA integrates LLM program translation and reasoning to support few-shot and zero-shot KBQA, while approaching supervised performance and offering flexibility in annotation, deployment, and domain use.

  • Conclusion and Future Work: FlexKBQA surpasses all baseline methods in few-shot settings and approaches supervised-model performance across diverse datasets.The conclusion reports effectiveness on diverse datasets, while the experiments include GrailQA, WebQSP, and KQA Pro.
  • Conclusion and Future Work: The framework pioneers exploration of zero-shot KBQA.
  • Conclusion and Future Work: FlexKBQA offers flexibility in data annotation, deployment, and domain-agnostic use.

A. Automatic Program Sampling

Automatic program sampling converts structured-query templates into executable programs by grounding variables step by step, with LLM prompting supporting question generation and reasoning.

  • A. Automatic Program Sampling: Program templates are constructed from annotated programs, and S-expressions are converted into SPARQL before sampling proceeds.The experiments use programs from few-shot annotated samples to construct templates.
  • A. Automatic Program Sampling: The sampling procedure represents query objects as variables and uses them to derive executable query structures.The example rephrases a template as a query selecting rel0, ent0, and ent1.
  • A. Automatic Program Sampling: Templates are instantiated by iteratively grounding variables, then filling the grounded values into the original template to form an executable program.The example grounding order is ent0 → rel0 → ent1.
  • A. Automatic Program Sampling: Prompt designs are provided for GrailQA, WebQSP, and KQA Pro, while more efficient techniques such as batch prompting remain unexplored.
  • A. Automatic Program Sampling: Inherent reasoning uses in-context prompts built directly from question-answer pairs collected from datasets or the web.The prompts are separate from the program-template construction process.

C. Execution-Guided Filtering

Execution-guided filtering validates pseudo-label programs using semantic similarity, answer-entity presence, and endpoint availability to improve training-data quality.

  • C. Execution-Guided Filtering: Semantic filtering compares generated questions with predicted-program relations using cosine similarity between 384-dimensional sentence embeddings.Similarity scores are averaged when multiple relations are present.
  • C. Execution-Guided Filtering: Inherent filtering accepts a pseudo-label when the predicted answer entity’s surface name appears in the LLM’s direct response.This criterion is intended to enhance the purity of pseudo-labeled training data.
  • C. Execution-Guided Filtering: Pairs are removed when retrieved entities lack surface names obtainable through the Freebase SPARQL endpoint because such entities often produce erroneous results.
  • C. Execution-Guided Filtering: Examples of underlying baselines include RnG-KBQA for GrailQA and WebQSP and BART-SPARQL for KQA Pro.RnG-KBQA ranks candidate programs before generation, whereas BART-SPARQL directly generates SPARQL queries.

D.2 Baselines

The baseline set includes strong KBQA systems and supervised models to assess FlexKBQA under limited-data conditions and against supervised performance.

  • D.2 Baselines: Pangu combines symbolic search with neural scoring and incrementally constructs valid plans to handle large search spaces.Its step-wise construction guarantees grammaticality and faithfulness of proposed plans.
  • D.2 Baselines: KB-BINDER uses LLM draft generation followed by knowledge-base binding to produce refined executable logical forms in few-shot KBQA.The draft captures semantic relationships but may contain mistakes before entities and relations are grounded.
  • D.2 Baselines: Supervised models such as ReTraCk and DecAF are included to show that FlexKBQA maintains usability while offering flexibility.

E. Implementation Details

The implementation uses established entity-linking and reasoning components, including LLM answers with NER for initially unanswerable questions. A supplementary table is referenced, but its contents are not provided.

  • E. Implementation Details: The GrailQA and WebQSP experiments use RnG-KBQA with its non-bootstrap ranking strategy and can run on a single GeForce RTX 3090.Other settings remain consistent with the original RnG-KBQA configuration.
  • E. Implementation Details: Entity linking follows the baseline settings, using TIARA results on GrailQA and ELQ results on WebQSP.
  • E. Implementation Details: For initially unanswerable questions, recognized entities from LLM textual responses are used as final answers through NER.The NER system is consistent with the one used in RnG-KBQA.
  • E. Implementation Details: Table 6 is identified as covering discrepancies between synthetic and real data, but the supplied passage does not report its contents.

F. Synthetic vs. Real Data

The supplementary experiment examines the gap between synthetic and real data by comparing training sources with matched data quantities. An accompanying in-context-example ablation evaluates how demonstration count affects GrailQA performance.

  • F. Synthetic vs. Real Data: The supplementary experiment is designed to characterize the synthetic–real data discrepancy and investigate reasons for the difference.
  • F. Synthetic vs. Real Data: The comparison trains and evaluates models using approximately 1,000 synthetic examples versus the same amount of randomly sampled real WebQSP data.A third condition combines real programs with synthetic questions, but its description is truncated in the supplied passage.
  • F. Synthetic vs. Real Data: The in-context-example ablation uses GrailQA development and test sets, omits inherent reasoning augmentation, and reports F1 scores in Table 7.The supplied passage states that the number of examples has little effect, but the sentence is truncated before the full observation.
  • F. Synthetic vs. Real Data: Figure 5 is labeled as showing model improvement in the PEGST process, without supplied axes or numerical values.

H. Progressive Execution-Guided Self-Training

PEGST progressively changes execution-guided filtering strategies across stages to inject diverse knowledge into the model. On WebQSP, it slightly outperforms EGST but requires more iterations and execution time, so EGST is used in the main study.

  • H. Progressive Execution-Guided Self-Training: PEGST changes its filtering strategy across three stages: error filtering, semantic filtering, and removal of pairs lacking retrievable entity surface names.The final stage checks whether retrieved entity names can be obtained through the Freebase SPARQL endpoint.
  • H. Progressive Execution-Guided Self-Training: PEGST achieves a slight advantage over EGST on WebQSP, but requires more self-training iterations and a longer execution period.Consequently, the authors select EGST for the main body of the study.
  • H. Progressive Execution-Guided Self-Training: The WebQSP F1 score increases substantially when PEGST changes filtering criteria between stages, while gains within a single stage are relatively minor.
  • H. Progressive Execution-Guided Self-Training: The authors interpret PEGST as injecting diverse knowledge into the model in distinct phases.
  • H. Progressive Execution-Guided Self-Training: The prompts convert structured programs into natural-language questions across GrailQA, WebQSP, and KQA Pro examples.The supplied examples include S-expressions and SPARQL queries covering joins, comparisons, aggregation, and yes/no questions.
Loading 2308.12060v3…