PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot In-context Learning for Knowledge Base Question Answering

Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, Wenhu Chen

arXiv:2305.01750v2cs.CLcs.AI

TL;DR

KB-BINDER addresses the data-intensive, dataset-specific challenges of KBQA with a training-free few-shot framework that combines LLM-generated logical-form drafts with knowledge-base grounding. Across four heterogeneous datasets, it achieves strong performance, surpassing prior state of the art on GraphQA and 3-hop MetaQA while remaining competitive elsewhere.

  • Problem

    KBQA requires extensive annotated data and often dataset-specific training because models struggle to generalize across heterogeneous knowledge-base schemas and unseen domains.

  • Method

    KB-BINDER uses few-shot demonstrations to prompt Codex for preliminary logical-form drafts, then binds their entities and schema items to an executable knowledge-base form.

  • Results

    Across four public datasets, KB-BINDER achieves strong performance, including 39.5 F1 on GraphQA, 99.5% Hits@1 on 3-hop MetaQA, and 74.4% F1 on WebQSP with KB-BINDER-R.

  • Takeaways & Limitations

    KB-BINDER provides a strong low-resource baseline for few-shot KBQA without heuristics customized to a target knowledge-base schema.

  • Takeaways & Limitations

    Performance can vary with randomly selected exemplars, and one-time draft generation limits the preliminary logical forms’ imaginary frame and schema items.

Abstract

from arXiv · show

Question answering over knowledge bases is considered a difficult problem due to the challenge of generalizing to a wide variety of possible natural language questions. Additionally, the heterogeneity of knowledge base schema items between different knowledge bases often necessitates specialized training for different knowledge base question-answering (KBQA) datasets. To handle questions over diverse KBQA datasets with a unified training-free framework, we propose KB-BINDER, which for the first time enables few-shot in-context learning over KBQA tasks. Firstly, KB-BINDER leverages large language models like Codex to generate logical forms as the draft for a specific question by imitating a few demonstrations. Secondly, KB-BINDER grounds on the knowledge base to bind the generated draft to an executable one with BM25 score matching. The experimental results on four public heterogeneous KBQA datasets show that KB-BINDER can achieve a strong performance with only a few in-context demonstrations. Especially on GraphQA and 3-hop MetaQA, KB-BINDER can even outperform the state-of-the-art trained models. On GrailQA and WebQSP, our model is also on par with other fully-trained models. We believe KB-BINDER can serve as an important baseline for future research. Our code is available at https://github.com/ltl3A87/KB-BINDER.

1 Introduction

KB-BINDER addresses the data, schema, and graph-scale challenges of training-free few-shot KBQA by generating preliminary logical forms and grounding them against the knowledge base. Across four datasets, it achieves competitive or superior results with only a few demonstrations.

  • Motivation: KBQA is difficult because large, heterogeneous knowledge bases require broad generalization while annotated data and schema-specific training are costly.The paper highlights data intensiveness, dataset specificity, and the challenge of conditioning on massive graphs with millions of nodes and billions of edges.
  • Approach: KB-BINDER uses Codex demonstrations to generate preliminary logical-form drafts, then binds their entities and relations to executable knowledge-base items.The draft may contain incorrect entities or relations because the language model lacks direct knowledge-base schema information.
  • Results: 39.5 F1 on GraphQA and 99.5% Hits@1 on 3-hop MetaQA surpass previous state of the art by 7.7 F1 and 0.6% Hits@1, respectively.On WebQSP, KB-BINDER-R reaches 74.4% F1, 4.4% below the state-of-the-art model.
  • Implication: KB-BINDER can serve as an important low-resource baseline because it requires no heuristics customized to a target knowledge-base schema.The authors evaluate the framework on WebQSP, GrailQA, GraphQA, and MetaQA.

2 Related Work

Prior KBQA systems generally rely on semantic parsing, substantial annotated data, or target-specific heuristics. This paper positions KB-BINDER as a training-free few-shot alternative that combines LLM reasoning with explicit schema binding.

  • Knowledge Base Question Answering: Existing KBQA methods map questions to logical forms over very large knowledge-base search spaces and often require substantial labeled data.Freebase is cited as containing 45 million entities and 3 billion facts.
  • Few-shot KBQA: Few-shot KBQA remained largely unexplored because handling the large logical-form search space with only a handful of examples was considered highly non-trivial.The cited meta-learning approach still requires 2,000 labeled questions to train its adaptation model.
  • In-Context Learning: In-context learning has demonstrated few-shot performance across question answering, information extraction, and numerical reasoning tasks.Prior analyses also study prompt construction and the mechanisms behind in-context learning.
  • Reasoning with LLMs: Prompted LLMs can generate structured programs, but thousands of relations and millions of entities make KBQA schema grounding substantially harder than SQL generation.KB-BINDER addresses this larger search space with a draft-generation and schema-binding pipeline.

3 Methodology

KB-BINDER first prompts Codex to generate semantically structured logical-form drafts, then grounds their entities and relations against the knowledge base. Candidate logical forms are executed after binding, with majority voting and retrieved exemplars providing optional robustness improvements.

  • Draft Generation: KB-BINDER reduces the search space by using an LLM to generate a semantically reasonable draft before retrieving real entities and schema terms.The draft is not guaranteed to be executable because it is not restricted to the knowledge base vocabulary or graph structure.
  • Drafts Generator: Codex receives randomly sampled question–logical-form exemplars and generates drafts using surface names instead of opaque machine identifiers.Surface names are substituted for MIDs so demonstrations expose semantic relationships among entities and schema items.
  • Knowledge Base Binder: Entity binding retrieves candidate knowledge-base entities from draft surface names, while relation binding uses BM25 similarity and local graph constraints to identify executable relations.Relation candidates are drawn from two-hop items beginning at the currently bound entity permutations.
  • Candidate Selection: The framework executes answerable logical-form candidates and repeats generation K times, using majority voting to improve prediction robustness.A single draft can produce hundreds of potential logical-form candidates before execution.
  • Retrieved Exemplars: KB-BINDER(K)-R retrieves similar training questions as demonstrations so their logical forms are more likely to cover schema items relevant to the target question.This retrieved-exemplar variant is intended to be especially advantageous for i.i.d. questions.

4 Experiment

KB-BINDER is evaluated in few-shot settings across four heterogeneous KBQA datasets against fully trained baselines and through ablations of demonstrations and draft voting. It achieves strong cross-dataset performance, with especially competitive results on GraphQA, MetaQA, and WebQSP, while showing trade-offs in GrailQA and inference cost.

  • 4.4 Main Result: KB-BINDER variants perform strongly across all selected datasets, with six-draft voting generally outperforming one draft and retrieved exemplars improving most cases.The retrieved-exemplar variant raises WebQSP F1 from 53.2 to 74.4 but slightly reduces GraphQA performance by 0.8 points.
  • 4.4 Main Result: 50.6 EM on GrailQA with 40 examples matches the BERT + Ranking setting trained on around 45k annotations, although supervised state-of-the-art methods retain an overall gap.KB-BINDER (6) also has 5.1- and 1.3-point EM advantages over BERT+Ranking on compositional and zero-shot questions, respectively.
  • 4.4 Main Result: The retrieved-exemplar variant reaches 99.5% Hits@1 on 3-hop MetaQA, setting a new state-of-the-art, and matches the prior state-of-the-art at 99.9% on 2-hop MetaQA.These results are reported for KB-BINDER (1)-R with five-shot demonstrations.
  • 4.4 Main Result: KB-BINDER is most competitive when annotated data are scarce, generalization is demanding, or the domain and schema search space are specific, but restricted logical-form coverage limits GrailQA.The authors identify WebQSP as a setting where retrieved exemplars help because test questions are I.I.D.-like, while GrailQA favors previous models when broad generalization and extensive training data are available.
  • 4.5 Ablation Study: Increasing demonstrations raises coverage and EM, while increasing drafts from one to six improves coverage by 19% and EM by 5.6%.The shot-number study samples 500 GrailQA development questions; larger shot and draft counts also increase inference time and cost.

5 Conclusion

KB-BINDER enables true few-shot KBQA by generating preliminary logical forms with an LLM and binding their entities and schema items to executable forms. Its strong performance across common KBQA datasets supports its use as a low-resource baseline.

  • KB-BINDER first enables challenging few-shot KBQA by generating preliminary logical forms and iteratively binding entities and schema items to the target knowledge base.The framework uses large-language-model reasoning to produce drafts, then grounds them into executable forms.
  • KB-BINDER and its derivatives achieve strong performance across the selected common-used KBQA datasets, supporting a strong baseline for low-resource research.The conclusion frames the method as a baseline for future KBQA work in low-resource settings.

Limitations

KB-BINDER’s performance can vary with exemplar selection in low-shot settings, and one-time draft generation limits its preliminary logical forms. The authors identify interactive generation, retrieval, and explanations as directions for mitigation.

  • Performance may vary with randomly sampled exemplars, especially in low-shot settings, although results remain strong across thousands of test examples.The authors describe this as a robustness limitation that is only partially mitigated by their evaluations.
  • One-time generated drafts restrict the preliminary logical forms, leaving potential improvements through interactive generation, retrieval, or explanations and instructions.These limitations concern the draft-generation stage rather than the knowledge-base binding stage.

A.1 Error Analysis

Error analysis separates failures in logical-frame generation from errors in entity and relation binding. On a 500-example GrailQA subset, the logic frame has the lowest reported top-1 recall.

  • The entity and relation binders reach recalls of 0.9 and 0.78, while the top-1 draft’s logic-frame recall is 0.66.The logic-frame component accounts for most error cases in this analysis.
  • The analysis evaluates KB-BINDER (1) on 500 randomly sampled GrailQA development examples using the same setting as Section 4.3.
Loading 2305.01750v2…