Source-linked AI summary
Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, Yu Su
TL;DR
Large-scale KBQA cannot rely solely on i.i.d. evaluation because user questions have broad, combinatorial coverage and may fall outside training distributions. The paper defines i.i.d., compositional, and zero-shot generalization, releases the 64,331-question GRAILQA dataset, and proposes a BERT-based model. Experiments show BERT’s critical role in compositional and zero-shot generalization while highlighting entity linking and search as remaining challenges.
Problem
Existing KBQA studies mainly assume matched training and test distributions, although broad and combinatorial user questions make sufficient coverage difficult on large-scale knowledge bases.
Method
The paper constructs GRAILQA with 64,331 questions for three generalization levels and proposes a BERT-based KBQA model.
Results
The model sets a new GRAPHQ state of the art by beating prior models by 3.5%, significantly outperforms a state-of-the-art model on GRAILQA, and demonstrates BERT’s critical role in compositional and zero-shot generalization.
Takeaways & Limitations
KBQA evaluation and model development should account for i.i.d., compositional, and zero-shot generalization rather than only standard matched-distribution performance.
Takeaways & Limitations
The paper identifies entity linking and brute-force logical-form search as remaining challenges for practical KBQA systems.
Abstract
from arXiv · showhide
Existing studies on question answering on knowledge bases (KBQA) mainly operate with the standard i.i.d assumption, i.e., training distribution over questions is the same as the test distribution. However, i.i.d may be neither reasonably achievable nor desirable on large-scale KBs because 1) true user distribution is hard to capture and 2) randomly sample training examples from the enormous space would be highly data-inefficient. Instead, we suggest that KBQA models should have three levels of built-in generalization: i.i.d, compositional, and zero-shot. To facilitate the development of KBQA models with stronger generalization, we construct and release a new large-scale, high-quality dataset with 64,331 questions, GrailQA, and provide evaluation settings for all three levels of generalization. In addition, we propose a novel BERT-based KBQA model. The combination of our dataset and model enables us to thoroughly examine and demonstrate, for the first time, the key role of pre-trained contextual embeddings like BERT in the generalization of KBQA.
1 INTRODUCTION
The paper argues that standard i.i.d. evaluation is insufficient for large-scale KBQA and proposes evaluating i.i.d., compositional, and zero-shot generalization. It introduces GRAILQA and a BERT-based model to study these settings.
- Motivation: Existing KBQA studies mainly assume that training and test questions share the same distribution.Large-scale KBs make this assumption problematic because user questions have broad, combinatorial coverage and new questions can repeatedly cause failures.
- Three Levels of Generalization: Practical KBQA models should generalize beyond i.i.d. questions to novel compositions of seen schema items and unseen schema items or domains.These levels are termed i.i.d., compositional, and zero-shot generalization.
- GRAILQA: 64,331 crowdsourced questions make GRAILQA a large-scale benchmark covering all three generalization levels.The questions involve up to 4 relations and functions including counting, comparatives, and superlatives.
- GRAILQA: GRAILQA covers all 86 domains in Freebase Commons and entities ranging from highly popular to long-tail examples.This breadth is intended to support realistic evaluation of entity linking and generalization.
- Model and Findings: The proposed BERT-based KBQA model achieves competitive results and enables analysis of search-space pruning, language-ontology alignment, and BERT’s role in compositional and zero-shot generalization.On GRAPHQ, it beats prior models by 3.5%; on GRAILQA, it significantly outperforms a state-of-the-art KBQA model.
- Contributions: The paper contributes a systematic three-level evaluation, a 64K-question dataset, and a BERT-based model for studying stronger KBQA generalization.It also reports that GRAILQA pre-training transfers to WEBQ, reaching similar performance with only 10% of the fine-tuning data.
2 BACKGROUND
The paper frames KBQA over Freebase Commons as structured query answering over a large ontology and fact base. It formally distinguishes i.i.d., compositional, and zero-shot generalization by the schema items and logical forms seen during training.
- Knowledge Base: A knowledge base is represented by an ontology of class-relation-class triples and relational facts involving entities, classes, and literals.The dataset uses Freebase Commons, which contains 86 domains, 2,038 classes, 6,265 relations, and over 45 million entities.
- Definitions: The schema-item set includes relations, classes, and language-specific constructs such as functions, while entities and literals are excluded.Training and question schema-item sets are compared to define the generalization levels.
- Definitions: I.i.d. generalization uses seen schema items, covered logical forms, and test questions following the training distribution.This is the standard baseline setting for KBQA evaluation.
- Definitions: Compositional generalization uses only seen schema items but requires logical-form compositions not covered during training.Zero-shot generalization requires at least one schema item in the question to be unseen during training.
3 DATA
GRAILQA is constructed by generating and validating complex logical forms, annotating canonical questions, crowdsourcing paraphrases, and sampling grounded combinations. Its resulting questions are large-scale, diverse, and designed to expose realistic entity-linking and language-variation challenges.
- Data Collection: The dataset construction process generates well-formed logical forms, validates them, annotates canonical questions, crowdsources paraphrases, grounds entities, and samples final pairs.The pipeline also mines common web surface forms for entities to make entity linking more realistic.
- Data Collection: Logical forms contain up to 4 relations and optionally one function from counting, superlatives, or comparatives.The exemplar logical form contains 3 relations and one function.
- Data Collection: 86.5% of generated logical forms pass validation against whether a real user could reasonably ask the corresponding question.This filtering targets artificial questions associated with generated logical forms.
- Data Collection: Crowdsourced paraphrases are cross-validated for fluency and semantic fidelity, with 17.4% discarded.Each paraphrase receives four judgments on average, and workers are monitored using control questions.
- Dataset Scale: 4,969 canonical logical forms and 29,457 paraphrases produce 64,331 final question-logical form pairs.The sampling design gives GRAILQA broad coverage and many unique canonical logical forms.
- Dataset Analysis: Manual analysis estimates 3% canonical-question error, 2.1% paraphrasing error, and 5.6% overall error.The examined paraphrases were all judged reasonably fluent.
- Dataset Analysis: GRAILQA’s paraphrases exhibit linguistic diversity, with average unigram and bigram Jaccard similarities of 0.569 and 0.286 to their canonical questions.The dataset was produced by 11 graduate students and 6,685 crowd workers with diverse demographics.
- Dataset Analysis: Mined entity surface forms are more colloquial than formal knowledge-base names, creating a practical entity-linking challenge largely neglected by existing datasets.This challenge is especially relevant for long-tail and ambiguous entities.
4 MODELING
The model combines a Seq2Seq KBQA architecture with BERT-based contextual representations, while addressing non-i.i.d. challenges through vocabulary design, inference strategies, and search-space control.
- 4.1 Model Overview: The proposed KBQA model uses an LSTM encoder-decoder Seq2Seq architecture to map questions to logical forms.The encoder processes the question, and the decoder autoregressively predicts tokens from a decoding vocabulary.
- 4.2 BERT Encoding: The model ties input and output embeddings through W, assigning semantic information from pre-trained embeddings to decoder vocabulary items.This embedding sharing is intended to facilitate open-vocabulary learning.
- 4.2 BERT Encoding: BERT jointly encodes each question with chunks of schema items because the full KBQA vocabulary exceeds BERT’s input limit.Question representations come from BERT outputs, while each vocabulary-item embedding averages the outputs of its constituent word-pieces.
- 4.2 BERT Encoding: Vocabulary construction balances zero-shot coverage against decoding and training costs by reducing the ontology-derived search space.Including every ontology schema item supports zero-shot generalization but creates many candidates and BERT chunks.
- 4.4 Inference: The model supports TRANSDUCTION and RANKING inference, trading flexible generation against more effective search-space pruning.RANKING scores enumerated logical forms within two hops of identified entities, whereas TRANSDUCTION predicts tokens autoregressively.
- 4.4 Inference: The candidate logical-form generator reaches 80% recall on GRAILQA, but often misses questions with superlatives and comparatives lacking a topic entity.RANKING prunes the search space more effectively, while TRANSDUCTION handles more question types.
5 EXPERIMENTS
Experiments evaluate KBQA models across three generalization levels, fine-grained question types, and error categories. RANKING performs best overall, while BERT chiefly improves compositional and zero-shot generalization and search-space pruning remains a major constraint.
- Experimental Setup: 44,337 training, 6,763 validation, and 13,231 test questions support evaluation of i.i.d., compositional, and zero-shot generalization.The validation and test sets contain 50% zero-shot, 25% compositional, and 25% randomly sampled questions.
- Overall Evaluation: RANKING achieves the best overall performance on GRAILQA, and both proposed models outperform QGG.Removing BERT or vocabulary pruning causes significant performance drops across the model variants.
- Generalization Analysis: BERT versus GloVe reduces TRANSDUCTION F1 by 17.5% for compositional and 26.2% for zero-shot generalization, while i.i.d. performance is similar.For RANKING, replacing BERT also causes a 21.9% drop in zero-shot generalization.
- Generalization Analysis: RANKING significantly outperforms TRANSDUCTION in compositional and zero-shot generalization, largely through more effective search-space pruning.RANKING prunes candidates using each identified entity’s neighboring facts, whereas TRANSDUCTION relies on less discriminating ontology information.
- Fine-Grained Evaluation: Performance degrades as structural complexity increases; TRANSDUCTION is stronger on comparatives and superlatives, while QGG gets zero F1 on counting questions.RANKING nevertheless outperforms TRANSDUCTION on questions with 3 relations, demonstrating the importance of candidate pruning.
- Error Analysis: Coverage limitation accounts for 34% of errors because RANKING generates candidates for only about 80% of questions and enumerates at most 2-relational logical forms.Entity linking accounts for 33% of errors, relation mis-classification for 26%, and other errors for 7%.
6 RELATED WORK
Prior KBQA datasets and models largely emphasize i.i.d. evaluation, while compositional and zero-shot generalization remain less developed. The paper positions GRAILQA and its BERT-based model as resources for studying all three levels.
- Existing KBQA datasets mainly evaluate i.i.d. generalization, whereas GRAPHQ and QALD primarily target compositional generalization.
- KBQA models include semantic-parsing methods that generate executable logical forms and information-retrieval methods that rank candidate entities directly.
- The best model on GRAPHQ achieved an F1 of 21.5, indicating that non-i.i.d. generalization had received limited attention.
- Earlier work connected pre-trained embeddings with cross-domain and compositional semantic parsing, including contextual embeddings for text-to-SQL.
7 CONCLUDING REMARKS
The paper frames its contribution as a systematic study of three KBQA generalization levels, supported by GRAILQA and a BERT-based model. It identifies contextual embeddings as especially important for compositional and zero-shot generalization while outlining unresolved practical challenges.
- The paper introduces a systematic study of i.i.d., compositional, and zero-shot generalization for KBQA.
- GRAILQA contains 64,331 questions and supports evaluation at all three generalization levels.
- The proposed BERT-based model and dataset expose challenges and potential solutions for non-i.i.d. KBQA generalization.
- Pre-trained contextual embeddings like BERT facilitate compositional and zero-shot generalization through better language-ontology alignment.
- Future work includes context-sensitive entity linking, guided search for complex questions, and deeper understanding of why BERT helps.
A CROWD WORKER DEMOGRAPHICS
The dataset collection involved 6,685 Amazon Mechanical Turk workers with diverse age and education backgrounds. Gender participation was fairly even, with somewhat higher female participation.
- 6,685 crowd workers were recruited through Amazon Mechanical Turk for data collection.
- Female workers represented 59.1% of participants, compared with 40.9% male participation.
- The workers showed diversity in age and completed education level.Figure 6 summarizes age and education categories, including high school, college, bachelor, master, and doctoral levels.
B S-EXPRESSION
The paper uses set-based S-expressions to represent entities, tuples, and values, with functions composing these denotations. The appendix also documents the annotation interface and dataset-statistics procedure.
- S-expressions use set-based semantics in which functions take and return sets of entities or entity tuples.
- The representation has three argument types: entity sets, entity-pair tuples, and entity-value tuples.
- Classes and single entities form entity sets, while relations form binary tuples that functions can combine into more complex structures.
- Table 4 lists the functions used in the S-expression grammar, including set and numerical operations.
- The appendix reports that dataset statistics are retrieved from logical-form and inferential-chain information across prior datasets.
D.1 Hyper-parameters
The models use Adam-based optimization with different batching and learning-rate settings for GloVe-based and BERT-based models. BERT training uses gradient accumulation to address memory consumption, with all encoder and decoder hidden sizes set to 768.
- GloVe-based models use batch size 32 and an initial learning rate of 0.001 with Adam.
- BERT parameters are fine-tuned with learning rate 2e-5, while the models’ own parameters use an initial learning rate of 0.001.
- All models use encoder and decoder hidden sizes of 768, and hyper-parameters are selected using the validation set.
D.2 Training Scheme
Training uses early stopping based on validation exact match, and compares GloVe-based and BERT-based Seq2Seq models.
- Training stops early with patience 3, selecting the model with the highest validation exact match.
- The experiments train two models: a GloVe-based Seq2Seq model and a BERT-based Seq2Seq model.
- Vocabulary pruning is unnecessary during training for the GloVe-based model.