Source-linked AI summary
RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Caiming Xiong
TL;DR
KBQA systems struggle with unseen schema items, while ranking-based methods additionally suffer from incomplete candidate coverage. RnG-KBQA combines a contrastive ranker over searched logical forms with a generator conditioned on the question and top-ranked candidates. It achieves new state-of-the-art results on GRAILQA and WEBQSP, with especially large gains in generalization settings.
Problem
Unseen KB schema items and compositions challenge KBQA generalization, while ranking-based methods can omit the desired logical form because exhaustive rule coverage is impractical.
Method
RnG-KBQA ranks searched candidate logical forms with a contrastive ranker and generates the final logical form from the question and top-k candidates.
Results
RnG-KBQA achieves new state-of-the-art performance on GRAILQA and WEBQSP and particularly outperforms prior methods in generalization settings.
Takeaways & Limitations
The interplay between ranking and generation supports strong performance across settings, especially compositional and zero-shot generalization.
Takeaways & Limitations
Candidate enumeration does not cover every possible logical form, and comprehensive enumeration can exceed 2,000,000 candidates for some queries.
Abstract
from arXiv · showhide
Existing KBQA approaches, despite achieving strong performance on i.i.d. test data, often struggle in generalizing to questions involving unseen KB schema items. Prior ranking-based approaches have shown some success in generalization, but suffer from the coverage issue. We present RnG-KBQA, a Rank-and-Generate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. Our approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a tailored generation model conditioned on the question and the top-ranked candidates to compose the final logical form. We achieve new state-of-the-art results on GrailQA and WebQSP datasets. In particular, our method surpasses the prior state-of-the-art by a large margin on the GrailQA leaderboard. In addition, RnG-KBQA outperforms all prior approaches on the popular WebQSP benchmark, even including the ones that use the oracle entity linking. The experimental results demonstrate the effectiveness of the interplay between ranking and generation, which leads to the superior performance of our proposed approach across all settings with especially strong improvements in zero-shot generalization.
1 Introduction
KBQA systems often struggle with unseen schema items and compositions: generation lacks reliable schema coverage, while ranking-based methods face incomplete candidate coverage. RnG-KBQA combines ranking and generation to address this gap and achieves strong results on GRAILQA and WEBQSP.
- Motivation: Unseen compositions and KB schema items remain challenging despite strong performance on i.i.d. benchmarks.Generation-based parsers struggle to generate unseen schema items, while ranking-based systems depend on predefined candidate-generation rules.
- Motivation: Ranking-based approaches improve generalization but suffer from coverage because exhaustive rule enumeration is impractical at KB scale.The desired logical form may be absent from the candidate set when the rules do not cover it.
- Approach: RnG-KBQA ranks searched logical-form candidates, then generates a final logical form from the question and top-ranked candidates.The ranker supplies semantically coherent schema ingredients, while the generator complements missing constructions or constraints.
- Approach: The ranker uses a BERT-based biencoder trained contrastively, while the generator uses a T5-based sequence-to-sequence model.The generator receives the question followed by a linearized sequence of top-k candidates.
- Results: 68.8 exact match and 74.4 F1 on GRAILQA surpass prior SOTA scores of 58.1 exact match and 65.3 F1.On WEBQSP, RnG-KBQA reaches 75.7 F1 and outperforms the best prior approach.
2 Generation Augmented KBQA
RnG-KBQA combines candidate logical-form ranking with generation to address incomplete candidate coverage while retaining ranking-based generalization. It also adapts ranking to entity disambiguation by using question and relation information.
- Candidate Enumeration: Candidates are enumerated from entities detected in the question by querying knowledge-base paths reachable within two hops and converting those paths into s-expressions.Enumeration omits some compositions, including comparative and argmin/max operations.
- Motivation: KBQA generation struggles with unseen schema items, while ranking methods face incomplete coverage because exhaustive logical-form enumeration is impractical.Comprehensive enumeration can exceed 2,000,000 candidates for some queries.
- Logical Form Ranking: The BERT-based ranker scores question–candidate pairs with a contrastive objective that promotes the ground-truth logical form and penalizes negative candidates.Its training contrasts candidates across the candidate set, helping distinguish correct logical forms from spurious ones.
- Logical Form Ranking: Negative candidates are bootstrapped by first warming up with random samples, then selecting confusing spurious logical forms for further ranker training.The paper reports better performance than using random negative samples alone.
- Target Logical Form Generation: A T5 generator receives the question and top-ranked candidates, then composes the final logical form by correcting, supplementing, or discarding candidate content.At inference, beam search produces candidate outputs, which are executed until a valid non-empty result is found.
- Entity Disambiguation: Entity disambiguation is recast as ranking by combining each candidate entity’s relations with the question, reusing the ranking architecture and loss.This addresses cases where surface matching and popularity scores cannot distinguish entities with the same name.
3 Experiments
RnG-KBQA achieves state-of-the-art results on GRAILQA and WEBQSP while addressing generalization across i.i.d., compositional, and zero-shot settings. Ablations and analyses show that ranking and generation complement one another, although generation is less effective for unseen relations.
- GRAILQA: GRAILQA: 68.8 EM and 74.4 F1 establish new state-of-the-art results, including a 16.1 F1 margin over ReTrack in zero-shot generalization.The dataset evaluates i.i.d., compositional, and zero-shot generalization, with zero-shot questions comprising 50% of its test set.
- WEBQSP: WEBQSP: 75.6 F1 surpasses QGG by 1.6 and remains stronger than prior approaches using oracle entity linking annotations.The approach also achieves the best EM score of 71.1, while QGG reports 85.2 entity-linking F1 versus 84.8 for RnG-KBQA.
- Ablation Study: Adding generation to ranking improves performance by 5.3 on GRAILQA and 2.9 on WEBQSP over the ranking-only variant.The generation stage remedies cases that the ranking model alone cannot address.
- Analysis: The generator fixes top-ranked candidates for 4.7% of GRAILQA and 8.9% of WEBQSP questions, while execution-guided inference can fall back to ranking results after generation errors.Generation is more helpful in i.i.d. and compositional settings than in zero-shot settings involving unseen relations.
4 Related Work
KBQA methods either retrieve and rank answer entities or parse questions into executable logical forms. RnG-KBQA belongs to the latter line while adding generation to address the coverage limitations of candidate-enumeration approaches.
- KBQA approaches can construct query-specific subgraphs and rank entity nodes to select answers.Subgraphs may be retrieved in one shot with heuristic rules or built iteratively with learned models.
- Logical-form methods represent queries with lambda-DCS, SPARQL, graph queries, or s-expressions.These methods parse questions into executable representations.
- Some systems enumerate logical-form candidates and select the candidate best matching the question’s intent.This candidate-ranking strategy appears across several logical-form representations and methods.
- RnG-KBQA adds a generation stage to remedy the coverage issue often suffered by candidate-enumeration approaches.
5 Conclusion
RnG-KBQA combines ranking and generation for knowledge-base question answering. Experiments on GRAILQA and WEBQSP report new state-of-the-art results, especially for generalization settings.
- RnG-KBQA combines a ranking step with a generation step for question answering over knowledge bases.
- The ranker uses iterative bootstrapping to distinguish correct logical forms from spurious ones better than a prior seq-to-seq ranker.
- The generator remedies uncovered operations or implicitly mentioned constraints in top-ranked logical forms.
- Experiments on GRAILQA and WEBQSP report new state-of-the-art performance, with especially large gains in generalization settings.
A Details of Entity Linking Performance
The entity disambiguation model improves GRAILQA entity linking and the resulting KBQA performance over a popularity-based baseline. Its advantage remains substantial even without the entity disambiguation modules.
- 7.4 entity linking F1 improvement over Bert Ranking boosts the final KBQA F1 score by 7.0.The comparison is against the popularity-based Bert Ranking baseline on GRAILQA.
- Without entity disambiguation modules, the method outperforms Bert Ranking by 11.4 F1 score.
- The method also beats ReTrack when built on a better entity linking model.