Source-linked AI summary
DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases
Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, Bing Xiang
TL;DR
KBQA must balance the accuracy of executable logical forms against the robustness of directly generated answers. DECAF jointly generates both from retrieved linearized-KB text and combines their outputs, achieving state-of-the-art results on three benchmarks and competitive performance on a fourth.
Problem
Logical-form methods can be more accurate but may fail to execute, while direct-answer methods avoid execution failures but are less accurate on complex questions.
Method
DECAF retrieves relevant passages from a linearized knowledge base and jointly generates logical forms and direct answers, combining executed and generated answers.
Results
DECAF achieves new state-of-the-art results on WebQSP, FreebaseQA, and GrailQA, with competitive results on ComplexWebQuestions.
Takeaways & Limitations
Combining logical-form-executed and directly generated answers can produce more accurate final answers across multiple KBQA benchmarks.
Takeaways & Limitations
Direct answer generation considers returning only one answer, although the paper explores multi-answer generation separately.
Abstract
from arXiv · showhide
Question answering over knowledge bases (KBs) aims to answer natural language questions with factual information such as entities and relations in KBs. Previous methods either generate logical forms that can be executed over KBs to obtain final answers or predict answers directly. Empirical results show that the former often produces more accurate answers, but it suffers from non-execution issues due to potential syntactic and semantic errors in the generated logical forms. In this work, we propose a novel framework DecAF that jointly generates both logical forms and direct answers, and then combines the merits of them to get the final answers. Moreover, different from most of the previous methods, DecAF is based on simple free-text retrieval without relying on any entity linking tools -- this simplification eases its adaptation to different datasets. DecAF achieves new state-of-the-art accuracy on WebQSP, FreebaseQA, and GrailQA benchmarks, while getting competitive results on the ComplexWebQuestions benchmark.
1 INTRODUCTION
KBQA methods either execute generated logical forms or predict answers directly: logical forms are often more accurate but may be non-executable, while direct prediction guarantees answers with lower accuracy on complex questions. DECAF jointly generates both and combines their outputs, using text retrieval instead of entity linking, and achieves strong results across four benchmarks.
- Logical-form methods often achieve higher answer accuracy but can produce non-executable outputs because of syntax and semantic restrictions.
- Direct-answer methods guarantee output answers but are usually less accurate, especially on complex multi-hop questions.
- DECAF jointly decodes logical forms and direct answers, then combines executed and directly generated answers into the final output.
- DECAF treats logical forms as regular text strings during generation rather than using constrained decoding or post-revision procedures.
- DECAF linearizes knowledge bases into text documents and uses free-text retrieval to locate relevant subgraphs without relying on entity linking.
- DECAF achieves new state-of-the-art results on WebQSP, FreebaseQA, and GrailQA, with competitive results on ComplexWebQuestions.
2 RELATED WORK
Prior KBQA research separates semantic parsing methods, which execute logical forms, from direct-answer methods, which predict answers without an executor. DECAF instead retrieves linearized knowledge-base text and uses a reader to generate both logical forms and answers.
- Semantic parsing methods generate logical forms such as SPARQL or S-expressions and execute them against knowledge bases.
- Direct-answer methods predict answers without relying on logical-form execution, using approaches ranging from graph neural networks to sequence-to-sequence generation.
- Entity linking is commonly used to ground questions onto relevant knowledge-base subgraphs, but prior systems often require dataset-specific linking methods.
- DECAF linearizes knowledge bases into text documents and applies simple text retrieval instead of entity linking to improve generality across datasets and knowledge bases.
- DECAF retrieves question-relevant information and uses different reader prefixes to generate logical forms and direct answers.
3 METHOD
DECAF retrieves linearized knowledge-base passages and jointly generates logical forms and direct answers with a shared sequence-to-sequence reader. It executes generated logical forms, combines those answers with directly generated answers, and selects a final answer set.
- Knowledge Base Retrieval: DECAF linearizes the knowledge base into text documents, retrieves relevant passages, and supplies them with the question to a generative reader.Triplets are converted into sentences and grouped by head entity to preserve one-hop structural information.
- Knowledge Base Retrieval: The retriever supports sparse BM25 and dense DPR retrieval over passages derived from the linearized knowledge base.BM25 uses sparse word matching, whereas DPR uses embedding-space similarity from fine-tuned BERT encoders.
- Reading: The FiD-based reader encodes each question–passage pair separately and jointly decodes their representations for cross-passage reasoning.This design avoids concatenating all passages before self-attention, reducing the inefficiency associated with quadratic computation over many retrieved passages.
- Joint Decoding: A shared sequence-to-sequence model uses different prefixes to generate logical forms and direct answers as regular text strings.The prefixes are “Question Answering:” for direct answers and “Semantic Parsing:” for logical forms, without constrained decoding.
- Evaluation: The method is evaluated on benchmark datasets using direct-answer-prediction and semantic-parsing baselines, including separate GrailQA F1 evaluation.The supplied table captions identify WebQSP, CWQ, FreebaseQA, and GrailQA evaluation settings.
- Joint Decoding: DECAF executes generated logical forms and combines their ranked answer sets with directly generated answers using weighted score aggregation.If no logical form is executable, the directly generated answer is used; otherwise, answer sets receive scores based on rank and the hyperparameter λ.
4 EXPERIMENT
DECAF is evaluated across four KBQA benchmarks, retrieval settings, answer-combination strategies, model variants, and baseline ensembles. It achieves strong benchmark results, while ablations show that executed logical-form answers and direct answers are complementary and retrieval quality affects performance.
- Main Results: DECAF achieves new state-of-the-art results on WebQSP and FreebaseQA, with competitive performance on ComplexWebQuestions.On WebQSP, it improves the previous highest Hits@1 by 3.0% and F1 by 2.3%; on FreebaseQA, it improves SOTA Hits@1 by 15.7%.
- Main Results: DECAF outperforms published GrailQA baselines, including a 1.6% overall F1 improvement over T5-large RnG-KBQA on the dev set.With FiD-3B, overall F1 improves by 2.7% over FiD-large and exceeds TIARA; compositional-question F1 is 5.3% higher than the best-performing method.
- Ablation Study: The default combination selects the first executed answer when available and otherwise the first generated answer.The ablation sets λ = 1.0, assigning full weight to LF-executed answers in the combination function.
- Ablation Study: Combining LF-executed and generated answers significantly outperforms either strategy alone.On GrailQA dev, DECAF’s overall F1 is 6.3% higher than LF Only and 24.0% higher than Answer Only.
- Ablation Study: A shared reader performs similarly to separate readers, while baseline ensembles still underperform DECAF.DECAFsep uses separate readers for answer and logical-form generation; ArcaneQA and RnG-KBQA are also ensembled with DECAF or each other for comparison.
- Ablation Study: Increasing retrieved passages from 5 to 100 improves performance across all four datasets, although retrieval is weaker on ComplexWebQuestions.Retrieval achieves over 80% hits and recall on WebQSP, GrailQA, and FreebaseQA, while CWQ’s multi-hop questions remain more difficult.
5 CONCLUSION
DECAF jointly generates direct answers and logical forms for KBQA, combines generated and LF-executed answers, and uses retrieval over linearized KB text instead of entity linking. It achieves state-of-the-art results on three benchmarks and competitive results on another.
- DECAF jointly generates direct answers and logical forms, then combines generated and LF-executed answers into the final answer.
- DECAF retrieves relevant information from a text-transformed knowledge base using sparse or dense retrieval, without entity linking.
- DECAF achieves new state-of-the-art results on WebQSP, FreebaseQA, and GrailQA, with competitive results on ComplexWebQuestions.
6 ETHICS STATEMENT
The authors identify risks that DECAF may propagate bias or unfairness from knowledge bases and may behave unexpectedly under adversarial attacks. They therefore recommend caution in practice.
- Bias or fairness problems in a knowledge base may be propagated when DECAF answers questions from it.
- Adversarial attacks may alter DECAF's behavior and performance unexpectedly, motivating caution in practical use.
A.1 KB LINEARIZATION: HYPER-TRIPLET
Knowledge-base linearization represents ordinary triplets and hyper-triplets as text. Hyper-triplets use a central CVT node to express relations involving multiple entities.
- CVT nodes allow complicated relations involving more than two entities to be expressed in Freebase.
- Ordinary KB facts are linearized as triplets containing two entities and one relation.
- Hyper-triplets represent multiple entities and relations connected through a central CVT node.
A.2 ENTITY NAME DISAMBIGUATION
Entity-name disambiguation must preserve mappings between entity IDs and names while handling cases where one name refers to multiple entities. Aliases are treated separately from the primary name mapping.
- An entity ID can usually map to one entity name, with other names treated as aliases.
- A single entity name may map to multiple entity IDs because names can be ambiguous.
A.3 MULTI-ANSWER GENERATION
DECAF evaluates generating multiple answers instead of returning only the top answer because KBQA questions may have several correct answers. Although multi-answer generation improves Answer Only performance, it harms logical-form generation and does not improve overall performance.
- Increasing K in beam search degrades performance because questions contain different numbers of correct answers.
- Multi-answer generation improves Answer Only performance but reduces logical-form-generation performance.
- DECAF retains single-answer generation because multi-answer generation does not improve overall performance.The paper leaves multi-answer generation for future work.
A.4 DIRECT COMPARISON BETWEEN LF-EXECUTED AND GENERATED ANSWERS
The relative strengths of LF-executed and directly generated answers depend on question structure. Direct generation is more advantageous for zero-shot and increasingly relational questions, whereas logical-form generation is more advantageous when questions have more answers.
- Direct-answer generation is more advantageous for zero-shot questions than for I.I.D. or compositional questions.The reported advantages are 15.3% for zero-shot, 4.4% for I.I.D., and 6.7% for compositional questions.
- Direct-answer generation’s advantage increases from 7.3% to 18.8% to 29.1% as the number of relations increases.More relations make logical forms more complicated and harder to generate.
- Logical-form generation’s advantage increases from 14.8% to 82.7% as the number of ground-truth answers increases.The paper attributes this to logical-form difficulty being less correlated with answer count than direct-answer generation difficulty.
A.5 ADDITIONAL ABLATION STUDIES
Additional ablations examine answer combination, training data, beam size, and failure cases. They find that the simple combination strategy is effective, more data and larger beams help primarily logical-form generation, and complex questions can defeat both outputs.
- Answer combination: λ = 1 is optimal, selecting the top-1 LF-executed answer when executable and otherwise using the directly generated answer.
- Answer combination: Oracle combination improves over the original method by no more than 3 points, indicating that the original LF-first strategy is effective.
- Training data: DECAF performance improves as GrailQA training data increases from 500 to 44,337 examples.The study separately examines effects on answer generation and logical-form generation.
- Beam size: Increasing beam size from 1 to 20 improves overall performance mainly through logical-form generation, while Answer Only performance barely changes.Beam size matters especially for long logical forms whose candidates are enumerated until one is executable.
- Error analysis: Case analysis shows answer combination can recover from either answer-generation or logical-form-generation errors, but not when both outputs are wrong.The remaining failure involves a compositional question with multiple relations, including a missed join operation and an incorrect release-date relation.