Source-linked AI summary
Leveraging Abstract Meaning Representation for Knowledge Base Question Answering
Pavan Kapanipathi, Ibrahim Abdelaziz, Srinivas Ravishankar, Salim Roukos, Alexander Gray, Ramon Astudillo, Maria Chang, Cristina Cornelio, Saswati Dana, Achille Fokoue, Dinesh Garg, Alfio Gliozzo, Sairam Gurajada, Hima Karanam, Naweed Khan, Dinesh Khandelwal, Young-Suk Lee, Yunyao Li, Francois Luus, Ndivhuwo Makondo, Nandana Mihindukulasooriya, Tahira Naseem, Sumit Neelam, Lucian Popa, Revanth Reddy, Ryan Riegel, Gaetano Rossiello, Udit Sharma, G P Shrivatsa Bhargav, Mo Yu
TL;DR
KBQA systems face challenges understanding complex questions, reasoning over knowledge bases, and obtaining large end-to-end training datasets. NSQA uses AMR parses, graph transformation, and a modular pipeline to produce KB-aligned logical queries and answers. It achieves state-of-the-art performance on QALD-9 and LC-QuAD 1.0, and the analysis identifies benefits of AMR for KBQA.
Problem
KBQA approaches must handle complex question understanding and reasoning, while end-to-end text-to-query systems require paired training data that is difficult to obtain at scale.
Source: §1 · p. 1Method
NSQA parses questions into AMR, transforms parses into candidate KB-aligned logical queries, and uses modular task-specific components with a neuro-symbolic reasoner.
Source: §1 · pp. 1–2Results
NSQA achieves state-of-the-art performance on QALD-9 and LC-QuAD 1.0, two prominent DBpedia-based KBQA datasets.
Source: §1 · p. 2Takeaways & Limitations
AMR offers a normalized question representation that supports resilience to subtle same-meaning changes and handles complex structures such as multi-hop questions.
Source: §5 · p. 9An AMR parse can fail to match the underlying knowledge graph when an extra intermediate variable adds a constraint to the query graph.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
Knowledge base question answering (KBQA)is an important task in Natural Language Processing. Existing approaches face significant challenges including complex question understanding, necessity for reasoning, and lack of large end-to-end training datasets. In this work, we propose Neuro-Symbolic Question Answering (NSQA), a modular KBQA system, that leverages (1) Abstract Meaning Representation (AMR) parses for task-independent question understanding; (2) a simple yet effective graph transformation approach to convert AMR parses into candidate logical queries that are aligned to the KB; (3) a pipeline-based approach which integrates multiple, reusable modules that are trained specifically for their individual tasks (semantic parser, entity andrelationship linkers, and neuro-symbolic reasoner) and do not require end-to-end training data. NSQA achieves state-of-the-art performance on two prominent KBQA datasets based on DBpedia (QALD-9 and LC-QuAD1.0). Furthermore, our analysis emphasizes that AMR is a powerful tool for KBQA systems.
1 Introduction
KBQA must map natural-language questions to structured queries over knowledge-base facts, while complex questions and limited end-to-end training data challenge existing approaches. NSQA uses AMR and modular query construction, and is reported to achieve state-of-the-art performance on two DBpedia datasets.
1 Introduction
- Structured KB queries can contain multiple triples representing steps needed to obtain an answer, making intermediate symbolic representations transparent and self-explanatory.Source: §1 · p. 1
- Existing KBQA approaches face complex question understanding and reasoning demands, while end-to-end text-to-query systems require paired examples that are challenging to label at scale.Pipeline approaches can delegate language understanding to semantic parsers, though they are considered vulnerable to error propagation.Source: §1 · p. 1
- NSQA achieves state-of-the-art performance on QALD-9 and LC-QuAD 1.0, two prominent DBpedia-based KBQA datasets.Source: §1 · p. 2
- A path-based graph transformation converts AMR parses into candidate logical queries aligned to the knowledge base.The intermediate logic form facilitates neuro-symbolic reasoning.Source: §1 · p. 2
- The modular pipeline trains reusable modules for individual tasks, including semantic parsing, entity linking, and relation linking, without end-to-end training data.Source: §1 · pp. 1–2
2 Approach Overview
NSQA turns natural-language questions into AMR graphs, transforms them into KB-aligned query graphs, and uses a reasoner to produce answers. Its modular pipeline separates question parsing, graph construction and logic generation, and reasoning.
2 Approach Overview
- NSQA parses a question into an AMR graph, transforms it into candidate KB-aligned logical queries, and uses a Logical Neural Network to reason over KB facts.The pipeline delegates question understanding to AMR parsing and uses separate modules for graph transformation and reasoning.Source: §2 · p. 2
2.1 AMR Parsing
- AMR represents concepts and their relations in a rooted, directed, acyclic graph, with a special amr-unknown node marking the missing concept sought by the question.AMR parsing also helps identify intermediate variables that behave as secondary unknowns.Source: §2.1 · pp. 2–3
2.2 AMR to KG Logic
- AMR-to-query transformation must reconcile n-ary PropBank predicates with binary KG relations and differences in graph structure, granularity, and vocabulary.Multiple AMR nodes and edges can jointly express one KG relation; AMR nodes may correspond to either entities or relationships.Source: §2.2 · p. 3
2.2.1 Path-based Graph Transformation
- The path-based method finds shortest paths from amr-unknown to KG-linked entities, retaining relevant edges and collapsing predicate nodes and adjacent labels into query-graph relations.Relation linking maps AMR-derived labels to KG relations and ranks candidate query graphs; NSQA selects the highest-ranked valid graph.Source: §2.2.1 · pp. 3–4, Figure 2 · p. 4
2.2.2 Logic Generation
- The query graph becomes an existential first-order logic form, enabling use of different FOL reasoners and supporting constructs such as SELECT, COUNT, ASK, and SORT.Rules derive query type, target variable, sorting, and counting from AMR features and the linked KG relation's range.Source: §2.2.2 · p. 5
2.3 Reasoner
- NSQA uses Logical Neural Networks for type-based and geographic reasoning, including type-hierarchy checks and manually added transitive axioms for dbo:birthPlace.The authors state that the intermediate logic and reasoning module allow NSQA to be extended to complex reasoning in future work.Source: §2.3 · p. 5
3 Experimental Evaluation
NSQA is evaluated on two DBpedia KBQA datasets, with state-of-the-art results reported on both. The analysis examines AMR-based query construction and supported question types, alongside evaluations of individual modules.
3.1 Datasets and Metrics
- The evaluation uses QALD-9 and LC-QuAD 1.0 on DBpedia, reporting precision, recall, and F-score for KBQA and modules and Smatch for AMR parsing.The individual-module development sets contain 98 QALD-9 and 200 LC-QuAD 1.0 questions.Source: §3.1 · p. 6
3.2 End-to-end Evaluation
- NSQA achieves state-of-the-art performance on QALD-9 and LC-QuAD 1.0, outperforming WDAqua and gAnswer on QALD-9 and QAmp by 11.45 percentage points on LC-QuAD 1.0 F1.Table 1 reports the end-to-end comparison.Source: §3.2 · p. 6, Table 1 · p. 6
- Liang et al.’s reported 68% F1 is excluded from the comparison because it uses a different subset and includes only questions returning an answer.On the common evaluation setup, its F1 reduces to 29.2%, but the authors still exclude it for lack of standard evaluation.Source: §3.3 · p. 6, §3.2 · p. 6
3.3 Performance Analysis of AMR
- On LC-QuAD 1.0 dev, AMR-based rules identify 64% of ASK questions correctly and exceed 80% accuracy for COUNT and SELECT, with constraint-count accuracies of 79% for single-hop and 70% for two-hop questions.The analysis reports that complex-question query structure is found almost as often as simple-question structure, independent of the knowledge graph.Source: Table 3 · p. 6, §3.3 · p. 7
- Figure 3 illustrates two intuitive AMR-to-query transformations and a failure in which the extra intermediate variable “side” creates an unmatched query constraint.The AMR parsing evaluation uses manually created train and development AMRs and a parser trained on human-annotated treebanks plus a synthetic corpus.Source: §3.3 · pp. 6–7, Figure 3 · p. 7
- The AMR-to-query transformation supports simple, multi-relational, count-based, and superlative questions; comparative and temporal reasoning remain future work.The LNN reasoner performs geographic and type-based reasoning to rank candidate logic forms.Source: §3.3 · p. 7
3.4 Individual Module Evaluation
- On LC-QuAD 1.0 dev, NMD+BLINK performs 24% better than Falcon on F1, while SemREL scores 0.55 F1 versus 0.43 for SLING and 0.42 for Falcon; on QALD-9, SemREL scores 0.54 versus 0.64 for SLING and 0.46 for Falcon.With LNN, NSQA scores 40.5 F1 on LC-QuAD 1.0 dev, compared with 37.6 using deterministic SPARQL translation.Source: §3.4 · p. 7
4 Related Work
Prior KBQA work spans parsing, end-to-end neural methods, graph-driven approaches, and modular systems. The paper positions NSQA against approaches whose question understanding or training requirements differ from its KG-independent AMR-based design.
Deep Learning Models
- End-to-end neural approaches translate questions into KG subgraphs, using embedding-based or hard-decision methods; some hard-decision systems evaluate sketches with gold entities while omitting target-variable or aggregation selection.Template-based approaches also report gains on these datasets, but their manually predefined templates make comparisons with generic, non-template systems inappropriate.Source: §4 · p. 8
- Earlier KBQA work mainly parsed questions into KB queries with hand-crafted features, while information-extraction exceptions relied directly on relation detection.The passage characterizes the performance of these earlier approaches as usually limited.Source: §4 · p. 8
- Template-based question-subgraph generation reports improved performance on QALD and LC-QuAD, but relies on manually created templates and is not a common comparison ground for generic systems.The passage names NSQA, WDAqua, and QAmp as generic, non-template approaches.Source: §4 · p. 8
Graph Driven Approaches
- Graph-driven systems include rule-based WDAqua, dependency-parse graph matching in gAnswer, and QAmp message passing over KG subgraphs; their subgraph learning is heavily KG-specific.The paper contrasts this with delegating question understanding to KG-independent AMR parsing.Source: §4 · p. 8
Modular Approaches
- Frankenstein emphasizes reusable components but does not use KG-independent AMR parsing or report results comparable to state-of-the-art approaches.Liang et al. (2021) use phrase-mapping ensembles and a TreeLSTM ranker requiring task-specific training data.Source: §4 · p. 8
5 Discussion
The discussion highlights benefits of AMR for question understanding and identifies parser, representation-granularity, and training-objective challenges. It also reports a link between improved AMR Smatch and overall system performance on LC-QUAD-1 dev.
Benefits of AMR
- A 1.4% AMR Smatch improvement on LC-QUAD-1 dev coincided with a 1.2% improvement in overall system performance.The discussion also notes that AMR parsing advances can improve NSQA and that multilingual and domain-specific parsers may expand its application domains.Source: §5 · p. 9
- AMR normalizes questions with the same meaning despite subtle wording changes and transparently represents complex structures such as multi-hop questions and imperative statements.The discussion also describes AMR as enabling independent parser advances to benefit overall system performance.Source: §5 · p. 9
Challenges
- AMR errors can propagate through NSQA and cause incorrect answers; granularity differences between AMR and SPARQL may require domain-specific adjustments to the generic path-based transformation.The passage reports over 84% Smatch for a state-of-the-art AMR parser and 83% inter-annotator agreement on newswire sentences.Source: §5 · p. 9
- Smatch may be suboptimal for KBQA because it weights all edge-node triples equally, while NSQA needs a particular subset of paths extracted correctly.The authors are exploring alternative metrics and ways to incorporate them into model training.Source: §5 · p. 9
6 Conclusion and Future Work
The conclusion presents NSQA as a modular KBQA system that uses generic AMR parsing and path-based mapping to the knowledge graph. Future work includes exploring its intermediate logic form and extending reasoning capabilities.
6 Conclusion and Future Work
- NSQA is presented as the first system to successfully use a generic semantic parser, particularly AMR, for KBQA.Its path-based mapping from AMR to a knowledge graph is described as first of its kind, with promising results on compositional queries.Source: §6 · p. 9
- NSQA trains its modules separately for their own tasks, so it does not require end-to-end KBQA training.Source: §6 · p. 9
- The authors plan to explore the more expressive intermediate logic form with a neuro-symbolic reasoner and extend NSQA to temporal reasoning and knowledge-base incompleteness and inconsistency.Source: §6 · p. 9