Source-linked AI summary
SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Bases
Yawei Sun, Lingling Zhang, Gong Cheng, Yuzhong Qu
TL;DR
Complex KBQA questions are difficult because dependency errors and grounding can produce inaccurate expressive formal queries. SPARQA introduces skeleton-based parsing and combines sentence- and word-level scoring, achieving state-of-the-art performance on GraphQuestions while both scorers contribute on ComplexWebQuestions.
Problem
Complex questions with multiple predicates challenge semantic parsing because dependency errors propagate downstream and grounding can change query structure.
Method
SPARQA uses a lightweight skeleton grammar with BERT-based parsing, generates structural query variants, and ranks grounded queries with sentence- and word-level scorers.
Results
SPARQA achieved a new state-of-the-art GraphQuestions result, improving F1 from 20.40 to 21.53, while removing either scorer reduced ComplexWebQuestions P@1.
Takeaways & Limitations
Skeleton-based parsing improves complex-question semantic parsing, and combining sentence- and word-level scoring improves grounded-query accuracy.
Takeaways & Limitations
The approach does not focus on aggregate questions, and structural heterogeneity remains an area for future graph-based methods.
Abstract
from arXiv · showhide
Semantic parsing transforms a natural language question into a formal query over a knowledge base. Many existing methods rely on syntactic parsing like dependencies. However, the accuracy of producing such expressive formalisms is not satisfying on long complex questions. In this paper, we propose a novel skeleton grammar to represent the high-level structure of a complex question. This dedicated coarse-grained formalism with a BERT-based parsing algorithm helps to improve the accuracy of the downstream fine-grained semantic parsing. Besides, to align the structure of a question with the structure of a knowledge base, our multi-strategy method combines sentence-level and word-level semantics. Our approach shows promising performance on several datasets.
1 Introduction
SPARQA addresses complex KBQA questions by replacing fragile dependency-driven parsing with skeleton-based structure and multi-strategy query ranking. It introduces a skeleton grammar, annotated training resource, and sentence- plus word-level scoring.
- Motivation: Complex KBQA questions require formal queries with multiple predicates, but dependency errors can propagate into downstream semantic parsing.The paper contrasts these questions with simpler single-predicate queries and illustrates missed long-distance dependencies.
- Motivation: Grounding can change query structure, such as adding a mediator node that increases two ungrounded predicates to three grounded predicates.The example concerns Freebase’s representation of actor-film-character n-ary relations.
- Approach: SPARQA identifies a high-level question skeleton before generating and grounding ungrounded-query variants, then executes the top-ranked formal query.The pipeline uses the skeleton to support ungrounded-query generation and ranks grounded candidates before execution.
- Contributions: The proposed skeleton grammar is a lightweight formalism whose parsing algorithm improves downstream semantic-parsing accuracy.It represents the high-level structure of complex questions rather than directly encoding the full fine-grained query.
- Contributions: The authors manually annotate skeleton structures for over 10K questions across two KBQA datasets and release the resource for future research.The annotations support both training and evaluation of skeleton parsing.
- Contributions: SPARQA combines sentence-level pattern matching with word-level similarity scoring to rank grounded queries.Sentence-level scoring mines and matches patterns, while word-level scoring processes bags of words with a neural model.
2 Overview of the Approach
SPARQA transforms questions into skeleton-guided ungrounded queries, expands them to address grounding heterogeneity, and ranks grounded candidates for execution.
- Skeleton-guided parsing: SPARQA uses skeleton grammar to represent complex-question structure as relations between text spans before downstream relation extraction.Standard dependency parses of the spans are joined and passed to NFF for ungrounded-query construction.
- Grounding: The system generates structural query variants by contracting class-node edges or subdividing edges with mediator nodes.Each variant is grounded by linking nodes to KB entities, classes, or literals and enumerating connecting predicates.
- Ranking: Candidate grounded queries are ranked using sentence-level and word-level scorers before the highest-ranked query is executed.This combines structural pattern evidence with lexical similarity in the overall pipeline.
3 Skeleton Parsing
Skeleton parsing builds a coarse tree over text spans, using BERT-based procedures to split spans, identify headwords, and classify attachment relations. The resulting structure supports more accurate fine-grained dependency parsing.
- Skeleton grammar: The skeleton grammar is a selected dependency-grammar subset designed to provide accurate coarse-grained parsing for complex questions.Its tree structure supports iterative removal of leaf spans while retaining maximal well-formed sentences.
- Skeleton grammar: A skeleton is a directed tree whose text-span nodes are connected by attachment relations from headwords in other spans.The grammar permits clause, noun-phrase, verb-phrase, and prepositional-phrase span types, while span typing is optional for the parser.
- Parsing algorithm: Algorithm 1 starts with the full sentence as a root, repeatedly splits a text span, finds its headword, classifies the attachment relation, and returns the completed tree.The loop stops when the remaining question no longer needs splitting.
- Parsing pipeline: The two-stage design joins dependency parses of simple spans into a full dependency tree, which is expected to improve downstream semantic parsing accuracy.The running example splits “named Tom Vaughan” and “that Miley Cyrus acted in” before stopping at the remaining simple sentence.
- BERT-based procedures: The parser uses BERT-based QA for text-span prediction and headword identification, and sentence-pair classification for attachment-relation prediction.Attachment classification selects among seven predefined relations; split decisions use single-sentence classification.
4 Multi-Strategy Scoring
SPARQA ranks candidate grounded queries by combining sentence-level pattern matching with a word-level neural scorer. The sentence-level method transfers query patterns from similar training questions, while the word-level method scores predicate-related word alignment.
- 4 Multi-Strategy Scoring: Candidate grounded queries are ranked by combining sentence-level and word-level scorer outputs.The two scores are combined only after each scorer evaluates the candidates.
- Sentence-Level Scorer: The sentence-level scorer matches question and formal-query patterns after replacing entity mentions with dummy tokens and placeholders.It mines question/query patterns from training data and compares them across examples.
- Sentence-Level Scorer: For a test question, the sentence-level scorer retrieves a training question with the same dummy-token count and transfers its entity-aligned query pattern.The transferred grounded query receives score 1.0 when it retrieves non-empty results, whereas other candidates receive 0.0.
- Word-Level Scorer: The word-level scorer represents questions and formal queries as bags of non-entity, non-stop words, emphasizing predicate-related terms.It uses a neural model to score each candidate grounded formal query.
- Word-Level Scorer: The word-level model forms an n × m GloVe cosine-similarity matrix, takes row- and column-wise maxima, and feeds the resulting vectors through linear layers to produce a score.Training uses underlying question/query pairs as positives and random combinations as negatives.
5 Experiments
SPARQA is evaluated on two complex-question KBQA datasets through benchmark comparisons, ablations, intrinsic parser evaluation, and error analysis. Results show gains from skeleton parsing and combining sentence-level with word-level scoring, alongside identifiable failure modes.
- Overall Results: F1 rose from 20.40 to 21.53 on GraphQuestions, establishing SPARQA as a new state-of-the-art result over known baselines.The increase was 5.5%; on 1,172 simple questions, SPARQA remained comparable with PARA4QA, scoring 27.68 versus 27.42.
- Overall Results: SPARQA outperformed most ComplexWebQuestions baselines, while two stronger systems used additional training examples or external search snippets.SPLITQA + data augmentation used 28,674 additional examples, and PullNet used search engine snippets unavailable to SPARQA.
- Ablation Study: Removing skeleton parsing reduced ComplexWebQuestions P@1 from 31.57 to 29.39, a 6.9% decrease.The ablation replaces skeleton-generated dependencies with Stanford CoreNLP dependencies before downstream query generation.
- Ablation Study: Removing the sentence-level or word-level scorer reduced P@1 from 31.57 to 26.45 or 26.11, respectively.The decreases were 16.2% without sentence-level scoring and 17.3% without word-level scoring, supporting the combined multi-strategy design.
- Accuracy of Skeleton Parsing: Skeleton parsing reached an overall LAS of 93.73, while each of its four components exceeded 97% accuracy.The intrinsic evaluation used manually annotated gold structures for 1,000 ComplexWebQuestions test questions.
6 Related Work
Prior KBQA research includes rule-based, transition-based, and neural semantic parsing, alongside methods for mapping questions to predicates using lexical, syntactic, and neural similarity signals.
- Semantic parsing research spans rule-based, transition-based, and neural encoder-decoder approaches for producing formal queries.
- Predicate mapping is a key KBQA step that matches questions with formal predicates.
- Mapping methods have progressed from lexicon and syntactic features to neural question–predicate similarity at character or word granularity.
7 Conclusion and Future Work
SPARQA’s skeleton parsing supports complex-question processing and can complement dependency-based KBQA, while the authors identify node linking, structural heterogeneity, and aggregate questions as limitations for future work.
- Skeleton parsing derives more accurate dependencies that benefit downstream fine-grained semantic parsing and can combine with other dependency-based KBQA methods.
- The approach’s major limitations are node recognition and linking, structural heterogeneity, and aggregate questions, which remain targets for future work.
- Future extensions include graph-based methods for structural heterogeneity and learned templates for aggregate questions.