PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Semantic Parsing

Aishwarya Kamath, Rajarshi Das

arXiv:1812.00978v3cs.CL

TL;DR

Semantic parsing addresses the challenge of translating natural-language utterances into executable meaning representations. This survey introduces the framework and synthesizes methods spanning rule-based systems, statistical and alternative supervision, and neural approaches. It identifies confidence estimation and performance across disparate domains with limited labeled data as directions for further work.

  • Problem

    Semantic parsing must map utterances to structured meaning representations that can execute in an environment, while learning such mappings can require complex annotated data.

    Source: §1 · p. 1

  • Method

    The survey introduces semantic parsing components and compares approaches across supervision levels, training strategies, feature engineering, and structural constraints.

    Source: §1 · p. 1

  • Results

    The survey traces the field from rule-based systems through statistical and denotation-supervised methods to neural approaches to program synthesis.

    Source: §1 · p. 1, §3.2 · p. 5, §4 · p. 6

  • Takeaways & Limitations

    The survey identifies confidence estimation and cross-domain performance with limited labeled data as promising research directions.

    Source: §8 · p. 15

    Performance across multiple disparate domains, especially with insufficient labeled data, remains a direction for further work.

    Source: §8 · p. 15

Abstract

from arXiv · show

A significant amount of information in today's world is stored in structured and semi-structured knowledge bases. Efficient and simple methods to query them are essential and must not be restricted to only those who have expertise in formal query languages. The field of semantic parsing deals with converting natural language utterances to logical forms that can be easily executed on a knowledge base. In this survey, we examine the various components of a semantic parsing system and discuss prominent work ranging from the initial rule based methods to the current neural approaches to program synthesis. We also discuss methods that operate using varying levels of supervision and highlight the key challenges involved in the learning of such systems.

1. Introduction

Semantic parsing maps natural-language utterances to meaning representations that support automated reasoning and execution in environments such as knowledge bases. The survey introduces the framework and traces its development while comparing approaches by supervision, training strategy, feature engineering, and structural constraints.

1. Introduction

  • Semantic parsing maps utterances to meaning representations that go beyond identifying roles and objects, enabling automated reasoning and execution in varied environments.Examples include database queries, robotic navigation commands, and queries for conversational agents.Source: §1 · p. 1
  • Unlike translation and language generation, semantic parsing predicts tree-like structured objects that must satisfy constraints to execute in a given environment.Source: §1 · p. 1
  • The survey introduces semantic parsing components and traces the field’s development, contrasting supervision, training strategies, feature engineering, and structural constraints.Source: §1 · pp. 1–2

2. Task and Salient Components

A semantic parsing system maps an utterance to a meaning representation, scores candidate forms, selects a high-scoring form, and uses an executor to produce an output in a context. Its representation language, grammar, and context shape what the system can express and process.

2. Task and Salient Components

  • A parser maps a natural-language query to a valid SQL program and executes it against a table to check whether it returns the correct answer.The framework includes a representation language or grammar, a model, a parser, a learning algorithm, and an execution environment.Source: §2 · p. 2

2.1 Language for Meaning Representation

  • Meaning representations include logic-based formalisms, graph-based forms, and programming languages, each offering different representational capabilities and applications.First-order logic cannot express set operations such as counting without augmentation; graph forms can be human-readable, abstract from syntax, and leverage graph algorithms.Source: §2.1 · p. 3

2.2 Grammar

  • A grammar defines candidate derivations, and its type affects both parser expressivity and computational complexity.Strict grammars can reduce candidate derivations for structured queries, while floating rules handle noisy utterances at the cost of more candidates.Source: §2.2 · pp. 3–4

2.3 Underlying context

  • The mapping from an utterance to a semantic representation is defined relative to an environment, such as a knowledge base, table, spreadsheet, image, or task-specific application.Source: §2.3 · p. 4

3. Early attempts

Early systems relied on hand-built rules, while later systems learned statistical mappings from examples. Statistical methods improved flexibility and data use, but complex annotation remained difficult to scale and limited systems to narrow domains.

3.1 Rule based systems

  • Rule-based systems used patterns or syntactic and semantic grammars to map utterances to queries, but domain-specific semantic knowledge made them difficult to adapt across domains.Pattern matching could be brittle, while syntactic information could be reused more readily across domains.Source: §3.1 · p. 4

3.2 Rise of statistical techniques

  • Zelle and Mooney [1996] trained a shift-reduce parser on sentence-query pairs and evaluated it by executing queries against a database.Thompson and Mooney [2003] learned phrase-to-meaning lexicons that performed comparably to hand-built lexicons and supported lexicons and parsers for multiple languages.Source: §3.2 · p. 5
  • Fully supervised statistical systems learned from sentence–logical-form pairs and used probabilistic CCG models to rank candidate parses.Zettlemoyer and Collins [2005] used a log-linear model with features and parameters to select likely parses.Source: §3.2 · p. 5
  • Factored lexicons decomposed entries into lexemes and lexical templates, sharing information across related entries to ease data sparsity and reduce the data needed to learn.Subsequent CCG work mostly adopted this approach.Source: §3.2 · p. 6
  • Complex annotation made supervised systems hard to scale and restricted them to narrow domains, motivating exploration of other forms of supervision.Source: §3.2 · p. 6

4. Semantic parsing from denotations

Denotation supervision trains semantic parsers from execution results without access to the logical form itself. The survey describes methods that address the resulting search and ambiguity challenges across knowledge bases, tables, and robot navigation.

4. Semantic parsing from denotations

  • Denotation supervision trains on the result of executing a program rather than access to the program, but faces large program search spaces and spurious programs that yield correct answers by chance.Source: §4 · p. 6
  • For Freebase question-answer pairs, Berant et al. [2013] first align phrases with predicates by shared entity co-occurrence, then use predicate neighbors to bridge weak or implicit relations.Source: §4 · pp. 6–7
  • Kwiatkowski et al. [2013] map utterances to domain-independent abstract forms with CCG, then match them to an ontology using target-domain knowledge and lexical similarity.Their model learns over parsing and ontology-matching derivations using final answers as supervision.Source: §4 · p. 7
  • For complex questions on disjoint train and test tables, Pasupat and Liang [2015] build a table-specific knowledge graph and rank candidate parses with a log-linear model.Beam search, type and denotation constraints, and pruning address the exponentially growing logical-form search space.Source: §4 · p. 7
  • For robot navigation, Artzi and Zettlemoyer [2013] use grounded CCG that incorporates logical-form execution in each state to learn while carrying out instructions.Source: §4 · p. 7

5. Alternate forms of supervision

The survey traces alternative supervision sources and training strategies for semantic parsing, from syntax- and database-based learning to conversations, videos, generated paraphrases, translation, and user feedback.

5. Alternate forms of supervision

  • Krishnamurthy and Mitchell [2012] use syntactic and semantic knowledge, without sentence-level annotations, to generate logical forms with 56% recall.Their method builds a lexicon from dependency paths and knowledge-base relations.Source: §5 · p. 7
  • Poon and Domingos [2009] learn logical forms unsupervised, while Poon [2013] grounds learning in a database schema to constrain executable parses.The earlier approach struggles to match induced clusters to existing ontologies; database grounding addresses more complex questions.Source: §5 · p. 8
  • Artzi and Zettlemoyer [2011] use conversational feedback to learn meaning representations from unannotated logs, while Ross et al. [2018] learn a semantic parser from captioned videos using visual-validation supervision.Source: §5 · p. 8
  • Wang et al. [2015] generate canonical utterance–logical-form pairs from a database schema, then use crowd paraphrases to create training data.The reverse construction reflects the grammar's logical functionality in the dataset.Source: §5 · pp. 8–9
  • Wong and Mooney [2006] and Andreas et al. [2013] treat semantic parsing as translation, with the latter reporting state-of-the-art-comparable GeoQuery results.The approaches align natural-language input with logical forms, which Andreas et al. linearize for decoding.Source: §5 · p. 9
  • Iyer et al. [2017] use binary user feedback to improve a neural semantic parser, while Lawrence and Riezler [2018] use interaction logs and a form of counterfactual learning.Source: §5 · p. 9

6. Enter Seq2Seq

The survey describes neural sequence-to-sequence parsing and methods that add intermediate representations or grammar constraints. It also reviews code-generation systems that model programming-language syntax during decoding.

6. Enter Seq2Seq

  • Seq2seq parsers map utterances directly to meaning representations, reducing feature and lexicon engineering but generally leveraging logic compositionality less effectively than traditional approaches.Tree-aware decoding generates matching parentheses and subtrees; attention-based copying and data recombination address rare words and support generalization.Source: §6 · p. 10

6.1 Employing Intermediate Representations

  • Intermediate representations can share structure across examples and narrow program search, while Goldman et al. [2017] use spatial-semantic clusters to generate training examples.Abstract representations may generalize across small datasets and reduce spurious programs; sketches also support higher-level decoding context.Source: §6.1 · p. 11

6.2 Decoding using a constrained grammar

  • Yin and Neubig [2017] constrain decoding with programming-language syntax, and Krishnamurthy et al. [2017] show that type constraints outperform unconstrained Seq2Tree.The latter also use entity-linking features to handle rare entities.Source: §6.2 · pp. 11–12

6.3 General Purpose Code Generation

  • Code-generation datasets align natural-language descriptions with programs, and later decoders model programming-language syntax to guarantee well-formed code.Rabinovich et al. [2017] compose decoder modules to generate abstract syntax trees top-down, while earlier syntax-agnostic models could produce ill-formed code.Source: §6.3 · pp. 12–13

7. Learning

The survey reviews maximum marginal likelihood, reinforcement learning, and maximum-margin reward methods for training semantic parsers. These approaches differ in their supervision signals, search procedures, and treatment of candidate programs.

7.1 Maximum marginal likelihood (MML)

  • In maximum marginal likelihood, fully supervised training marginalizes over derivations of the annotated logical form, while weak supervision scores programs by whether execution matches the observed denotation.The weakly supervised objective retains programs that produce the correct denotation.Source: §7.1 · p. 13
  • MML search commonly uses beam search; offline dynamic programming on denotations collapses programs with the same result and recovers consistent forms up to a size bound.Online search may require wide beams and offers no guarantee that all denotation-consistent forms will be included.Source: §7.1 · pp. 13–14

7.2 Reinforcement learning

  • Reinforcement learning trains a stochastic policy to maximize expected reward, but sampling from large program spaces is difficult and policy-based exploration can reinforce spurious candidates.Randomized beam search and memory buffers are described as ways to broaden exploration; the buffer method outperforms prior state of the art on WikiTableQuestions.Source: §7.2 · p. 14

7.3 Maximum margin reward methods (MMR)

  • Maximum-margin reward methods compare the highest-scoring correct program with margin-violating alternatives and update the scores of the best program and most violating program.Misra et al. [2018] relate these training paradigms and provide a generalized update equation.Source: §7.3 · p. 15

8. Future directions

The survey identifies open directions for semantic parsing, including uncertainty handling, cross-domain performance, and smarter use of structure. It also points to applications of logical forms in natural language understanding and machine translation evaluation.

8. Future directions

  • Confidence estimation could help conversational agents recognize uncertain predictions and involve people in query refinement.The survey connects this direction to improving user experience.Source: §8 · p. 15
  • Cross-domain parsers that perform well across disparate domains with little labeled data remain a promising direction.The survey suggests multi-task learning, co-training, and fine-tuning across datasets to leverage shared representations and generalize to new domains.Source: §8 · p. 15
  • Using known structure more effectively may reduce the representational burden on models and help them generalize.Source: §8 · p. 15
  • Logical forms could be explored in natural language understanding studies such as recognizing textual entailment, either within models or for evaluation.Source: §8 · p. 15
  • Semantic parsing could supplement surface-form metrics such as BLEU in machine translation evaluation, including by checking whether translated queries yield the same denotation.The survey also proposes checking whether translations produce semantically meaningful sentences.Source: §8 · p. 15

Related papers

Loading 1812.00978v3…