Source-linked AI summary
Data Recombination for Neural Semantic Parsing
Robin Jia, Percy Liang
TL;DR
Semantic parsing requires crisp logical regularities that task-agnostic neural models may not capture. The paper induces a high-precision synchronous context-free grammar to recombine training fragments, then trains an attention-copying sequence-to-sequence RNN on generated examples. Data recombination improves accuracy across three datasets, including a 4.3-point GEO gain and state-of-the-art performance under comparable supervision.
Problem
RNNs make few domain-specific assumptions but lack the built-in awareness of logical compositionality that helps standard semantic parsers generalize.
Method
Data recombination induces a high-precision generative model, samples recombinant examples, and trains a sequence-to-sequence RNN with attention-based copying on them.
Results
Data recombination consistently improves accuracy across GEO, ATIS, and Overnight, including a 4.3 percentage point GEO improvement over the baseline RNN and 89.3% GEO test accuracy.
Takeaways & Limitations
Structured SCFG-generated examples can inject crisp compositional structure into an RNN while retaining a domain-general modeling approach.
Takeaways & Limitations
The system only explores learning from logical forms, cannot directly learn from denotations, and assumes conditional independence that phenomena such as anaphora can violate.
Abstract
from arXiv · showhide
Modeling crisp logical regularities is crucial in semantic parsing, making it difficult for neural models with no task-specific prior knowledge to achieve good results. In this paper, we introduce data recombination, a novel framework for injecting such prior knowledge into a model. From the training data, we induce a high-precision synchronous context-free grammar, which captures important conditional independence properties commonly found in semantic parsing. We then train a sequence-to-sequence recurrent network (RNN) model with a novel attention-based copying mechanism on datapoints sampled from this grammar, thereby teaching the model about these structural properties. Data recombination improves the accuracy of our RNN model on three semantic parsing datasets, leading to new state-of-the-art performance on the standard GeoQuery dataset for models with comparable supervision.
1 Introduction
Semantic parsing translates natural-language utterances into logical forms, but RNNs’ domain-general flexibility can hinder the logical-compositional generalization that standard parsers provide. Data recombination addresses this by inducing a high-precision SCFG, sampling recombinant examples, and training a sequence-to-sequence RNN.
- Semantic parsing translates natural-language utterances into logical forms for applications including question answering, instruction following, and regular expression generation.
- RNNs require few domain-specific assumptions and little feature engineering, but lack standard semantic parsers’ built-in awareness of logical compositionality.
- The system represents semantic parsing examples as tokenized input utterances and logical-form outputs for sequence-to-sequence modeling.
- Data recombination induces a high-precision synchronous context-free grammar from a dataset, samples recombinant examples, and uses them to train a sequence-to-sequence RNN.
- On GEO, data recombination improves test accuracy by 4.3 percentage points over the baseline RNN and yields state-of-the-art results without a seed lexicon for predicates.
2 Problem statement
The paper formulates semantic parsing as sequence-to-sequence prediction over tokenized utterances and logical forms, evaluating the approach on three established datasets. Its scope is limited to learning from logical forms rather than denotations.
- Semantic parsing is cast as sequence-to-sequence prediction from input words x_1, . . . , x_m to output logical-form tokens y_1, . . . , y_n.
- An RNN can predict tree-structured outputs in a linear token sequence, supporting the paper’s sequence-to-sequence formulation.
- The evaluation uses GeoQuery, ATIS, and Overnight, which pair natural-language queries with Prolog, lambda-calculus, or varied-domain logical forms.
- The system explores learning from logical forms only and cannot directly learn from denotations, though it could rerank candidate derivations from other systems.
3 Sequence-to-sequence RNN Model
The model encodes utterances with a bidirectional RNN and generates logical-form tokens with attention, while copying input words to handle entity names. Its decoder learns over both writing and copying actions.
- 3.1 Basic Model: A bidirectional RNN converts input words into context-sensitive embeddings for the attention-based decoder.The forward and backward RNNs process the input in opposite directions, and their hidden states form each input representation.
- 3.1 Basic Model: The decoder generates the output logical form one token at a time using its current state and attention over the input.Attention weights form a probability distribution over input words at each decoding step.
- 3.2 Attention-based Copying: The basic softmax model struggles with long-tail entity names that often appear directly in both input and output.For example, an input entity such as “iowa” can correspond directly to the output token iowa.
- 3.2 Attention-based Copying: Attention-based copying lets the decoder either write an output-vocabulary word or copy an input word directly.The copy probability is determined by the attention score on the selected input word.
- 3.2 Attention-based Copying: Training maximizes the output log-likelihood while marginalizing over the latent write-or-copy action sequence.The decoder chooses among all write and copy actions with a softmax, then maps each action deterministically to an output token.
4 Data Recombination
Data recombination induces a high-precision SCFG from training data to generate recombinant examples that inject task-relevant prior knowledge into a sequence-to-sequence model. The framework captures conditional independence through compositional grammar strategies and trains on newly sampled examples.
- Data recombination induces a high-precision generative model from training data, then samples recombinant examples to inject prior knowledge into a domain-general structured prediction model.The two-stage design separates declaring task properties from the model architecture.
- An SCFG represents aligned joint derivations of utterances and logical forms, defining a distribution over their pairs by uniformly sampling production rules.The grammar uses rules X →⟨α, β⟩ with aligned nonterminals on both sides.
- Grammar induction starts with one ROOT →⟨x, y⟩ rule per training pair and maps an input grammar to an output grammar, enabling symbolic strategy composition.Composed strategies can be applied directly to grammar rules without sampling intermediate grammars.
- Grammar induction strategies: ABSENTITIES replaces entity mentions with inferred types and adds rules mapping those types back to entities, supporting entity-level recombination.For example, an entity such as “texas” can be abstracted to its type and regenerated through a corresponding type-to-entity rule.
- Grammar induction strategies: ABSWHOLEPHRASES abstracts entities and whole expressions with inferred types, while also stripping question identifiers to produce more natural examples.It can create rules mapping a type to a phrase whose logical form evaluates to a set of that type.
- The framework relies on conditional independence of a semantically coherent phrase’s meaning from the rest of the sentence, though anaphora can violate this assumption.The authors state that this property holds in most existing semantic parsing datasets.
- Grammar induction strategies: CONCAT-k creates a ROOT rule expanding to k SENT symbols and converts each root-level input rule into a SENT rule, generating longer examples.Unlike abstraction strategies, concatenation is applicable to any sequence transduction problem but does not encode semantic-parser-specific independence properties.
5 Experiments
Experiments evaluate copying, data recombination, and longer examples across GEO, ATIS, OVERNIGHT, and artificial data. Copying and composed recombination generally improve results, while longer examples provide additional gains.
- Evaluation setup: The system is evaluated on GEO, ATIS, and OVERNIGHT, using denotation match for GEO and OVERNIGHT and logical-form exact match for ATIS.
- Copying mechanism: Copying improves test accuracy by 10.4 percentage points on GEO and 6.4 points on ATIS, but slightly hurts OVERNIGHT.The authors attribute the OVERNIGHT result partly to its small number of distinct entities.
- Main results: Data recombination consistently improves accuracy across all three domains, with the strongest results coming from composing multiple strategies.
- Main results: 89.3% test accuracy on GEO surpasses the previous state-of-the-art excluding Liang et al. (2011), which used a seed lexicon for predicates.
- Main results: 83.3 test accuracy on ATIS from ABSENTITIES composed with CONCAT-3 beats the baseline by 7 percentage points and is competitive with the state-of-the-art.
- Longer examples: Independent and recombinant examples both improve performance, while longer examples help more than same-length examples despite the short test distribution.In every domain, the best accuracy numbers involved some form of concatenation, and ABSENTITIES alone was generally less effective.
6 Discussion
The discussion presents data recombination as a way to combine structured prior knowledge with domain-general neural modeling. The authors argue that this framework may extend beyond semantic parsing.
- Framework: Data recombination generates new training examples from a high-precision generative model induced from the original training data.
- Framework: The framework improves a sequence-to-sequence RNN on three semantic parsing datasets using a synchronous context-free grammar as its generative model.
- Generalization: Some induced grammars generate examples outside the test distribution that nonetheless aid generalization.
- Broader perspective: The approach combines structured synchronous context-free grammars for crisp structure with RNNs suited to soft linguistic relationships.