Source-linked AI summary
Language to Logical Form with Neural Attention
Li Dong, Mirella Lapata
TL;DR
Semantic parsing requires mapping natural language to formal meaning representations, while many existing systems rely on task-specific resources and features. The paper uses an attention-enhanced recurrent encoder-decoder that generates logical forms as sequences or trees, and reports competitive performance across four datasets without domain- or representation-specific features.
Problem
Semantic parsing lacks a broadly adaptable approach because many existing systems rely on high-quality lexicons, templates, and domain- or representation-specific features.
Method
The model encodes utterances with recurrent neural networks and generates logical forms as sequences or trees conditioned on the encoded representations, using attention.
Results
The approach achieves similar or better performance than previous methods across datasets and meaning representations without hand-engineered domain- or representation-specific features.
Takeaways & Limitations
A common encoder-decoder framework can be used across datasets and meaning representations without modification.
Takeaways & Limitations
The attention model ignores alignment history, so some question words, especially in longer questions, may be ignored during decoding.
Abstract
from arXiv · showhide
Semantic parsing aims at mapping natural language to machine interpretable meaning representations. Traditional approaches rely on high-quality lexicons, manually-built templates, and linguistic features which are either domain- or representation-specific. In this paper we present a general method based on an attention-enhanced encoder-decoder model. We encode input utterances into vector representations, and generate their logical forms by conditioning the output sequences or trees on the encoding vectors. Experimental results on four datasets show that our approach performs competitively without using hand-engineered features and is easy to adapt across domains and meaning representations.
1 Introduction
The paper addresses semantic parsing with a general neural encoder-decoder approach that maps utterances to logical forms using attention. It aims to avoid domain- and representation-specific engineering while achieving competitive performance.
- Semantic parsing translates text into formal meaning representations such as logical forms or structured queries.
- Previous systems often depend on predefined templates and manually designed features, making them domain- or representation-specific.
- An attention layer learns soft alignments between input utterances and their logical forms.
- The model encodes utterances with recurrent neural networks and decodes logical forms as sequences or trees conditioned on the encodings.
- The approach achieves similar or better performance than previous methods across four datasets and meaning representations without hand-engineered domain- or representation-specific features.
2 Related Work
The paper situates its contribution at the intersection of semantic parsing and neural encoder-decoder modeling. It replaces much task-specific machinery with a general framework intended to transfer across domains and meaning representations.
- The paper synthesizes semantic parsing with encoder-decoder architectures based on neural networks.
- Semantic parsing research includes models that learn mappings from sentences paired with logical forms using diverse strategies.
- Unlike systems relying on high-quality lexicons, templates, and domain- or representation-specific features, the proposed method is designed for adaptation across settings.
- Prior neural work includes sequence-to-sequence mapping of navigational instructions to actions.
- The model targets well-defined representations such as Prolog and lambda calculus with a conceptually simpler architecture than one related alternative.
3 Problem Formulation
The paper formulates semantic parsing as mapping a natural-language input to a logical-form output with an encoder-decoder model. It develops sequence, tree, and attention-based decoding variants, trains them by likelihood maximization, and generates outputs with search.
- 3 Problem Formulation: The model maps natural-language input q to a logical-form representation a by encoding q and generating output tokens conditioned on the encoding.The conditional probability p(a|q) is decomposed over output tokens.
- 3.1 Sequence-to-Sequence Model: SEQ2SEQ uses separate recurrent encoder and decoder networks with LSTM units, processing input and output tokens sequentially.The encoder processes the first |q| time steps, and the decoder processes the following |a| steps.
- 3.1 Sequence-to-Sequence Model: The decoder initializes its hidden states from encoded input tokens and uses the topmost decoder hidden vector to predict each output token.Sequences include start-of-sequence and end-of-sequence tokens, with generation terminating at </s>.
- 3.2 Sequence-to-Tree Model: SEQ2TREE addresses logical-form hierarchy by generating trees top-down and recursively decoding subtrees when it emits a nonterminal token.A parent-feeding connection supplies the parent nonterminal’s hidden vector to the recurrent decoder.
- 3.2 Sequence-to-Tree Model: In the tree decoder, tokens at one depth are generated as a sequence, then child sequences are generated conditioned on the corresponding nonterminal’s hidden vector.For the example “A B (C)”, the first sequence produces y1 ··· y4 and the subtree produces y5, y6.
- 3.3 Attention Mechanism: Attention computes a context vector as a weighted sum of encoder hidden vectors and uses it to predict the current decoder output.Attention scores compare the current decoder hidden state with encoder hidden states.
- 3.4 Model Training: Training maximizes the likelihood of generated logical forms over natural-language–logical-form pairs, using RMSProp and dropout regularization.Dropout is applied between LSTM layers and before softmax classifiers to reduce overfitting, especially on small datasets.
- 3.5 Inference: At test time, the model predicts a logical form by decomposing its probability and generating tokens incrementally with greedy or beam search.Both decoders have time complexity O(|a|); SEQ2TREE additionally maintains a nonterminal queue.
4 Experiments
Experiments evaluate the model across four datasets, domains, and meaning representations, comparing full and ablated variants with prior systems. Results show that hierarchical decoding, attention, and argument identification contribute to performance, while data quality and coverage constrain some gains.
- Datasets: Experiments cover four datasets spanning job listings, U.S. geography, flight booking, and IFTTT automation, with varied meaning representations.The datasets use Prolog-style queries, lambda-calculus expressions, and recipe programs.
- Settings: The evaluation compares SEQ2SEQ and SEQ2TREE with variants removing attention or argument identification, using parsing accuracy as the main metric.Accuracy is the proportion of inputs mapped to gold logical forms, except for systems that output answers directly.
- Results: SEQ2TREE outperforms SEQ2SEQ by 2.9% on JOBS and 2.5% on GEO, where logical forms contain nested structures.The advantage is smaller on ATIS, which lacks complex nested structures.
- Results: Attention substantially improves performance on all three datasets, while argument identification is especially important for small-scale datasets with rare entities.About 92% of city names occur fewer than four times in GEO training data, making reliable parameter estimation difficult.
- Results: Across datasets and meaning representations, the models achieve comparable or better performance than previous systems without domain- or representation-specific hand-engineered features.SEQ2TREE ranks second on JOBS and ties for first with ZC07 on ATIS.
- Results: On IFTTT, the improvement from the argument identification method is smaller, likely because user-curated descriptions align loosely with their corresponding abstract syntax trees.The authors attribute this pattern to low-quality descriptions in the dataset.
5 Conclusions
The paper presents an encoder-decoder neural model that maps natural-language utterances to logical forms as sequences or trees. Its hierarchical tree decoder and attention mechanism improve performance, while comparisons show competitive results without domain- or representation-specific features.
- The authors present an encoder-decoder neural network that maps natural-language descriptions to meaning representations.The model encodes utterances into vectors and generates corresponding logical forms as sequences or trees using recurrent neural networks with long short-term memory units.
- A hierarchical tree decoder and attention mechanism improve performance across the evaluated settings.
- The approach performs competitively without domain- or representation-specific features.The conclusion reports extensive comparisons with previous methods.