Source-linked AI summary
Grammar as a Foreign Language
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, Geoffrey Hinton
TL;DR
Constituency parsing needs accurate methods that are less domain-specific, complex, and inefficient than traditional parsers. The paper applies an attention-enhanced sequence-to-sequence model with human-annotated and high-confidence synthetic data, matching standard parsers with little human data and reaching state-of-the-art performance with synthetic data. The approach also has a concrete scope boundary: malformed trees remain for rare sentences or fragments lacking proper punctuation.
Problem
Accurate constituency parsers are domain-specific, complex, and inefficient despite decades of research and engineering.
Method
The paper uses an attention-enhanced sequence-to-sequence model that encodes input words, decodes linearized parse trees, and trains on human-annotated or parser-agreement-selected synthetic data.
Results
92.5 F1: attention-trained parsing on a high-confidence synthetic corpus achieved a new state-of-the-art on WSJ section 23 without an ensemble.
Takeaways & Limitations
Attention models matched BerkeleyParser on a small human-annotated dataset, while imperfect synthetic labels substantially improved over the models that generated them.
Takeaways & Limitations
Malformed trees occurred for 0.8% of high-confidence-corpus development sentences, especially sentences or fragments without proper punctuation.
Abstract
from arXiv · showhide
Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.
1 Introduction
The paper recasts constituency parsing as a sequence-to-sequence task and evaluates generic models with standard and synthetic training data. Attention enables strong performance with limited human annotation, while high-confidence synthetic data yields a new state-of-the-art result.
- 1 Introduction: Constituency parsing is important but existing accurate parsers are domain-specific, computationally demanding, and difficult to transfer to other problems.Traditional parsers require cubic time in sentence length, while linear-time alternatives have not matched state-of-the-art accuracy.
- 1 Introduction: The authors formulate parsing as sequence-to-sequence prediction by linearizing parse trees and construct artificial training data using existing parsers.Early experiments label a large corpus with BerkeleyParser after poor performance on a small human-annotated dataset.
- 1 Introduction: 90.5 F1 with an ensemble: attention-trained models matched BerkeleyParser's 90.4 when trained only on the small human-annotated dataset.A single attention model achieved 88.3 F1, while the ensemble reached 90.5 without using an ensemble-trained BerkeleyParser baseline.
- 1 Introduction: 92.5 F1: training attention models on high-confidence synthetic trees produced a new state-of-the-art on WSJ section 23 without an ensemble.The high-confidence trees were selected using agreement between two parsers; an ensemble further reached 92.8.
2 LSTM+A Parsing Model
The LSTM+A parsing model encodes an input sentence and generates an invertible linearization of its parse tree, using attention over encoder states to guide output generation.
- 2 LSTM+A Parsing Model: The model uses separate LSTMs to encode input words and decode output symbols into a distribution over output sequences.The decoder’s output distribution is defined over successive symbols, ending with a special end-of-sequence token.
- 2.1 Attention Mechanism: Attention assigns normalized scores to encoder hidden states and concatenates the resulting context with the decoder state before prediction.The attention scores measure how much weight to place on each encoder state, and the normalized vector is recomputed at each output step.
- 2 LSTM+A Parsing Model: Parsing converts each tree into an invertible sequence by following a depth-first traversal order.The network first consumes the sentence left to right, then outputs the linearized tree from the vectors stored in memory.
- 2.3 Parameters and Initialization: The reported LSTM+A configuration uses 3 LSTM layers, 256 units per layer, a 90K input vocabulary, and 128 output symbols.For small-data training, the authors additionally use two dropout layers and call the resulting model LSTM+A+D.
- 2.3 Parameters and Initialization: POS-tag normalization improves F1 by about 1 point, while reversing input sentences has an approximately 0.2-point negative effect when omitted.The experiments use normalized POS tags and reversed input sentences.
- 2.3 Parameters and Initialization: The framework maps words beyond its 90K vocabulary to a single UNK token and avoids task-specific preprocessing such as tree binarization or special unary handling.The authors note that this potentially underestimates final results while preserving task independence.
3 Experiments
Experiments show that attention-enhanced sequence-to-sequence parsing matches domain-specific parsers on small human-annotated data and reaches state-of-the-art performance with high-confidence synthetic data. The model also generalizes beyond news text, parses quickly, and degrades less with sentence length than BerkeleyParser.
- Training data: 40K sentences in the WSJ training set still yielded results matching domain-specific parsers.The paper characterizes this dataset as very small by neural-network standards.
- Evaluation: 92.5 F1 on WSJ section 23 was achieved by a single attention model trained on the large high-confidence corpus.An ensemble increased the score to 92.8, exceeding previously reported single-model and ensemble results.
- Evaluation: 90.5 F1 on WSJ section 23 matched BerkeleyParser’s 90.4 when attention was trained on the small human-annotated dataset with an ensemble.Without an ensemble, the attention model achieved 88.3.
- Evaluation: 14 malformed trees occurred on the full high-confidence dataset, representing 0.8% of development-set cases.All 14 cases involved sentences or fragments without proper final punctuation, and the authors balanced them by adding brackets.
- Evaluation: A 0.7-point F1 difference across the evaluated length range shows less degradation for LSTM+A than for BerkeleyParser.The comparison covers development-set sentences partitioned by length.
- Evaluation: 95.7 F1 on QTB and 84.6 on WEB show that the news-trained high-confidence model generalized to other text types.The WEB score exceeded the best reported score and the in-house BerkeleyParser trained on human-annotated data, while QTB remained below that parser’s best score.
- Evaluation: Over 120 sentences per second were parsed on a multi-core CPU with batch size 128 and beam-size 1.The decoder was generic and unoptimized, and the speed covered sentences of all lengths.
4 Analysis
The attention mechanism was especially important for learning from small datasets and produced a sharply focused, monotonic alignment while decoding parse trees. The learned procedure resembles a stack but is not a simple stack decoder.
- Analysis: Attention was especially important when the model learned from a relatively small dataset.The authors report less overfitting, faster learning of the parsing function, and better generalization than a plain LSTM without attention.
- Attention behavior: The attention model focuses sharply on one input word while producing the parse tree and moves monotonically from the first word to the last.The focus advances deterministically to the right when a word is consumed.
- Attention behavior: The learned attention procedure resembles a stack but is not simple stack decoding.Because the inputs are reversed, a focused state also contains information about all words after its position; some examples skip words.
5 Related Work
Earlier constituency parsers relied on grammar-specific structure, linguistic assumptions, or complex inference. The paper positions its LSTM sequence-to-sequence model as a domain-agnostic alternative related to linear-time and attention-based neural approaches.
- Related work: Traditional constituency parsers rely on probabilistic context-free grammars, smoothing rare lexicalized events, or carefully designed model structures.Latent-variable approaches partially reduce reliance on manual linguistic modeling.
- Related work: Early neural parsers incorporated strong linguistic insights, while incremental sigmoid belief networks avoided independence assumptions at the cost of intractable inference.Another recurrent approach decomposed trees into independent stack levels to avoid complex inference.
- Related work: Linear-time incremental parsers typically require task-specific constraints or multiple passes, whereas a related left-to-right parser requires an explicit stack.The paper contrasts these design requirements with its sequence-to-sequence formulation.
- Related work: Attention mechanisms evolved from monotonic alignment for handwritten text synthesis to more general alignment models for sequence-to-sequence tasks.The paper adapts the general attention model introduced for machine translation.
6 Conclusions
Generic sequence-to-sequence models achieve excellent constituency-parsing results with relatively little effort or tuning. Attention improves data efficiency, while synthetic data enables a single model to reach state-of-the-art performance and high speed.
- Attention-enhanced sequence-to-sequence models match BerkeleyParser performance on a small human-annotated parsing dataset.
- Synthetic datasets with imperfect labels substantially improve performance beyond the models that produced their training data.The authors suggest the student model can treat teacher errors as noise and ignore them.
- A single attention model achieves a new state-of-the-art result while remaining exceedingly fast.