Source-linked AI summary
Constituency Parsing with a Self-Attentive Encoder
Nikita Kitaev, Dan Klein
TL;DR
Constituency parsers have predominantly used LSTM encoders, motivating investigation of whether self-attention can improve sentence encoding and make information propagation analyzable. The paper combines a self-attentive encoder with a parsing decoder, factors positional and content information, and evaluates lexical representations. It reports 93.55 F1 without external word representations, 95.13 F1 with pre-trained representations, and better published results on 8 of 9 SPMRL languages.
Problem
Recent constituency parsers predominantly use LSTM encoders, leaving alternative encoder architectures for summarizing global sentence context less explored.
Method
The parser combines a self-attentive sentence encoder with a chart decoder, analyzes positional and content attention, and evaluates alternative lexical representations.
Results
The parser achieves 93.55 F1 on Penn Treebank WSJ without external word representations, 95.13 F1 with pre-trained representations, and exceeds previous best-published results on 8 of 9 SPMRL languages.
Takeaways & Limitations
Separating positional and content information can improve parsing accuracy, while self-attention can outperform an LSTM-based encoder under the same decoding procedure.
Takeaways & Limitations
A self-attentive lexical model was empirically slower, did not parallelize better than a character-level LSTM, and initially underperformed the LSTM.
Abstract
from arXiv · showhide
We demonstrate that replacing an LSTM encoder with a self-attentive architecture can lead to improvements to a state-of-the-art discriminative constituency parser. The use of attention makes explicit the manner in which information is propagated between different locations in the sentence, which we use to both analyze our model and propose potential improvements. For example, we find that separating positional and content information in the encoder can lead to improved parsing accuracy. Additionally, we evaluate different approaches for lexical representation. Our parser achieves new state-of-the-art results for single models trained on the Penn Treebank: 93.55 F1 without the use of any external data, and 95.13 F1 when using pre-trained word representations. Our parser also outperforms the previous best-published accuracy figures on 8 of the 9 languages in the SPMRL dataset.
1 Introduction
The paper replaces the predominantly LSTM-based encoder in constituency parsers with self-attention, making sentence-level information flow explicit. It analyzes positional and content attention, lexical representations, and achieves strong results on Penn Treebank and SPMRL benchmarks.
- Motivation and architecture: Self-attention replaces the predominantly recurrent encoder architecture used in recent constituency parsers.The parser combines a self-attentive sentence encoder with a parsing-customized decoder.
- Motivation and architecture: Attention exposes how information transfers between sentence locations, including effects based on positions and word content.This explicit information flow supports analysis of which contextual information matters for parsing.
- Attention analysis: Factoring positional and content attention noticeably improves parsing accuracy after the model predominantly relies on position-based attention.The model first learns an implicit trade-off between the two attention types.
- Lexical representations: Character-based subword representations can outperform external part-of-speech tags, while character LSTMs and contextualized word representations provide further gains.The paper reports that deep contextualized representations boost parsing accuracy.
- Benchmark results: 93.55 F1 is achieved on the Penn Treebank WSJ test set without external word representations, rising to 95.13 F1 with pre-trained representations.The parser also outperforms previous best-published results on 8 of 9 SPMRL languages.
2 Base Model
The base parser combines a self-attentive word encoder with a chart-based span-scoring decoder. Self-attention propagates context between sentence positions, while the model scores trees compositionally and decodes them efficiently.
- Architecture: The parser uses an encoder-decoder architecture with a self-attentive encoder adapted from Vaswani et al. and a chart decoder based on Stern et al.The encoder produces context-aware word representations, which the chart component combines into span scores.
- Tree Scores and Chart Decoder: The chart decoder assigns each tree a score by summing labeled constituent scores, handles unary chains and binarized n-ary trees, and finds the optimal tree with CKY-style inference.Dummy labels introduced by binarization receive zero scores, preserving the tree-score decomposition across binarizations.
- Context-Aware Word Representations: The encoder adds word, part-of-speech, and position embeddings at each sentence position before processing them through eight identical layers.The layers contain multi-headed attention and position-wise feed-forward sublayers with residual connections and Layer Normalization.
- Self-Attention: Multi-headed self-attention is the only mechanism that propagates information between different sentence positions.Each attention head uses queries, keys, and values to distribute attention across words and aggregate attended values.
- Results: 92.67 F1 was achieved on the Penn Treebank WSJ development set, compared with 92.24 for an otherwise comparable LSTM-based encoder.These results show that the parser does not require an RNN-based encoder and that self-attention can perform better in this setting.
3 Content vs. Position Attention
The parser separates content-based and position-based attention because intermingling them can cause one type to dominate. Explicit factoring improves development-set parsing accuracy while retaining both information types.
- Factored attention: The factored encoder explicitly separates content and position information throughout its computations, including attention and feed-forward sublayers.Query-key dot products decompose into content and position contributions, whose log-probabilities are combined before value lookup.
- Motivation: The encoder’s attention can use both word content and sentence position to transfer information between locations.The model was motivated by the possibility that intermingling these information types compromises their balance.
- Accuracy and parameters: 93.15 development-set F1 is achieved by the factored scheme, improving almost 0.5 F1 over the unfactored model.Equal vector halves preserve vector sizes while roughly halving the number of trainable parameters.
- Control experiment: A block-sparse model that retains component-wise addition reaches 92.63 F1, supporting information factoring rather than sparsity alone as the source of improvement.This control result is not much different from the unfactored model.
- Relative importance: Position-based attention contributes most to parsing, while content-based attention remains useful, especially in later encoder layers.The model’s later layers use a greater balance between the two attention types.
4 Analysis of our Model
The analysis tests how the self-attentive parser uses content, position, and sentence-wide context. Windowing experiments show that long-distance dependencies are essential for maximum accuracy and benefit from direct global access.
- Content and position attention: Position-based attention is the most important contributor, while content-based attention is especially helpful in the encoder’s final layers.This conclusion comes from selectively disabling either attention component at test time across subsets of layers.
- Windowed attention: Strict test-time windowing produces poor results, with even a window of size 40 reducing parsing accuracy relative to the original model.Relaxed windowing leaves designated boundary tokens globally accessible for sentence-wide pooling.
- Global pooling: Some attention heads almost always attend to the start token, suggesting that it serves as a location for sentence-wide pooling or processing.This observation arose while investigating how the model uses long-distance attention.
- Training with windows: Long-distance dependencies remain essential during retraining: a per-layer window of size 10 across 8 layers still fails to recover full-parser performance.The effective context size is around 80, which remains insufficient under either windowing approach.
- Global computation: Relaxed windowing consistently improves accuracy over strict windowing by allowing global information pooling, but small windows still limit performance.The results suggest that direct access to long-distance information avoids an intermediate bottleneck.
5 Lexical Models
The paper compares lexical representations for handling morphology, including character-based features, external taggers, and pretrained contextual word representations. Character LSTMs perform best among the subword approaches, while ELMo produces the strongest overall encoder configuration.
- 5.1 Models with Subword Features: Character LSTMs outperform predicted part-of-speech tags as lexical representations.Removing word embeddings does not hurt and can improve parsing accuracy.
- 5.1 Models with Subword Features: Prefix/suffix character concatenation outperforms pretagging and works without word embeddings, but remains below a character LSTM.The scheme represents each word using its first and last eight letters, with padding for short words.
- 5.1 Models with Subword Features: Self-attention is slower and initially underperforms a character-level LSTM for lexical modeling.The paper attributes this partly to lexical models producing one vector per word, whereas self-attention is suited to context-aware summaries at multiple positions.
- 5.2 External Word Representations: The best-performing parser uses a factored self-attentive encoder over ELMo word representations.ELMo vectors are projected to the dimensionality required by the model.
6 Results
The parser achieves strong English and multilingual constituency-parsing results. It reaches state-of-the-art performance on WSJ and exceeds previous best-published results on 8 of 9 SPMRL languages, with Swedish as the exception.
- 6.1 English (WSJ): 93.55 F1 on the Penn Treebank WSJ test set exceeds previous single-system parsers trained only on WSJ data.This result uses the CharLSTM parser without external word representations.
- 6.1 English (WSJ): 95.13 F1 on the WSJ test set is a new state-of-the-art when the parser is augmented with ELMo word representations.The WSJ-only parser takes 18 hours to train, while the ELMo version takes 13 hours excluding embedding pretraining.
- 6.2 Multilingual (SPMRL): The model is evaluated across the nine languages in the SPMRL 2013/2014 shared tasks using character-LSTM lexical models.Models with and without learned word embeddings are considered for these morphologically rich languages and smaller treebanks.
- 6.2 Multilingual (SPMRL): The parser exceeds previous best-published results on 8 of 9 SPMRL languages, while Swedish remains state-of-the-art for an ensembled reranked model.The paper’s model was evaluated only in the single-system condition.
7 Conclusion
The paper concludes that encoder choice substantially affects parser performance and that factored self-attention achieves state-of-the-art parsing results. Its findings motivate further study of utterance encoding for parsing and other NLP tasks.
- 7 Conclusion: Encoder choice can have a substantial effect on parser performance.The conclusion identifies encoder design as an important source of variation in parsing accuracy.
- 7 Conclusion: A factored self-attentive encoder produces state-of-the-art parsing results.The paper separates different kinds of information in the architecture rather than relying only on additional information sources.
- 7 Conclusion: The results suggest that alternative ways of encoding utterances may yield additional improvements in parsing and other natural language processing tasks.This conclusion extends the paper’s architectural focus beyond the specific parser evaluated.
A.1 Model Hyperparameters
The model’s hyperparameters are tuned on the English development set and summarized in Table 8.
- A.1 Model Hyperparameters: Hyperparameters are reported in Table 8.The table summarizes the settings used for the model.
- A.1 Model Hyperparameters: Hyperparameters are tuned on the development set for English.The tuning procedure is specified for the English setting.
- A.1 Model Hyperparameters: The reported tuning scope is English rather than the multilingual evaluation.The passage specifically identifies English as the development-set tuning context.
A.2 Optimizer Parameters
Training used Adam with a large batch, learning-rate warmup, development-based halving, and best-iterate selection. These choices were intended to stabilize optimization.
- Adam trained the model with batches of 250 sentences.
- The learning rate warmed linearly from 0 during the first 160 batches, then was halved after five unimproved epochs.Development performance was evaluated four times per epoch.
- Optimizer stability depended on a large batch size, learning-rate warmup, and an appropriate learning rate.
A.3 Position Embeddings
All model variants used learned position embeddings. Sinusoidal embeddings consistently performed worse in the authors’ experiments.
- All model variations used learned position embeddings.
- Sinusoidal position embeddings consistently performed worse than learned embeddings.The sinusoidal alternative was proposed by Vaswani et al. (2017).
- Table 9 lists the learning rates used in the experiments.