Source-linked AI summary
Deep Biaffine Attention for Neural Dependency Parsing
Timothy Dozat, Christopher D. Manning
TL;DR
Dependency parsers are important for downstream NLP but incorrect parses can hurt performance. This paper develops a larger, more regularized graph-based parser with biaffine classifiers, achieving state-of-the-art or near-state-of-the-art results across several treebanks while approaching the strongest transition-based parser.
Problem
Incorrect dependency parses can severely inhibit downstream NLP performance, motivating higher-quality parsers.
Method
The parser combines a larger, more regularized neural graph-based architecture with deep biaffine classifiers for arcs and labels and dimension-reducing MLPs.
Results
The model achieves state-of-the-art or near-state-of-the-art performance on standard treebanks for six languages and comparable performance to the state-of-the-art transition-based parser.
Takeaways & Limitations
Empirical comparisons identify architecture and hyperparameter choices that enable large gains over other neural graph-based parsers while preserving graph-based simplicity.
Takeaways & Limitations
The model appears to lag in LAS, possibly because it captures phrasal compositionality less effectively or has limitations inherent to graph-based parsing.
Abstract
from arXiv · showhide
This paper builds off recent work from Kiperwasser & Goldberg (2016) using neural attention in a simple graph-based dependency parser. We use a larger but more thoroughly regularized parser than other recent BiLSTM-based approaches, with biaffine classifiers to predict arcs and labels. Our parser gets state of the art or near state of the art performance on standard treebanks for six different languages, achieving 95.7% UAS and 94.1% LAS on the most popular English PTB dataset. This makes it the highest-performing graph-based parser on this benchmark---outperforming Kiperwasser Goldberg (2016) by 1.8% and 2.2%---and comparable to the highest performing transition-based parser (Kuncoro et al., 2016), which achieves 95.8% UAS and 94.6% LAS. We also show which hyperparameter choices had a significant effect on parsing accuracy, allowing us to achieve large gains over other graph-based approaches.
1 INTRODUCTION
Dependency parsing supports many natural-language understanding tasks, but incorrect parses can damage downstream performance. This paper modifies a neural graph-based parser to approach transition-based state-of-the-art performance while retaining much of graph-based simplicity.
- Dependency parses are useful for many NLP tasks, especially those involving natural language understanding.
- Frequent incorrect parses can severely inhibit downstream performance, motivating improvements in parser quality.
- The parser uses a larger, more regularized network, biaffine attention and labeling classifiers, and dimension-reducing MLPs.
- The resulting graph-based parser maintains most of the approach’s simplicity while approaching the performance of the state-of-the-art transition-based parser.
2 BACKGROUND AND RELATED WORK
Dependency parsing can be organized around sequential transition decisions or global graph scoring. Prior neural work explored both paradigms, including attention-based graph parsers and increasingly complex transition-based systems.
- Transition-based parsers build trees left to right by manipulating a stack and buffer and predicting the next parser action.
- Neural transition-based parsers use embeddings and recurrent or search-based mechanisms to improve action prediction and can achieve state-of-the-art performance.
- Graph-based parsers score every possible dependency edge and construct a maximum spanning tree from those scores.
- Kiperwasser and Goldberg’s neural graph parser uses BiLSTM representations and an MLP to score each possible head-dependent arc, with labels predicted analogously.
- Later graph-based models replaced MLP attention with bilinear attention or introduced additional objectives and architectural changes.
3 PROPOSED DEPENDENCY PARSER
The proposed parser combines BiLSTM representations with deep biaffine classifiers for arc and dependency-label prediction. Dimension-reducing MLPs and extensive regularization support the larger architecture.
- 3.1 DEEP BIAFFINE ATTENTION: The parser replaces MLP or bilinear attention with biaffine attention, uses a biaffine label classifier, and applies dimension-reducing MLPs to recurrent outputs.
- 3.1 DEEP BIAFFINE ATTENTION: The biaffine arc classifier separately models head-dependent interactions and head priors, while the label classifier scores labels using the dependent, head, and their interaction.
- 3.1 DEEP BIAFFINE ATTENTION: Reducing recurrent-state dimensionality before scoring removes information irrelevant to each decision, improving parsing speed and reducing overfitting risk.
- 3.1 DEEP BIAFFINE ATTENTION: At training time, each word selects its highest-scoring head; at test time, the MST algorithm ensures a well-formed tree.
- 3.2 HYPERPARAMETER CONFIGURATION: The configuration uses three 400-dimensional-per-direction BiLSTM layers, 100-dimensional word and tag vectors, ReLU MLPs, and dropout throughout the model.
4.1 DATASETS
The evaluation covers English and Chinese Penn Treebanks together with CoNLL 09 languages, following standard dataset practices. Punctuation is excluded for the PTB-SD and CTB evaluations.
- The evaluation uses PTB-SD 3.3.0, PTB-SD 3.5.0, the Chinese Penn Treebank, and the CoNLL 09 shared-task dataset.
- Punctuation is omitted from evaluation for the English PTB-SD and Chinese Treebank datasets.
4.2 HYPERPARAMETER CHOICES
The paper finds that architecture, recurrent-cell choice, regularization, and optimization settings substantially affect dependency-parsing accuracy and speed. Deep biaffine models benefit from larger networks, stronger regularization, LSTM-based recurrence, and a lower Adam β2 value.
- Attention mechanism: Deep bilinear models outperform alternative classifier architectures in both speed and accuracy, while shallow variants can overfit and run slower.The shallow model matches the deep model's unlabeled performance but uses a much larger label classifier.
- Network size: Three or four recurrent layers significantly outperform two, and increasing LSTM size from 200 to 300 or 400 dimensions also significantly improves performance.The 400-dimensional model significantly outperforms the 300-dimensional model on validation data but not on the test set.
- Recurrent cell: LSTM cells substantially outperform GRU cells, while Cif-LSTM slightly trails LSTM but runs faster than GRU with the same parameter count.The authors hypothesize that Cif-LSTM's output gate helps maintain sparse recurrent outputs under high dropout.
- Embedding dropout: Stronger regularization is required for the larger parser: independent 33% word and tag dropout prevents overfitting, whereas partial dropout harms accuracy.Using tags without dropout performs worse than using no tags at all.
- Optimizer: Setting Adam's β2 to .9 instead of .999 makes a large positive impact on final performance by allowing updates to adapt more quickly to recent gradient changes.The paper attributes the problem with β2=.999 to excessive influence from gradients far in the past.
4.3 RESULTS
The model matches leading unlabeled parsing performance across several benchmarks, while its labeled accuracy trails the state-of-the-art model and may reflect architectural or input-quality limitations.
- The model achieves nearly the same UAS as Kuncoro et al. (2016) on PTB-SD 3.3.0, despite its substantially simpler architecture.
- It reaches state-of-the-art UAS on CTB 5.17 and state-of-the-art performance across all CoNLL 09 languages.CoNLL 09 includes many non-projective dependencies, which graph-based parsers can predict more readily than transition-based parsers.
- A bug in the original code affected the CTB 5.1 dataset.
- The model appears to lag behind the state-of-the-art model in LAS, potentially because of embeddings, tagging, compositionality, or graph-based access to syntactic information.Addressing compositionality or graph-based limitations would require a more innovative architecture than current neural graph-based parsers.
5 CONCLUSION
The paper presents modified bilinear attention and a larger, more regularized network that improve graph-based dependency parsing speed and performance. It also identifies remaining gaps in labeled accuracy and out-of-vocabulary handling.
- Modified bilinear attention increases parsing speed without hurting performance.
- The larger but more regularized network outperforms other neural graph-based parsers and performs comparably to the current state-of-the-art transition-based parser.
- The paper provides empirical motivation for its architecture and configuration choices over similar approaches in existing literature.
- Future work targets the gap between labeled and unlabeled accuracy and improved handling of out-of-vocabulary tokens in morphologically richer languages.