Source-linked AI summary
Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations
Eliyahu Kiperwasser, Yoav Goldberg
TL;DR
Dependency parsing traditionally requires substantial manual feature engineering, and existing neural approaches still rely on carefully chosen core features or external resources. The paper uses context-sensitive BiLSTM token representations, concatenates a few of them into parser features, and trains the encoder jointly with simple transition-based and graph-based parsers. The resulting systems achieve competitive accuracies, including 93.1 UAS on English and 86.6 UAS on Chinese without external resources.
Problem
Dependency parsers require difficult manual feature engineering, while prior neural approaches can rely on many core features or external pre-trained embeddings.
Method
The paper represents each token with a BiLSTM encoding, concatenates a minimal set of encodings into features, and trains the encoder jointly with transition-based and graph-based parser objectives.
Results
93.1 UAS on English and 86.6 UAS on Chinese are achieved with a first-order two-feature parser trained solely on Treebank data.
Takeaways & Limitations
Simple BiLSTM-based feature extractors yield very competitive parsing accuracies in both greedy transition-based and globally optimized first-order graph-based models.
Takeaways & Limitations
Adding external word embeddings degrades the graph-based parser, and the authors leave effective semi-supervised graph-based parsing for future work.
Abstract
from arXiv · showhide
We present a simple and effective scheme for dependency parsing which is based on bidirectional-LSTMs (BiLSTMs). Each sentence token is associated with a BiLSTM vector representing the token in its sentential context, and feature vectors are constructed by concatenating a few BiLSTM vectors. The BiLSTM is trained jointly with the parser objective, resulting in very effective feature extractors for parsing. We demonstrate the effectiveness of the approach by applying it to a greedy transition-based parser as well as to a globally optimized graph-based parser. The resulting parsers have very simple architectures, and match or surpass the state-of-the-art accuracies on English and Chinese.
1 Introduction
The paper replaces extensive manual feature engineering for dependency parsing with context-sensitive BiLSTM representations trained jointly with simple parsers. Across greedy transition-based and graph-based architectures, this approach achieves competitive or state-of-the-art-level English and Chinese accuracies.
- Motivation: Manual feature-function design remains a major challenge because parsers depend on selecting core components and their combinations.Traditional systems use hand-crafted templates, while neural approaches reduce but do not eliminate the need to define core features.
- Approach: The proposed feature function concatenates a minimal set of BiLSTM encodings, each representing a word together with its sentential context.The resulting vector is passed to a multilayer perceptron, while the BiLSTM is trained jointly with the parsing objective.
- Architectures: The BiLSTM extractor is integrated into both greedy transition-based and globally optimized graph-based dependency parsers.The graph-based model jointly trains structured prediction and recurrent feature extraction end to end.
- Results: 93.1 UAS on English and 86.6 UAS on Chinese are achieved with a first-order parser using two features and only Treebank training data.These results are reported without pre-trained embeddings, word clusters, or tri-training.
- Results: 93.9 UAS on English and 87.6 UAS on Chinese are obtained by the greedy transition-based parser with 11 features and pre-trained word embeddings.With four features and embeddings, the same parser reaches 93.6 UAS on English and 87.4 UAS on Chinese.
2 Background and Notation
The background contrasts hand-crafted dependency-parser features with neural alternatives and introduces recurrent representations that encode sequence context. BiRNN and BiLSTM encodings provide token representations using information from both the preceding and following sentence context.
- Feature Functions: Traditional dependency parsers use linear models over hand-crafted feature functions built from core components and feature templates.Examples include stack, buffer, modifier, parent, span-length, word, and part-of-speech information.
- Related Research: Designing effective parser feature sets is hard and time-consuming, motivating methods that learn feature combinations or encode parser states neurally.Prior work uses low-rank tensors, kernels, dense encodings, stack-LSTMs, and compositional representations.
- Related Research: The proposed approach uses simpler feature functions without elaborate network architectures or compositional tree representations.It represents words with BiLSTM encodings and concatenates a small number of them for parsing.
- Related Research: All neural-network approaches discussed in the related work initialize word vectors with pre-trained embeddings from corpora external to the training data.This makes it difficult to separate the effects of automatic feature combination from semi-supervised information.
- Recurrent Networks: An RNN maps a sequence of input vectors to output vectors, with each output conditioned on the input prefix through the corresponding position.The final output can summarize the full sequence, according to the notation introduced in the paper.
- Bidirectional Representations: A bidirectional RNN combines forward and reverse RNNs so each token representation incorporates its entire preceding and following context.The paper describes this as an effectively infinite context window around the represented item.
3 Our Approach
The approach replaces hand-crafted parser features with minimally defined concatenations of learned BiLSTM representations, trained jointly with the parsing objective. These context-sensitive vectors are scored by a nonlinear MLP, while sentence encodings can be precomputed in linear time.
- Core representation: BiLSTM representations replace hand-crafted feature functions as the parser’s learned feature representation.Each word is represented by a context-sensitive BiLSTM vector, and only a small number of such vectors are concatenated.
- Input representation: Each input vector concatenates a word embedding with a POS-tag embedding, with optional external word embeddings.The embeddings are trained together with the model; the paper assumes POS tags are supplied by an external model.
- Scoring: The concatenated BiLSTM feature vector is scored by a one-hidden-layer multilayer perceptron.The MLP is parameterized by W1, W2, b1, and b2 and applies tanh to its hidden layer.
- Training: Joint training propagates parsing-objective signals through the MLP and BiLSTM, encouraging representations suited to dependency parsing.The BiLSTM is trained jointly with the rest of the parsing objective rather than as a separate feature extractor.
- Parsing-time computation: After training, all sentence BiLSTM vectors are computed in linear time and reused during feature extraction.Parsing then concatenates a small number of precomputed vectors for each parser configuration.
4 Transition-based Parser
The transition-based parser combines standard greedy transition-system machinery with a compact BiLSTM feature function. It scores actions from a few stack and buffer representations, trains with margin-based dynamic-oracle methods, and can support compact state signatures for dynamic-programming search.
- Greedy parsing: At each configuration, greedy parsing scores all possible transitions and applies the highest-scoring action until reaching a terminal configuration.The configuration is represented by a feature function and passed to a scoring function over configuration-transition pairs.
- Transition system: The parser uses the arc-hybrid transition system with SHIFT, LEFTℓ, and RIGHTℓ actions over stack, buffer, and dependency-arc configurations.SHIFT moves the buffer front to the stack; LEFTℓ and RIGHTℓ add labeled dependency arcs while removing the top stack item.
- Scoring function: Replacing the traditional linear score with an MLP lets the parser score the compact BiLSTM feature vector nonlinearly.The conventional linear score requires combination features, whereas the MLP is used with the learned BiLSTM representations.
- Feature function: The minimal transition-based feature function concatenates BiLSTM vectors for the top three stack items and the first buffer item.It includes the items affected by possible transitions plus one additional stack-context item, and excludes the already built structure T.
- Scope and implications: Using only four word indices yields compact state signatures that may suit transition-based parsers with dynamic-programming search.The paper presents this as an unexplored potential benefit rather than an evaluated result.
- Feature variants: The extended feature set adds modifier encodings to reach 11 BiLSTM vectors, but alternative feature configurations were not evaluated.The authors note that fewer additional child encodings might produce similar or better gains.
- Training: Training uses a margin-based objective, dynamic-oracle error exploration, and backpropagation through the full MLP-BiLSTM network.Local losses are summed across a sentence, and error exploration follows an incorrect action with probability pagg = 0.1 under the stated margin condition.
5 Graph-based Parser
The graph-based parser uses BiLSTM-derived arc representations in an arc-factored scoring model, then decodes the highest-scoring valid dependency tree. Training jointly optimizes the structured parser and recurrent feature encoder, while labeling is performed after unlabeled structure prediction.
- Graph-based model: Each candidate arc is scored by an MLP over endpoint BiLSTM representations, and shared MLP parameters are used across arcs.The implementation uses a two-layer BiLSTM, although the illustration shows one layer.
- Inference and training: The model computes scores for all possible n^2 arcs and uses dynamic programming to find the best-scoring tree.This organizes global tree selection from locally scored arcs.
- Graph-based model: Arc-factored parsing scores each head-modifier arc independently and sums local scores to obtain a tree score.The highest-scoring projective tree can be found efficiently with Eisner’s decoding algorithm.
- Graph-based model: The parser replaces complex hand-designed arc features with a concatenation of the BiLSTM encodings of the head and modifier words.These contextual encodings are passed to a nonlinear MLP scoring function.
- Inference and training: A margin-based hinge objective trains the gold tree above the highest-scoring incorrect tree, with gradients propagated through the full network.Gradients reach both the BiLSTM encoder and word embeddings; loss-augmented inference addresses early overfitting.
- Labeled parsing: The labeled parser first predicts unlabeled structure and then labels each resulting arc, keeping the number of scored parts small for faster parsing.The labeler uses the same feature representation with a different MLP predictor, and shares the BiLSTM encoder with the parser.
6 Experiments and Results
Experiments on English and Chinese evaluate simple BiLSTM-based parsers against prior systems and ablate key components. The parsers achieve competitive accuracy, while external embeddings help the greedy parser but degrade the graph-based parser.
- The evaluation uses English Stanford Dependencies and Chinese Treebank 5.1 with established train/dev/test splits and comparison settings.
- The parsers are competitive despite simple architectures and minimal feature extractors, with the four-feature greedy parser matching or surpassing several stronger baselines.
- The extended 11-feature set improves accuracy for both languages, producing the best reported Chinese results and the second-ranked English results.
- External word embeddings improve the greedy parser but degrade the graph-based parser, whose behavior remains unexplained by the authors.
- Loss-augmented inference is crucial for the graph-based parser, while multi-task arc labeling improves unlabeled scores and dynamic-oracle training benefits both languages.
7 Conclusion
The paper presents a jointly trained BiLSTM feature extractor integrated into simple greedy transition-based and globally optimized graph-based parsers. Both models achieve very competitive dependency-parsing accuracies.
- A BiLSTM encoder trained jointly with the parser provides feature extraction for both greedy transition-based and globally optimized first-order graph-based models.
- The resulting parsers achieve very competitive parsing accuracies in both architectures.