Source-linked AI summary
Neural Semantic Role Labeling with Dependency Path Embeddings
Michael Roth, Mirella Lapata
TL;DR
Existing SRL systems struggle with rare and complex dependency structures, including control and raising constructions. This paper learns embeddings for lexicalized dependency paths with LSTMs, and reports improved performance over prior state-of-the-art systems across English and other languages.
Problem
Feature-based SRL systems often fail on infrequent linguistic phenomena because lexical and syntactic indicators do not model their triggered interactions well.
Method
The model decomposes lexicalized predicate-argument dependency paths into words and relations and uses LSTMs to learn full-path representations.
Results
PathLSTM improves over mate-tools by an order of magnitude on unseen paths, from 0.9% to 9.6% recall, and achieves state-of-the-art results across evaluated settings.
Takeaways & Limitations
Dependency path embeddings can alleviate sparsity and help cover linguistic phenomena missed by other SRL methods.
Takeaways & Limitations
The paper does not list results for models involving nominal predicates or dependency annotations.
Abstract
from arXiv · showhide
This paper introduces a novel model for semantic role labeling that makes use of neural sequence modeling techniques. Our approach is motivated by the observation that complex syntactic structures and related phenomena, such as nested subordinations and nominal predicates, are not handled well by existing models. Our model treats such instances as sub-sequences of lexicalized dependency paths and learns suitable embedding representations. We experimentally demonstrate that such embeddings can improve results over previous state-of-the-art semantic role labelers, and showcase qualitative improvements obtained by our method.
1 Introduction
Semantic role labeling identifies predicate arguments using labels that abstract beyond syntax, but feature-based systems struggle with rare and structurally complex phenomena. The paper addresses this by learning representations of lexicalized dependency paths with neural sequence modeling.
- Semantic roles label predicate arguments independently of syntactic alternations and can also apply to nominal predicates.
- Existing SRL systems rely heavily on lexical and syntactic indicators, which often fail on infrequent control, raising, recursive, and nominal-predicate phenomena.These features do not adequately model the interactions triggered by such structures.
- In “He had trouble raising funds,” four competitive SRL tools fail to identify He as an argument of raise because of the complex dependency path.The correct role is the agent argument of raise.
- The proposed neural model decomposes lexicalized dependency paths into words and dependency relations, then uses LSTMs to reconstruct full-path representations.It targets control predicates and related phenomena in sentence dependency structures.
- Experiments show that path embeddings address data sparsity and outperform state-of-the-art SRL systems in English while improving performance in Chinese, German, and Spanish.The evaluation uses the CoNLL-2009 benchmark dataset and extends across multiple languages.
2 Dependency Path Embeddings
The paper represents dependency paths as sequences of words and dependency relations, then uses an LSTM to produce path embeddings for SRL. These embeddings are combined with binary features for classification.
- An embedding is a real-valued vector produced by transforming an input within a neural network.
- A dependency path is the sequence of word nodes and relational edges traversed between a predicate and an argument.For SRL, paths run from the predicate to a potential argument.
- 2.1 Recurrent Neural Networks: An LSTM recurrently processes path items and returns a final embedding state for the complete sequence.Its memory state is updated from the current input and previous state, with gates controlling memory modification.
- The model decomposes lexicalized paths into word forms, part-of-speech tags, and dependency relations rather than treating each full path as one unit.This lets words, categories, and relations influence role-labeling decisions and addresses path sparsity.
- 2.2 Embedding Dependency Paths: The path embedding is the LSTM’s final memory output after processing the argument-head word.The sequence begins with the predicate’s part-of-speech tag and word form, followed by intervening relations and words.
- 2.3 Joint Embedding and Feature Learning: The SRL model combines the final path embedding with binary features in a hidden layer before softmax classification.The hidden layer uses rectified linear units, and path embeddings are learned jointly with traditional feature detectors.
3 System Architecture
The SRL system uses a pipeline for predicate processing, argument identification and classification, and global reranking. Neural-network components operate in the final three stages, while predicate identification and disambiguation use logistic regression classifiers.
- The pipeline performs predicate identification and disambiguation, argument identification, argument classification, and global reranking.Its architecture follows previous SRL systems and is depicted in Figure 4.
- Predicate Identification and Disambiguation: Predicate identification and disambiguation focus on verbal and nominal predicates in syntactically analyzed sentences.Both tasks use logistic regression classifiers with lexico-syntactic indicator features.
- Argument Identification and Classification: LSTM-based neural networks separately model nominal and verbal predicates and the identification and classification of arguments.These models address the next two pipeline stages after sense-disambiguated predicates are available.
- Argument Identification and Classification: Argument decisions use lexico-syntactic, local-context, and other features, including dependency paths, sibling words, relative position, and intervening part-of-speech tags.The listed features combine predicate and candidate-argument information with surrounding syntactic context.
- Global Reranking: The global reranker selects the best overall argument structure from n-best argument sets and label assignments using logistic regression.Its features include hidden and embedding-layer states of identified arguments.
4 Experiments
Experiments evaluate PathLSTM on CoNLL-2009 in-domain and out-of-domain settings, examining overall performance, sparsity, sentence length, syntactic phenomena, and predicate or role categories.
- PathLSTM reaches 87.7% single-model and 87.9% ensemble F1 on in-domain data, exceeding published best results by 0.4 and 0.2 points.
- PathLSTM achieves 76.1% single-model and 76.5% ensemble F1 out of domain, surpassing the previous best by 0.2 and 0.6 points.
- For unseen dependency paths, PathLSTM raises recall from 0.9% to 9.6%, while paths seen 1–10 times improve from 12.8% to 24.2%.
- PathLSTM improves short sentences containing expletives or nominal predicates by 0.8% absolute F1 and longer sentences by up to 1.0%.
- Embedding-space projections cluster syntactic configurations and largely separate A0 from A1, with example cases correctly labeled by PathLSTM but not mate-tools.
- PathLSTM improves precision for all nominal-predicate argument types, improves verbal-predicate A0 and A1 recall, and trades modifier-role recall for precision.
5 Path Embeddings in other Languages
The experiments test whether PathLSTM's English improvements generalize to Chinese, German, and Spanish. PathLSTM outperforms the compared system in all cases and achieves the best overall F1-scores for German and Chinese.
- The experiments train and test separate SRL models for Chinese, German, and Spanish using the architecture and hyperparameters established earlier.Evaluation uses CoNLL-2009 data and the official scorer.
- PathLSTM performs better than Bj¨orkelund et al. (2009) in all reported language comparisons.The comparison uses automatic syntactic preprocessing shared with the CoNLL test data.
- 80.1% is PathLSTM's best overall F1-score for German, while 79.4% is its best overall F1-score for Chinese.These are the best overall F1-scores reported for those two languages in Table 7.
6 Related Work
Related SRL work ranges from neural models without syntactic information to systems using sparse parse-derived features. This paper instead represents dependency paths explicitly in a neural architecture to address sparse lexico-syntactic features.
- Early neural SRL models used feed-forward or convolutional architectures, while later work improved them with binary dependency-path indicators.These approaches differ in how much syntactic information they incorporate.
- Some neural SRL systems use no syntactic parse information, whereas others represent parse structure through binary constituency or dependency features.The cited approaches define two contrasting uses of syntax in neural SRL.
- Our approach falls between these extremes by using syntactic parse information while explicitly modeling dependency paths in a neural network.It replaces reliance on solely sparse binary path features with neural path modeling.
- Alternative SRL architectures include sigmoid belief networks and low-rank tensor models, while prior work also explored latent variables and selection-preference models for sparse lexical features.Henderson et al. learn incremental path representations, whereas this paper uses syntactically preprocessed text rather than building parses incrementally.
- The paper revisits familiar features by representing them in a novel way intended to generalize better and overcome sparse lexico-syntactic features.This motivation combines feature-sparsity concerns with the success of neural SRL models.
- Dependency-path embeddings had previously been applied to relation classification and sentiment analysis, but this work models an individual path for SRL.Earlier entity-pair approaches used pooling layers to detect relation-indicating parts of a path.
7 Conclusions
The paper concludes that jointly learned dependency-path and feature-combination embeddings improve SRL performance and cover linguistic phenomena missed by other methods. It also identifies potential uses beyond SRL.
- The proposed neural architecture jointly learns embeddings for dependency paths and feature combinations.
- The experimental results indicate substantially increased classification performance and new state-of-the-art results.
- Qualitative analysis found that the model covers varied linguistic phenomena missed by other methods.
- Dependency path embeddings may benefit semantic and discourse parsing, while the jointly learned feature space may support cross-lingual transfer methods.These are presented as expected applications beyond SRL rather than evaluated results in this section.