Source-linked AI summary
Deep Semantic Role Labeling with Self-Attention
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, Xiaodong Shi
TL;DR
RNN-based SRL models face challenges with long-range dependencies and structural information. The paper introduces DEEPATT, a self-attention architecture that directly connects arbitrary tokens. It reports state-of-the-art improvements on CoNLL-2005 and CoNLL-2012, with efficient parsing and a constituent-finding bottleneck.
Problem
RNN-based SRL models struggle with long sentences and lack a mechanism for directly handling sentence tree structure.
Method
DEEPATT uses self-attention to directly capture global dependencies between arbitrary tokens, with RNN, CNN, and FFN variants for representation enhancement.
Results
1.8 and 1.0 F1 score improvements over previous state-of-the-art systems are reported on CoNLL-2005 and CoNLL-2012, respectively.
Takeaways & Limitations
The feed-forward variant enables significantly more parallelization and reaches 50K tokens per second on a single Titan X GPU.
Takeaways & Limitations
Finding the right constituents remains a bottleneck because most improvements come from classifying semantic roles.
Abstract
from arXiv · showhide
Semantic Role Labeling (SRL) is believed to be a crucial step towards natural language understanding and has been widely studied. Recent years, end-to-end SRL with recurrent neural networks (RNN) has gained increasing attention. However, it remains a major challenge for RNNs to handle structural information and long range dependencies. In this paper, we present a simple and effective architecture for SRL which aims to address these problems. Our model is based on self-attention which can directly capture the relationships between two tokens regardless of their distance. Our single model achieves F$_1=83.4$ on the CoNLL-2005 shared task dataset and F$_1=82.7$ on the CoNLL-2012 shared task dataset, which outperforms the previous state-of-the-art results by $1.8$ and $1.0$ F$_1$ score respectively. Besides, our model is computationally efficient, and the parsing speed is 50K tokens per second on a single Titan X GPU.
Introduction
SRL provides an intermediate semantic representation useful for NLP applications, but RNN-based end-to-end models struggle with long sentences and sentence structure. DEEPATT uses self-attention to address these challenges and reports stronger benchmark performance with efficient parsing.
- Motivation: SRL identifies who did what to whom, when, and where, providing an intermediate semantic representation for NLP applications.The paper names information extraction, question answering, machine translation, and multi-document abstractive summarization as applications.
- Motivation: RNNs compress an entire sentence history into one fixed-size vector, making long sentences difficult while wasting memory on shorter ones.Their sequential processing also lacks a direct mechanism for handling tree-structured inputs.
- Approach: DEEPATT applies self-attention to connect arbitrary tokens directly, shortening distant interactions from O(n) to O(1) paths.The architecture also includes RNN, CNN, and FFN variants for enhancing representations.
- Results: 1.8 and 1.0 F1 score improvements over previous state-of-the-art systems are reported on CoNLL-2005 and CoNLL-2012, respectively.The paper also reports a 2.0 F1 improvement over the previous end-to-end approach on an out-of-domain dataset.
- Results: 50K tokens per second is achieved by the feed-forward DEEPATT variant on a single Titan X GPU.The feed-forward design allows significantly more parallelization.
Semantic Role Labeling
The paper frames SRL as identifying and classifying predicate arguments into semantic roles. Its approach treats this process as BIO tagging using utterances and predicate masks as neural inputs.
- Task Definition: SRL identifies and classifies the arguments of each target verb into semantic roles.The paper illustrates roles such as ARG0, ARG1, ARG2, AM-TMP, and V with a borrowing example.
- Task Definition: In the example, ARG0 marks the borrower, ARG1 the thing borrowed, ARG2 the entity borrowed from, AM-TMP the timing adjunct, and V the verb.
- Pipeline: Conventional SRL typically identifies arguments, assigns their roles, prunes obvious non-candidates, and applies post-processing to inconsistent predictions.A dynamic programming algorithm is often used to find a global optimum for the sequence-labeling problem.
Deep Attentional Neural Network for SRL
DEEPATT consists of repeated layers combining nonlinear processing with self-attention, followed by a softmax classification layer. This design makes attention the central mechanism while allowing multiple nonlinear sub-layer variants.
- Architecture: DEEPATT contains N identical layers, each comprising a nonlinear sub-layer followed by an attentional sub-layer.
- Architecture: The topmost layer is a softmax classification layer.
Self-Attention
DEEPATT uses self-attention to model relationships within a single sequence and combines multiple attention heads to produce richer representations. Compared with recurrent or convolutional alternatives, self-attention offers direct token connections, flexible receptive fields, easier gradient propagation, and greater parallelization.
- Self-attention operates on a single sequence and computes relationships among its tokens.
- Dot-product attention supports faster computation through matrix multiplication, while all attention heads can be computed in parallel.
- Multi-head attention projects input vectors into queries, keys, and values, then uses parallel heads to focus on different value-vector channels.
- The outputs of parallel heads are concatenated and linearly mapped to mix information across channels.
- Self-attention gives any input-output pair distance 1, avoids fixed convolutional windows, and produces outputs through weighted sums.
Nonlinear Sub-Layers
The architecture supplements attention with nonlinear sub-layers to increase representational power. It explores recurrent, convolutional, and feed-forward alternatives, including bidirectional LSTMs, GLUs, and ReLU-based feed-forward networks.
- Nonlinear sub-layers address the limited representational power of weighted-sum attention and come in recurrent, convolutional, and feed-forward forms.
- The recurrent sub-layer uses bidirectional LSTMs and sums opposite-direction representations to preserve input-output dimensionality.
- The convolutional sub-layer uses Gated Linear Units with two filters, and the filter width is set to 3 in all experiments.
- The feed-forward sub-layer contains two linear layers with a hidden ReLU nonlinearity; its hidden size hf is set to 800 unless otherwise noted.
Deep Topology
DEEPATT uses residual connections to ease training in its deep network and applies layer normalization afterward to stabilize activations.
- Residual connections are used to ease training of the deep attentional neural network.
- Layer normalization follows each residual connection to stabilize activations in the deep neural network.
Position Encoding
Because attention alone cannot distinguish token positions, the model adds timing signals to input embeddings. This approach encodes positions without introducing additional parameters.
- Attention cannot distinguish different positions, so the model encodes the positions of input words.
- Timing signals are added to input embeddings and, unlike position embeddings, introduce no additional parameters.
Pipeline
The SRL pipeline embeds utterances with predicate masks, processes them through DEEPATT, and predicts labels from the topmost attention representation. It treats SRL as classification while learning latent label dependencies in the attention layers.
- Input representation: Original utterances and binary predicate masks are projected into real-valued embeddings before neural processing.Each mask value is 1 for a predicate and 0 otherwise; word and mask embeddings are concatenated as input features.
- Deep attentional processing: DEEPATT learns sequential and structural information from the embedding feature maps through its deep attentional network.The network is designed to capture nested sentence structures and latent dependencies among semantic labels.
- Label prediction: SRL is performed as a classification problem rather than using a separate transition model for label probabilities.Latent dependency information is embedded in the topmost attention sub-layer, simplifying implementation relative to prior transition-based approaches.
- Label prediction: The model predicts each label yt from the representation ht produced by DEEPATT’s topmost attention sub-layer.A softmax output layer assigns the probability of the predicted label, and training maximizes correct-label log probabilities over the training set.
Experiments
Experiments evaluate DEEPATT on CoNLL-2005 and CoNLL-2012, compare architectural and decoding choices, and analyze detailed labeling behavior. The model achieves strong benchmark performance, while span identification remains a bottleneck and larger widths trade speed for small F1 gains.
- Benchmark results: 83.4 F1 is achieved by the single FFN-sub-layer model on CoNLL-2005, exceeding the previous best by 1.8 F1.The corresponding single-model RNN and CNN variants each achieve 82.3 F1.
- Benchmark results: 1.0 F1 is the single FFN variant’s improvement over the previous state of the art on CoNLL-2012.Five-model ensembles achieve F1 scores of 84.6 and 83.9 on CoNLL-2005 and CoNLL-2012, respectively.
- Benchmark results: 2.0 F1 is the improvement achieved on the out-of-domain dataset over the previous end-to-end approach.The model reaches 74.1 F1 on that dataset.
- Architectural analysis: 10 layers give the best development-set performance: four-layer DEEPATT reaches 79.9 F1, while twelve layers slightly reduce performance by 0.1 F1.Increasing depth consistently improves performance up to the ten-layer model.
- Architectural analysis: 83.4 F1 is achieved with 600 hidden units, but larger widths slow training and parsing because of increased parameter counts.The improvement from increasing model width is described as slight.
- Architectural analysis: 79.6 to 83.1 F1 is the increase from using pretrained GloVe embeddings.The embeddings are initialized from Wikipedia- and Gigaword-trained GloVe vectors but remain trainable.
- Decoding and labeling analysis: Constrained decoding slightly lowers performance and significantly slows decoding, whereas the model is described as sufficiently powerful to capture label relationships directly.DEEPATT performs SRL as classification while embedding latent dependency information in its topmost attention sub-layer.
Related work
SRL research progressed from syntax-heavy feature engineering toward end-to-end neural models. Self-attention emerged as a related technique across several language-processing tasks.
- Traditional SRL methods focused on designing feature templates to represent utterance structure.
- Syntactic features and parser combinations were used to capture sentence structure and reduce prediction risk.
- End-to-end SRL advanced through stacked LSTMs, highway LSTMs, and constrained decoding.
- Self-attention was applied to machine reading, natural language inference, sentence embedding, and related language tasks.
Conclusion
The paper proposes DEEPATT, a 10-layer deep attentional network for SRL, and evaluates it on two CoNLL shared-task datasets. The experiments report substantial performance improvements and a new state-of-the-art.
- The paper proposes a deep attentional neural network for semantic role labeling.
- The SRL models use a depth of 10 and are evaluated on the CoNLL-2005 and CoNLL-2012 shared-task datasets.
- The experimental results substantially improve SRL performance, leading to a new state-of-the-art.