PaperlayerView exact revision on arXiv

Source-linked AI summary

Linguistically-Informed Self-Attention for Semantic Role Labeling

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallum

arXiv:1804.08199v3cs.CL

TL;DR

Semantic role labeling could benefit from explicit syntax, but prior models did not maximize auxiliary syntactic information. LISA combines multi-task learning with multi-head self-attention to jointly model linguistic tasks and SRL, achieving state-of-the-art results on two benchmark datasets and benefiting from injected parses at test time.

  • Problem

    Prior models did not maximize the benefits of auxiliary syntactic information for semantic role labeling.

  • Method

    LISA combines multi-task learning with stacked multi-head self-attention to jointly predict POS tags and predicates, perform parsing, attend to syntactic parents, and assign semantic role labels.

  • Results

    LISA out-performs syntax-free state-of-the-art models on CoNLL-2005 and CoNLL-2012, including out-of-domain evaluation, while test-time injection of predicted parses improves performance.

  • Takeaways & Limitations

    LISA can incorporate rich linguistic information from raw tokens and leverage improved external parses without retraining.

  • Takeaways & Limitations

    The parsing attention head usually emits a tree, but LISA does not enforce that its output is a tree.

Abstract

from arXiv · show

Current state-of-the-art semantic role labeling (SRL) uses a deep neural network with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL decoding, suggesting the possibility of increased accuracy from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech tagging, predicate detection and SRL. Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates. Syntax is incorporated by training one attention head to attend to syntactic parents for each token. Moreover, if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art performance for a model using predicted predicates and standard word embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art on newswire and more than 3.5 F1 on out-of-domain data, nearly 10% reduction in error. On ConLL-2012 English SRL we also show an improvement of more than 2.5 F1. LISA also out-performs the state-of-the-art with contextually-encoded (ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on out-of-domain text.

1 Introduction

LISA addresses the tension between syntax-free state-of-the-art SRL and evidence that syntactic information can improve accuracy. It combines multi-task learning with syntactically informed self-attention in an end-to-end model and achieves strong gains across benchmark and out-of-domain evaluations.

  • 1 Introduction: Prior neural SRL models surpassed syntax-informed systems without explicitly modeling syntax, but richer linguistic features typically required auxiliary preprocessing.This created a gap for an end-to-end architecture that could use syntax without a separate linguistic-analysis pipeline.
  • 1 Introduction: LISA jointly predicts POS tags and predicates, performs dependency parsing, attends to syntactic parents, and assigns semantic role labels.Its earlier layers predict prerequisite POS and predicate information for later SRL scoring.
  • 1 Introduction: The model encodes each sentence once, then predicts predicates, POS tags, parses, and semantic roles for all predicates in parallel.This contrasts with prior approaches that re-encode sentences for separate tasks and predicates.
  • 1 Introduction: 2.5 F1 points absolute improvement over the previous state of the art was achieved on the CoNLL-2005 WSJ test set with predicted predicates and standard word embeddings.On the Brown out-of-domain test set, the improvement exceeded 3.5 F1, corresponding to nearly 10% error reduction.
  • 1 Introduction: More than 2.5 F1 absolute improvement was achieved on CoNLL-2012, while ELMo-based models improved by nearly 1.0 F1 on news and more than 2.0 F1 out of domain.The reported gains span both standard and contextually encoded word representations.

2 Model

LISA combines a multi-head self-attention encoder with multi-task learning to perform parsing, tagging, predicate detection, and SRL end-to-end. It incorporates syntax by supervising one attention head to attend to syntactic parents while jointly producing predicate- and role-specific representations.

  • Architecture: LISA combines supervised syntactic attention with multi-task learning across dependency parsing, POS tagging, predicate detection, and SRL.The model shares earlier representations across related tasks and uses syntax to pass linguistic information to later layers.
  • Self-attention token encoder: The network uses stacked residual multi-head self-attention and feed-forward layers to transform token embeddings into contextual representations.Positional encodings are added because self-attention has no innate notion of token position.
  • Prediction and role labeling: LISA predicts POS tags and predicates jointly, then scores roles for every predicted predicate using predicate and role embeddings with a bilinear model.Role scores are produced for each token and semantic frame, avoiding re-encoding the sentence separately for each predicate.
  • Syntactically-informed self-attention: The syntactic attention head composes a weighted average of dependency-aware value representations, while other heads operate normally before all outputs are concatenated and projected.This produces a syntactically informed token representation for downstream processing.
  • Syntactically-informed self-attention: One attention head is replaced with a deep bi-affine dependency model whose attention distribution predicts each token’s syntactic parent.The head uses distinct key, query, and value projections, and its highest-weight token defines the predicted parent; roots receive self-loops.
  • External syntax: External parse parents can be injected at test time by setting the syntactic attention weights, allowing improved parsers to benefit the model without retraining.The model therefore retains access to auxiliary syntactic information beyond its internally predicted dependencies.

3 Related work

Related work spans feature-rich and neural SRL, syntax-aware models, contextual representations, and multi-task learning, while motivating LISA’s broader syntax integration.

  • Early SRL systems used rich linguistic features with linear models and sometimes complex constrained inference.
  • Later neural SRL models largely used syntax-free architectures, including deep LSTMs, self-attention, and end-to-end predicate prediction.
  • ELMo-based contextual representations produced significant gains on PropBank SRL but were described as computationally expensive alternatives to typical word embeddings.
  • Existing syntax-aware neural SRL methods encoded dependency paths or predicted syntax trees, whereas LISA incorporates the entire parse.
  • Multi-task learning studies have explored combining subsets of related NLP tasks and where and when to combine them effectively.
  • Training with gold versus predicted labels connects to learning to search, scheduled sampling, and teacher forcing, which the authors interpret their approach as extending to multi-task learning.

4 Experimental results

Experiments show that LISA improves SRL across CoNLL-2005 and CoNLL-2012, especially when supplied with stronger predicted parses. Analyses connect the gains to syntactic accuracy and persistent span-boundary difficulties.

  • Semantic role labeling: LISA improves over syntax-free state of the art on CoNLL-2005 and CoNLL-2012 with predicted predicates.The evaluation uses GloVe and ELMo representations and compares against strong prior baselines.
  • Parsing, POS and predicate detection: D&M parses improve performance in every tested setting, while parse-quality differences likely explain much of the SRL gain.D&M achieves the best parsing scores, and its advantage over LISA’s parses corresponds to higher SRL performance.
  • Semantic role labeling: Nearly 1.0 F1 higher on WSJ and more than 2.0 F1 higher on Brown with ELMo and D&M parses than the previous state of the art.ELMo improves all scores, while narrowing the gap between models using LISA’s and D&M’s parses.
  • Semantic role labeling: More than 2 F1 higher on both WSJ and Brown when LISA receives D&M parses with gold predicates.These experiments use GloVe embeddings and predicate indicators, re-encoding the sequence relative to each gold predicate.
  • Semantic role labeling: More than 2.5 F1 higher than the state of the art on CoNLL-2012 with stronger D&M parses.With its own parses, LISA slightly under-performs the syntax-free model; gold parses provide a four-point ELMo upper bound.
  • Parsing, POS and predicate detection: Above 97 predicate-detection F1 on both in-domain datasets, with 1.5–2 F1-point gains over He et al. on Brown.The result supports the effectiveness of multi-task learning for predicate detection.
  • Analysis: Both parsers are correct on 26% of sentences, while both make mistakes on 57%; D&M’s less-wrong parses yield about 1.5 higher average SRL F1.On difficult sentences, D&M averages 87.5 LAS versus 85.7 for LISA.

5 Conclusion

The conclusion presents LISA as a multi-task model that incorporates rich linguistic information into SRL and outperforms state-of-the-art systems on two benchmark datasets, including out-of-domain data.

  • LISA is a multi-task neural network model for incorporating rich linguistic information into semantic role labeling.
  • LISA outperforms the state-of-the-art on two benchmark SRL datasets, including out-of-domain data.
  • Future work targets improved parsing accuracy, better training techniques, and adaptation to more tasks.

A.1 Supplemental analysis

Supplemental analyses examine decoding, sentence length, and predicate–argument distance. Gold parses especially help longer sentences, while better parses improve performance across distances.

  • Decoding and label consistency: Gold parses provide little improvement over predicted parses for BIO label consistency, and Viterbi decoding has nearly the same impact across evaluated systems.The same trends are reported across both datasets.
  • Sentence length: Gold parses are particularly helpful for sentences longer than 10 tokens, likely because syntactic parsers perform worse on longer sentences.
  • Predicate–argument distance: For predicate–argument distance, all distances perform better with better parses, especially gold parses, without the sentence-length trend.Figure 6 plots CoNLL-2005 F1 as a function of predicate distance from the argument span.

A.2 Supplemental results

Supplemental results report CoNLL-2005 development performance with predicted predicates and state that these results follow the same trends as the test data.

  • Additional experimental results are provided because of space constraints in the main paper.
  • Table 9 lists CoNLL-2005 development scores with predicted predicates.
  • The development results follow the same trends as the test data.

A.3 Data and pre-processing details

The experiments use pretrained GloVe embeddings, CoNLL evaluation scripts for SRL and predicate detection, and specified parsing resources and data splits.

  • Embeddings and evaluation: Word embeddings are initialized with 100d pretrained GloVe vectors trained on 6 billion Wikipedia and Gigaword tokens.
  • Embeddings and evaluation: The CoNLL-2005 evaluation script computes segment-level precision, recall, and F1 for SRL and reports predicate detection scores.
  • Parsing resources: Parsing uses distinct dataset-specific D&M parsers, Stanford dependencies v3.5, Stanford CoreNLP POS tags, and pretrained ELMo models.The parsers use the same SRL data splits, with CoNLL-2005 development section 22 replacing section 24.

A.3.1 CoNLL-2012

The CoNLL-2012 evaluation follows a prior split and spans seven domains, with annotations covering multiple linguistic and discourse properties.

  • The evaluation uses the CoNLL-2012 split from He et al. (2018), based on annotations and documents from the CoNLL-2012 coreference split.
  • The dataset covers seven domains, including newswire, web, broadcast news and conversation, magazines, telephone conversations, and biblical text.
  • The text includes gold annotations for part-of-speech, syntactic constituencies, named entities, word sense, speaker, and coreference.

A.3.2 CoNLL-2005

The CoNLL-2005 setup uses PropBank-based verbal predicate data with an out-of-domain Brown Corpus test set, while training uses Nadam and a scheduled learning rate.

  • Dataset: The CoNLL-2005 data derives from PropBank and includes Wall Street Journal predicate-argument structures plus a Brown Corpus out-of-domain test set.
  • Dataset: The dataset contains verbal predicates, 28 role-label types, and 105 SRL labels after encoding predicate-argument segment boundaries with BIO tags.
  • Optimization: The model is trained with Nadam, an adaptive stochastic-gradient method combining Adam with Nesterov momentum.
  • Optimization: The learning-rate schedule increases the rate linearly during warmup and then decays it proportionally to the inverse square root of the step number.
  • Hyperparameters: Experiments use 10 or 12 self-attention layers with 8 heads per layer, 800d feed-forward projections, and dropout rate 0.1.
Loading 1804.08199v3…