Source-linked AI summary
Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling
Luheng He, Kenton Lee, Omer Levy, Luke Zettlemoyer
TL;DR
Existing BIO-based SRL models typically assume gold predicates and lack span-level features. This paper jointly predicts predicates, argument spans, and their relations with contextualized span representations and independent span-pair decisions. It achieves state-of-the-art PropBank performance without gold predicates, while independent decisions create global consistency weaknesses.
Problem
BIO-based SRL models are typically evaluated with gold predicates and cannot incorporate span-level features, motivating end-to-end predicate and argument prediction.
Method
The model predicts labeled relations between all token predicate candidates and continuous argument spans using contextualized span representations and independent pairwise classifiers.
Results
The model achieves state-of-the-art performance on PropBank without assuming that predicates are given.
Takeaways & Limitations
Span-based SRL supports joint predicate identification, span-level features, and overlapping spans across different predicates in one output structure.
Takeaways & Limitations
Independent predicate-argument classifiers weaken global consistency, producing more structural constraint violations and duplicate core arguments than previous systems.
Abstract
from arXiv · showhide
Recent BIO-tagging-based neural semantic role labeling models are very high performing, but assume gold predicates as part of the input and cannot incorporate span-level features. We propose an end-to-end approach for jointly predicting all predicates, arguments spans, and the relations between them. The model makes independent decisions about what relationship, if any, holds between every possible word-span pair, and learns contextualized span representations that provide rich, shared input features for each decision. Experiments demonstrate that this approach sets a new state of the art on PropBank SRL without gold predicates.
1 Introduction
Existing neural SRL systems typically label arguments for gold predicates one predicate at a time, limiting deployment and span-level modeling. The paper proposes an end-to-end span-graph approach that jointly predicts predicates, argument spans, and their relations.
- Motivation: Recent high-performing BIO-based SRL models label argument spans for one predicate at a time and are typically evaluated with gold predicates.Deployment therefore requires a separate predicate-identification model.
- Approach: The proposed model predicts all predicates and their argument spans jointly in one forward pass.It uses contextualized span representations to predict SRL graphs directly over text spans.
- Approach: The span-graph structure models overlapping argument spans across different predicates in a single output.This addresses a limitation of semi-Markov and BIO-based formulations.
- Approach: Span representations let the model dynamically select spans and roles without previously standard syntactic features.They generalize the token-level representations used in BIO-based models.
- Results: The model is presented as the first span-based SRL system that does not assume given predicates and achieves state-of-the-art PropBank performance in that setting.The authors also suggest the span-span modeling style may extend to other relation tasks.
2 Model
The model represents SRL as labeled relations between every token candidate predicate and every continuous argument span. Independent predicate-argument classifiers score these relations, while beam pruning makes the cubic candidate space tractable.
- Structured formulation: The model considers every input token as a possible predicate and every continuous text span as a possible argument.Each predicate-argument pair may receive a semantic role label or a null label indicating no relation.
- Structured formulation: The SRL output is the set of non-empty labeled predicate-argument relations over tokens and spans.The formulation represents the output as a labeled relation set rather than BIO tags.
- Scoring: Each predicate-argument pair receives an independently scored role decision conditioned on the input.The score combines predicate and argument unary scores with a label-specific relation score.
- Scoring: The null relation receives a constant score of 0.This makes the no-relation option analogous to the null baseline in logistic regression.
- Efficiency: O(n^3|L|) possible relations are reduced through predicate and argument beams, yielding O(n^2|L|) evaluated relational factors.A fixed maximum span width further reduces computed unary factors to O(n).
3 Neural Architecture
The architecture builds contextualized representations for predicates and argument spans, then scores predicate–argument labels with feed-forward span-pair classifiers.
- Representation construction: Predicate and argument representations are built from contextualized BiLSTM outputs for all candidate predicates and argument spans.Argument vectors combine span endpoints, a soft head word, and span-width features; predicate vectors use the BiLSTM output at the predicate position.
- Representation construction: Argument span representations combine BiLSTM endpoint vectors, an attention-based soft head word, and embedded span-width features.The soft head attends over word inputs within the argument span, with weights computed from BiLSTM outputs.
- Span-pair scoring: The model feeds predicate and argument representations into feed-forward scoring functions for predicate–argument role decisions.The span-pair classifier computes a softmax over the label space L.
- Span-pair scoring: The resulting span-pair classifier assigns a role label, including the possibility of no relation, to each predicate–argument pair.The classifier uses the concatenated predicate and argument representations as input.
4 Experiments
Experiments evaluate joint end-to-end and gold-predicate settings on CoNLL 2005 and CoNLL 2012 using tuple-level micro-averaged F1. The joint model improves over the previous best pipeline across end-to-end settings, while ELMo yields the strongest gold-predicate comparisons.
- Setup: Experiments use CoNLL 2005 and CoNLL 2012 in end-to-end and gold-predicate settings, evaluating micro-averaged F1 on predicate–argument–label tuples.The end-to-end setup predicts all predicates and arguments from tokenized sentences.
- Setup: ELMo representations are added to the BiLSTM input as an orthogonal enhancement applicable to most previous neural systems.Results distinguish comparable single-model systems from ELMo-augmented or ensemble systems.
- End-to-end results: 1.3–6.0 F1 points separate the joint model from the previous best pipeline system in every reported end-to-end setting.The improvement is larger on the out-of-domain Brown test set and on CoNLL 2012, which contains nominal predicates.
- End-to-end results: Over 40% of sentences are predicted completely correctly on all datasets in the end-to-end evaluation.This result is reported alongside the joint model’s improvements over the previous best pipeline system.
- Gold-predicate results: With gold predicates, the model surpasses He et al. (2017), trails Tan et al. (2018), and exceeds prior systems after adding ELMo representations.The comparison includes Peters et al. (2018), which applies ELMo to He et al.’s SRL model.
5 Analysis
The analyses examine pruning, long-range dependencies, syntactic agreement, and global consistency, revealing strengths in span modeling alongside weaknesses from independent decisions.
- Effectiveness of beam pruning: Over 99.7% of argument-bearing predicates are retained with λp = 0.4 predicates per word, while beam pruning reduces considered edges.Joint pruning provides a soft trade-off between efficiency and predicate recall compared with POS-based filtering.
- Long-distance dependencies: The model predicts farther-away arguments more accurately than an ensemble with higher overall F1, likely because predicate-argument pairs connect directly.In BIO taggers, predicate information passes through many LSTM timesteps before reaching long-distance arguments.
- Experimental conditions: Analyses (2)-(4) use gold predicates on the CoNLL 2005 development set, limiting direct interpretation of end-to-end predicate prediction.The analyses are also performed without ELMo embeddings.
- Agreement with syntax: The model achieves 95.0% agreement with gold syntactic span boundaries without explicitly modeling syntax.This is comparable to the cited syntax-based system’s agreement while directly modeling span information.
- Global consistency: Complete-predicate accuracy is 62.6%, below BIO systems’ 64.3%-66.4%, reflecting weaker global consistency.Independent predicate-argument classifiers struggle to track decisions across arguments sharing a predicate and produce more duplicate core arguments.
- Global consistency: Constrained decoding eliminates all core-role inconsistencies but does not significantly improve final SRL F1.The constraint is enforced with dynamic programming at test time, while F1 evaluates piece-wise predicate-argument structures.
6 Conclusion and Future Work
The proposed SRL model jointly predicts predicates and argument spans while incorporating span-level features. It performs better on long-range dependencies and syntactic-boundary agreement but remains weaker on global consistency because of its independence assumption.
- The model jointly predicts all predicates and argument spans using a span-based architecture generalized from a coreference system.
- Compared with BIO systems, it supports joint predicate identification and incorporates span-level features.
- Future work could use higher-order inference to relax the independence assumption and self-attention for more effective contextualization.
A.1 Hyperparameters
The model uses contextual neural representations, bounded span inference, and regularized training with specified optimization and batching settings.
- Word embeddings are fixed 300-dimensional GloVe vectors, while the character CNN uses 8-dimensional embeddings and 50 filters at widths 3, 4, and 5.
- The network uses 3 stacked bidirectional LSTMs with highway connections and 200-dimensional hidden states.
- Inference considers spans up to length 30 and uses dynamic programming to predict nonoverlapping arguments for each predicate.
- Training uses Adam with initial learning rate 0.001, scheduled decay, dropout, and variational dropout masks shared across LSTM timesteps.
- Training batches contain at most 40 sentences and 700 words, with early stopping within at most 320,000 steps.
A.2 OntoNotes Data Statistics
The end-to-end experiments use cleaner CoNLL 2012 splits because some OntoNotes 5.0 training and development sentences have missing predicates.
- Some OntoNotes 5.0 train/dev sentences have missing predicates, making that split unsuitable for end-to-end SRL training.
- End-to-end SRL models are trained on the smaller but cleaner CoNLL 2012 splits.
- Gold-predicate experiments use the full OntoNotes 5.0 train/dev split and the CoNLL 2012 test set, following previous work.