Source-linked AI summary
Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
Swabha Swayamdipta, Sam Thomson, Chris Dyer, Noah A. Smith
TL;DR
Frame-semantic parsers have relied on syntax for candidate-span filtering and argument features, despite parsing costs and overly strict filters. This paper introduces a recall-oriented segmental RNN and a multitask syntactic scaffold trained on Penn Treebank constituent annotations, with no parser required at training or test time. The scaffold reaches state-of-the-art performance and offers a cheaper alternative to syntactic pipelining.
Problem
Existing frame-semantic parsers use syntactic filters and features, although parsing is costly and strict filters can limit recall.
Source: §1 · p. 1Method
A recall-oriented softmax-margin segmental RNN is paired with a multitask constituent-identification scaffold trained on Penn Treebank annotations, without parsing at training or test time.
Source: §1 · pp. 1–2Results
The best scaffolding model is within 0.2% absolute F1 of state-of-the-art, without a syntactic parser or external semantic resources.
Source: §6.6 · p. 8Takeaways & Limitations
Syntactic scaffolding is a cheaper alternative to syntactic features because it requires no syntactic parsing at training or test time.
Source: §8 · p. 9The full-text test data do not overlap with the Penn Treebank, but 86 training and development sentences do.
Source: §5 · p. 6
Abstract
from arXiv · showhide
We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates. Built using an extension to the segmental RNN that emphasizes recall, our basic system achieves competitive performance without any calls to a syntactic parser. We then introduce a method that uses phrase-syntactic annotations from the Penn Treebank during training only, through a multitask objective; no parsing is required at training or test time. This "syntactic scaffold" offers a cheaper alternative to traditional syntactic pipelining, and achieves state-of-the-art performance.
1 Introduction
Frame-semantic parsing identifies text-evoked frames and their arguments. The paper tests whether syntax is needed for this task, presenting a competitive syntax-free model and two syntax-informed approaches, including a syntactic scaffold that requires no parser at training or test time.
1 Introduction
- Existing parsers use syntax both to prune candidate spans and as argument-identification features, but parsing is costly and filtering can be too strict.Dependency-based filtering heuristics bounded recall below 72.6%.Source: §1 · p. 1
- The softmax-margin SegRNN provides a competitive syntax-free approach to argument identification and is designed to encourage recall over precision.It forgoes both syntactic filtering and syntactic features.Source: §1 · p. 1
- Syntactic pipelining improves over the syntax-free model and achieves state-of-the-art performance.The approach adds features from either a dependency parser or a phrase-structure parser.Source: §1 · p. 1
- The syntactic scaffold achieves state-of-the-art performance without syntactic parsing during training or testing.It uses a multitask objective with unlabeled constituent identification trained on the Penn Treebank.Source: §1 · p. 2
2 Frame-Semantic Parsing Task
Frame-semantic parsing identifies the frames evoked by targets and the text spans that fill their semantic roles. This paper focuses on argument identification, representing each instance as a constrained segmentation for a given frame.
2 Frame-Semantic Parsing Task
- FrameNet associates frames with frame elements, and targets are words or phrases that evoke frames through lexical units.The lexicon maps frames to the roles they may take.Source: §2 · p. 2
- Frame-semantic parsing is commonly organized as target identification, frame identification, and argument identification; this paper focuses on the last task.The paper evaluates with gold-standard frames and with output from a state-of-the-art frame identifier.Source: §2 · p. 2
2.1 Formal Notation
- For each frame, the input includes a sentence, POS tags, target span, lexical unit, and frame, and the output segments the sentence into role-labeled or NULL spans.Segments cover the sentence without overlap, and a separate segmentation is produced for each frame evoked.Source: §2.1 · p. 3
3 Segmental RNN for Argument Identification
The SegRNN scores labeled spans and uses semi-Markov inference to identify arguments for a frame. Its span representations combine token context with frame and role information, while softmax-margin training biases learning toward recall.
3 Segmental RNN for Argument Identification
- The SegRNN models labeled segmentations with span representations computed by bidirectional RNNs, using O(n^2) inference or O(nb) when spans are capped at 20 tokens.The 20-token cap prunes less than 1% of gold arguments.Source: §3 · p. 3
3.1 Input Span Embeddings
- A token biLSTM contextualizes sentence tokens, and a second biLSTM embeds candidate spans from those token representations.Candidate spans have length at most 20 tokens, and their embeddings share computation where possible.Source: §3.1 · pp. 3–4
3.2 Input Target and Frame Embeddings
- The target and frame representation combines learned frame and lexical-unit embeddings with a contextual representation of the target and its neighboring tokens.The target context is formed from token states over the target span and one neighboring token on each side.Source: §3.2 · p. 4
3.3 Segment Scores
- Each segment score uses a concatenation of its span embedding, frame-element label embedding, and features for span length and position relative to the target.NULL-labeled spans are handled like other labeled spans.Source: §3.3 · p. 4
3.4 Softmax-Margin Segmental RNNs
- Softmax-margin training modifies the partition function with a cost that more strongly penalizes false negatives, encouraging recall over precision.The cost factors by predicted span for tractable inference; restricting NULL segments to length 1 significantly lowered performance.Source: §3.4 · p. 4
4 Syntactic Features
The section tests whether parse-derived syntactic features benefit the SegRNN despite the computational cost of parsing. The features are added to span or token representations, and candidate spans are not hard-filtered.
4 Syntactic Features
- Phrase-structure features from an RNNG parse are concatenated to span representations, and 84% of gold argument spans correspond to predicted constituents.The model still considers candidate spans rather than filtering them based on syntax.Source: §4 · p. 5
- Dependency-parser features are added at both token and span levels, with word-level representations providing dependency head and label information to the path LSTM.The out# features capture information similar to prior span-filtering heuristics; out#=1 approximates the is-phrase-constituent feature.Source: §4 · p. 5
- The syntax-based variants let the learner use syntactic features when scoring spans, potentially creating a soft filter instead of pruning candidates.The motivation is to test syntax’s benefit despite the computational cost of parsing, especially phrase-structure parsing.Source: §4 · p. 5
5 Syntactic Scaffolding
The syntactic scaffold trains the frame-semantic parser alongside a constituent-span predictor, sharing span representations and using Penn Treebank annotations as additional supervision. The scaffold is removed at prediction time, so parsing is not needed for inference.
5 Syntactic Scaffolding
- The auxiliary task predicts whether text spans are syntactic constituents, sharing span embeddings with the frame-semantic parser.It uses Penn Treebank annotations, which do not significantly overlap with FrameNet; 86 training and development sentences occur in both datasets.Source: §5 · p. 6
- For FrameNet, any span annotated as a frame element for any frame is positive; other spans are negative.Source: §5 · p. 6
- The scaffold scores each span-label segment using learned label and scoring parameters with the basic model’s span embedding.Source: §5 · p. 6
- The scaffold objective uses a binary logistic-regression loss for each span, combined with the frame-semantic objective in a joint multitask loss.The scaffold-task weight δ is less than 1; the frame-semantic loss does not apply to Penn Treebank sentences, which lack frame-semantic annotations.Source: §5 · p. 6
- The scaffold is removed at prediction time, so no syntactic prediction is needed, including at test time.Source: §5 · p. 6
6 Experiments
Experiments evaluate argument identification and end-to-end parsing on FrameNet 1.5, comparing the basic model, syntactic-feature variants, the scaffold, and published systems. The best ensembled results approach or match state of the art, while the scaffold avoids syntactic parsing.
6 Experiments
- The systems are compared on argument identification and joint frame-and-argument identification on the FrameNet 1.5 test set.The latter evaluation uses predicted frames from FitzGerald et al. (2015).Source: §6 · p. 6, Table 3 · p. 8
6.1 Data
- The dataset uses FrameNet 1.5 full-text sentences, the Das and Smith test set, and eight randomly selected files for development.It contains 62,716 training sentences with 17,141 targets and 2,420 test sentences with 4,427 targets.Source: §6.1 · p. 6
6.2 Hyperparameters
- Most hyperparameters were selected by intuition or prior work, and the authors note that more careful tuning could improve performance.The remaining hyperparameters were selected using held-out development-set F1.Source: §6.2 · p. 7
6.3 Self-Ensembling
- Ensembling five models with different random initializations aggregates their local segment scores before decoding.The authors report that self-ensembling markedly improves performance across models, presumably by reducing variance from the non-convex objective.Source: §6.3 · p. 7
6.4 Evaluation
- Evaluation reports micro-averaged precision, recall, and F1, using gold frames for argument identification and predicted frames for end-to-end parsing.Source: §6.4 · p. 7
6.5 Baselines
- Published baselines include syntax-dependent systems using parsed features or span pruning, while the scaffold does not rely on a syntactic parser.The compared systems include SEMAFOR, its extension, and neural and reranking approaches with syntactic features.Source: §6.5 · pp. 7–8
6.6 Results
- The basic model beats original SEMAFOR without syntax, while the best ensembled system ties state of the art and the scaffold comes within 0.2% absolute F1 without a parser or external semantic resources.All three syntactic additions improve the basic model; the scaffold beats dependency features after ensembling.Source: Table 3 · p. 8, §6.6 · p. 8
- The recall-oriented softmax-margin loss is reported as a consistently good choice across models compared with plain log loss.Source: Figure 3 · p. 7, §6.6 · p. 8
7 Related Work
Related work includes joint syntax-semantic models, latent-syntax approaches, and syntax-free neural methods. This paper uses partial syntax annotations, potentially from a separate corpus, and presents its basic model as the first non-syntactic frame-semantic parser.
Joint syntax and semantics
- Joint syntax-semantic models can represent syntax-role connections more directly, but require syntactic and semantic annotations over the same corpora.Source: §7 · p. 9
- This work uses partial syntax annotations, specifically bracketing, which need not come from the frame-semantic corpus.Source: §7 · p. 9
Latent syntax
- The authors characterize their basic model as the first non-syntactic frame-semantic parser, distinguishing it from latent-syntax approaches that require no syntactic supervision.Source: §7 · p. 9
8 Conclusion
The paper presents a softmax-margin semi-Markov model for predicting frame-semantic arguments, then adds syntax through feature pipelines or a training-only multitask scaffold. Both approaches improve over the baseline and achieve state-of-the-art performance; the scaffold avoids parsing at training and test time.
8 Conclusion
- The paper presents a softmax-margin semi-Markov model that uses representation learning to predict frame-semantic arguments.Source: §8 · p. 9
- Both syntactic feature pipelines and the multitask scaffold improve over the baseline and achieve state-of-the-art performance.Source: §8 · p. 9
- The scaffold is a cheaper alternative to syntactic features because it requires no syntactic parsing during training or testing.Applying scaffolding to other tasks that rely on syntactic parsing pipelines is identified as a future direction.Source: §8 · p. 9
A Frame Identification
The paper presents a syntax-free frame identification system that classifies the frame evoked by each target and lexical unit. For fair comparison with prior systems, it uses gold targets and lexical units.
A Frame Identification
- Frame identification is treated as an independent multiclass classification task for each target.Given a sentence, target, and corresponding lexical unit, the system predicts the frame evoked by that target.Source: A Frame Identification · p. 12
- The system is released as a syntax-free frame identification complement to the argument identification system.Source: A Frame Identification · p. 12
- For fair comparison with Hermann et al. (2014) and FitzGerald et al. (2015), evaluation uses gold targets and lexical units.Source: A Frame Identification · p. 12
A.1 Model
The model predicts a frame from a sentence, predicted part-of-speech sequence, target span, and lexical unit. It builds contextual token and target representations, then aggregates frame scores with a five-model ensemble.
A.1 Model
- The model input combines a sentence, predicted part-of-speech sequence, target span, and lexical unit; FrameNet specifies possible frames for each lexical unit.Source: §A.1 · p. 12
- Each token representation uses learned word-type and part-of-speech embeddings alongside a fixed pretrained word embedding, processed by a token biLSTM.Source: §A.1 · p. 12
- The target representation uses token states over the target span and a size-one neighboring context window as input to a forward LSTM.Source: §A.1 · p. 12
- A learned lexical-unit embedding is combined with the target representation and passed through a rectified linear unit to produce model scores.Source: §A.1 · p. 12
- Training minimizes negative log likelihood with ADAM, and five independently initialized models are ensembled by summing their segment scores at test time.Pretrained GloVe vectors have dimension 100.Source: §A.1 · p. 12
A.2 Results
The syntax-free frame identification model significantly outperforms SEMAFOR and performs similarly to Hartmann et al. (2017), despite using no external resources. In end-to-end parsing, performance is lower than argument identification with gold frames.
A.2 Results
- The syntax-free model significantly outperforms SEMAFOR and performs similarly to Hartmann et al. (2017), while using no external resources.Hermann et al. (2014) reuse most SEMAFOR features and add WSABIE embeddings; FitzGerald et al. (2015) report slightly better scores, possibly due to initialization or syntactic parses.Source: §A.2 · p. 12
- The model’s end-to-end performance with frame and argument identification is lower than its argument-identification results with gold frames.The authors note this illustrates the importance of frame identification in the parsing pipeline.Source: §A.2 · p. 12