PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Diego Marcheggiani, Ivan Titov

arXiv:1703.04826v4cs.CLcs.LG

TL;DR

Semantic role labeling uses predicate-argument structure, and the paper exploits its relationship to syntax with graph convolutional sentence encoders for syntactic dependency graphs. Combining GCN and LSTM layers improves over LSTM SRL models, achieving reported state-of-the-art results for Chinese and English.

  • Problem

    Semantic role labeling identifies a sentence’s predicate-argument structure, while recent neural SRL models have largely set aside syntactic information despite close syntax-semantics relations.

  • Method

    The paper introduces GCNs for labeled directed syntactic dependency structures and uses them with LSTM layers as syntax-aware sentence encoders for SRL.

  • Results

    89.1% F1 was achieved by a three-model product-of-experts ensemble, improving 1.2% over the best previous result; the best model also exceeded the Chinese state-of-the-art by 3.1%.

  • Takeaways & Limitations

    GCN and LSTM layers are complementary for syntax-aware SRL, with one GCN layer outperforming the LSTM version by 1.9% for Chinese and 0.6% for English.

  • Takeaways & Limitations

    The evaluated model makes locally conditionally independent argument decisions and leaves beneficial global modeling for future work.

Abstract

from arXiv · show

Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard NLP pipeline. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of neural networks operating on graphs, suited to model syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark (CoNLL-2009) both for Chinese and English.

1 Introduction

SRL identifies predicate–argument structure, and the paper addresses the challenge of incorporating syntactic information into neural sequence models by introducing syntax-aware GCNs. Combining GCNs with LSTMs substantially improves SRL performance on English and Chinese CoNLL-2009.

  • 1 Introduction: SRL identifies predicates, their senses, and arguments assigned to semantic roles such as agents and patients.It is regarded as an important NLP pipeline step supporting downstream tasks including information extraction and question answering.
  • 1 Introduction: Semantic and syntactic representations are closely related, motivating the use of syntactic information for predicting semantics.The paper notes that many syntactic dependency arcs are mirrored in semantic dependency graphs, while the syntax–semantics interface remains nontrivial.
  • 1 Introduction: The paper introduces GCNs for encoding syntactic information at the word level and modeling labeled directed dependency structures.Each node receives a real-valued representation of relevant neighborhood information.
  • 1 Introduction: GCN and LSTM layers have complementary modeling power in the proposed SRL model.Stacking GCNs on LSTM layers substantially improves an already state-of-the-art LSTM SRL model.
  • 1 Introduction: The paper focuses on SRL, leaving combined syntactic-semantic graph encoders for machine translation and question answering as future work.GCNs are not constrained to tree-structured graphs, but those broader applications are not evaluated here.
  • 1 Introduction: The paper reports state-of-the-art results on the English and Chinese portions of CoNLL-2009.The introduction presents this as a central contribution alongside the syntax-aware GCN formulation.

2 Graph Convolutional Networks

GCNs produce node representations from graph neighborhoods, with stacked layers extending the modeled context beyond immediate neighbors. The paper’s syntactic setting requires adapting this framework to dependency graphs and omitting normalization factors used in the original formulation.

  • 2 Graph Convolutional Networks: GCNs are neural networks that induce real-valued node features from graph-neighborhood properties.The induced features can support tasks such as node classification.
  • 2 Graph Convolutional Networks: One convolution layer captures immediate neighbors, while K stacked layers capture nodes at most K hops away.Stacking layers therefore incorporates progressively higher-degree neighborhoods.
  • 2 Graph Convolutional Networks: A node representation combines neighboring input features through a weight matrix, bias, and ReLU activation.Self-loops ensure that the node’s own input feature also affects its induced representation.
  • 2 Graph Convolutional Networks: The formulation drops the normalization factors used in Kipf and Welling’s GCN because they are not used in the syntactic GCNs.This is an implementation choice specific to the paper’s syntactic variant.
  • 2 Graph Convolutional Networks: The paper’s syntactic GCN adapts graph convolution to directed, labeled dependency structures.Figure 2 distinguishes syntactic-function parameters and reverse information flow from dependents to heads.

3 Syntactic GCNs

The paper generalizes GCNs to directed, labeled syntactic dependency graphs and adds edge-wise gating to manage direction, labels, and unreliable parser edges. Combined with LSTMs, this design helps capture long-distance predicate–argument dependencies.

  • 3.1 Incorporating directions and labels: The proposed GCN generalization propagates information along directed, labeled dependency edges in both syntactic directions, while distinguishing self-loops.Edge labels encode syntactic functions and direction relative to dependency arcs.
  • 3.1 Incorporating directions and labels: The model reduces parameterization by sharing direction-specific matrices and encoding syntactic functions in label-dependent feature vectors.The three edge types are head-to-dependent, dependent-to-head, and self-loop.
  • 3.2 Edge-wise gating: Edge-wise gates let the model up-weight useful syntactic messages and down-weight potentially erroneous edges from automatically predicted parses.The motivation is that uniformly accepting all neighboring information is inappropriate for SRL.
  • 3.3 Complementarity of GCNs and LSTMs: GCNs capture only K-hop neighborhoods with K layers, but LSTM states provide broader sequence information before graph convolution.This combination is intended to address long predicate–argument paths without requiring many GCN layers.
  • 3.3 Complementarity of GCNs and LSTMs: 20% of English and 30% of Chinese arguments are more than 5 tokens from their predicates, falling to 9% and 13% after one longest-edge teleportation.The figures motivate complementary LSTM and GCN encoders for long-distance dependencies.

4 Syntax-Aware Neural SRL Encoder

The SRL encoder combines word representations, a BiLSTM, a syntax-based GCN, and a role classifier. It uses automatically predicted syntax to re-encode contextual word states and predicts predicate-specific semantic roles for candidate arguments.

  • 4 Syntax-Aware Neural SRL Encoder: The benchmark marks predicate positions in advance, while predicate disambiguation uses an off-the-shelf model and arguments are processed separately for each predicate.Each predicate-specific task is treated as sequence labeling over the sentence.
  • 4 Syntax-Aware Neural SRL Encoder: The SRL model consists of word embeddings, a BiLSTM encoder, a syntax-based GCN encoder, and a role classifier.The GCN re-encodes BiLSTM representations using automatically predicted syntactic structure.
  • 4.1 Word representations: Each word representation concatenates trainable random word, part-of-speech, and predicate-lemma embeddings with a fixed pre-trained word embedding.The lemma embedding is active only when the word is a predicate.
  • 4.4 Semantic role classifier: The classifier concatenates candidate-argument and predicate representations from the GCN and predicts a semantic role, including the special NULL role.The classifier relies on GCN-provided word representations.
  • 4.4 Semantic role classifier: Predicate lemmas and semantic roles are jointly embedded so role predictions remain predicate-specific while supporting representations for roles linked to infrequent predicates.Training uses categorical cross-entropy.

5 Experiments

Experiments evaluate the GCN-based SRL model on English and Chinese CoNLL-2009 data, including development-set ablations, test-set comparisons, dependency-distance effects, and robustness to out-of-domain syntax.

  • 5.1 Datasets and parameters: The model is evaluated on English and Chinese CoNLL-2009 splits using provided or automatically predicted parses and external word embeddings.Predicate disambiguation models and predicted POS tags are supplied separately; model selection uses the English development set.
  • 5.2 Results and discussion: One GCN layer improves SRL-only development F1 over the LSTM version by 1.9% for Chinese and 0.6% for English.The comparison uses a syntax-agnostic model whose hyperparameters were tuned separately for best development performance.
  • 5.2 Results and discussion: Removing edge-wise gates lowers F1 by 0.3% for English and 0.6% for Chinese, while stacking two GCN layers provides no benefit when BiLSTMs remain.Without BiLSTM layers, two GCN layers instead produce gains of 3.8% F1 for English and 3.0% for Chinese; a third layer improves performance further.
  • 5.2 Results and discussion: GCNs are more beneficial for long-distance predicate–argument dependencies, while shorter dependencies are already captured accurately by the LSTM encoder.The distance analysis reports F1 on the English development set as a function of token distance.
  • 5.2 Results and discussion: On test data, the model improves over the best previous English result by 1.2% to 89.1% F1 with a three-model ensemble and exceeds the Chinese state-of-the-art model by 3.1%.The English model also outperforms prior global models and ensembles without using global modeling or ensembles; the 89.1% result uses product-of-experts combination.
  • 5.2 Results and discussion: On English out-of-domain data, the model substantially outperforms previous syntax-aware systems but does not outperform the syntax-agnostic model.The authors attribute the setting’s difficulty partly to lower-quality predicted syntax and interpret the result as evidence of robustness to syntactic mistakes.

6 Related Work

Prior work incorporated syntax into neural SRL and other NLP models in several ways. This paper contrasts those approaches with word-level syntactic representations that can complement or replace recurrent models.

  • Earlier SRL systems used RNNs with shift-reduce parsers to model dependencies across joint syntactic and semantic parsing actions.
  • Later neural SRL models incorporated syntax through CRF features or LSTM representations of syntactic paths between predicates and arguments.
  • Tree-structured recursive networks have also been used to incorporate syntax in NLP, including by stacking them over sequential BiLSTM representations.
  • The presented approach differs by encoding syntactic information at the word level rather than primarily inducing sentence- or constituent-level representations.The authors identify this as attractive because it can be used instead of or alongside recurrent neural networks.

7 Conclusions and Future Work

The paper demonstrates GCNs as a way to build syntax-aware SRL models and reports state-of-the-art results for Chinese and English. It also identifies broader graph-structured NLP applications and joint argument modeling as future directions.

  • GCNs are used to incorporate syntactic information and construct a syntax-aware SRL model.
  • The model achieves state-of-the-art results on Chinese and English SRL.
  • Jointly modeling arguments is identified as a likely way to improve performance beyond independently labeling arguments.
  • GCNs may incorporate syntactic, semantic, discourse, and coreference structures in other NLP tasks because they apply to general graph structures.

A Hyperparameter values

This appendix section lists hyperparameter values.

  • The section consists of a table of hyperparameter values.
Loading 1703.04826v4…