PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multitask Learning for Semantic Dependency Parsing

Hao Peng, Sam Thomson, Noah A. Smith

arXiv:1704.06855v2cs.CL

TL;DR

Semantic dependency parsing is hindered by limited annotations within competing graph formalisms. The paper introduces neural parsing and multitask approaches that share information across formalisms, improving performance without syntax and achieving a new state of the art.

  • Problem

    Competing semantic theories fragment annotations, leaving limited data in each formalism and making neural semantic dependency parsing difficult.

  • Method

    The paper combines bidirectional-LSTM and multi-layer perceptron parsing with efficient inference, then models multiple formalisms through parameter sharing and cross-task output structures.

  • Results

    Both multitask approaches improve performance across formalisms on average, while the base system significantly improves over state-of-the-art systems without syntax.

  • Takeaways & Limitations

    Deep multitask learning can improve graph-based semantic dependency parsing by exploiting relationships among parallel formalisms without syntactic parsing.

  • Takeaways & Limitations

    The benefit of syntactic information remains unclear, although automatically generated parses have produced more than 1% absolute improvements in related work.

Abstract

from arXiv · show

We present a deep neural architecture that parses sentences into three semantic dependency graph formalisms. By using efficient, nearly arc-factored inference and a bidirectional-LSTM composed with a multi-layer perceptron, our base system is able to significantly improve the state of the art for semantic dependency parsing, without using hand-engineered features or syntax. We then explore two multitask learning approaches---one that shares parameters across formalisms, and one that uses higher-order structures to predict the graphs jointly. We find that both approaches improve performance across formalisms on average, achieving a new state of the art. Our code is open-source and available at https://github.com/Noahs-ARK/NeurboParser.

1 Introduction

The paper treats three parallel semantic dependency formalisms as related tasks and uses multitask learning to exploit their overlap. A neural single-task baseline improves prior systems, while parameter sharing and cross-task structures further improve performance.

  • 1 Introduction: The 2015 SDP shared task supplies parallel English annotations for three semantic graph representations, enabling multitask learning across formalisms.The paper presents this as the first treatment of SDP as a multitask learning problem.
  • 1 Introduction: The base system combines a bidirectional LSTM, multilayer perceptron, and efficient nearly arc-factored inference, without hand-engineered features or syntax.It parses each formalism separately and significantly improves over state-of-the-art methods.
  • 1 Introduction: The two multitask extensions use parameter sharing and cross-task higher-order structures to model relationships among formalisms.Both train on all three formalisms and use AD3 for decoding.
  • 1 Introduction: Both multitask techniques improve over the basic model, with a smaller additional gain when combined.The improvement in unlabeled F1 is greater for the two structurally more similar formalisms.

2 Broad-Coverage Semantic Dependency Parsing (SDP)

Broad-Coverage Semantic Dependency Parsing represents predicate-argument relations as labeled directed graphs over sentence tokens. Its three English formalisms differ in linguistic source and structural properties, with PAS closest to syntax and PSD harder to predict.

  • 2 Broad-Coverage Semantic Dependency Parsing (SDP): SDP converts three semantic formalisms into bilexical dependencies whose graphs represent predicate-argument relations between content-bearing words.The formalisms come from varied linguistic traditions but share this central goal.
  • 2 Broad-Coverage Semantic Dependency Parsing (SDP): Semantic dependencies abstract over alternative syntactic realizations while distinguishing different senses expressed by similar syntax.They are therefore closer in purpose to semantic role labeling or abstract meaning representation than to ordinary syntactic dependencies.
  • 2 Broad-Coverage Semantic Dependency Parsing (SDP): The representations are labeled directed graphs over sentence tokens rather than necessarily trees, allowing tokens to participate in multiple predicate-argument relations.Analyses may omit non-content-bearing tokens while remaining connected over content-bearing tokens.
  • 2 Broad-Coverage Semantic Dependency Parsing (SDP): DM, PAS, and PSD derive from different English resources, and this study trains and evaluates only on English annotations.DM comes from DeepBank, PAS from the Enju Treebank, and PSD from the Prague Czech-English Dependency Treebank.
  • 2 Broad-Coverage Semantic Dependency Parsing (SDP): PAS follows syntax most closely and is considered easiest to predict, whereas PSD has the largest label set and substantially lower prior performance.The supplied passage establishes a contrast in prediction difficulty across the formalisms.

3 Single-Task SDP

The single-task system parses each semantic formalism separately with a neural graph-based model that scores predicates and arcs using contextual token representations. Efficient constrained decoding and syntax-free training yield significant improvements over published systems on in-domain and out-of-domain data.

  • 3.1 Problem Formulation: The scoring function decomposes into local part scores over possible labeled graphs, yielding a graph-based all-pairs formulation.The model differs across variants primarily in which structures are scored and how the local scoring function is parameterized.
  • 3.2.1 Basic Structures: AD3 decodes the highest-scoring internally consistent graph while enforcing predicate, labeling, and determinism constraints over overlapping structures.Without the predicate and determinism constraints, decoding would be arc-factored; overlapping structures motivate approximate dual-decomposition inference.
  • 3.2.2 Basic Scoring: The model scores predicates, unlabeled arcs, and labeled arcs with shared BiLSTM representations and structure-specific MLPs.The BiLSTM contextualizes each token, while separate MLPs process each structure type; the architecture uses two layers in practice.
  • 3.3 Learning: Training uses an ℓ2-regularized max-margin objective with weighted Hamming costs that favor recall over precision.False-negative arc predictions receive cost 0.6, whereas false positives receive cost 0.4.
  • 3.4 Experiments: Evaluation uses SemEval 2015 Task 18 closed-track data without syntactic analyses, split into 33,964 training, 1,692 development, 1,410 in-domain test, and 1,849 out-of-domain test sentences.The closed track excludes both machine-generated and gold syntactic analyses from training and testing.
  • 3.4 Experiments: A 1.1% in-domain and 1.6% out-of-domain absolute labeled F1 improvement over the best published results establishes the basic model as a stronger baseline.Table 2 reports labeled F1 on both test settings and compares against two published systems.

4 Multitask SDP

The paper extends a strong single-task semantic dependency parser with parameter sharing and cross-task higher-order structures across three formalisms. These multitask models improve performance overall, with gains linked to structural similarity and a computational cost for joint inference.

  • 4.2–4.3 Multitask Models: The multitask models use all three formalisms’ training data, either through shared representations with separate inference or cross-task higher-order structures with joint inference.Both approaches use AD3 for decoding and the same margin-based training objective as the single-task model.
  • 4.2 Multitask SDP with Parameter Sharing: FREDA augments task-specific BiLSTM encoders with a shared encoder, whereas SHARED uses only the shared encoder and retains task-specific output representations.The shared features are intended to capture global patterns while preserving formalism-specific components.
  • 4.3 Multitask SDP with Cross-Task Structures: Cross-task higher-order scoring models interactions among arcs from different formalisms using low-rank tensors, limiting parameters that would otherwise become unreasonably numerous.The modeled structures involve arcs sharing the same head and modifier, and joint decoding predicts all three formalisms together.
  • 4.4 Implementation Details: Joint inference reduces throughput to roughly 5.2 sentences, or 15.5 task-specific graphs, per second on a single Xeon E5-2690 CPU.This cost is reported for the third-order model.
  • 4.5 Experiments: By 1.7% absolute F1 over the best closed-track system and 0.9% over the best open-track system, SHARED3 and FREDA3 achieve the strongest reported results without syntax.Three of four multitask variants improve over the basic model, while the out-of-domain PSD advantage over the open-track system is not statistically significant.
  • 4.5 Experiments: FREDA3 improves unlabeled performance on DM and PAS but degrades it on PSD relative to FREDA1, supporting a connection between multitask gains and formalism similarity.DM and PAS are more structurally similar, while PSD differs notably in edge directions.

5 Related Work

The paper situates its graph-based semantic parser among work on graph parsing and multitask learning in NLP. It builds on representation learning, BiLSTM encoders, parameter sharing, and joint learning across tasks.

  • Related work: The related work spans two complementary traditions: graph-based semantic parsing and multitask learning across NLP tasks.These strands provide the context for combining neural graph parsing with multitask learning across semantic formalisms.
  • Graph-based parsing: Graph-based parsing adapted methods from non-projective syntactic parsing to semantic parsing, increasingly replacing hand-engineered features with learned representations.The paper’s approach is closest to prior work using BiLSTMs to encode inputs.
  • Multitask learning in NLP: Multitask learning in NLP often replaces pipelines through joint learning, motivated partly by concerns about cascading errors.Prior work includes shared word representations, continuous stacking, multilingual parameter sharing, and domain-adaptation-inspired multitask learning.

6 Conclusion

The paper applies deep multitask learning to graph-based parsing through shared token encoders and cross-formalism output interactions. These approaches outperform state-of-the-art semantic dependency parsers without syntactic parsing and can extend to additional graph formalisms.

  • Conclusion: The two multitask approaches outperform state-of-the-art semantic dependency parsing systems that use syntax, despite using no syntactic parsing.The approaches share recurrent token-encoding parameters or introduce interactions between output structures across formalisms.
  • Conclusion: The first approach shares recurrent neural network parameters for encoding input tokens, while the second models interactions between output structures across formalisms.The paper characterizes these as two orthogonal ways to apply deep multitask learning to graph-based parsing.
  • Conclusion: Because the techniques apply to labeled directed graphs generally, they can be extended to additional semantic or non-semantic formalisms.The paper identifies cross-task scoring and inference without parallel annotations as future work.
Loading 1704.06855v2…