PaperlayerView exact revision on arXiv

Source-linked AI summary

AMR Parsing via Graph-Sequence Iterative Inference

Deng Cai, Wai Lam

arXiv:2004.05572v2cs.CLcs.LG

TL;DR

AMR parsing must map text to a semantic graph despite lacking explicit alignments between graph concepts and words, while concept and relation decisions interact. This paper uses iterative inference to make dual decisions over the input sequence and partial graph, reporting best-reported SMATCH results on two AMR benchmarks.

  • Problem

    AMR parsing lacks explicit alignments between graph concepts and text words, and one-pass prediction may fail to model important interactions between concept and relation decisions.

    Source: §1 · p. 1

  • Method

    The model incrementally constructs the graph while iteratively deciding which input text to abstract and where to place each concept in the partial graph.

    Source: §1 · pp. 1–2

  • Results

    80.2% SMATCH on LDC2017T10 and 75.4% on LDC2014T12 are the model’s best-reported scores, surpassing previous state-of-the-art models by large margins.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The model is aligner-free and can be effectively trained with limited labeled data.

    Source: §1 · p. 2

Abstract

from arXiv · show

We propose a new end-to-end model that treats AMR parsing as a series of dual decisions on the input sequence and the incrementally constructed graph. At each time step, our model performs multiple rounds of attention, reasoning, and composition that aim to answer two critical questions: (1) which part of the input \textit{sequence} to abstract; and (2) where in the output \textit{graph} to construct the new concept. We show that the answers to these two questions are mutually causalities. We design a model based on iterative inference that helps achieve better answers in both perspectives, leading to greatly improved parsing accuracy. Our experimental results significantly outperform all previously reported \textsc{Smatch} scores by large margins. Remarkably, without the help of any large-scale pre-trained language model (e.g., BERT), our model already surpasses previous state-of-the-art using BERT. With the help of BERT, we can push the state-of-the-art results to 80.2\% on LDC2017T10 (AMR 2.0) and 75.4\% on LDC2014T12 (AMR 1.0).

1 Introduction

The paper frames AMR parsing as graph construction from text without explicit concept-word alignments, and argues that concept and relation decisions need to interact. It proposes iterative graph-sequence inference and reports new best-reported results on two AMR benchmarks.

1 Introduction

  • AMR represents sentence meaning as a rooted, directed, labeled graph, but its lack of explicit concept-word alignments complicates parsing.The passage notes that this alignment challenge affects concept prediction and is closely tied to relation prediction.Source: §1 · p. 1
  • Prior approaches include latent-variable alignments and attention-based sequence-to-sequence or sequence-to-graph models; incremental graph construction is presented as computationally efficient and cognitively plausible.The passage associates incremental construction with first grasping core ideas and then adding details.Source: §1 · p. 1
  • The paper identifies one-pass prediction’s limited modeling of concept-relation interactions as a possible reason existing systems lag human-level performance, especially on long, informative sentences.The authors describe the proposed model as aligner-free and trainable with limited labeled data.Source: §1 · pp. 1–2
  • The model treats each graph expansion as a dual decision about which input text to abstract and where to construct a concept in the partial graph.It uses iterative inference to alternate between finding supporting text and reading the partially constructed graph.Source: §1 · pp. 1–2
  • The aligner-free parser outperforms previous best parsers on both AMR benchmarks, achieving best-reported SMATCH F1 scores of 80.2% on LDC2017T10 and 75.4% on LDC2014T12, surpassing prior state-of-the-art models by large margins.Source: §1 · p. 2

2 Related Work & Background

The section groups AMR parsers into two-stage pipelines and one-stage graph construction, then distinguishes several one-stage designs. It notes that reciprocal effects between concept and relation prediction have not been well utilized.

2 Related Work & Background

  • Two-stage parsing predicts concepts before relations, while one-stage parsing builds the graph incrementally.The cited examples include Flanigan et al. (2014), Lyu and Titov (2018), and Zhang et al. (2019a) for two-stage parsing.Source: §2 · p. 2
  • One-stage methods include transition-based parsing, seq2seq linearization, and graph-based parsing that jointly decides a new node and its connections.Graph-based systems may decide connections in order or in parallel.Source: §2 · p. 2
  • The authors state that reciprocal causation between relation prediction and concept prediction had not been closely studied or well utilized.This is the gap motivating the paper’s focus on their interaction.Source: §2 · p. 2
  • The section also lists hyperedge-replacement grammar, machine translation, combinatory categorical grammar, and AM algebra approaches outside its main taxonomy.The cited examples are Peng et al. (2015), Pust et al. (2015), Artzi et al. (2015), Groschwitz et al. (2018), and Lindemann et al. (2019).Source: §2 · p. 2

3 Motivation

The motivation frames each incremental graph expansion as choosing both text to abstract and a graph location for the new concept. It argues that checking these choices against each other can guide more harmonious decisions.

3 Motivation

  • Each expansion adds a concept abstracted from input text to complete a missing element in the partially constructed semantic graph.The example contrasts adding the concept boy for the subject of go-02 with adding negation triggered by “not.”Source: §3 · pp. 2–3
  • The overview alternates graph-side and sequence-side inference, selecting graph nodes and input tokens via attention mechanisms.The chain begins with an initial concept decision and repeats graph and sequence inference.Source: Figure 2 · p. 3
  • The two decisions can constrain one another: choosing “not” suggests a polarity relation, while seeking the action’s subject points toward “boy” rather than a function word.A graph review can reject a proposed concept that has no place to attach, and a sentence review can reject information absent from the text.Source: §3 · p. 3
  • The paper proposes iterative inference to exploit these mutual effects, with multiple rounds of reasoning intended to refine expansion decisions.The authors compare the process to forming and checking hypotheses across the graph and sequence before converging.Source: §3 · p. 3

4 Proposed Model

The model incrementally expands an AMR graph by iteratively refining paired decisions about what to abstract from the input and where to connect the new concept. Sequence and graph memories support dedicated concept and relation solvers whose decisions inform one another.

4.1 Overview

  • The overview combines text and graph memories with concept and graph solvers, so each solver's hypothesis can refine the other's decision.Text memories are computed once, while graph memories are built incrementally as parsing proceeds.Source: §4.1 · pp. 3–4

4.2 Sequence Encoder

  • A Transformer sequence encoder creates contextualized representations of the input tokens for use as text memories throughout parsing.It adds a BOS token whose representation serves as an overall input representation.Source: §4.2 · p. 4

4.3 Graph Encoder

  • The graph encoder represents the incrementally built graph as an ordered concept sequence, using masked self-attention and source-attention.This design supports training parallelization and incremental test-time computation, but inherently neglects edge information; auxiliary attention-head supervision did not improve performance.Source: §4.3 · p. 4

4.4 Concept Solver

  • The concept solver attends to input tokens and combines vocabulary prediction, lemma copying, and raw-token copying to generate a concept label.The copy channels support labels drawn from input lemmas and original token strings, including attribute values such as names or numbers.Source: §4.4 · p. 5

4.5 Relation Solver

  • Multiple attention heads identify existing source nodes for the new concept, allowing several nodes to be selected simultaneously.AMR reentrancies allow a concept to participate in multiple semantic relations.Source: §4.5 · p. 5

4.6 Iterative Inference

  • Iterative inference alternates between sequence and graph hypotheses to refine the concept to add and its connections before expanding the graph.After N reasoning steps, the model predicts edge labels and adds the concept and edges unless the concept is the termination symbol EOG.Source: §4.1 · p. 3, §4.6 · p. 6

5 Training & Prediction

Training maximizes decomposed step-wise likelihood using a breadth-first node order, with a mixed sibling-order strategy that becomes deterministic in final training steps. At test time, beam search selects graphs by step-wise log-likelihood, with complexity linear in graph size for fixed beam size.

5 Training & Prediction

  • Breadth-first traversal sets the reference node order, while sibling order is first mixed random and frequency-sorted, then deterministic in final training steps.The authors report that the deterministic-after-random strategy slightly improves performance.Source: §5 · pp. 6–7
  • O(k|V|) is the reported test-time complexity, with beam size k and graph node count |V|; beam search uses step-wise log-likelihood.The training objective is standard maximum likelihood over decomposed step-wise concept, edge, and edge-label probabilities.Source: §5 · pp. 6–7

6 Experiments

Experiments on AMR 2.0 and AMR 1.0 show strong benchmark results, while analyses examine the effects of iterative inference, sentence length, beam size, and attention over successive reasoning steps.

6.1 Experimental Setup

  • The evaluation uses AMR 2.0 and AMR 1.0, whose training set is about one-third the size, and ablations assess BERT and graph re-categorization.Hyperparameters are chosen on AMR 2.0 development data; the default uses four inference steps, and testing uses beam size 8.Source: §6.1 · p. 7

6.2 Experimental Results

  • On AMR 2.0, the model exceeds Zhang et al. (2019b) by 3.2% and surpasses 80% SMATCH; on AMR 1.0, it improves the best-reported result by 4.1% to 75.4%.Without BERT, it also outperforms prior BERT-based state-of-the-art systems on both datasets.Source: §6.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8
  • Fine-grained AMR 2.0 scores are highest on almost all subtasks, with roughly 2%–3% gains over Zhang et al. (2019b).The ablation study reports lower NER scores with graph re-categorization, attributed to rule-based NER not generalizing well to unseen entities.Source: §6.2 · p. 8, Table 1 · p. 8

6.3 More Analysis

  • SMATCH generally rises as inference steps increase, with the largest change from one step to two and diminishing later gains; model size stays constant across step counts.Figure 4 plots SMATCH by inference-step count, with sentences grouped by length.Source: §6.3 · p. 8, Figure 4 · p. 8
  • Iterative inference helps more on longer sentences, while shorter sentences reach peak performance earlier, motivating input-dependent step counts as future work.Figure 4 includes separate curves for three sentence-length classes.Source: Figure 4 · p. 8, §6.3 · p. 9
  • Larger test-time beams generally improve performance, but beam size 2 already captures most of the gains.Figure 6 plots SMATCH scores across beam sizes.Source: §6.3 · p. 9, Figure 6 · p. 9
  • In the case study, the parser corrects first-step mistakes over later reasoning steps, whose attention distributions become sharper.Figure 5 visualizes attention-score intensity during iterative reasoning.Source: §6.3 · p. 9, Figure 5 · p. 9
  • Parsing takes about 300 tokens/s with BERT and 330 tokens/s without BERT on one Nvidia P4 GPU; theoretical complexity is O(kbn), linear in graph size.Here k is the number of iterative steps, b the beam size, and n the number of graph nodes; absolute speed depends on implementation and hardware.Source: §6.3 · p. 9

7 Conclusion

The paper concludes that AMR parsing can be framed as iterative, node-by-node graph construction guided by dual sequence and graph decisions. The method advances state-of-the-art results on two AMR corpora, with adaptive inference steps and broader structured prediction proposed as future directions.

7 Conclusion

  • The method builds AMR graphs node by node, iteratively deciding what to abstract from the sequence and where to construct in the graph.It leverages the mutual causalities between these decisions.Source: §7 · p. 9
  • The method significantly advances state-of-the-art results on two AMR corpora.Source: §7 · p. 9
  • Future work may adapt the number of inference steps to each input sentence and apply the approach to structured prediction with two interdependent output parts.Source: §7 · p. 9

A Hyper-parameter Settings

The full models share hyper-parameter settings for the character-level CNNs and Transformer layers in both encoders. Training uses dropout, input masking, Adam optimization, a warm-up learning-rate schedule, and development-set early stopping.

A Hyper-parameter Settings

  • The sentence and graph encoders use shared hyper-parameter settings for their character-level CNNs and Transformer layers.The BERT model is the Hugging Face bert-base-cased implementation.Source: A Hyper-parameter Settings · p. 12
  • Training applies dropout at rate 0.2 and randomly replaces input lemmas, POS tags, and NER tags with UNK at rate 0.33.Source: A Hyper-parameter Settings · p. 12
  • Optimization uses Adam with β1 = 0.9 and β2 = 0.999, a learning-rate schedule with 2K warm-up steps, and development-set early stopping.Source: A Hyper-parameter Settings · p. 12

B AMR Pre- and Post-processing

The preprocessing and post-processing follow Zhang et al. (2019a,b). Preprocessing anonymizes entities, removes wiki links and polarity attributes, and compresses certain subgraphs; post-processing restores the original AMR format and adds back wiki links and polarity attributes.

B AMR Pre- and Post-processing

Related papers

Loading 2004.05572v2…