Source-linked AI summary
Neural AMR: Sequence-to-Sequence Models for Parsing and Generation
Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, Luke Zettlemoyer
TL;DR
AMR parsing and generation are constrained by limited labeled data and the non-sequential structure of AMR graphs. The paper combines paired training on millions of unlabeled sentences with AMR graph preprocessing for seq2seq models, achieving 62.1 SMATCH parsing and 33.8 BLEU generation. Its analyses indicate robustness to graph-to-sequence ordering variations, while future work targets other meaning representations and multilingual applications.
Problem
AMR annotation is expensive, and graph non-sequentiality and limited labeled data constrain seq2seq parsing and generation.
Method
The approach combines paired self-training and generator pre-training on millions of unlabeled sentences with preprocessing that simplifies AMR graphs and supports graph-isomorphic linearizations.
Results
The models achieve competitive parsing performance of 62.1 SMATCH and state-of-the-art generation performance of 33.8 BLEU.
Takeaways & Limitations
Sequence-based AMR models can use varied graph linearization orders while achieving strong results without significant reliance on external semantic resources.
Takeaways & Limitations
The evaluation reports human-ordering results, although practical realization systems require consistent or random-derived linearization orders.
Abstract
from arXiv · showhide
Sequence-to-sequence models have shown strong performance across a broad range of applications. However, their application to parsing and generating text usingAbstract Meaning Representation (AMR)has been limited, due to the relatively limited amount of labeled data and the non-sequential nature of the AMR graphs. We present a novel training procedure that can lift this limitation using millions of unlabeled sentences and careful preprocessing of the AMR graphs. For AMR parsing, our model achieves competitive results of 62.1SMATCH, the current best score reported without significant use of external semantic resources. For AMR generation, our model establishes a new state-of-the-art performance of BLEU 33.8. We present extensive ablative and qualitative analysis including strong evidence that sequence-based AMR models are robust against ordering variations of graph-to-sequence conversions.
1 Introduction
AMR represents meaning as directed semantic graphs, but expensive annotation and graph linearization complicate neural modeling. The paper addresses these constraints with paired training, graph preprocessing, and seq2seq models for parsing and generation.
- 1 Introduction: AMR encodes natural-language meaning as a directed graph of semantic dependencies while abstracting away surface forms.It supports applications including machine translation, summarization, sentence compression, and event extraction.
- 1 Introduction: Expensive AMR annotation limits the use of neural network models.
- 1 Introduction: Seq2seq models achieve strong results for both text-to-AMR parsing and AMR-to-text generation despite graph linearization and data-sparsity challenges.The models can use any graph-isomorphic linearization, while unlabeled text reduces sparsity.
- 1 Introduction: Paired training bootstraps a parser on millions of unlabeled Gigaword sentences, then uses automatically parsed graphs to pre-train the generator before supervised fine-tuning.
- 1 Introduction: Graph preprocessing anonymizes entities and dates, groups entity categories, and concisely encodes nesting information to reduce sparsity and graph complexity.Any depth-first traversal, including a different random order for each example, can then serve as an effective linearization.
- 1 Introduction: 62.1 SMATCH for parsing is competitive, while 33.8 BLEU establishes a new state of the art for generation.The experiments use the LDC2015E86 AMR corpus and include ablations and qualitative analysis.
2 Related Work
Prior AMR systems use alignment-based, grammar-based, and optimization-based approaches, often relying on external resources. This paper instead uses a neural approach trained directly on parallel data with large unannotated English corpora.
- 2 Related Work: Alignment-based parsers identify concepts and relations using graph algorithms and alignment features derived from lexical cues and hand-written rules.
- 2 Related Work: Grammar-based parsers perform graph transformations or grammar induction, commonly depending on externally trained parsers, pretrained categories, or semantic resources.
- 2 Related Work: Earlier neural parsers address sparsity by anonymizing named entities and typing low-frequency words, producing compact vocabularies.The paper instead introduces unlabeled sentences to reduce out-of-vocabulary rates without reducing vocabulary size.
- 2 Related Work: AMR generation has been formulated with tree-to-string transduction, machine-translation decoding, or traveling-salesman optimization to determine graph linearization.
- 2 Related Work: The paired training procedure differs from back-translation by pre-training on the external corpus first and fine-tuning on the original dataset afterward.
3 Methods
The method trains separate attention-based seq2seq models for parsing and generation, converts AMR graphs into recoverable token sequences, and uses paired self-training with unlabeled text. Graph preprocessing and staged training address sparsity while preserving graph structure.
- 3.1 Tasks: The tasks use paired sentences and rooted directed acyclic AMR graphs containing concept nodes and labeled semantic-relation edges.
- 3.1 Tasks: The parser predicts an AMR graph from a sentence, while the generator predicts a sentence from an AMR graph.
- 3.2 Sequence-to-sequence Models: Both tasks use independent global-attention seq2seq predictors, implemented with stacked bidirectional-LSTM encoders and stacked LSTM decoders.The architecture includes unknown-word replacement and encoder modifications involving hidden-state concatenation and first-layer dropout.
- 3.3 Graph-to-sequence Conversion: AMR graphs are converted into linear token sequences using a linearization order and rendering function, with parsing requiring graph recovery from the sequence.
- 3.4 Paired Training: The paired-training procedure uses an unlabeled sentence corpus to reduce sparsity because jointly annotated sentence–AMR pairs are expensive and limited to thousands.
- 3.4 Paired Training: Self-training repeatedly parses sampled unlabeled sentences, retrains the parser on the resulting pairs, and fine-tunes it on the original annotated data.The sample size grows by an order of magnitude after each iteration.
- 3.4 Paired Training: The final self-trained parser labels a larger unlabeled sample, whose AMR–sentence pairs pre-train the generator before supervised fine-tuning.
4 AMR Preprocessing
The paper preprocesses AMR graphs and sentence-graph pairs to reduce sequence complexity and vocabulary sparsity while preserving recoverable information. It combines simplification, anonymization, clustering, scope encoding, and depth-first linearization.
- Preprocessing goals: Preprocessing targets shorter, easier-to-learn sequences and sparsity in open-class vocabulary such as named entities and quantities.The procedure aims to reduce graph complexity while retaining enough original information.
- Graph simplification: Graph simplification removes variable names and instance-of relations, replaces re-entrant variables with concepts, and removes senses for generation.These changes reduce the length of linearized graphs, although replacing re-entrant nodes can lose information.
- Anonymization: Named-entity anonymization replaces named-entity subgraphs with fine-grained type-and-index tokens and records alignments for restoring entity names during generation.The method is designed to reduce sparsity and handle unseen entities.
- Anonymization: Dates receive dedicated anonymization tokens for date components and formats, with the corresponding format rendered during generation.The paper distinguishes year, month, day, and textual-versus-numeric forms.
- Anonymization: Named-entity clustering maps fine-grained AMR types to four coarse Stanford NER categories during generation to reduce sparsity.The categories are person, location, organization, and misc.
- Linearization: The graph is linearized by depth-first node traversal with backward edges, while rendering emits node and edge tokens with bracketed scope markers.Scope markers are omitted when a node has only one child, reducing generated tokens.
5 Experimental Setup
Experiments use the LDC2015E86 AMR corpus, Gigaword-derived unlabeled data, and SMATCH and BLEU evaluation. Models are tuned on development performance and trained with specified optimization, self-training, and decoding settings.
- Data: The LDC2015E86 corpus contains 16,833/1,368/1,371 train/dev/test examples, while Gigaword supplies vocabulary-matched unlabeled sentences without overlap with AMR train or test sets.Gigaword samples are used for the paired training procedure.
- Reported results: Table 1 reports parsing SMATCH scores, including comparisons whose starred results come from the newswire portion of LDC2014T12.The table caption identifies the dataset-release distinction for starred values.
- Evaluation: AMR parsing is evaluated with SMATCH, and AMR generation is evaluated with BLEU.The reported corpus experiments use these task-specific metrics.
- Optimization: Embedding and RNN hidden sizes are selected from {128, 256, 500, 1024}, with both ultimately set to 500 using development performance.Training uses cross-entropy loss, stochastic gradient descent, batch size 100, and dropout 0.5.
- Training procedure: Self-training begins with a 200,000-sentence Gigaword sample and runs for a maximum of three iterations, using deeper two-layer models after initialization.The best development-set model is retained during training choices.
- Reported results: Table 2 reports corpus and Gigaword statistics, including NL-side OOV@1 and OOV@5 rates and vocabulary sizes of 13027 AMR tokens and 17319 NL tokens.OOV thresholds are 1 and 5, respectively.
6 Results
Self-training and preprocessing improve seq2seq AMR performance across parsing and generation. The final systems reach competitive parsing and state-of-the-art generation results, while ablations identify anonymization and scope markers as especially important.
- Parsing: The parsing system outperforms the original JAMR but remains behind parser-dependent CAMR and resource-heavy SBMT approaches.The comparison concerns the full models reported in the paper.
- Sparsity: GIGA-20M reduces the OOV rate at threshold 5 almost fivefold relative to the smaller data setting.The paper attributes this reduction to incorporating increasing amounts of Gigaword sentences.
- Ablations: Each preprocessing component is required for generation, with scope markers and anonymization contributing most to performance.The authors associate scope markers with capturing long-range semantic relationships in AMR graphs.
- Ablations: Seq2seq AMR parsing is largely ineffective without anonymization in the reported ablation analysis.This finding is stated as consistent with previous work.
7 Linearization Evaluation
The evaluation compares human-authored, global-random, and per-example random AMR linearization orders for generation. Seq2seq models are largely insensitive to ordering variation, although human order performs better and appears to encode additional sentence-order information.
- 7.1 Linearization Orders: Seq2seq models are largely agnostic to AMR linearization order, with per-example random ordering performing nearly identically to global random ordering.Random ordering performs somewhat worse than human ordering, but the two random strategies produce nearly identical results.
- 7.2 Results: Human linearization order outperforms random ordering, suggesting it carries information not apparent in the AMR graph structure.The authors investigate whether edge ordering correlates with the order in which children are realized in generated sentences.
- 7.2 Results: 57.6% of AMR edge pairs always occurred in the same relative order, revealing no additional generation-order information for those pairs.This ordering is consistent with constraints in the AMR annotation tool, such as :ARG0 preceding :ARG1.
- 7.2 Results: Among variable edge pairs, 70.3% appeared in an order consistent with their realization order in the sentence.The relative ordering of location and time edges was 17% more indicative of generation order than the majority pattern of locations preceding time.
- 7.2 Results: Practical systems intended to generate AMRs for later realization should train with either consistent or random-derived linearization orders.The paper reports human orderings for comparison with previous work, while noting this practical constraint.
8 Qualitative Results
Qualitative analysis shows that the generator can produce nearly correct text but still omits graph information and makes attachment errors. Across sampled development examples, coverage problems are the dominant error type, followed by fluency mistakes.
- 8 Qualitative Results: The first qualitative example is nearly perfect except for a small grammatical error attributed to anonymization.The example illustrates that preprocessing can leave a localized realization problem even when the generated text is otherwise strong.
- 8 Qualitative Results: In a challenging deep right-branching graph, the generator omits the concepts terrorist and virus.The example also contains a coordination of stabilize and push in a subordinate clause headed by state.
- 8 Qualitative Results: A more difficult example omits the subgraph headed by expert and makes attachment errors in the final two subgraphs.The incorrect attachments assign unimpeachable and irrefutable to equipment rather than evidence.
- 8 Qualitative Results: Coverage issues dominate the generator’s errors, followed by fluency mistakes, while attachment errors are less frequent.The analysis is based on 50 randomly selected development-set examples and links the lower attachment-error rate to robustness with graph linearization and long-range dependencies.
9 Conclusions
The paper applies seq2seq models to AMR parsing and generation using graph preprocessing and pretraining on millions of unlabeled Gigaword sentences. It achieves competitive parsing performance and state-of-the-art generation performance, and proposes extending the approach to other meaning representations.
- 9 Conclusions: Seq2seq models address AMR parsing and generation through graph preprocessing and pretraining on millions of unlabeled Gigaword sentences.The approach avoids relying on knowledge bases and externally trained parsers.
- 9 Conclusions: SMATCH 62.1 is competitive for parsing, while BLEU 33.8 establishes state-of-the-art performance for generation.These results are reported for the AMR parsing and generation tasks, respectively.
- 9 Conclusions: Future work proposes extending the models to Minimal Recursion Semantics and applying them to semantics-based machine translation.The proposed direction includes studying MRS as an intermediate representation, especially for distant language pairs such as English and Japanese.