Source-linked AI summary
A Graph-to-Sequence Model for AMR-to-Text Generation
Linfeng Song, Yue Zhang, Zhiguo Wang, Daniel Gildea
TL;DR
AMR-to-text generation seeks to recover text with the same meaning as an input graph, while serialized sequence encoders can lose graph structure on large inputs. The paper introduces a graph-state LSTM that directly encodes AMR graphs, and it outperforms sequence-to-sequence baselines and prior work on standard and enlarged-data benchmarks.
Problem
AMR-to-text generation must recover sentence meaning from an AMR graph, but sequence-to-sequence encoders can lose graph-structure information after serialization, especially for large graphs.
Method
A graph-state LSTM directly encodes AMR structures through recurrent information exchange between connected node states, paired with an attention-based LSTM decoder and copy mechanism.
Results
The model surpasses a strong sequence-to-sequence baseline by 2.3 BLEU points and achieves 23.3 BLEU on the standard test set, 1.3 points above existing state of the art.
Takeaways & Limitations
Direct graph encoding outperforms sequence-to-sequence models on AMR-to-text generation and remains more effective than the compared prior model with scaled Gigaword training data.
Takeaways & Limitations
The authors do not exhaustively search for the optimal number of graph-state transition steps, and further increases may improve performance.
Abstract
from arXiv · showhide
The problem of AMR-to-text generation is to recover a text representing the same meaning as an input AMR graph. The current state-of-the-art method uses a sequence-to-sequence model, leveraging LSTM for encoding a linearized AMR structure. Although being able to model non-local semantic information, a sequence LSTM can lose information from the AMR graph structure, and thus faces challenges with large graphs, which result in long sequences. We introduce a neural graph-to-sequence model, using a novel LSTM structure for directly encoding graph-level semantics. On a standard benchmark, our model shows superior results to existing methods in the literature.
1 Introduction
AMR-to-text generation must recover a sentence’s meaning from a rooted, directed graph, but sequence models can lose graph structure when serialization separates related nodes. The paper introduces a graph-to-sequence model with a graph-state LSTM and reports improved benchmark performance.
- Problem: AMR-to-text generation produces text with the same meaning as an input AMR graph, whose concepts and relations are represented as nodes and edges.AMR abstracts away word tenses and function words, and node-to-phrase translation can be nonliteral.
- Motivation: Sequence-to-sequence models require AMR graph serialization, which can place closely related parents, children, and siblings far apart in the input sequence.This makes graph-structure information harder to represent, especially for large graphs.
- Contribution: The proposed graph-to-sequence model uses a graph-state LSTM to encode AMR structures directly.The graph state comprises node states, and connected nodes exchange information through recurrent transition steps.
- Model: The attention-based LSTM decoder includes a copy mechanism that helps reproduce sparse input tokens such as numbers and named entities.These tokens can appear in AMR graphs and require copying during generation.
- Results: 2.3 BLEU points separate the model from a strong sequence-to-sequence baseline on LDC2015E86, while its final test score is 23.3 BLEU.The final score is 1.3 points above the existing state of the art trained on the same dataset.
2 Baseline: a seq-to-seq model
The baseline linearizes each AMR graph with depth-first traversal and encodes the resulting token sequence using a bidirectional LSTM. An attention-based LSTM decoder uses encoder representations, coverage, and recurrent context to generate the output sequence.
- Baseline: The baseline follows an encoder-decoder sequence-to-sequence framework.It is described as following the encoder-decoder framework of Konstas et al. (2017).
- Input representation: Depth-first traversal linearizes each AMR graph into tokens, turning directly connected concepts such as “describe” and “genius” into sequence positions 14 apart.The example serialization preserves graph content but increases the distance between related nodes.
- Input representation: Token representations begin with word embeddings, optionally concatenated with the final hidden state of a forward character LSTM.The character encoder is adopted to alleviate data sparsity and improve input word representations.
- Encoder: A bidirectional LSTM encodes the linearized graph, generating forward and backward hidden states from each token and its neighboring sequence states.The encoder follows the depth-first-traversal representation used by Konstas et al. (2017).
- Decoder: The attention decoder generates output tokens recurrently from attention memory, prior hidden and context states, token embeddings, and coverage information.Attention memory concatenates vectors built from encoder states in both directions and the input token representation.
3 The graph-to-sequence model
The model directly encodes AMR graph structure with recurrent node-state transitions, allowing information to propagate through connected nodes without serialization. An attention-based LSTM decoder with copying generates text while supporting sparse tokens such as dates, numbers, and named entities.
- 3.1 The graph encoder: The recurrent graph encoder represents each AMR node with a hidden state and models graph structure directly without serialization.The graph state is the set of node states, and edge representations distinguish incoming from outgoing connections.
- 3.2 Recurrent steps: Repeated state transitions exchange information between connected nodes, enabling non-local propagation across the graph.LSTM cells are used during transitions to avoid gradient diminishing or bursting.
- 3.2 Recurrent steps: In a worst-case chain, information may require as many transition steps as the graph has nodes to reach an arbitrary node.The model evaluates different transition-step settings to study global encoding.
- 3.2 Recurrent steps: Node-state updates can be parallelized, making the graph encoder potentially efficient on GPUs and applicable to sequences, syntactic trees, and cyclic structures.This contrasts with the sequence LSTM encoder's sequential representation process.
- 3.3 The decoder: The attention-based LSTM decoder uses the last graph state, concatenates each node's last state with its attention representation, and averages node states for initialization.These changes adapt the decoder to the graph encoder's sequence of graph states.
- 3.5 Integrating the copy mechanism: The copy mechanism interpolates vocabulary generation with attention-based copying, favoring dates, numbers, and named entities appearing in the input graph.A learned switch controls whether each word is generated from the vocabulary or copied from the graph.
4 Training and decoding
The models are trained with cross-entropy loss and optimized with Adam, selecting the best development-set model for test evaluation. Dropout and beam search are used during training and decoding, respectively.
- 4 Training: Training minimizes cross-entropy loss over each gold-standard output sequence.The input is an AMR graph and the model parameters are optimized for the target sequence.
- 4 Training: Adam uses a learning rate of 0.001, and the model with the best devset performance is selected for test evaluation.Dropout with rate 0.1 is applied during training.
- 4 Decoding: Beam search with beam size 5 is used for decoding on Tesla K80 GPUs.Both training and decoding use Tesla K80 GPUs.
5 Experiments
Experiments evaluate the graph-to-sequence model on LDC2015E86, with additional Gigaword data, component comparisons, transition analyses, and qualitative cases. The graph encoder and copy mechanism improve BLEU and encoding efficiency over sequence baselines, while scaled automatic data further improves results.
- Data and setup: The experiments use LDC2015E86 with 16,833 training, 1,368 development, and 1,371 test instances, supplemented in some settings with Gigaword sentences parsed into AMRs.Training with external data follows pretraining on Gigaword followed by AMR-corpus fine-tuning.
- Development results: The graph encoder improves development BLEU by 1.6 points over Seq2seq alone and by 2.3 points when both models use copying.The authors attribute the larger gap to better node representations that support attention and copying.
- Development results: 3.4 BLEU points separate Graph2seq+copy from the Seq2seq baseline, and the graph encoder remains better with or without character LSTMs.Graph2seq+copy combines graph encoding with the copy mechanism.
- Development results: The graph encoder provides consistently better encoding efficiency than the sequence encoder, reflecting parallelization.The comparison uses encoding times because the two decoders are similar.
- Component effects: Character LSTMs improve both systems by roughly 0.6 BLEU points, while copying is effective in both graph-to-sequence and sequence-to-sequence models.Character LSTMs help handle unseen words that share substrings with vocabulary items.
- Graph state transitions: 22.8 BLEU is reached with 9 graph-state transitions, compared with 21.5 at 5 transitions and 14.1 at 1 transition.At 5 transitions, each node incorporates information within distance 5; at 9, nodes within distance less than 10 are incorporated.
- Graph state transitions: Using both incoming and outgoing edges is important because one-direction transitions omit either ancestors or descendants, whereas bidirectional exchange also includes siblings.The experiments report a large performance drop when only incoming or outgoing edges are used.
- Test results: Graph2seq+charLSTM+copy achieves 23.3 BLEU, 1.3 points above MSeq2seq+Anon on the same AMR corpus, and reaches 28.2 and 33.0 with 200K and 2M additional Gigaword sentences.With 2M additional sentences, the model is reported to outperform all existing methods; exhaustive transition-number search was not performed.
6 Related work
Prior AMR-to-text work includes statistical graph or tree transformations, graph neural methods, and copy mechanisms. The paper distinguishes its graph LSTM from related approaches by directly and concurrently encoding AMR structure.
- Statistical systems converted AMRs into trees, synchronous grammars, or breadth-first linearizations before generating text.
- Graph neural approaches exchanged information between neighboring nodes, but the paper’s graph LSTM retained a cell vector for each node across transitions.
- Unlike Peng et al.’s sequential DAG LSTM, this model updates states in parallel and applies LSTM encoding directly to AMR graphs without splitting them into separate DAGs.
- The recurrent exchange mechanism resembles loopy belief propagation, but it passes gated neural values to learn representation features rather than estimate marginal probabilities.
- The paper also investigates copy mechanisms for AMR-to-text generation, building on their use in neural translation, dialogue, summarization, and question generation.
7 Conclusion
The paper presents a graph-to-sequence model that directly models full AMR structure and can update graph states in parallel. In experiments, it outperforms a strong sequence-to-sequence model and achieves the best reported performance.
- The graph-to-sequence model uses a graph LSTM to directly model full AMR structure instead of linearizing AMRs before decoding.
- Parallel graph encoding makes the graph encoder more efficient than the sequence encoder.
- The graph model outperforms a strong sequence-to-sequence model and achieves the best performance in the experiments.