Source-linked AI summary
Semantic Neural Machine Translation using AMR
Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, Jinsong Su
TL;DR
NMT had received relatively little attention from semantic representation research, despite the relevance of meaning representations to translation. This paper incorporates AMR as additional knowledge through graph-based encoding and dual attention, improving English-to-German translation over a strong baseline, especially with limited data.
Problem
Semantic representations had been used in non-neural translation, but leveraging them for NMT had received relatively little attention.
Method
The model encodes AMR graphs with a graph recurrent network and combines them with sequential source encoding through a doubly-attentive LSTM decoder.
Results
25.5 vs 23.7 BLEU on 4.17M WMT16 English-German examples shows that incorporating AMR improves a strong attention-based sequence-to-sequence baseline.
Takeaways & Limitations
AMR provides complementary structural semantic information and is more effective than dependency trees and semantic roles in the reported benchmark experiments.
Takeaways & Limitations
The study leaves comparisons among GRN, GCN, and GGNN for this task to future work.
Abstract
from arXiv · showhide
It is intuitive that semantic representations can be useful for machine translation, mainly because they can help in enforcing meaning preservation and handling data sparsity (many sentences correspond to one meaning) of machine translation models. On the other hand, little work has been done on leveraging semantics for neural machine translation (NMT). In this work, we study the usefulness of AMR (short for abstract meaning representation) on NMT. Experiments on a standard English-to-German dataset show that incorporating AMR as additional knowledge can significantly improve a strong attention-based sequence-to-sequence neural translation model.
1 Introduction
The paper motivates AMR as a richer semantic representation for NMT and investigates it as complementary knowledge to textual inputs. Experiments show that AMR improves translation quality, especially with limited training data.
- Research gap: The work addresses limited prior exploration of semantics in NMT, extending earlier SRL-based efforts toward richer AMR representations.The authors state that no existing work had exploited AMR for enhancing NMT.
- Motivation and approach: AMR encodes sentence meaning as a rooted, directed graph whose concept nodes and relation edges capture more relations than semantic roles.AMR also captures entity relations while abstracting away inflections and function words.
- Motivation and approach: Structural information from AMR can reduce data sparsity when training data are insufficient for large-scale training.The smaller-data experiment reports a larger AMR advantage than the full-data experiment.
- Proposed direction: The proposed system uses a graph recurrent network to encode AMR graphs and an external attention model to incorporate graph knowledge alongside the baseline source encoder.Graph nodes exchange information through recurrent transitions, allowing each node to receive increasingly broad context.
- Empirical motivation: 25.5 vs 23.7 BLEU shows that AMR significantly improves a strong attention-based English-to-German NMT baseline on 4.17M WMT16 examples.With 226K examples, the scores are 19.2 vs 16.0 BLEU, indicating a larger improvement in the smaller-data setting.
2 Related work
Prior semantic NMT work mainly used predicate-argument information from SRL, while this paper applies AMR and GRN to NMT. The authors leave systematic comparisons among graph neural network variants for future work.
- Semantic translation: Earlier semantic translation research focused largely on predicate-argument structures, while prior graphical meaning-representation experiments used only 880 sentences.The paper positions AMR-based NMT against these narrower or limited-scale precedents.
- Semantic NMT: The paper differs from SRL-based NMT by exploiting AMR and encoding it with a graph recurrent network rather than graph convolutional layers.The GRN is described as formally consistent with the recurrent sentence encoder.
- Graph recurrent networks: The authors extend prior GRN work on graph encoding and text generation by investigating whether AMR representations improve machine translation.They state that they are the first to use GRN for machine translation.
- Open comparison: Comparisons among GRN, GCN, and GGNN for this task are left to future work because the study focuses on AMR effectiveness.This is an explicit scope limitation of the related-work comparison.
3 Baseline: attention-based BiLSTM
The baseline is an attention-based encoder-decoder that represents the source with a bidirectional LSTM and generates the target recurrently with an attention-based LSTM decoder.
- BiLSTM encoder: The source encoder is a bidirectional LSTM that produces forward and backward states for each input word.Each word embedding is processed in both directions to form its source representation.
- Attention-based decoder: The decoder generates the target sequence recurrently, using attention over the concatenated forward and backward encoder states.The attention memory H contains vectors formed from both directional states for every source token.
- Attention-based decoder: At each decoding step, the LSTM updates its hidden state from the current input embedding and previous attention context, then computes attention over source vectors.The previous decoder state and context also participate in calculating attention probabilities.
- Output generation: The decoder produces a target-vocabulary probability distribution from its current state using learnable parameters.The supplied baseline description identifies the output distribution and its learnable parameters.
4 Incorporating AMR
The model encodes the source sentence and AMR graph separately, then uses dual attention in an LSTM decoder to generate the target sequence. AMR encoding preserves graph structure through recurrent node-state transitions.
- Overall architecture: The architecture combines a source BiLSTM, a GRN for AMR, and an LSTM decoder with attention over both encoder memories.The graph attention memory comes from the final graph-transition state.
- AMR encoding: A graph recurrent network represents an AMR graph as node states and repeatedly exchanges information between directly connected nodes.The graph state is a set of node states, and recurrent transitions enable non-local interaction across multiple steps.
- AMR encoding: The number of graph state transitions T is a hyperparameter controlling the sequence of recurrent graph states.The transition sequence is denoted g0 through gT.
- AMR encoding: The model preserves the original AMR graph structure while allowing node representations to incorporate information from increasingly broader graph contexts.Each transition communicates directly between a node and all nodes connected to it.
t. We use an input gate ij
The graph recurrent network encodes AMR structure by propagating edge and neighboring-node information through recurrent state transitions. Its final graph state supplies additional attention context to the sequence-to-sequence decoder.
- t. We use an input gate ij: Each node receives representations of incoming and outgoing edges together with summed hidden states from incoming and outgoing neighbors.Incoming and outgoing edge inputs are aggregated separately before entering the recurrent cell and gates.
- t. We use an input gate ij: Information propagates between neighboring AMR nodes through recurrent state transitions, so each node can incorporate ancestors, descendants, and siblings after several steps.The model varies the number of transition steps to study the effectiveness of global encoding.
- t. We use an input gate ij: AMR edge representations encode relation labels and source-node embeddings, making labeled relations part of graph-state updates.The paper identifies edge labels as important because they represent relations between connected nodes.
- t. We use an input gate ij: The graph encoder's last state forms an attention memory, and its contextual vector is incorporated into the decoder's target-vocabulary probability calculation.This external attention mechanism adds AMR-derived context to the baseline model.
5 Training
Training uses paired source-target instances and cross-entropy over gold target sequences. Inputs are either source sentences alone or source sentences paired with automatically parsed AMR graphs.
- 5 Training: Models are trained with cross-entropy loss over each gold-standard target sequence.The training instances contain paired inputs and target sequences.
- 5 Training: The dataset and vocabulary statistics are reported after preprocessing, including BPE-based token counts and vocabulary sizes.The tables summarize dataset statistics and vocabulary sizes used for training.
- 5 Training: The baseline receives a source sentence, whereas the AMR model receives a source sentence paired with an automatically parsed AMR graph.The model parameters are represented by θ.
6 Experiments
Experiments show that Dual2seq’s AMR-based graph encoding improves translation quality over strong baselines, with larger gains when training data are limited. Analyses indicate that graph relations, AMR structure, and parsing quality each affect the model’s effectiveness.
- 6.2 Development experiments: Increasing graph transitions improves Dual2seq by roughly +1.3 BLEU from 1 to 5 steps and 0.2 BLEU from 5 to 10, followed by a slight drop at 12.Dual2seq (self) shows only small improvements and does not outperform Seq2seq, indicating gains are not explained by parameter count.
- 6.3 Main results: Dual2seq is consistently better than competing systems under BLEU, TER, and Meteor on both the NC-v11 subset and the full training set.It also outperforms OpenNMT-tf and Transformer-tf, with AMRs providing complementary information to textual inputs.
- 6.3 Main results: +3.2 BLEU on small-scale data and +1.7 BLEU on large-scale data over Seq2seq demonstrate larger AMR gains when training data are limited.Dual2seq is significantly better than Seq2seq in both settings.
- 6.3 Main results: Removing AMR relations substantially weakens performance, showing that encoding concept-to-concept structure is important beyond using AMR concepts alone.Dual2seq-LinAMR only slightly outperforms Seq2seq, whereas full graph modeling can enhance target word ordering.
- 6.3 Main results: AMRs outperform dependency trees more strongly on small-scale data, with BLEU scores of 19.2 versus 17.8, compared with 25.5 versus 25.0 on large-scale data.The authors associate this pattern with AMR lemmatization, direct concept relations, and reduced function-word distances.
- 6.4 Analysis: Gold AMRs improve Little Prince BLEU by +0.7 over automatic AMRs, while the authors note that better AMR parsing could further improve translation.The evaluation also involves a literary-domain test set and news-domain training data.
7 Conclusion
The paper concludes that AMRs improve neural machine translation by adding abstract structural semantic information alongside source text. Across experiments, AMRs help across sentence lengths and outperform dependency trees and semantic roles.
- 7 Conclusion: AMRs improve neural machine translation by complementing source text with higher-level structural semantic information.The model combines a GRN AMR encoder, sequential LSTM source encoder, and doubly-attentive LSTM decoder.
- 7 Conclusion: AMRs help across sentence lengths and are more effective than dependency trees and semantic roles.