Source-linked AI summary
Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks
Diego Marcheggiani, Jasmijn Bastings, Ivan Titov
TL;DR
Semantic representations were proposed as a way to preserve meaning and generalize across surface forms, but semantic structures had not been incorporated into neural machine translation. This paper injects predicate-argument information through semantic GCNs in NMT and finds BLEU improvements on English–German, including gains over syntax-aware models and further gains from combining syntax and semantics.
Problem
Semantic representations may help preserve meaning and generalize across surface realizations, but prior NMT work had not incorporated semantic structures.
Method
The paper integrates source-sentence predicate-argument structures into attention-based NMT using semantic GCNs over encoder representations.
Results
On English–German WMT16, semantic GCNs improve over the BiRNN baseline, outperform syntactic GCNs by 0.6 BLEU with full data, and reach 24.9 BLEU when syntax and semantics are combined.
Takeaways & Limitations
Semantic structures are beneficial for English–German NMT, while combined syntax and semantics provide a further gain.
Abstract
from arXiv · showhide
Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods. In this work, we are the first to incorporate information about predicate-argument structure of source sentences (namely, semantic-role representations) into neural machine translation. We use Graph Convolutional Networks (GCNs) to inject a semantic bias into sentence encoders and achieve improvements in BLEU scores over the linguistic-agnostic and syntax-aware versions on the English--German language pair.
1 Introduction
The paper introduces semantic-role information as a previously unexplored source of linguistic structure for neural machine translation. It uses semantic GCNs to integrate predicate-argument structure into attention-based NMT and reports gains over baseline, syntactic, and combined alternatives.
- Motivation: Semantic representations abstract over different surface realizations of the same meaning, potentially improving meaning preservation and generalization in translation.
- Research gap: Neural machine translation had not previously incorporated semantic structures, motivating integration of source-sentence predicate-argument information into attention-based models.
- Semantic representations: PropBank-style semantic-role representations mark predicate arguments and categorize them by roles such as giver, recipient, and item given.
- Graph structure: The semantic dependency graphs are not trees, so they cannot be processed bottom-up or easily linearized using methods designed for syntactic structures.
- Results: 24.5 BLEU was achieved with a semantic GCN versus 23.3 BLEU for the semantics-agnostic BiRNN baseline on English–German WMT16.
- Results: 24.9 BLEU from jointly using syntactic and semantic GCNs exceeded 23.9 BLEU for the syntactic GCN, suggesting complementary information.
2 Model
The model starts with attention-based encoder-decoder NMT and augments BiRNN or CNN sentence encoders with labeled, directed semantic GCN layers. These GCNs aggregate neighbor representations using direction-specific parameters, semantic-role embeddings, and gates that can reduce the effect of erroneous predicted edges.
- Encoder-decoder models: The attention-based encoder-decoder encodes each source word, while an RNN decoder generates the target sentence word by word using attention.
- Encoder architectures: The experiments use both recurrent and convolutional encoders to test whether semantic-role structures benefit different encoder architectures.
- Graph Convolutional Networks: A GCN updates each node representation using representations of neighboring nodes in a graph.
- Graph Convolutional Networks: The semantic GCN operates on directed, labeled dependency graphs whose edge directions and semantic-role labels are incorporated into the update.
- Graph Convolutional Networks: Direction-specific matrices, semantic-role label embeddings, and scalar gates determine how each neighboring node influences the updated representation.
- Graph Convolutional Networks: Gates can downweight erroneous edges because the semantic graphs are automatically predicted and may contain mistakes.
- Architecture: GCN layers are stacked over BiRNN or CNN representations to incorporate higher-order neighborhoods.
3 Experiments
Experiments evaluate semantic GCNs across encoder architectures, data settings, and combinations with syntax. Semantic structure improves translation quality, with larger gains on full WMT data and complementary benefits when combined with syntax.
- Experimental Setup: The experiments used English-to-German WMT16 data, measured with cased BLEU, and selected hyperparameters on a validation set.The full training data contained approximately 4.5 million sentence pairs; News Commentary provided approximately 226,000 additional sentence pairs.
- Results and Discussion: Semantic GCNs improved baseline performance by 0.7 BLEU with BiRNN and 0.8 BLEU with CNN on News Commentary.The same trend held for both encoder types, although syntactic GCNs appeared stronger in these experiments.
- Results and Discussion: Combining syntactic and semantic GCNs produced further improvement over the semantic GCN model and substantial improvement over the syntactic CNN model.
- Results and Discussion: On full WMT data, semantic structures yielded a +1.2 BLEU improvement, while semantic GCNs exceeded syntactic GCNs by 0.6 BLEU.The joint model reached 24.9 BLEU, and the authors describe syntax and semantic dependencies as complementary information.
- Ablation and Syntax-Semantics GCNs: Linguistic-agnostic GCNs performed on par with the baseline, indicating that the gains came from encoded linguistic structure rather than an extra nonlinear layer.The ablation used only self-loop edges in the GCN updates.
- Ablation and Syntax-Semantics GCNs: Syntax and semantics combined in one GCN layer gave no improvement over using either alone, whereas stacking semantic and syntactic GCNs was more successful.The authors attribute the single-layer result to limited interaction between the two signals under simpler aggregation.
A0 AM-DIR A0 A1
The example contrasts baseline and semantic-GCN translations, showing a more faithful German rendering of the sentence with semantic information.
- The semantic-GCN translation preserves the intended walking and drinking relations more clearly than the BiRNN output.
A1 A2 A0 A1 AM-LOC
Qualitative examples show that semantic structure can improve some translations, especially preposition choice and grammaticality, while the semantic GCN also makes a case where the BiRNN is preferable.
- A1 A2 A0 A1 AM-LOC: The first two examples are translations where the semantic structure helps, while both translations in the last example are problematic.The table identifies these contrasting outcomes across the three examples.
- A1 A2 A0 A1 AM-LOC: Semantic GCN uses correctly assigned semantic roles to choose the appropriate German preposition for Mark and produce a grammatical translation in another example.The model selects an rather than nach when Mark is labeled A2 rather than AM-DIR.
- A1 A2 A0 A1 AM-LOC: The semantic GCN ignored the verbs sit and play, whereas the BiRNN translation was preferable because it was grammatically correct.Thus, the qualitative comparison includes a case where the baseline is better despite lower fluency or precision.
4 Conclusions
The paper proposes injecting predicate-argument structure into NMT and reports benefits for English–German translation, while identifying evaluation and scope limitations.
- 4 Conclusions: The model injects predicate-argument structures into neural machine translation models.The approach targets semantic information as an input to NMT.
- 4 Conclusions: Semantic structures are reported as beneficial for the English–German language pair.The conclusion does not claim the same result for other language pairs.
- 4 Conclusions: The evaluation used BLEU only, leaving the benefits of semantics, their responsible components, and other language pairs for future work.The authors explicitly identify these as open directions.
A Hyperparameters
The experiments use separate BPE settings for News Commentary and full En–De data, with shared recurrent, dropout, regularization, and decoding choices.
- A Hyperparameters: News Commentary uses 8000 BPE merges, while full En–De experiments use 16000 BPE merges.All experiments use bidirectional GRUs, embedding size 256, word and edge dropout retain probability 0.8, L2 value 10^-8, and greedy decoding.
B Datasets Statistics
The appendix provides dataset sentence-count statistics and vocabulary-size statistics in separate tables.
- B Datasets Statistics: Table 5 reports the number of sentences in the datasets.The supplied table text does not include the individual counts.
- B Datasets Statistics: Table 6 reports vocabulary sizes.The supplied table text does not include the individual vocabulary values.