Source-linked AI summary
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation
Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, Khalil Sima'an
TL;DR
The paper asks how to add useful syntactic structure to NMT systems that lack explicit syntax modeling. It uses syntactic GCNs over predicted dependency trees, integrated as word-level encoder layers, and reports consistent improvements across English-German and English-Czech experiments. The study focuses on dependency syntax rather than more general graph-based linguistic structures.
Problem
Sequential NMT systems lack explicit syntax modeling, while existing methods may incorporate syntax indirectly or restrict its interface with translation.
Method
Syntactic GCNs use predicted source dependency trees to produce word representations sensitive to syntactic neighborhoods, including as layers over RNN or CNN encoders.
Results
The approach yields consistent BLEU4 improvements on English-German and English-Czech translation, including +1.2 and +0.7 BLEU points over bidirectional RNN encoders.
Takeaways & Limitations
GCNs provide a simple way to integrate dependency structure into NMT while allowing end-to-end training to capture syntax relevant to translation.
Takeaways & Limitations
The study focuses solely on dependency syntax and leaves broader investigation of other graph-based linguistic structures for future work.
Abstract
from arXiv · showhide
We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoder-decoder models for machine translation. We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data. Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e. hidden states of the encoder) that are sensitive to their syntactic neighborhoods. GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidirectional RNNs or convolutional neural networks). We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial improvements over their syntax-agnostic versions in all the considered setups.
1 Introduction
The paper addresses the limited explicit syntax modeling in sequential NMT by integrating syntactic GCNs into standard encoders. These word-level representations capture syntactic neighborhoods and improve translation across English-German and English-Czech setups.
- Motivation: Sequential NMT systems lack explicit modeling of syntax, motivating simpler methods for incorporating structured information into neural encoders.Prior approaches either incorporate syntax indirectly or restrict how syntax interfaces with translation.
- Motivation: Dependency trees provide syntactic neighborhoods for source words, which can enrich the latent feature vectors used by attention-based NMT.For example, dependency syntax represents monkey as the subject of eats and banana as its object.
- Approach: Syntactic GCNs compute node representations from k-hop graph neighborhoods and produce word-level outputs suitable for attention-based encoder-decoder models.They are used on top of recurrent or convolutional encoders to enrich encoder states with syntactic information.
- Results: +1.2 and +0.7 BLEU points are obtained on English-German and English-Czech, respectively, by adding syntactic GCNs to bidirectional RNN encoders.The improvement occurs in a setup where RNNs already capture some syntactic phenomena without explicit treebank supervision.
- Contributions: The paper introduces syntactic GCNs for NMT and shows that GCNs work with both RNN and CNN encoders.The reported benefits cover English-Czech and English-German translation.
2 Background
The section introduces encoder-decoder NMT and graph convolutional networks, then explains how directed, labeled, gated GCNs propagate information over graph neighborhoods.
- Neural machine translation: NMT models translate source sequences with an encoder, decoder, and conditioning mechanism such as attention.Training maximizes the likelihood of the next target word given its context.
- Encoders: RNN encoders summarize sequential inputs, while CNN encoders capture local context through fixed-size windows and stacked layers.Bidirectional RNNs read sentences in both directions; CNNs trade non-local context for faster parallel computation.
- Graph convolutional networks: GCNs encode node neighborhoods by passing messages along graph edges, with k layers reaching nodes at most k hops away.Each node includes itself among its neighbors, and stacked layers integrate progressively larger neighborhoods.
- GCN computation: The GCN computation uses node features such as word embeddings, applies learned transformations to neighboring representations, and stacks layers for multi-hop propagation.The formulation omits Kipf and Welling’s normalization factor, following the syntactic GCN implementation described here.
- Syntactic GCNs: Syntactic GCNs extend GCNs to directed, labeled dependency graphs and use edge-wise gates to regulate individual edge contributions.Direction-specific weights distinguish incoming, outgoing, and self-loop connections, while labels can specialize bias terms without multiplying weight matrices.
3 Graph Convolutional Encoders
The proposed encoders place syntactic GCNs over progressively richer word representations, while keeping the attention-based decoder fixed. The designs range from BoW to CNN and BiRNN encoders, with residual connections for deeper GCNs.
- Design: All models incorporate source-side syntax in the encoder while keeping the Bahdanau decoder constant across experiments.The intended effect is to enrich word representations used as decoder context vectors.
- BoW + GCN: BoW + GCN combines word and position embeddings with a GCN to add structural information beyond crude linear-order encoding.The paper presents this as the simplest proposed encoder.
- Convolutional + GCN: Convolutional + GCN enriches one-layer CNN representations with dependency-based information instead of stacking multiple CNN layers.Although Figure 2 illustrates window size 3, the experiments use a window size of 5.
- BiRNN + GCN: BiRNN + GCN applies a GCN to BiGRU hidden states, allowing information to pass along dependency edges between otherwise distant words.The GCN also exploits dependency relation types, making this the most challenging setup because RNNs already capture some syntax implicitly.
- Depth: For more than one GCN layer, the authors add residual connections to ease optimization while hypothesizing that the first layer captures most syntactic context.They expect further layers to propagate information farther but modify representations only modestly.
4 Experiments
The experiments test GCN-enhanced encoders on artificial reordering and English-German and English-Czech translation, finding consistent gains over syntax-agnostic baselines.
- 4.1 Reordering artificial sequences: After 6 epochs, the BiRNN + GCN reorders permuted sequences with validation BLEU 99.2.The model uses a bidirectional GRU with a one-layer GCN.
- 4.1 Reordering artificial sequences: GCN edge-wise gating separates real and fake dependency labels, supporting the model’s ability to identify useful graph connections.The mean gate-bias values for real and fake edges are far apart; removing gating would make reordering mediocre.
- 4.2.1 Results: GCN-enhanced encoders improve English-German translation: CNN+GCN gains +1.9 BLEU1 and +1.1 BLEU4, while BiRNN+GCN gains +2.3 BLEU1 and +1.2 BLEU4.On the full English-German dataset with beam 12, BiRNN reaches 23.3 BLEU4 and BiRNN+GCN reaches 23.9.
- 4.2.1 Results: On English-Czech, CNN+GCN improves over CNN by +1.0 BLEU1 and +0.6 BLEU4, while BiRNN+GCN improves over BiRNN by +1.9 BLEU1 and +0.7 BLEU4.The BiRNN baseline has BLEU4 8.9, and the GCN-enhanced model exceeds it.
- 4.2.1 Results: Adding GCN layers expands syntactic neighborhoods; English-German gains +1.3 BLEU1 and +0.73 BLEU4 from the second layer, while English-Czech improves with each added layer.Validation BLEU is reported for one- and two-layer GCNs.
- 4.2.1 Results: GCN-based models outperform their respective baselines fairly uniformly across sentence-length buckets, while syntax-aware representations improve BLEU4, Kendall tau, and BLEU1.The authors associate BLEU1 and Kendall tau gains with word-order and lexical/BPE selection phenomena that depend on syntax.
5 Related Work
Prior work incorporates syntax through explicit features, tree-based representations, multitask learning, latent structures, and convolutional encoders, but these approaches differ in how directly and flexibly they connect syntax to translation.
- Syntactic features and/or constraints: Explicit syntactic features augment word embeddings with POS tags, lemmas, and dependency labels.
- Syntactic features and/or constraints: Tree-based approaches encode constituency structures or predict linearized parse trees for neural translation.
- Multi-task Learning: Multitask methods share NMT parameters with syntactic parsers or taggers to obtain syntactically aware representations.
- Latent structure: Latent-structure models learn syntax-inspired representations, including relaxed dependency-tree encodings and treebank-pretrained encoders.
- Convolutional encoders: Convolutional encoders expand contextual receptive fields by stacking layers or using dilation, whereas this work uses dependency-informed GCNs.
6 Conclusions
The paper presents a simple approach for integrating syntax into neural machine translation and reports consistent BLEU4 improvements on English-German and English-Czech. It identifies broader graph-based linguistic structures as future directions.
- 6 Conclusions: The proposed approach integrates syntax into neural machine translation models and improves BLEU4 consistently on English-German and English-Czech.
- 6 Conclusions: Future work extends beyond syntax to semantic role labeling, AMR, and co-reference chains.
- 6 Conclusions: Because GCNs encode graph-based structure, the authors identify them as suitable for these broader semantic annotations.