Source-linked AI summary
Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder
Huadong Chen, Shujian Huang, David Chiang, Jiajun Chen
TL;DR
The paper addresses the limited use of syntax in sequential NMT by incorporating precomputed source-side syntactic trees into both encoder and decoder. It proposes a bidirectional tree encoder and a tree-coverage attention model, achieving +3.54 BLEU over standard attention and +1.90 BLEU over a stronger tree-based coverage baseline.
Problem
Sequential NMT largely processes sentences as word sequences without explicitly exploiting syntactic structure, despite persistent syntactic errors such as attachment.
Method
The model uses a precomputed source-side syntactic tree in a bidirectional tree encoder and a tree-coverage attention mechanism.
Results
+3.54 BLEU over a standard attentional NMT system and +1.90 BLEU over a stronger baseline with a Tree-LSTM encoder and coverage.
Takeaways & Limitations
Source-side syntax improves NMT performance and can better control translation, with especially strong benefits reported for long sentences.
Takeaways & Limitations
The current model uses only syntactic tree structure, without node labels or target-side syntactic information.
Abstract
from arXiv · showhide
Most neural machine translation (NMT) models are based on the sequential encoder-decoder framework, which makes no use of syntactic information. In this paper, we improve this model by explicitly incorporating source-side syntactic trees. More specifically, we propose (1) a bidirectional tree encoder which learns both sequential and tree structured representations; (2) a tree-coverage model that lets the attention depend on the source-side syntax. Experiments on Chinese-English translation demonstrate that our proposed models outperform the sequential attentional model as well as a stronger baseline with a bottom-up tree encoder and word coverage.
1 Introduction
The paper augments sequential NMT with explicit source-side syntax in both encoder and decoder, targeting syntactic errors, long-distance dependencies, and phrasal cohesion. Experiments on Chinese-English translation report improvements over standard attention and a stronger tree-based baseline.
- Sequential NMT representations omit explicit syntactic structure, while syntactic errors such as attachment remain a problem.
- Explicit syntax can help the encoder represent long-distance dependencies, especially when translating long sentences.
- The decoder can use source syntax to guide reordering and preserve phrasal cohesion that word-level attention and coverage cannot capture.
- The bidirectional tree encoder combines information from below and above each source-tree node, summarizing sequential and syntactic context.
2 Neural Machine Translation
The attentional encoder-decoder NMT framework encodes source words with bidirectional GRU annotations and generates the target sentence word by word. Its attention mechanism selects relevant source annotations through weighted context vectors.
- Model objective: The framework directly models translation probability conditioned on the source sentence and previously generated target words.The model follows the encoder-decoder framework with attention described by Bahdanau et al. (2015).
- Source encoding: The bidirectional GRU reads the source sentence in both directions, producing annotations that summarize preceding and following words.Each source-word annotation concatenates the forward and backward hidden states, and the full annotation sequence is passed to the decoder.
- Source encoding: The source representations begin with word embeddings processed by gated recurrent units.The paper identifies si as the i-th source-word embedding and GRU as a gated recurrent unit.
- Target decoding: The decoder is a forward GRU that predicts the translation one target word at a time.Its state uses the previous target-word embedding and the current context vector.
- Attention: Attention computes each context vector as a weighted sum of source annotations.The attention score measures how well the previous decoder state matches each source annotation.
3 Tree Structure Enhanced Neural Machine Translation
The paper incorporates source-side syntactic trees into both NMT encoding and attention. It proposes a bidirectional tree encoder for fuller contextual representations and a tree-coverage mechanism to reduce redundant attention and over-translation.
- Preliminaries: The model uses precomputed source-side phrase-structure trees, currently exploiting tree structure without syntactic labels.The encoder assumes strictly binary branching trees whose leaves correspond to source words.
- Tree-GRU Encoder: The Tree-GRU encoder recursively combines the hidden states of each interior node’s left and right children.Tree-GRU units use gates to control information flow without separate memory cells.
- Bidirectional Tree Encoder: The bidirectional tree encoder adds top-down information to bottom-up representations, allowing tree structure to flow from the root to the leaves.Each node annotation combines bottom-up and top-down hidden states, giving words full sequence and syntactic-tree context.
- Bidirectional Tree Encoder: Unlike the bottom-up encoder, the bidirectional encoder supplies syntactic information to word representations rather than relying only on each node’s subtree.The bottom-up representation of a leaf remains sequential and contains no syntactic information from higher in the tree.
- Tree-Coverage Model: The tree-coverage model makes attention depend on source syntax and helps prevent redundant use of overlapping parent and child nodes.When a child contributes to translation, its parent’s coverage reflects that use, reducing repeated translation of the same source phrase.
- Tree-Coverage Model: The tree-coverage model produces more concentrated attention on non-leaf nodes and corrects the over-translation of the Chinese phrase zhu manila.The mechanism addresses redundant information supplied by both child and parent nodes.
4 Experiments
Experiments on Chinese–English NMT evaluate the proposed tree encoders and tree-coverage model against sequential, Tree-LSTM, Tree-GRU, and coverage baselines. The combined bidirectional tree encoder and tree-coverage model achieves the strongest reported performance, with larger gains for longer sentences.
- 4.1 Data: The experiments use NIST Chinese–English translation data, including 1.6M training sentence pairs, NIST MT02 development data, and NIST MT03–06 test data.The corpora are primarily newswire and exclude sentence pairs longer than 50 tokens.
- 4.3 Tree Encoders: The bidirectional tree encoder outperforms the original Tree-LSTM encoder under matched annotation-size comparisons.It also outperforms a head-lexicalized bidirectional tree encoder, suggesting head-word information may be less helpful for translation than parsing.
- 4.4 Tree-Coverage Model: The tree-coverage model consistently improves performance beyond word coverage when paired with the proposed tree encoders.The authors attribute the difference to word coverage neglecting relationships among tree nodes, such as parent–child relationships.
- 4.4 Tree-Coverage Model: +3.54 BLEU is achieved over the standard attentional model, and +1.90 BLEU over the stronger bottom-up tree-encoder-plus-coverage baseline.These gains come from combining the bidirectional tree encoder with the proposed tree-coverage model.
- 4.5 Analysis By Sentence Length: The bidirectional tree encoder outperforms sequential NMT and Tree-GRU across all sentence lengths, with larger improvements beyond 20 words.The biggest improvement occurs for sentences longer than 50 words.
5 Related Work
Related work has used explicit syntax for sentence representations and structural biases in attention. This paper applies bidirectional tree representations and direct structural attention with coverage to machine translation.
- Sentence Representations: Bidirectional Tree-LSTMs and Tree-GRUs have previously been applied to sentence classification, motivating their use for machine translation.The paper draws on work by Teng and Zhang and by Kokkinos and Potamianos.
- Structural Attention: Prior NMT work extended attention with structural biases from alignment models or richer structural distributions.These approaches differ from the paper’s direct use of structural information in attention combined with coverage.
- Structural Attention: The paper’s decoder contribution is to exploit source syntactic structure directly within the attention model and a coverage mechanism.This extends prior word-level attention and coverage approaches toward tree-structured information.
6 Conclusion
The proposed syntax-aware encoder-decoder model uses explicit source-side syntactic trees, improving encoding and decoder control; benefits are especially strong for long sentences. The work currently uses tree structure without node labels.
- The syntax-aware encoder-decoder model incorporates explicit source-side syntactic trees to improve NMT representations and translation control.The experiments support a bidirectional tree encoder and decoder-side syntactic structure as useful enhancements.
- A top-down encoder enhances the original bottom-up tree encoder, while decoder-side syntactic information better controls translation.
- The benefits of source-side syntax are especially strong for long sentences.
- The current model uses only syntactic tree structure, without node labels.Future work could incorporate tree labels or target-side syntactic information.