Modeling Source Syntax for Neural Machine Translation

Junhui Li; Deyi Xiong; Zhaopeng Tu; Muhua Zhu; Min Zhang; Guodong Zhou

Source-linked AI summary

Modeling Source Syntax for Neural Machine Translation

Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu, Min Zhang, Guodong Zhou

arXiv:1705.01020v1May 2, 2017cs.CL

TL;DR

NMT can learn some source syntax implicitly, but it still misses structural details and sometimes produces syntactically disrespectful translations. This paper linearizes source parse trees into structural label sequences and introduces three syntax-aware RNN encoders. On Chinese-to-English translation, all improve accuracy, with Mixed RNN achieving the best result and a 1.4 BLEU-point improvement.

Problem
NMT can learn some source syntax implicitly but still fails to capture many deep structural details and often produces translations that do not respect source syntax.
Method
The paper linearizes source phrase parse trees into structural label sequences and incorporates them using Parallel, Hierarchical, and Mixed RNN encoders.
Results
1.4 BLEU points: Mixed RNN achieved the best performance, while all three proposed encoders improved over a state-of-the-art baseline NMT system on Chinese-to-English translation.
Takeaways & Limitations
Explicit source syntax can improve NMT translation accuracy, with the simplest Mixed RNN encoder producing the strongest reported result.
Takeaways & Limitations
NMT systems perform especially poorly on sentences longer than 50 words, partly because training limits source length to 50 and models tend to stop early.

Abstract

from arXiv · show

Even though a linguistics-free sequence to sequence model in neural machine translation (NMT) has certain capability of implicitly learning syntactic information of source sentences, this paper shows that source syntax can be explicitly incorporated into NMT effectively to provide further improvements. Specifically, we linearize parse trees of source sentences to obtain structural label sequences. On the basis, we propose three different sorts of encoders to incorporate source syntax into NMT: 1) Parallel RNN encoder that learns word and label annotation vectors parallelly; 2) Hierarchical RNN encoder that learns word and label annotation vectors in a two-level hierarchy; and 3) Mixed RNN encoder that stitchingly learns word and label annotation vectors over sequences where words and labels are mixed. Experimentation on Chinese-to-English translation demonstrates that all the three proposed syntactic encoders are able to improve translation accuracy. It is interesting to note that the simplest RNN encoder, i.e., Mixed RNN encoder yields the best performance with an significant improvement of 1.4 BLEU points. Moreover, an in-depth analysis from several perspectives is provided to reveal how source syntax benefits NMT.

1 Introduction

Although NMT can learn some source syntax implicitly, it often fails to respect structural relationships. The paper explicitly incorporates source syntax through linearized parse-tree labels and shows improved Chinese-to-English translation accuracy.

1 Introduction: NMT can learn some implicit source syntax but still misses deep structural details and often produces translations that violate source syntax.The paper identifies syntactic disrespect as a limitation of linguistics-free sequence-to-sequence NMT.
1 Introduction: The paper addresses NMT’s syntactic weaknesses generally rather than targeting only discontinuous or over translation.The approach aims to incorporate source syntax explicitly to improve translation accuracy and potentially alleviate both issues.
1 Introduction: The method linearizes a phrase parse tree into a structural label sequence and lets the model learn useful syntactic information automatically.This differs from assigning manually designed syntactic labels to individual source words.

2 Attention-based NMT

The attention-based NMT baseline encodes source words and uses a recurrent decoder with attention-derived context vectors to predict target words. Its encoder–decoder structure is the foundation for incorporating source syntax later in the paper.

2 Attention-based NMT: The model consists primarily of a bidirectional RNN encoder and an RNN decoder with attention.The encoder reads the source in both directions, while the decoder predicts the target sequence.
2 Attention-based NMT: Each source word is represented by a word annotation vector formed by concatenating forward and backward encoder states.These annotation vectors summarize source information for downstream decoding.
2 Attention-based NMT: At each decoding step, attention computes a weighted sum of source annotation vectors as the context vector used to predict the next target word.The decoder prediction also depends on its recurrent state and the previously predicted target word.

3 NMT with Source Syntax

The paper incorporates source syntax into NMT by linearizing parse trees into structural-label sequences and combining their representations with word encodings. It compares parallel, hierarchical, and mixed RNN encoders, which differ in how strongly word and label sequences are coupled.

3 NMT with Source Syntax: The three encoders use linearized parse-tree labels to augment word representations with structural context.The label sequence follows a depth-first traversal, and each word receives the annotation vector of its corresponding POS tag.
3 NMT with Source Syntax: The Parallel RNN encoder independently encodes words and structural labels, then concatenates each word vector with its POS-tag label vector.The two RNNs do not exchange backpropagated error signals.
3 NMT with Source Syntax: The Hierarchical RNN encoder places a structural-label RNN below a word RNN, feeding label representations into the word-level encoder.The word RNN is uppermost because each word maps to an item in the structural-label sequence.
3 NMT with Source Syntax: The Mixed RNN encoder stitches words and structural labels into one sequence, learning annotations for both while decoding uses only word annotations.Backpropagation through the word sequence updates structural-label annotations even though they are not directly decoded.
3 NMT with Source Syntax: Coupling increases from Parallel to Hierarchical to Mixed, while the Mixed encoder remains the simplest architecture.Its input sequence is approximately three times longer than that of conventional NMT encoders.

4 Experimentation

Experiments on Chinese-to-English translation compare three syntax-aware NMT encoders with SMT and an attentional NMT baseline. All syntax-aware systems improve translation accuracy, while the Mixed RNN performs best despite modest training overhead.

Experiment Results: All three proposed NMT models improve BLEU translation accuracy over RNNSearch across the test sets, with considerable differences among variants.Table 1 reports translation performance in BLEU score; the experiments compare Parallel, Hierarchical, and Mixed RNN systems with RNNSearch and cdec.
Parameters: Mixed RNN requires no additional parameters, whereas Parallel and Hierarchical RNN add parameters for structural-label processing.The Mixed RNN keeps the vocabulary size unchanged despite its longer input sequence.
Speed: Parallel and Hierarchical RNN increase training time by about 6%, while Mixed RNN increases it by 26% because its input sequence is roughly three times longer.The baseline runs at 153 minutes per epoch on a single GeForce GTX 1080 GPU.
Comparison with the baseline NMT model: Mixed RNN achieves the best overall accuracy except on NIST MT 02, outperforming RNNSearch by 1.4 BLEU points and the other improved models by 0.3∼0.4 BLEU points.Parallel and Hierarchical RNN achieve similar accuracy, such as 36.6 versus 36.7.
Comparison with the SMT model (cdec): All NMT systems outperform the cdec SMT system on the evaluated Chinese-to-English test sets.

5 Analysis

The analysis examines how source syntax affects alignment, phrase continuity, over-translation, rare-word translation, and performance across sentence lengths. Mixed RNN generally improves syntactic consistency, while long inputs remain challenging.

5.1 Effects on Long Sentences: Mixed RNN outperforms RNNSearch across all tested input-length groups, although both systems degrade substantially as sentences become longer.NMT systems perform especially poorly beyond length 50, compared with cdec; cited explanations include the training length limit and early stopping.
5.2 Analysis on Word Alignment: Source syntax improves attention alignment by maintaining annotation vectors that summarize structural information for each source word.The alignment evaluation uses 900 manually aligned Chinese-English sentence pairs and reports alignment error rate, where lower scores are better.
5.3 Phrase Alignment Analysis: Mixed RNN translates syntactic phrases more consistently than RNNSearch, addressing discontinuous or untranslated realizations of phrases such as PPs and NPs.The analysis categorizes phrase translations as continuous, discontinuous, or untranslated; RNNSearch translates 57.3% of PPs continuously, 33.6% discontinuously, and 9.1% not at all.
5.4 Analysis on Over Translation: ROT measures average over-translation frequency as total repeated translation counts divided by the number of source words considered.For a source word, over-translation count equals the target translation length minus its number of unique words.
5.4 Analysis on Over Translation: 18%: Mixed RNN reduces ROT from 5.5% to 4.5%, while RNNSearch shows especially high over-translation for proper nouns and cardinal numbers.The paper attributes these patterns to unknown words and determiners that can be translated repeatedly, including multiple English “the” tokens.
5.5 Analysis on Rare Word Translation: Mixed RNN is more likely to translate source-side rare words as target-side UNK rather than as non-UNK words.The paper explains that rare source words tend to correspond to rare target words, making correct non-UNK translation difficult after UNK replacement.

6 Related Work

Prior work established syntax benefits in SMT and explored syntactic information in NMT, but approaches differ in how linguistic structure enters the model. This paper learns from structural labels rather than specifying a fixed feature inventory.

Prior Syntax-Based Translation: Syntax-based SMT has substantially improved translation by integrating varied forms of syntactic knowledge, whereas syntax’s effective use in NMT remained unsettled.
Syntax in NMT: Tree-to-sequence NMT learns annotation vectors for terminal and non-terminal nodes, while this paper’s Mixed RNN stitchingly encodes syntactic labels and words within a sequence-to-sequence model.
Feature-Based Linguistic Input: Unlike feature-based NMT that concatenates manually selected lemmas, subword tags, POS tags, or dependency labels with word embeddings, this approach learns from structural label sequences.Related work also examined implicit syntax learning and target-side linguistic factors, but those studies did not focus on improving NMT with linguistic input or extended beyond word-level features.

7 Conclusion

The paper concludes that explicit source syntax can improve NMT when learned from linearized parse trees. Across Chinese-to-English tests, all three encoders help, with Mixed RNN delivering the strongest gain, while further syntax-aware modeling remains open.

7 Conclusion: 1.4 BLEU points: the simplest encoder, Mixed RNN, achieves the best performance on NIST MT 02 to 05.All three proposed models improve over the state-of-the-art baseline NMT system.
7 Conclusion: Linearizing parse trees into structural label sequences lets the models automatically learn useful source-syntactic information for NMT encoding.
7 Conclusion: The analysis finds substantial remaining room for NMT translations to become consistent with source syntax, motivating richer syntactic features and syntax-constrained attention in future work.