Source-linked AI summary
Towards String-to-Tree Neural Machine Translation
Roee Aharoni, Yoav Goldberg
TL;DR
The paper addresses whether target-side syntactic information can benefit neural machine translation. It translates source sentences into linearized, lexicalized constituency trees and reports better translation outcomes, more reordering, and an advantage in a small human evaluation than a syntax-agnostic baseline.
Problem
Neural machine translation usually does not explicitly use syntactic information, motivating investigation of target-side syntax in NMT.
Method
The method translates source sentences into linearized, lexicalized constituency trees using a conventional attention-based sequence-to-sequence NMT model.
Results
The syntax-aware bpe2tree model improves BLEU relative to the bpe2bpe baseline in the reported experiments, performs more reordering, and is preferred in 37.2% versus 30.8% of human-evaluated cases.
Takeaways & Limitations
Target-side syntactic information can benefit NMT and is associated with more reordering during translation.
Takeaways & Limitations
The linearized approach has a long-sequence problem, and the analysis notes inconsistent differences between models in distortion bins 4–6.
Abstract
from arXiv · showhide
We present a simple method to incorporate syntactic information about the target language in a neural machine translation system by translating into linearized, lexicalized constituency trees. An experiment on the WMT16 German-English news translation task resulted in an improved BLEU score when compared to a syntax-agnostic NMT baseline trained on the same dataset. An analysis of the translations from the syntax-aware system shows that it performs more reordering during translation in comparison to the baseline. A small-scale human evaluation also showed an advantage to the syntax-aware system.
1 Introduction and Model
The paper proposes neural string-to-tree translation to incorporate target-side syntax into NMT by predicting linearized, lexicalized constituency trees. The approach preserves sequence-to-sequence learning while adding syntactic structure to the output.
- 1 Introduction and Model: The method is motivated by syntax-based translation approaches that permit flexible reordering across parse-tree levels.The paper contrasts this target-side syntactic modeling with syntax-agnostic NMT and related source-side or alternative target-side approaches.
- 1 Introduction and Model: The proposed neural string-to-tree model translates source sentences into linearized, lexicalized constituency trees.The predicted trees contain translated words as terminals rather than part-of-speech tags.
- 1 Introduction and Model: Unlike earlier sequence representations of trees, the model omits part-of-speech information to avoid significantly longer output sequences.Its terminals are the words of the translated sentence.
- 1 Introduction and Model: Target-side syntax is incorporated while retaining the conventional attention-based sequence-to-sequence architecture.Automatically parsed target sentences provide the tree-structured training targets.
2 Experiments & Results
The experiments compare a syntax-agnostic bpe2bpe baseline with a bpe2tree model under resource-rich and low-resource conditions. The models differ primarily in whether the target includes syntactic annotation, while evaluation derives surface translations from the predicted trees.
- Experimental Setup: The study evaluates German-to-English translation in WMT16 news data and low-resource News Commentary v8 settings.The resource-rich experiment uses 4.5 million sentence pairs, while the low-resource setting includes German, Russian, and Czech to English.
- Experimental Setup: Each experiment compares bpe2bpe, which predicts English sentences, with bpe2tree, which predicts English linearized constituency trees.The models use nearly identical parameter counts; the syntax-aware system additionally represents tree brackets.
- Experimental Setup: The models are trained with attention-based NMT, selected using development-set BLEU, and evaluated after removing tree nonterminals and merging subwords.BLEU is computed with the Moses mteval-v13a.pl script.
- Results: The syntax-aware model performs similarly on resource-rich newstest2015 and has an advantage on newstest2016, while consistently outperforming the baseline in the low-resource setting.This occurs despite bpe2tree predicting longer tree-augmented output sequences.
3 Analysis
The analysis examines whether the syntax-aware model changes reordering and syntactic choices, and whether these differences correspond to translation quality and model behavior. It reports more reordering, sensible extracted patterns and alignments, more relative constructions, and a human preference for bpe2tree outputs.
- The Resulting Trees: 5970 of 6003 development-set sentences received valid trees, with choices generally appearing syntactically reasonable.The authors note that they did not perform an in-depth error analysis.
- Quantifying Reordering: 2.73 versus 1.27: in an example, bpe2tree had the higher distortion score because it performed more reordering.For bpe2tree, distortion was computed only over tokens corresponding to tree terminals.
- Quantifying Reordering: The bpe2tree model produced more translations in higher distortion bins and significantly fewer in the least-reordering bin than bpe2bpe.Differences in bins 0–3 were consistent across runs, while bins 4–6 showed a slight, inconsistent bpe2bpe excess.
- Rule Extraction: 22,914 of 233,657 extracted rules, or 9.8%, contained reordering, revealing sensible German-English ordering patterns.The most common rule occurred in 184 development-set sentences and reordered a German sequence into an English verb phrase.
- Relative Constructions: The bpe2tree model produced more relative constructions than bpe2bpe, while both systems produced more than the reference translations.The counted pronouns were who, which, that, whom, and whose; “that” was not distinguished as a determiner versus a relative pronoun.
- Qualitative Analysis and Human Evaluations: Human evaluators preferred bpe2tree in 186 of 500 cases (37.2%), compared with 154 cases (30.8%) for bpe2bpe, with 30% neutral.The evaluation used paired system outputs judged against the English reference.
4 Conclusions and Future Work
The paper presents a simple string-to-tree neural translation model that outperforms a neural string-to-string model. It identifies several directions for extending target-side syntactic modeling.
- The proposed string-to-tree neural translation model produces better results than a neural string-to-string model.
- The linearized approach may benefit from models that alleviate long sequences or support more natural tree decoding.
- Future work includes comparisons with other syntax-aware NMT models and contrastive evaluation against syntax-agnostic systems.
A Supplementary Material
The supplementary material describes preprocessing, model-training settings, human evaluation, attention visualization, and additional translation examples. The examples report improved translation, alignment, and tree quality, with some attachment errors.
- The WMT16 data use Moses tokenization and truecasing, BPE segmentation, and large source and target vocabularies.The WMT16 experiment used 89,500 BPE operations, producing vocabularies of 84,924 input and 78,499 output tokens.
- The WMT16 bpe2tree model was trained with larger embeddings and encoder-decoder dimensions than the other experiments, using Adadelta and beam search.The WMT16 model used 500-dimensional embeddings, 1024-dimensional encoder and decoder states, minibatches of 40, and beam size 12.
- Human evaluation compared randomly ordered system outputs against an English reference using two annotators per sentence.Annotators chose which output better portrayed the reference, with options distinguishing stronger and weaker preferences, equal quality, and disagreement.
- The supplementary examples report improved translation and alignment quality and generally high structural quality in tree-based outputs.The remaining syntactic mistakes are described as SBAR and PP attachment errors, which can also challenge dedicated parsers.