Source-linked AI summary
Neural Machine Translation with Reconstruction
Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, Hang Li
TL;DR
NMT often lacks adequacy because it can repeat source content, ignore other content, and favor short candidates during decoding. The paper adds a reconstructor that rebuilds the source from target-side decoder states and combines reconstruction with likelihood. The framework improves translation performance over strong NMT and SMT baselines, while introducing substantial training overhead and leaving validation across more language pairs and architectures as future work.
Problem
NMT translations often lack adequacy because systems may repeatedly translate some source words while ignoring others.
Method
An encoder-decoder-reconstructor framework reconstructs the source sentence from target-side decoder hidden states and uses reconstruction alongside likelihood.
Results
2.3 BLEU points over a strong attention-based NMT system and 4.5 BLEU points over a state-of-the-art SMT system were achieved on the same data.
Takeaways & Limitations
The combined objective improves parameter training and consistently improves translation performance as the decoding space increases.
Takeaways & Limitations
Training speed decreases substantially, and the approach still requires validation on more language pairs and other NMT architectures.
Abstract
from arXiv · showhide
Although end-to-end Neural Machine Translation (NMT) has achieved remarkable progress in the past two years, it suffers from a major drawback: translations generated by NMT systems often lack of adequacy. It has been widely observed that NMT tends to repeatedly translate some source words while mistakenly ignoring other words. To alleviate this problem, we propose a novel encoder-decoder-reconstructor framework for NMT. The reconstructor, incorporated into the NMT model, manages to reconstruct the input source sentence from the hidden layer of the output target sentence, to ensure that the information in the source side is transformed to the target side as much as possible. Experiments show that the proposed framework significantly improves the adequacy of NMT output and achieves superior translation result over state-of-the-art NMT and statistical MT systems.
Introduction
NMT improves translation across challenging language pairs but often produces inadequate translations because it may omit or repeat source content. The proposed encoder-decoder-reconstructor framework adds reconstruction to improve candidate adequacy during training and testing.
- Motivation: NMT has significantly improved translation, especially for morphologically rich language pairs and substantial word reordering.Long Short-Term Memory enables long-distance reordering, a major challenge for statistical machine translation.
- Motivation: NMT often repeats some source content while ignoring other content, causing over-translation and under-translation.The paper attributes this to the lack of a mechanism ensuring complete source-to-target information transfer.
- Motivation: Likelihood-based decoding favors short candidates and fails to distinguish adequate translations from inadequate ones in large decoding spaces.Likelihood captures only the source-to-target dependency, which does not correlate well with translation adequacy.
- Approach: The proposed encoder-decoder-reconstructor model reconstructs the source from target-side decoder representations and linearly combines reconstruction with likelihood.The combined scoring supports parameter learning during training and candidate reranking during testing.
- Results: 2.3 BLEU points over a strong attention-based NMT system and 4.5 BLEU points over a state-of-the-art SMT system were achieved on the same data.The approach also consistently improves performance as the decoding space increases.
Background
End-to-end NMT models target probabilities word by word, but their selective use of encoder information provides no guarantee of complete source coverage. Reconstruction is introduced as an auxiliary adequacy objective to complement likelihood.
- Encoder-Decoder based NMT: End-to-end NMT models translation probability word by word using an encoder-decoder framework.The encoder produces latent source representations, while the decoder generates the target sentence sequentially.
- Encoder-Decoder based NMT: The decoder adaptively selects partial encoder information for each target word, without guaranteeing complete source-to-target information transfer.This selective process is described as a greedy choice of the most useful information for each generated word.
- Decoding Limitation: NMT performance decreases as the decoding space increases because likelihood favors short but inadequate candidates.These candidates appear alongside good candidates in larger decoding spaces, and length normalization does not solve the problem.
- Reconstruction: An auxiliary objective is needed to measure translation adequacy and complement likelihood.Likelihood measures the target-generation objective, while reconstruction evaluates faithfulness to the original input.
Approach
The proposed encoder-decoder-reconstructor framework augments NMT with a reconstructor that uses target-side hidden states to reconstruct the source and improve translation adequacy. During training and testing, likelihood and reconstruction scores jointly guide translation selection.
- Framework components: The framework combines a standard encoder-decoder with an added reconstructor that reads target-side decoder hidden states.The encoder-decoder produces translation candidates and likelihood scores; the reconstructor produces reconstruction scores.
- Reconstruction mechanism: The reconstructor reconstructs the source sentence word by word from inverse context vectors formed by weighted target-side hidden states.An independent inverse attention model computes the weights for each target-side hidden state.
- Training objective: The training objective combines likelihood for translation fluency with reconstruction for translation adequacy, balanced by λ.Both the encoder-decoder and reconstructor are trained jointly on paired source-target examples.
- Testing: At test time, beam search first generates translation candidates with hidden states and likelihood scores, then reranks them using interpolated reconstruction scores.Reconstruction operates as a reranking technique over the decoder’s k-best candidates.
Experiments
Experiments on Chinese–English translation evaluate reconstruction as an auxiliary adequacy objective across decoding, training, reranking, error types, and sentence lengths. The approach improves translation quality, especially with larger decoding spaces and longer inputs, but substantially slows training.
- Experimental Setup: Experiments use 1.25M Chinese–English sentence pairs, NIST validation and test sets, and case-insensitive 4-gram NIST BLEU.The models are compared with MOSES and an attention-based RNNSEARCH baseline.
- Correlation between Reconstruction and Adequacy: Reconstruction scores correlate more strongly with human-rated translation adequacy than fluency across two evaluators.The study evaluates 200 randomly sampled source sentences using Pearson correlation.
- Learning Curves: As reconstruction improves during training, translation BLEU also rises; the best validation performance occurs at iteration 110K when likelihood and reconstruction are balanced.Figure 4 tracks translation and reconstruction BLEU on the validation set over training.
- Effect of Reconstruction in Large Decoding Space: Increasing beam size generally improves BLEU for the proposed model, with a significant gap between k = 10 and k = 100, while larger beams reduce decoding efficiency.The combined objective measures both fluency and adequacy; further beam expansion yields limited accuracy gains.
- Main Results: The proposed method gains 1.1 BLEU points over the baseline at k = 10 and a further 1.2 points when expanding the decoding space.Reconstruction also improves the Oracle score of k-best candidates across beam sizes.
- Contribution and Efficiency Analysis: Reconstruction contributes through both parameter training and candidate reranking, but training speed falls from 960 to 500 target words per second on a Tesla K80.Decoding with beam=10 changes from 2.28 to 2.60 seconds per sentence.
- Translation Error and Length Analysis: Reconstruction reduces under-translation errors by 11.0% and over-translation errors by 38.5%, while improving performance across all sentence-length segments.The gains are reported as especially significant for long sentences, where inadequate translations are a major baseline weakness.
Related Work
The work situates its encoder-decoder-reconstructor approach alongside methods for improving translation adequacy, beam search, and bidirectional dependency. It differs by integrating reconstruction into a unified NMT framework and reports that combining approaches can further improve performance.
- Enhancing Translation Adequacy: The comparison table summarizes earlier adequacy methods using coverage and context-gate mechanisms.
- Enhancing Translation Adequacy: Prior work addressed NMT’s fluent-but-inadequate translations, whereas this paper introduces an encoder-decoder-reconstructor framework.
- Enhancing Translation Adequacy: The paper states that combining its reconstruction approach with previous adequacy methods can further improve translation performance.
- Improving Beam Search: Related beam-search methods rescore candidates with language-model, SMT, or sentence-level mutual-information features.
- Capturing Bidirectional Dependency: Standard NMT captures source-to-target dependency, while prior work also combined directional models or reconstructed monolingual corpora.
Conclusion
The proposed framework adds reconstruction-based adequacy scoring to NMT and is reported to improve translation performance as decoding space increases. The authors note remaining gaps in candidate quality and the need for broader validation.
- The encoder-decoder-reconstructor framework adds an auxiliary adequacy score and consistently improves translation performance as decoding space increases.
- A remaining gap separates de facto translations from oracle k-best candidates, especially when decoding space increases.
- The authors plan richer candidate-quality features and validation across more language pairs and NMT architectures.