Source-linked AI summary
Dual Learning for Machine Translation
Yingce Xia, Di He, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, Wei-Ying Ma
TL;DR
Machine translation requires costly, large parallel corpora, motivating methods that exploit abundant monolingual data. The paper introduces dual-NMT, a reinforcement-learning framework in which dual translation models communicate in a closed loop and improve from feedback. With only 10% bilingual data, it achieves accuracy comparable to vanilla NMT trained with 100% bilingual data for French-to-English translation.
Problem
Machine translation systems rely on costly, limited parallel corpora, while existing monolingual-data methods either leave the shortage unresolved or lack quality control for pseudo-pairs.
Method
Dual-NMT trains two dual-direction translation models from monolingual data through a communication game using naturalness and reconstruction rewards with reinforcement learning.
Results
With only 10% bilingual data, dual-NMT achieves comparable translation accuracy to vanilla NMT using 100% bilingual data for French-to-English translation.
Takeaways & Limitations
The framework reduces aligned-bilingual-data requirements and provides a way to extract reward signals for reinforcement learning in machine translation.
Takeaways & Limitations
Experiments use bilingual data to warm-start dual-NMT, so learning translations entirely from monolingual data remains future work.
Abstract
from arXiv · showhide
While neural machine translation (NMT) is making good progress in the past two years, tens of millions of bilingual sentence pairs are needed for its training. However, human labeling is very costly. To tackle this training data bottleneck, we develop a dual-learning mechanism, which can enable an NMT system to automatically learn from unlabeled data through a dual-learning game. This mechanism is inspired by the following observation: any machine translation task has a dual task, e.g., English-to-French translation (primal) versus French-to-English translation (dual); the primal and dual tasks can form a closed loop, and generate informative feedback signals to train the translation models, even if without the involvement of a human labeler. In the dual-learning mechanism, we use one agent to represent the model for the primal task and the other agent to represent the model for the dual task, then ask them to teach each other through a reinforcement learning process. Based on the feedback signals generated during this process (e.g., the language-model likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e.g., using the policy gradient methods). We call the corresponding approach to neural machine translation \emph{dual-NMT}. Experiments show that dual-NMT works very well on English$\leftrightarrow$French translation; especially, by learning from monolingual data (with 10% bilingual data for warm start), it achieves a comparable accuracy to NMT trained from the full bilingual data for the French-to-English translation task.
1 Introduction
Machine translation depends heavily on costly, limited parallel corpora. The paper proposes dual-NMT, which uses two translation agents and reinforcement feedback to learn from monolingual data while reducing this dependence.
- Parallel training corpora are costly to collect and limited in scale, constraining machine-translation research and applications.
- Existing monolingual-data methods either train target-language models without resolving the parallel-data shortage or generate pseudo-pairs without quality guarantees.
- Dual-NMT uses source- and target-language monolingual data through a two-agent communication game that provides translation feedback.
- The agents assess naturalness and reconstruction consistency, then iteratively improve both translation models through reinforcement learning.
- Dual learning trains translation models from unlabeled data, substantially reduces aligned-bilingual-data requirements, and may support learning without parallel data.
- Dual learning also offers a way to extract reward signals for deep reinforcement learning in complex applications such as machine translation.
2 Background: Neural Machine Translation
NMT commonly uses an RNN encoder-decoder to model conditional translation probabilities. The encoder represents the source sentence, while the decoder generates target words from prior words and source context.
- NMT learns a probabilistic mapping P(y|x) from a source sentence x to a target sentence y.
- The encoder reads the source sentence and generates one hidden state for each source position using a recurrent neural network.
- The recurrent unit f may be implemented as a Long Short-Term Memory or Gated Recurrent Unit.
- The decoder computes each target-word probability conditioned on preceding target words and the source sentence.
- Decoder context c_t can summarize the full source sentence or use local attention over encoder hidden states.
- Training seeks parameters Θ* that maximize the log probability of target sentences given source sentences in dataset D.
3 Dual Learning for Neural Machine Translation
Dual-NMT trains two directional translation models through a dual-learning game using monolingual corpora, language-model feedback, reconstruction rewards, and policy-gradient updates. The process alternates games beginning in each language and repeats until convergence.
- Dual-NMT uses two neural translation models to improve A-to-B and B-to-A translation with separate, potentially unrelated monolingual corpora.The corpora need not be aligned or topically related, and the models are initialized as weak translation systems.
- Each game generates middle translations with one directional model, evaluates their naturalness with a target-language model, and scores reconstruction through reverse translation.The language-model reward is LMB(smid), while the communication reward is log P(s|smid; ΘBA).
- The total reward combines language-model and communication rewards as r = αr1 + (1 − α)r2, with α controlling their relative weighting.The combined reward is optimized by policy-gradient methods with respect to both translation-model parameter sets.
- Beam search produces K high-probability middle translations whose averaged rewards approximate the expected gradient during training.Beam search is used because random sampling can have high variance and produce unreasonable machine-translation outputs.
- Training alternates updates from sentences sampled from both monolingual corpora and repeats the two directional games until convergence.The algorithm symmetrically updates the models for games beginning in language A and language B.
4 Experiments
Experiments evaluate dual-NMT against standard NMT and pseudo-NMT on English↔French translation using Large and Small warm-start settings. Dual-NMT outperforms the baselines across settings, with especially strong gains for French→English and improved reconstruction and length-robust BLEU performance.
- Experimental Setup: Experiments compare dual-NMT with standard NMT and pseudo-NMT on English→French and French→English using Large and Small warm starts.Large uses all 12M bilingual sentence pairs, while Small uses 10%.
- Translation Results: Dual-NMT outperforms both baseline algorithms in every reported setting.For English→French, gains over NMT are about 2.1/3.4 BLEU points and over pseudo-NMT about 1.7/3.1 points across the two warm-start settings.
- Translation Results: For French→English, dual-NMT improves over NMT by about 2.3/5.2 points and over pseudo-NMT by about 2.1/4.3 points across the two warm-start settings.The improvements are larger for French→English than for English→French.
- Translation Results: With only 10% bilingual data, dual-NMT achieves comparable French→English accuracy to vanilla NMT trained with 100% bilingual data.This result comes from the Small warm-start setting.
- Observations: Dual-NMT improves more when parallel bilingual data are small, indicating effective use of monolingual data.The paper reports that this pattern supports applying dual-NMT to language pairs with less labeled parallel data.
- Length Analysis: Dual-NMT outperforms the baselines across all source-sentence length ranges in Figure 1.The figure plots BLEU scores against source-sentence length.
- Reconstruction Analysis: Dual-NMT has higher self-reconstruction BLEU than NMT and pseudo-NMT, with gains over NMT reaching about 20.7/17.8 points under 10% parallel-data warm starts.Self-reconstruction translates each test sentence forth and back before comparing it with the original using BLEU.
- Conclusion: The reported results collectively indicate that dual learning better utilizes monolingual data.The authors summarize the experiments as evidence that the dual-learning mechanism is promising.
5 Discussions
The discussion extends dual learning beyond two-task machine translation loops and identifies broader closed-loop applications, while outlining current scope boundaries and future directions.
- Dual learning can apply to any pair of tasks in dual form, using reinforcement learning to train both from unlabeled data.
- Potential applications beyond machine translation include speech recognition and text-to-speech, image captioning and image generation, and question answering and question generation.
- A generalized close-loop learning scheme could connect more than two associated tasks and use input-output similarity as feedback for all models.The paper illustrates English→Chinese→French→English translation and proposes testing this extension in future work.
- The reported dual-NMT experiments used bilingual data to warm-start training, leaving direct learning from monolingual data as a future direction.
- Future work also includes applying the idea to phrase-based SMT and jointly training translation models for 3+ languages.