Source-linked AI summary
A Study of Reinforcement Learning for Neural Machine Translation
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, Tie-Yan Liu
TL;DR
Applying reinforcement learning effectively to deep, real-world neural machine translation remains challenging, and its combination with monolingual data is insufficiently understood. This paper systematically studies RL training choices, proposes a method using source and target monolingual data, and achieves competitive results across major translation tasks, including state-of-the-art performance on WMT17 Chinese-English.
Problem
Previous work had not effectively applied reinforcement learning to real-world NMT systems with powerful deep models, and the benefits of combining RL with monolingual data remained unclear.
Method
The paper systematically evaluates RL training strategies and proposes a method that combines RL with large-scale source and target monolingual data.
Results
The approach achieves competitive translation quality across WMT14 English-German, WMT17 English-Chinese, and WMT17 Chinese-English, surpassing a strong WMT17 Chinese-English baseline by nearly 1.5 BLEU points.
Takeaways & Limitations
The study provides practical findings for training deep NMT models with RL, including that multinomial sampling outperforms beam search and combining MLE with RL objectives is important.
Abstract
from arXiv · showhide
Recent studies have shown that reinforcement learning (RL) is an effective approach for improving the performance of neural machine translation (NMT) system. However, due to its instability, successfully RL training is challenging, especially in real-world systems where deep models and large datasets are leveraged. In this paper, taking several large-scale translation tasks as testbeds, we conduct a systematic study on how to train better NMT models using reinforcement learning. We provide a comprehensive comparison of several important factors (e.g., baseline reward, reward shaping) in RL training. Furthermore, to fill in the gap that it remains unclear whether RL is still beneficial when monolingual data is used, we propose a new method to leverage RL to further boost the performance of NMT systems trained with source/target monolingual data. By integrating all our findings, we obtain competitive results on WMT14 English- German, WMT17 English-Chinese, and WMT17 Chinese-English translation tasks, especially setting a state-of-the-art performance on WMT17 Chinese-English translation task.
1 Introduction
The paper studies how to apply reinforcement learning effectively to large-scale, deep NMT systems, including when source- and target-side monolingual data are available. It compares RL training choices and reports competitive or state-of-the-art translation results across three WMT tasks.
- Motivation: Previous RL studies had not established how to apply RL effectively to deep, real-world NMT systems with large datasets.The paper identifies both limited evaluation on shallow models and unresolved training instability as gaps.
- Approach: The study comprehensively evaluates reward setup, MLE–RL objective weighting, and variance-reduction strategies for practical RL training.These factors target efficient rewards, training stability, and gradient-estimation variance.
- Monolingual data: The paper proposes combining RL with source- and target-side monolingual data to further improve NMT models.This addresses the open question of whether RL remains beneficial in semi-supervised NMT settings.
- Findings: Experiments on WMT17 Zh-En, WMT17 En-Zh, and WMT14 En-De find that multinomial sampling outperforms beam search for reward computation.The study also reports that combining RL and monolingual data significantly enhances NMT performance.
- Results: Nearly 1.5 BLEU points above a Transformer big model with back translation, the method achieves state-of-the-art translation quality on WMT17 Zh-En.The paper also reports strong competitive results on WMT14 En-De and WMT17 En-Zh.
2 Background
The background introduces attention-based encoder–decoder NMT and frames RL as sequence-level optimization that directly targets translation evaluation measures. Because the candidate-sentence space is exponentially large, REINFORCE approximates expected reward through sampling.
- Neural Machine Translation: Attention-based NMT encodes a source sentence into continuous representations, then decodes the target sentence token by token.At each step, the decoder conditions on the source and preceding target tokens.
- Neural Machine Translation: MLE trains NMT by maximizing the likelihood of target tokens over bilingual sentence pairs.The objective sums token-level log probabilities, with m denoting the target sentence length.
- Neural Machine Translation: Transformer achieves strong translation quality using self-attention rather than recurrent or convolutional operations.This distinguishes it from earlier RNNSearch and ConvS2S architectures.
- Reinforcement Learning: RL views NMT as an agent whose policy selects words and receives a sentence-level BLEU reward after generating a complete translation.The training objective maximizes expected reward over generated candidate sentences.
- Reinforcement Learning: Because the candidate translation space is exponentially large, REINFORCE estimates expected reward by sampling translations from the policy.The paper uses REINFORCE as its policy-optimization method throughout.
3 Strategies for RL Training
RL training for NMT requires practical choices that address unstable gradients, sparse rewards, and the trade-off between likelihood and sequence-level objectives. The study examines sampling, reward shaping, variance reduction, and objective combination.
- RL training is unstable and inefficient because gradient estimation and reward computation are noisy, with no consensus on effective configurations for large-scale NMT.
- Reward computation: Beam search exploits likely translations, whereas multinomial sampling explores more diverse candidates when generating sentences for BLEU rewards.
- Reward computation: Reward shaping adds intermediate decoding-step rewards because a single terminal sentence reward is sparse and treats all token actions identically.
- Variance reduction of gradient estimation: Baseline rewards subtract an estimated average reward from training signals to reduce the high variance of REINFORCE gradient estimation.The study uses a learned function, such as a multilayer perceptron, to estimate the baseline.
- Combine MLE and RL Objectives: A linear combination of MLE and RL objectives is used as a stabilization strategy, with α controlling their trade-off.The paper empirically evaluates how different α values affect translation accuracy.
4 RL Training with Monolingual Data
The paper investigates whether RL can benefit from source- and target-side monolingual data, despite missing reference or source sentences. It proposes pseudo-data construction methods and sequential or unified combinations for RL training.
- Combining RL training with monolingual data remains an open problem because prior NMT RL work typically uses only bilingual data.
- Source-side monolingual data: Source-side monolingual data lacks reference translations needed for sentence-level rewards, so the method generates pseudo target references with an NMT model trained on bilingual data.The model beam-searches a pseudo reference, then obtains the sampled translation with multinomial sampling for reward computation.
- Target-side monolingual data: Target-side monolingual data lacks source sentences, so a reverse NMT model back-translates each target sentence into a pseudo source sentence.The pseudo source-target pair is then used for RL training like a genuine bilingual pair.
- Combining both data sources: The sequential method combines both monolingual sources by using them successively in MLE and REINFORCE-based training.
5 Experiments
Experiments across three translation tasks systematically evaluate RL training strategies and monolingual-data integration. Combining MLE and RL objectives, multinomial sampling, and monolingual data produced stronger NMT results, including 26.73 BLEU on WMT17 Chinese-English.
- Experimental settings: Experiments evaluate RL strategies on WMT14 English-German, WMT17 English-Chinese, and WMT17 Chinese-English bilingual datasets, with monolingual-data experiments on Chinese-English.The monolingual datasets contain 4M Chinese and 7M English sentences after preprocessing.
- Reward computation: Multinomial sampling outperformed beam search by roughly 0.2-0.3 BLEU points, with significant test scores (ρ < 0.05).The authors conjecture that multinomial sampling improves data diversity and exploration during reward computation.
- Variance reduction: Baseline-reward learning did not help RL training, suggesting that additional baseline-reward steps were unnecessary for NMT in these experiments.The authors relate this result to concentrated probability mass in the target-side language space.
- Combining MLE and RL objectives: 27.48 En-De, 34.63 En-Zh, and 25.04 Zh-En were obtained when combining MLE and RL objectives with α = 0.3.Further increasing α brought no additional gain; α = 0.3 was the best trade-off in these experiments.
- Monolingual data: Sequential training improved BLEU by 0.7 points, while unified RL training improved a strong MLE baseline from 26.13 to 26.73.The sequential improvement was significant at ρ < 0.01, as was the unified improvement.
- Overall comparison: Combining source-side and target-side monolingual data with RL achieved a state-of-the-art 26.73 BLEU score on WMT17 Chinese-English.The result surpassed the best ensemble model reported for that translation challenge.
6 Related Work
Prior work applied reinforcement learning to directly optimize sequence-level or evaluation objectives in neural machine translation. Related research also explored source- and target-side monolingual data, while reported benefits varied across methods and language pairs.
- Reinforcement learning for NMT: Representative NMT studies used REINFORCE, minimum risk training, or actor-critic methods to optimize sequence-level objectives.These approaches directly target evaluation-related objectives rather than only token-level likelihood.
- Reinforcement learning for NMT: Earlier work reported improved BLEU for English-French but not English-German using a simple reinforcement-learning method with stacked LSTMs.
- Monolingual data for NMT: Monolingual-data research leveraged source-side data, target-side data through back-translation, and related communication-game formulations for NMT.
7 Conclusion
The conclusion identifies practical findings for reinforcement-learning training of NMT models across multiple translation tasks. Combining reinforcement learning with monolingual data achieved state-of-the-art BLEU on WMT17 Chinese-English.
- Study scope: The study evaluates reinforcement-learning strategies for NMT across German-English, English-Chinese, and Chinese-English translation tasks using large-scale bilingual datasets.
- RL training findings: Multinomial sampling performs better than beam search for reinforcement-learning reward computation.
- RL training findings: Reward shaping and baseline reward do not make a significant difference, whereas combining maximum-likelihood and reinforcement-learning objectives is important.
- Monolingual data: Combining reinforcement learning with source- and target-side monolingual data achieves the state-of-the-art BLEU score on WMT17 Chinese-English translation.