Source-linked AI summary
On the Weaknesses of Reinforcement Learning for Neural Machine Translation
Leshem Choshen, Lior Fox, Zohar Aizenbud, Omri Abend
TL;DR
RL is attractive for MT because it can optimize nondifferentiable scores and address exposure bias. The paper analyzes RL methods and uses experiments and simulations to examine their learning behavior, finding theoretical and convergence problems and gains that may arise from distribution peakiness rather than the reward signal.
Problem
RL is used in MT to optimize nondifferentiable evaluation scores and address exposure bias.
Source: §1 · p. 1Method
The paper analyzes RL approximations theoretically and uses naturalistic experiments and simulations to examine observed gains and training behavior.
Source: §1 · p. 1Results
Takeaways & Limitations
Observed gains may stem from fine-tuning already probable tokens or distribution effects such as peakiness, rather than from the reward signal.
Source: §7 · p. 7In MT, even effective exploration may not cover the large state-action space, where almost all sentences are not rewarding.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, but rather from changes in the shape of the distribution curve.
1 Introduction
RL is motivated by its ability to optimize nondifferentiable MT scores and address exposure bias, but little is known about the source of reported gains or the training dynamics. The introduction outlines theoretical and empirical concerns about how current RL methods learn.
1 Introduction
- Common approximation methods may converge to parameters that neither minimize risk nor are local minima of it.Source: §1 · p. 1
- Observed gains may reflect increased output-distribution peakiness rather than making target tokens most probable.The authors compare informative and constant rewards using naturalistic experiments and simulations.Source: §1 · p. 1
- Promoting the target token to the mode is likely prohibitively slow, with likely improvements limited to cases where the pretrained model is nearly correct.The target token is among the pretrained model’s first 2–3 most probable tokens in the identified favorable case.Source: §1 · p. 1
- The paper also discusses RL practices in MT to avoid and alternative approaches intended to address a broader class of pretrained-model errors.Source: §1 · p. 1
2 RL in Machine Translation
The section defines expected-reward optimization for autoregressive MT and contrasts REINFORCE with Contrastive MRT (CMRT). It argues that CMRT does not optimize the expected reward and that its sampled-set weighting requires careful tuning.
2.1 Setting and Notation
- MT is modeled autoregressively with token rewards conditioned on source, prefix, and reference; the analysis assumes one valid target and rewards independent of following tokens.The paper identifies negative log-likelihood and metrics such as BLEU as common reward choices.Source: §2.1 · p. 2
2.2 REINFORCE
- REINFORCE samples k tokens per context; its update is an unbiased estimator of the expected-reward gradient, so constant reward yields zero expected update.The section notes that adding a constant baseline to the reward does not affect the optimal parameters.Source: §2.2 · p. 2
2.3 Minimum Risk Training
- CMRT reweights sampled-set support rather than the full vocabulary; lowering α makes Q more uniform and increases improbable-token weight, but α near zero vanishes gradients, making tuning consequential.The paper reports simulations with CMRT showing trends similar to REINFORCE.Source: §2.3 · pp. 2–3
- CMRT does not optimize the expected reward or its own estimated objective, and may converge to parameters that are not local maxima of expected reward.The analysis therefore characterizes CMRT as theoretically ill-founded.Source: §2.3 · p. 2
3 Motivating Discussion
Pretrained policies tend to be peaked, limiting exploration and slowing reinforcement-learning fine-tuning. This can initially raise the probability of likely tokens rather than more rewarding ones, producing a peakiness effect and sample-inefficient training.
3 Motivating Discussion
- REINFORCE is guaranteed to converge to a stationary point under broad conditions, but its convergence rate under prevailing NMT conditions is not well understood.The section later reports empirical evidence that convergence is extremely sample-inefficient.Source: §3 · p. 3
- Pretrained RNNs quickly form peaky distributions; because RL samples from the policy, this limits exploration of potentially more rewarding tokens and can hamper convergence..Source: §3 · p. 3
- With k typically between 1 and 20 and 1–100 epochs, only a few unique tokens are likely sampled, so REINFORCE initially raises likely candidates even when more rewarding tokens rank lower..Source: §3 · p. 3
- The peakiness-effect (PKE)—increased probability mass on the most probable tokens—appears in simulations and full-scale NMT experiments, while training is extremely sample-inefficient.The paper reports that the peakiness increase occurs at least in the first phase of training.Source: §3 · p. 3
4 The Peakiness Effect
The peakiness effect (PKE) is an early increase in probability mass for the most probable tokens, even when reward does not favor them. It appears in controlled simulations and NMT experiments.
4 The Peakiness Effect
- Initially most-probable tokens gain probability mass even when they are not the most rewarding, producing the peakiness effect (PKE).The authors report this effect in both simulations and full-scale NMT experiments.Source: §4 · p. 3, §4.2 · p. 5
4.1 Controlled Simulations
- REINFORCE did not approach convergence after 1M steps in simulations where it had not converged by 50K steps.The simulations used deterministic rewards and were designed to make learning easier, so these results are optimistic about convergence.Source: §4.1 · p. 4
- With constant reward, the mode and top-10 tokens gained probability while entropy fell from 2.9 to 2.85 after one step.Under simulated reward, entropy fell from 3 to about 0.001 in 100K steps; PKE appeared within a few hundred steps and was stronger than with constant reward.Source: §4.1 · p. 4
4.2 NMT Experiments
- The NMT experiments sampled expected-BLEU rewards and contexts from pretrained and reinforced models, whose parameters are shared across contexts.Unlike the controlled simulations, a conditional distribution is rarely sampled more than a handful of times in NMT.Source: §4.2 · pp. 4–5
- In NMT, average entropy fell from 3.45 before reinforcement to 2.82 afterward, alongside a shift toward higher mode probabilities.The authors suggest the more modest entropy reduction may indicate that training did not reach the optimal parameters.Source: §4.2 · p. 5
5 Performance following REINFORCE
The section tests when REINFORCE improves NMT, using controlled simulations and translation experiments. Gains are modest, and target-token improvements are concentrated among tokens already highly ranked by the pretrained model.
5.1 Controlled Simulations
- Only when ybest began second-most probable did REINFORCE make it the mode within 100K steps; lower-ranked targets showed little gain even after 1M steps.These were among the easiest simulated settings, tested with a high learning rate.Source: §5.1 · p. 6
5.2 NMT Experiments
- About half of NMT contexts ranked ybest fourth or lower before training, where simulations predict little probability gain.The reinforced model ranked more target tokens first, but showed little consistent shift elsewhere among the ten highest ranks.Source: Figure 4 · p. 6, §5.2 · p. 6, Figure 5 · p. 7
- 30.31 pretraining BLEU rose to 30.73 after expected-BLEU REINFORCE on the news2014 test set.With constant reward, BLEU reached 30.72, a nearly identical improvement.Source: §5.2 · p. 6
- The similar BLEU improvement with constant reward suggests gains may partly reflect reward-independent factors such as increased peakiness.The authors also report a similar pattern in target-token rank changes under constant reward.Source: §5.2 · pp. 6–7
6 Experiments with Contrastive MRT
Contrastive MRT (CMRT) matches REINFORCE's difficulty when the target token is not already highly ranked, while producing only a small peakiness effect.
6 Experiments with Contrastive MRT
- CMRT pushed ybest to the highest rank when it began second, but struggled when it began third or lower.The controlled simulations averaged 100 trials using α = 0.005 and k = 20.Source: §6 · p. 7
- CMRT showed only a small peakiness effect, likely because unsampled tokens do not lose probability mass under its contrastive effect.The authors say this contrast may counter the decrease in probability mass for non-sampled tokens.Source: §6 · p. 7
- Across several other reward functions, BLEU scores ranged from 30.73 to 30.84.The authors describe the improvement as stable across metrics, trials, and pretrained models.Source: §6 · p. 7
7 Discussion
The discussion argues that existing RL practices often cannot promote low-ranked target tokens efficiently and that observed gains may reflect peakiness rather than reward learning. It also identifies practices that may hinder convergence or destabilize training.
7 Discussion
- Promoting a target token to the mode is likely prohibitively slow except when the pretrained model is already nearly correct.The authors suggest improvements may instead come from fine-tuning the pretrained model’s most probable tokens or from reward-independent effects such as peakiness.Source: §7 · p. 7
- Several observations support reward-independent effects: constant-reward NMT produced similar gains, reinforced distributions became peakier, and simulations rarely converged outside easy conditions.The authors present these as reasons to believe improvements may not be due to the reward function.Source: §7 · p. 7
- Gradient clipping is expected to hinder REINFORCE convergence because it violates the method’s assumptions.Source: §7 · p. 7
- Adding reference translations to samples may reduce missed-target problems, but can destabilize training on outputs unlike those the model assigns high probability.The cited concern is that improving such outputs may come at the cost of other outputs.Source: §7 · pp. 7–8
8 Conclusion
MT poses unusually difficult conditions for reinforcement learning: enormous discrete action spaces, sparse rewards, and limited exploration from peaky pretrained policies. Off-policy and exploration techniques may help, but even diverse sampling may not cover MT's state-action space.
8 Conclusion
- MT combines high-dimensional discrete action spaces, sparse rewards, and peaky pretrained models that limit on-policy exploration.On-policy actions are sampled from the model being optimized.Source: §8 · p. 8
- Off-policy sampling can in principle use a more exploratory policy, while state/action usefulness measures and parameter-space noise offer other exploration approaches.Off-policy sampling can also provide smoothing while retaining convergence guarantees.Source: §8 · p. 8
- Even effective diverse sampling may not cover MT's large state-action space, where almost all sentences are unrewarding.The paper identifies diversity-based and multi-goal RL methods as a promising path for NLP.Source: §8 · p. 8
A Contrastive MRT does not Maximize the Expected Reward
A simple three-outcome example shows that Contrastive MRT can converge to values that do not maximize expected reward. The section also derives the estimator gradient for sampled outcomes.
A Contrastive MRT does not Maximize the Expected Reward
- With a sufficiently small learning rate, initial θ∈(0,0.5] converges to γ≈0.295, while θ=0 stays fixed; neither value maximizes R(θ).The expected gradient is positive below γ and negative above it.Source: A Contrastive MRT does not Maximize… · p. 11
- Using g(θ) can place γ arbitrarily far from θ*, and maximizing E[eR] instead yields a maximum near θ≈0.32, not the maximum of R(θ).The passage notes g could map to (−inf, inf), as often used for neural-network parameters.Source: A Contrastive MRT does not Maximize… · p. 11
B Deriving the Gradient of eR
- For k=2, Table 1 lists each possible sample S and its corresponding eR gradient.The table supports the section's derivation of the gradient for sampled outcomes.Source: Table 1 · p. 11, B Deriving the Gradient of eR · p. 11
C NMT Implementation Details
The NMT experiments use a six-layer encoder and decoder with 512-dimensional embeddings and specified preprocessing and optimization settings. Figure 7 depicts token probabilities in constant-reward simulations for different initial target-token ranks.
C NMT Implementation Details
- The MT experiments use six encoder and decoder layers, 512-dimensional embeddings, BPE with 30,715 tokens, and discard sentences longer than 50 tokens.Pretraining used gradient clipping at 5; residual dropout was 0.1 and attention dropout was not used.Source: C NMT Implementation Details · p. 11
- Pretraining and training stopped after BLEU failed to rise on the development set for 10 evaluations; pretraining took about 7 days on 4 GPUs.RMSProp used learning rate 0.01 for pretraining and Adam with decay used 0.005 for training; Monte Carlo used 20 sentence rolls per word.Source: C NMT Implementation Details · pp. 11–12
- Figure 7 shows token probabilities after REINFORCE in constant-reward simulations, with target tokens initially ranked second, third, or fourth.The green line is the target token, yellow lines are medium-reward tokens, and red lines have reward zero.Source: Figure 7 · p. 12
D Detailed Results for Constant Reward Setting
The constant-reward results are reported as following trends similar to the simulated-reward setting. Figure 8 represents differences between reinforced and pretrained rank probabilities for the target token.
D Detailed Results for Constant Reward Setting
- The constant-reward setting shows trends similar to those in the simulated-reward setting.Source: D Detailed Results for Constant Reward… · p. 12
- Figure 8 compares the probability that ybest occupies each rank in the reinforced model with that probability in the pretrained model.Source: Figure 8 · p. 12
- Each Figure 8 column x gives the difference in probability of ybest being ranked x in the reinforced versus pretrained model.Source: Figure 8 · p. 12