PaperlayerView exact revision on arXiv

Source-linked AI summary

An Actor-Critic Algorithm for Sequence Prediction

Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, Yoshua Bengio

arXiv:1607.07086v3cs.LG

TL;DR

Sequence prediction trained by log-likelihood suffers from a mismatch between ground-truth-conditioned training and self-conditioned test-time generation. The paper uses actor-critic training with a ground-truth-conditioned critic to optimize task-specific scores, achieving improvements over maximum likelihood on synthetic and machine-translation tasks. The approach offers lower-variance gradient estimates than REINFORCE, but critic imperfection introduces bias and optimization can suffer when the actor becomes deterministic.

  • Problem

    Log-likelihood training conditions on ground-truth histories, whereas test-time generation conditions on the model’s own guesses, which can compound errors.

  • Method

    An actor generates sequences while a critic predicts token values using the ground-truth output, enabling actor updates toward expected task-specific scores.

  • Results

    The method significantly improves over maximum-likelihood training on a synthetic task and machine translation, including a 2.3 BLEU-point gain with greedy decoding in one translation experiment.

  • Takeaways & Limitations

    Actor-critic training brings task objectives into supervised sequence prediction and fits machine-translation training data faster than REINFORCE.

  • Takeaways & Limitations

    The critic lowers variance at the cost of significant bias, and actor distributions can saturate, causing vanishing gradients and requiring further investigation.

Abstract

from arXiv · show

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a \textit{critic} network that is trained to predict the value of an output token, given the policy of an \textit{actor} network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

1 INTRODUCTION

Sequence models trained with teacher forcing face a train–test mismatch because test-time generation conditions on their own guesses. The paper proposes actor-critic training to optimize task-specific test metrics directly while using ground-truth outputs to improve the critic.

  • Teacher forcing trains RNNs to predict each correct token after the previous correct tokens.
  • At test time, models condition on their own guesses, so errors can compound, especially in longer sequences.
  • The proposed critic predicts each token’s expected task-specific score under the actor’s continuation policy.
  • Actor-critic training uses critic values to train the actor toward higher expected task-specific scores rather than only log-likelihood.
  • The method applies actor-critic reinforcement-learning ideas to supervised structured-output problems and improves over maximum likelihood and REINFORCE on synthetic and translation tasks.

2 BACKGROUND

The paper formulates sequence prediction with conditioned recurrent networks and defines actor-critic value estimates for sampled continuations. Its training algorithm alternates critic and actor updates, with pretraining before joint optimization.

  • An RNN generates an output sequence Y conditioned on input X, with tokens drawn from an alphabet A.
  • A stochastic output layer defines next-token probabilities and the probability of complete sequences, including an end-of-sequence token.
  • Attention conditions the RNN on encoded input vectors by assigning weights and forming a context vector for each time step.
  • Value functions: The value of a candidate action is its expected future return after that token, under the actor’s policy and reward function.
  • Actor-Critic Training: Actor-critic training initializes actor and critic networks, samples sequences, computes critic targets, updates the actor, and slowly updates delayed networks.
  • Actor-Critic Training: The complete procedure pretrains the actor with log-likelihood, pretrains the critic with a fixed actor, and then runs joint actor-critic training.

3 ACTOR-CRITIC FOR SEQUENCE PREDICTION

The method treats sequence prediction as supervised actor-critic learning: a critic estimates token values under the actor’s policy, and those estimates train the actor toward task-specific returns. Ground-truth outputs aid critic training, while temporal-difference learning, target networks, delayed actors, variance penalties, and reward shaping address optimization challenges.

  • Actor-critic objective: The actor is trained using critic-estimated token values to increase probabilities of high-value actions and decrease probabilities of low-value actions.This provides an unbiased gradient formulation when summing over actions, although replacing exact values with a parametric critic introduces bias.
  • Critic architecture: The critic is a separate RNN that runs alongside the actor, predicts Q values for all candidate tokens, and receives the ground-truth output during training.The ground-truth sequence can be used because the critic is needed only during training, not at test time.
  • Critic training: Temporal-difference learning trains the critic with a Bellman-equation target instead of full future returns, reducing the high variance of naïve Monte Carlo targets.The target combines the immediate reward with the actor’s probability-weighted value estimates for subsequent actions.
  • Stability mechanisms: A slowly updated target critic and a delayed actor are used to reduce instability caused by nonlinear temporal-difference learning and actor-critic feedback.The delayed networks provide more stable predictions and sampled trajectories during joint training.
  • Large action spaces: For large vocabularies, the critic objective penalizes variance across action values because rare actions can otherwise be overestimated, biasing gradients and causing divergence.The penalty drives rare-action values toward their mean and was found necessary for convergence.
  • Reward shaping: Potential-based reward shaping supplies intermediate rewards from score differences while preserving the optimal policy.This makes the critic’s training signal less sparse than awarding the complete task score only at the final step.
  • Training procedure: The method combines the actor-critic estimate with a weighted log-likelihood gradient because actor-critic training alone can cause early policy determinization and vanishing gradients.Jointly initializing randomly would also leave actor and critic without adequate training signals for one another.

4 RELATED WORK

The paper relates its approach to REINFORCE, imitation learning, expected-score gradient methods, and beam-search optimization. These alternatives differ in their use of ground truth, variance properties, expert policies, search procedures, or inference-specific objectives.

  • REINFORCE: REINFORCE optimizes sequence-level BLEU or ROUGE while gradually transitioning from maximum likelihood, but has high variance and does not exploit ground-truth outputs like the critic.The actor-critic estimator instead uses learned token values, trading lower variance for bias from imperfect simultaneous critic training.
  • Imitation learning: Imitation-learning methods such as SEARN and DAGGER train a policy to imitate an expert, whereas this approach learns a critic that plays a similar role without requiring an expert policy.Constructing an expert policy for a task-specific score may be difficult or impossible.
  • Expected-score optimization: Direct Loss Minimization and Minimum Bayes Risk Training also approximate expected task scores, respectively by adapting inference or restricting the expectation to a small candidate subset.These methods target task-specific objectives through different approximations than the actor-critic procedure.
  • Search optimization: Learning-as-search-optimization methods optimize global sequence cost through beam-search selection and pruning, making them specific to the precise inference procedure.This is presented as an alternative to the paper’s approach.

5 EXPERIMENTS

Experiments evaluate actor-critic training on spelling correction and machine translation, comparing it with log-likelihood and REINFORCE-based methods. Results show improved task-specific performance, with important effects from critic design and training choices.

  • Experimental setup: The experiments cover synthetic spelling correction and German-English machine translation, with an additional WMT14 English-French evaluation.The spelling task uses corrupted natural-text strings; translation experiments use IWSLT14 and WMT14 datasets.
  • Spelling correction: Character error rate improves over log-likelihood training in all four spelling-correction settings.Without simultaneous log-likelihood training, actor-critic outperforms REINFORCE-critic in three of four settings; the fourth is similar.
  • Machine translation: 2.3 BLEU points separate the proposed methods from the log-likelihood baseline under greedy IWSLT14 decoding.REINFORCE-critic performs 0.6 BLEU points above actor-critic; with beam search, the ranking remains but the margin over log-likelihood narrows.
  • Learning dynamics: Generated-prediction methods regularize validation performance, while actor-critic fits the training set better than REINFORCE.The learning curves compare log-likelihood, REINFORCE, and actor-critic on training and validation BLEU.
  • Ablation study: A target network was crucial, and the value-penalty coefficient λ was effective in the range [10^-3, 10^-6].Reward shaping and a delayed actor produced moderate gains.
  • Machine translation: 1.5 BLEU improvement with greedy search and 0.4 BLEU improvement with beam search are reported for actor-critic on WMT14 English-French.In this experiment, actor-critic also has a significant advantage over REINFORCE-critic.

6 DISCUSSION

The discussion reports optimization challenges, regularization effects, and evidence that actor-critic training can improve sequence prediction while using low-variance, potentially high-bias estimates.

  • Actor-critic training fits machine-translation training data much faster than REINFORCE, while sometimes achieving better test error with the critic as REINFORCE’s baseline.
  • Generated-prediction methods strongly regularize training, making actor-critic expectations harder to assess because noisier gradients can improve test performance.The authors attribute this effect to increased training-data diversity from conditioning on sampled outputs.
  • Evidence from virtually infinite spelling data and large WMT14 data supports actor training, while larger machine-translation datasets remain future work.
  • Critic variance penalties address inflated values for low-probability actions, while policy saturation can cause vanishing gradients; combining RL and log-likelihood objectives can help.
  • The actor-critic approach targets gradient estimation from a different angle, using low-variance but potentially high-bias estimates and ground-truth outputs.Future work could reduce bias with a multi-sample critic training criterion.

A HYPERPARAMETERS

The appendix specifies training hyperparameters, model-input variants, decoding length penalties, and an ablation study reported in Table 5.

  • Machine-translation training used λ = 10^-4 and γθ = γφ = 10^-4; REINFORCE with the critic used γθ = 1.
  • Spelling correction used λ = 10^-3 with the same γθ and γφ, while the combined criterion assigned λLL = 0.1.
  • Actor states were supplied to the critic in spelling correction and WMT14 machine translation, but not in the other reported results.
  • Beam-search decoding subtracted candidate length times ρ from log-likelihood cost; ρ was 0.8 for log-likelihood and REINFORCE, and 1.0 for actor-critic and REINFORCE-critic.
  • An ablation study examined some hyperparameters, with results reported in Table 5.

B DATA

The datasets use bounded vocabularies and sentence lengths for IWSLT14 and WMT14, with reported validation and test sizes for IWSLT14.

  • IWSLT14 contains 6,969 validation examples and 6,750 test examples, with English and German vocabularies capped at 22,822 and 32,009 words.
  • WMT14 uses 30,000-word vocabularies for both languages, and both datasets impose a maximum sentence length of 50.

C GENERATED Q-VALUES

Figure 3 lists the critic’s three highest-valued next words at intermediate translation steps, alongside their predicted values and a reference translation.

  • At each intermediate step, the critic ranks three candidate words and reports each candidate’s predicted value ˆQ.
  • For the example translation, the reference is “and there’s one I want to talk about,” while the critic’s rankings vary across generation steps.

D PROOF OF EQUATION (7)

The passage presents an equation involving the derivative of a sequence probability and a reward term, but does not provide enough context to explain the proof.

  • The expression differentiates a product of conditional token probabilities with respect to θ.
  • The sequence probability is written as p(ŷ1)p(ŷ2|ŷ1) … p(ŷT|ŷ1 … ŷT−1).
  • The expression includes the reward term R(Ŷ).
Loading 1607.07086v3…