PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning For Sequence to Sequence Models

Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, Chandan K. Reddy

arXiv:1805.09461v4cs.LGstat.ML

TL;DR

Seq2seq models face exposure bias and a mismatch between training objectives and test-time metrics. This survey reviews reinforcement-learning methods for addressing these issues, reports varied empirical outcomes, and provides an open-source summarization library.

  • Problem

    Traditional seq2seq models suffer from exposure bias and inconsistency between training objectives and testing objectives.

  • Method

    The paper surveys RL methods combined with seq2seq training and presents RL formulations, algorithms, and implementations for abstractive summarization.

  • Results

    The experiments report that SCPG improves the pointer-generator model only marginally, while the Actor-Critic model achieves superior ROUGE results and converges in a few hours.

  • Takeaways & Limitations

    The survey provides an overview for selecting RL algorithms for seq2seq problems and open-source code for training summarization models with different RL techniques.

Abstract

from arXiv · show

In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-of-the-art performance in a wide variety of tasks such as machine translation, headline generation, text summarization, speech to text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoder-decoder models produce competitive results, many researchers have proposed additional improvements over these sequence-to-sequence models, e.g., using an attention-based model over the input, pointer-generation models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with sequence-to-sequence models that enable remembering long-term memories. We present some of the most recent frameworks that combine concepts from RL and deep neural networks and explain how these two areas could benefit from each other in solving complex seq2seq tasks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization.

I. INTRODUCTION

Seq2seq models use encoder–decoder neural networks to transform input sequences into outputs, but their training and evaluation rely on distinct procedures. This survey introduces reinforcement-learning approaches for training such models and reviews their challenges.

  • The survey reviews reinforcement-learning methods that combine seq2seq models with decision-making algorithms to address training challenges.It aims to provide a broad overview and guide researchers in choosing RL algorithms for seq2seq problems.
  • The paper provides source code for implementing several RL models for abstractive text summarization.
  • Seq2seq models commonly solve tasks including translation, headline generation, summarization, speech-to-text, and image captioning.
  • An encoder reads an input sequence, while a decoder generates the output sequence from the encoder representation and its current state.The decoder may also use an additional context vector.
  • Teacher forcing trains the decoder with ground-truth outputs using a maximum-likelihood cross-entropy objective.At test time, the model generates outputs sequentially, optionally using beam search, and is evaluated with measures such as ROUGE, BLEU, METEOR, or CIDEr.

B. Problems with Seq2seq Models

Seq2seq training can suffer from exposure bias and a mismatch between differentiable training losses and discrete test metrics. Reinforcement learning formulates generation as sequential decision-making, with task-specific actions, rewards, and state representations.

  • Exposure bias arises because training conditions each decoder step on ground-truth outputs, whereas testing conditions generation on the model’s own previous outputs.
  • Scheduled sampling addresses exposure bias through random selection between ground-truth and model outputs but is statistically inconsistent because error back-propagation is unclear.
  • Cross-entropy training can mismatch discrete, non-differentiable evaluation measures such as BLEU and ROUGE.
  • In the RL formulation, a seq2seq model acts as a policy that selects task-specific actions and receives application-dependent rewards.For summarization, actions can be next tokens and rewards can include ROUGE or BLEU; the decoder output state usually defines the state.
  • The value function evaluates states, the Q function evaluates state–action pairs, and the advantage function measures action importance as Q minus V.

D. Paper Organization

This section frames the paper as a broad survey of reinforcement-learning methods for sequence-to-sequence training, emphasizing challenges, method-selection guidance, and an open-source implementation framework.

  • The paper formulates seq2seq training as a problem of avoiding exposure bias and directly optimizing evaluation measures.
  • The survey explains how RL and seq2seq learning can benefit one another across complex tasks beyond robotics and vision.
  • It surveys RL methods, their strengths and weaknesses, and guidelines for improving RL-based seq2seq training.
  • The paper provides an open-source library for implementing different RL techniques in complex seq2seq models.
  • It discusses seq2seq applications including translation, summarization, speech-to-text, image captioning, sentiment analysis, relation extraction, and pronoun resolution.
  • Attention-based, pointer-generator, and transformer models are related seq2seq approaches, while the latter advanced models are outside this paper’s scope.

A. Evaluation Measures

The paper describes discrete evaluation measures used across seq2seq applications and surveys datasets supporting tasks such as translation, summarization, dialogue, semantic parsing, and captioning.

  • Evaluation Measures: Seq2seq models are trained with cross-entropy but evaluated with discrete, application-specific measures, creating a training–evaluation mismatch.
  • Evaluation Measures: ROUGE, BLEU, and METEOR evaluate text generation, while CIDEr and SPICE are used primarily for image and video captioning.
  • Evaluation Measures: Word Error Rate measures speech-recognition output through substitutions, deletions, insertions, and corrections relative to the ground truth.
  • Datasets: Table II summarizes seq2seq applications by describing the sequence units provided as inputs and generated as outputs.
  • Datasets: The surveyed datasets cover machine translation, text summarization, headline generation, question answering, dialogue generation, and semantic parsing.
  • Datasets: Additional datasets support sentiment analysis, natural language inference, semantic role labeling, relation extraction, pronoun resolution, image captioning, and video captioning.

III. REINFORCEMENT LEARNING METHODS

The paper presents reinforcement learning as a framework for maximizing sequence-level rewards and organizes methods around policy gradients, actor-critic learning, Q-learning, and later Q-learning improvements.

  • RL trains an agent to maximize the expected reward received from its actions.
  • The surveyed approaches optimize sequence rewards, discounted rewards, advantage functions, or Q functions.
  • Policy Gradient: Policy Gradient methods directly target the expected sequence-level reward objective.
  • Actor-Critic: Actor-Critic methods improve policy-gradient models by using an expansion of the Q function to solve an advantage-function objective.
  • Q-learning: Q-learning methods maximize the Q function and are presented as improving policy-gradient and actor-critic models.
  • Q-learning: The paper then discusses recent methods designed to improve Q-learning performance.

A. Policy Gradient

Policy-gradient training treats seq2seq generation as selecting actions from a policy to maximize sequence-level reward, but single-sample REINFORCE has high variance and delayed feedback. Baselines, self-critic rewards, multiple samples, and pretraining are presented as mitigation strategies.

  • Policy-gradient objective: Seq2seq policy gradients maximize the expected reward of a generated action sequence under policy πθ.The loss is defined as the negative expected full-sequence reward.
  • REINFORCE: REINFORCE estimates the gradient using sampled sequences and a baseline reward that need not depend on θ.The baseline encourages actions above the baseline reward and reduces gradient-estimator variance.
  • Variance reduction: Single-sample REINFORCE suffers from high variance, motivating averaging gradients over N sampled action sequences.The baseline can be set to the mean reward of the sampled sequences.
  • Self-Critic: Self-Critic uses the model’s greedy inference output as the baseline instead of estimating it from current samples.The sampled output and greedy output are compared through their rewards to update the loss.
  • Delayed feedback: REINFORCE observes reward only after sampling the full sequence, so poor early actions cannot receive immediate partial-reward feedback.The paper notes this can produce poor results or slower convergence, especially during random initialization.
  • Training strategy: Pretraining with cross-entropy for a few epochs before gradually switching to REINFORCE is suggested to alleviate early-training difficulties.The transition is intended for the phase when a randomly initialized model selects arbitrary actions.

B. Actor-Critic Model

Actor-Critic models learn a critic to estimate baseline or advantage values for a seq2seq actor. The critic’s estimates are used to update the actor while avoiding the full inner expectation through sampling.

  • Actor-Critic objective: Actor-Critic training learns an estimator for the baseline reward and maximizes the actor’s advantage function.The approach extends the policy-gradient objective with advantage estimation.
  • Advantage estimation: The advantage equals Qπ(st, yt) − Vπ(st), combining the action value with the state value.The displayed definition also expresses it using the immediate reward and expected next-state value.
  • Advantage estimation: Actor-Critic methods approximate the advantage by sampling once rather than computing the expensive inner expectation.The approximation is used during inference at each decoding time step.
  • Actor and critic roles: A neural-network critic estimates Vπ(s; Ψ), while the actor supplies successive policy states and uses the critic’s estimates to update itself.The critic returns value estimates for calculating the actor’s advantage approximation.

1) Training Critic Model:

Critic training is formulated as value-function regression, with the actor generating state-reward pairs. Off-policy replay reduces sample correlation, while critic error creates bias and bias–variance methods trade these effects.

  • On-policy and off-policy training: On-policy Actor-Critic collects state-value pairs from the current actor, but correlated samples can cause poor critic generalization.Off-policy training stores samples in a memory buffer and trains on selected mini-batches.
  • Batch training: Batch Actor-Critic trains the critic from batches of actor-generated state-reward pairs, whereas online Actor-Critic uses one sample and has higher variance.Synchronous and asynchronous advantage methods are described as alternatives for reducing online-training variance.
  • Bias–variance trade-off: Actor-Critic generally has lower variance but becomes biased when the critic is erroneous, unlike policy gradients’ high-variance unbiased estimator.The paper presents combining the two approaches as a possible bias–variance trade-off.
  • Bias–variance trade-off: Generalized Advantage Estimation controls a bias–variance trade-off through λ, with larger λ increasing variance and reducing bias.Smaller λ has the opposite effect.
  • Actor update: The actor uses critic estimates for VΨ(st) and VΨ(st′) to calculate its loss and update its parameters.The critic is trained from collected state-value pairs before actor updates use those estimates.

C. Actor-Critic with Q-Learning

Q-learning replaces advantage maximization with Q-function estimation for seq2seq actions. Q-networks are trained from actor-generated experience and replay, but neural approximation removes convergence guarantees and uses bootstrapped estimates.

  • Q-learning formulation: Q-learning simplifies advantage maximization to estimating a Q function because the value function does not depend on the selected action.The paper identifies Q-learning as a common reinforcement-learning algorithm.
  • Q-function training: Q-learning trains the Q-function estimator with mean squared error, using network-based estimates rather than ground-truth reward-to-go targets.Value estimation uses ground-truth reward-to-go, whereas Q-learning uses the approximator’s estimate.
  • Scope and limitation: Q-value estimates can support faster training without collecting ground-truth reward at every step, but their distance from true Q-values lacks rigorous analysis.The paper reports that these estimators are empirically useful while acknowledging this unresolved question.
  • Training procedure: The Q-learning pipeline collects experiences et = (st, yt, st′, rt) from the actor and trains the Q-network from replay-buffer mini-batches.The sequence-to-sequence actor is then updated using estimated Q values.
  • Scope and limitation: Neural-network Q-function approximation removes the original Q-learning convergence guarantee, and correlated samples can destabilize training.The paper notes that small Q-network updates may substantially change the data distribution.
  • DQN: DQN is presented as a Q-learning algorithm whose training uses experience data and a Q-network objective.Algorithm 4 provides its pseudocode.

D. Advanced Q-Learning

Double DQN addresses DQN’s Q-value overestimation by separating next-action selection from Q-value estimation with current and target networks, while retaining a remaining estimation limitation.

  • Double Q-Learning: DQN can overestimate Q-values because the same network estimates QΨ(si, yi) and supplies the regression targets.Double DQN separates these roles across current and target networks.
  • Double Q-Learning: Double DQN uses a target network to select the next best action and the current network to estimate its Q-value.The target network is periodically updated or Polyak-averaged from the current network.
  • Double Q-Learning: The Double DQN procedure trains an actor with sampled sequences while critics learn from experience-buffer transitions and estimated Q-values.Its pseudocode initializes actor, current critic, and target critic models before alternating these updates.
  • Double Q-Learning: Double DQN resolves DQN’s overestimation problem and provides more realistic estimations, but lacks a relation between true Q-values and network estimates.The method uses EOS as the End-Of-Sequence action.

2) Dueling Networks:

Dueling networks decompose action values into state value and action advantage, improving target normalization without reducing the number of available actions.

  • 2) Dueling Networks:: Dueling networks address large action spaces by estimating state value and action advantage rather than directly estimating every Q-value.This is especially relevant when only a small portion of many discrete actions is suitable.
  • 2) Dueling Networks:: The dueling representation is unidentifiable because a constant can shift between VΨ(st) and AΨ(st, yt).The proposed normalization forces the advantage estimator to have zero at the selected action.
  • 2) Dueling Networks:: Replacing the max operator with an average is proposed as an alternative intended to make the dueling model more stable.The accompanying formulation combines value and advantage estimates through averaging.
  • 2) Dueling Networks:: Dueling networks do not decrease the number of actions but provide better normalization over the target distribution.Like DQN and DDQN, they still lack a relation between true Q-values and network estimates.

IV. COMBINING RL WITH SEQ2SEQ MODELS

The survey reviews reinforcement-learning approaches for combining seq2seq training with task rewards, emphasizing train/test mismatch and the difficulty of applying Q-learning to large action spaces. It describes REINFORCE-based methods, including scheduled and mixed transitions from cross-entropy training.

  • IV. COMBINING RL WITH SEQ2SEQ MODELS: Most RL-enhanced seq2seq models aim to solve the mismatch between training objectives and test-time evaluation by adding a reward function.The survey notes that REINFORCE has been used more often than Actor-Critic, DQN, or DDQN for these tasks.
  • IV. COMBINING RL WITH SEQ2SEQ MODELS: Q-learning remains difficult for seq2seq models because text generation requires estimating actions across a vocabulary-sized action space.Researchers therefore often use REINFORCE despite its own limitations.
  • A. Policy Gradient and REINFORCE Algorithm: Exposure bias arises because teacher-forced training uses ground-truth previous outputs, whereas inference requires the decoder to use its own predictions.Imitation-learning approaches let the model generate actions during training but may require an oracle that is infeasible for text summarization.
  • A. Policy Gradient and REINFORCE Algorithm: REINFORCE avoids the cross-entropy objective by using sampled sequences as training targets, typically after cross-entropy pre-training.The sequence-level reward is observed at the end and back-propagated across decoding steps.
  • A. Policy Gradient and REINFORCE Algorithm: MIXER gradually switches from cross-entropy to REINFORCE, using cross-entropy for the first T−δ steps and REINFORCE for the remaining δ steps.It was used for text summarization, image captioning, and machine translation.
  • A. Policy Gradient and REINFORCE Algorithm: A mixed loss controls the transition with η, beginning at η = 0 for cross-entropy and reaching η = 1 for REINFORCE.This training strategy has been used across summarization, captioning, speech recognition, dialogue, and question-answering tasks.

B. Actor-Critic Models

Actor-Critic methods address policy-gradient difficulties by estimating value during sequence generation, while RL-based seq2seq systems still face efficiency, action-space, and reward-design challenges.

  • Actor-Critic Models: Policy-gradient models must sample complete action sequences before observing the final reward, causing accumulated generation error and slow convergence on long sequences.Actor-Critic models are introduced to avoid relying only on delayed sequence-level feedback.
  • Actor-Critic Models: Delayed actors and target-style value estimation are used to estimate the best action, paralleling the idea of Double Q-Learning.The delayed Actor provides the true Q estimation used by the method described.
  • Actor-Critic Models: Some Actor-Critic models use semantic matching and context coverage to estimate value, but require a fully trained seq2seq model and apply the value network only during beam-search inference.The value network is not used during seq2seq training in this approach.
  • Current RL-Based Model Issues: RL methods for seq2seq models face sample inefficiency and high variance, motivating intermediate approaches such as RAML, SPG, and entropy-regularized policy optimization.RAML adds reward-aware perturbations to MLE, whereas SPG uses reward distributions for policy-gradient sampling.
  • Current RL-Based Model Issues: Seq2seq RL applications have massive action spaces because actions commonly correspond to vocabulary items in translation, summarization, and image captioning.Large output spaces make these applications harder to combine with RL techniques.
  • Current RL-Based Model Issues: BLEU and ROUGE provide standard feedback but can misrepresent abstractive summaries, motivating interest in learning reward functions through inverse reinforcement learning.The paper states that no such IRL work had been done for this area at the time described.

V. RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS

RLSeq2Seq is an open-source library implementing diverse RL and seq2seq training options, with experiments focused on abstractive summarization. The reported results compare quality and training speed across methods.

  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: The library implements multiple RL techniques for abstractive text summarization and is released as open-source software.The authors encourage users to share trained models because individual configurations can require days of GPU training.
  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: Available components include temporal and intra-decoder attention, scheduled sampling, differentiable exposure-bias methods, adaptive REINFORCE, self-critic baselines, and Actor-Critic variants.Actor-Critic options include asynchronous Value Network, DQN, DDQN, and Dueling Net training.
  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: Experiments pre-train the model for 15 epochs with cross-entropy and then apply RL training for another 10 epochs.The setup follows the pointer-generator model and treats RL primarily as a fine-tuning technique.
  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: Scheduled sampling and E2E models outperform the pointer-generator on ROUGE, while Self-Critic Policy Gradient improves it only marginally in these experiments.The authors report that this contrasts with the original Self-Critic Policy Gradient paper and attribute the discrepancy partly to different hyperparameters.
  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: Actor-Critic training takes only a few hours before and after RL activation, whereas pointer-generator training requires more than 3 days and Self-Critic Policy Gradient increases batch time from 2–3 to 5–6 seconds.The paper reports that Actor-Critic training also converges faster and has superior ROUGE results in the comparison.
  • RLSEQ2SEQ: AN OPEN-SOURCE LIBRARY FOR TRAINING SEQ2SEQ MODELS WITH RL METHODS: The paper surveys RL methods for seq2seq models and provides a library demonstrating how different RL techniques can train abstractive summarization systems.The broader framework targets training problems including exposure bias and mismatch between training and testing objectives.
Loading 1805.09461v4…