Source-linked AI summary
Deep Reinforcement Learning for Dialogue Generation
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, Dan Jurafsky
TL;DR
Neural dialogue models often optimize one response at a time, leaving long-term conversational success under-modeled. This paper uses policy-gradient reinforcement learning with simulated two-agent dialogues and rewards for informativity, coherence, and ease of answering. The resulting system generates more interactive responses and sustains conversations longer in simulation, while remaining limited by heuristic rewards and restricted conversational history.
Problem
SEQ2SEQ models optimize next-turn likelihood, which may not reflect the goal of producing diverse, informative responses that sustain engaging human conversations.
Method
The model uses policy-gradient reinforcement learning to optimize developer-defined long-term rewards during simulated conversations between two virtual agents.
Results
The proposed model generates more interactive responses and fosters more sustained dialogue than standard MLE-trained SEQ2SEQ models.
Takeaways & Limitations
Integrating SEQ2SEQ generation with reinforcement learning can produce diverse, interactive responses that support more sustained simulated conversations.
Takeaways & Limitations
The approach is constrained by manually defined heuristic rewards that cannot cover all crucial aspects of an ideal conversation, and by limited conversational history.
Abstract
from arXiv · showhide
Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.
1 Introduction
Neural SEQ2SEQ dialogue models can use rich context but are shortsighted: maximum-likelihood training often produces generic, repetitive responses. The paper introduces reinforcement learning with developer-defined long-term rewards and reports more interactive, sustained simulated conversations.
- SEQ2SEQ models maximize next-turn likelihood, but this objective does not clearly match the goal of engaging, diverse, informative human conversation.
- Generic responses such as “I don’t know” arise frequently, while repetitive turns can trap dialogues in infinite loops.
- The proposed framework combines developer-defined rewards with modeling a generated response’s long-term influence on the ongoing dialogue.
- Policy-gradient learning optimizes an encoder-decoder policy through simulated conversations between two virtual agents, targeting interactive, informative, and coherent dialogue.
- The model produces more interactive responses and fosters more sustained dialogue than standard MLE-trained SEQ2SEQ models.
2 Related Work
Prior statistical dialogue research mainly treats response generation as source-to-target transduction or builds task-oriented systems with explicit dialogue-state models. This work combines neural SEQ2SEQ generation with reinforcement learning, extending earlier efforts to merge the paradigms.
- One research line treats dialogue generation as source-to-target transduction, including SMT systems and neural encoder-decoder models trained from large corpora.
- Another line develops task-oriented dialogue systems using MDPs, POMDPs, and statistically learned generation rules for domain-specific tasks.
- Reinforcement learning has also been applied to language understanding tasks involving delayed rewards, text-based games, instructions, and navigation dialogues.
- The paper seeks to integrate SEQ2SEQ and reinforcement learning, building on prior work that combines reinforcement learning with task-oriented or neural dialogue generation.
3 Reinforcement Learning for Open-Domain Dialogue
The framework models open-domain dialogue as policy learning over utterance actions in simulated two-agent conversations. Its reward combines ease of answering, information flow, and semantic coherence using neural representations and SEQ2SEQ probabilities.
- The learning system simulates two agents whose alternating generated sentences are actions selected by an encoder-decoder recurrent neural network policy.
- Policy search maximizes expected future reward, with policy gradients preferred because MLE initialization already produces plausible responses.
- 3.1 Action: An action is an utterance to generate, giving the model an infinite action space because responses may have arbitrary length.
- The state uses the previous two dialogue turns, encoded by an LSTM, while the stochastic policy maps dialogue history to a distribution over utterance actions.
- 3.4 Reward: Ease of answering is approximated through the negative log likelihood of responding with manually listed dull responses, while similar vector-space responses are also discouraged.
- 3.4 Reward: Information flow penalizes semantic similarity between consecutive turns from the same agent using the negative log cosine similarity of their encoded representations.
- 3.4 Reward: Semantic coherence is encouraged through mutual information between a candidate utterance and preceding dialogue turns, using forward and backward SEQ2SEQ probabilities.
- 3.4 Reward: The final action reward is a weighted sum of the component rewards, with λ1 + λ2 + λ3 = 1 and a reward observed after each sentence.
4 Simulation
The dialogue system simulates two virtual agents taking turns, using reinforcement learning to explore responses and optimize long-term expected reward. Training combines supervised initialization, mutual-information rewards, policy gradients, and progressively longer dialogue simulations.
- Two virtual agents take turns generating responses to explore the dialogue state-action space and learn a policy for expected reward.
- The reinforcement-learning policy is initialized from supervised encoder-decoder models trained on OpenSubtitles dialogue pairs.The supervised model uses roughly 80 million source-target pairs, with two previous sentences as input.
- Mutual-information rewards encourage responses that improve informativeness and reduce dull, generic outputs.Candidate responses receive mutual-information scores from forward and backward SEQ2SEQ models, which are back-propagated as rewards.
- Optimization: Policy gradients optimize expected future reward rather than the maximum-likelihood objective used by standard SEQ2SEQ models.The objective sums expected rewards for generated actions, with gradients estimated using the likelihood ratio trick.
- Dialogue Simulation: Curriculum learning begins with two simulated turns and gradually increases simulations to five turns, generating five candidate responses at each step.The number of candidates grows exponentially with the number of simulated turns, motivating the five-turn limit.
5 Experimental Results
The evaluation compares dialogue systems using automatic metrics and human judgments, finding that reinforcement learning improves sustained, interactive conversation while exposing trade-offs in BLEU and residual failure modes.
- Evaluation Setup: The evaluation uses conversation length, diversity, and human judgments to assess dialogue quality.Human studies cover single-turn quality, ease to answer, and multi-turn general quality.
- Automatic Evaluation: 1.28 BLEU is reported for RL models, versus 1.44 for mutual-information models and 1.17 for vanilla SEQ2SEQ models.The authors attribute the lower RL BLEU score to optimization for future reward rather than maximum likelihood.
- Automatic Evaluation: The proposed RL model achieves the best dialogue-length evaluation score among the compared systems.The test set contains 1,000 input messages, with simulated turns limited to fewer than eight.
- Automatic Evaluation: The RL model generates more diverse outputs than the compared response-generation systems.Diversity is measured with unigram and bigram type-token ratios scaled by generated-token count.
- Human Evaluation: RL wins 52% versus 23% losses on ease to answer and wins 72% in multi-turn general quality, but shows no significant single-turn quality boost.For single-turn response quality, RL wins 40% and loses 36% of pairwise judgments.
- Qualitative Analysis and Discussion: The model can still enter cycles longer than one and start less relevant topics because conversational history is limited and rewards trade off relevance against repetition.The authors also note that manually defined heuristic rewards cannot cover every aspect of ideal conversation, while exploration is limited by exponential growth in candidates and simulated turns.
6 Conclusion
The paper introduces a reinforcement learning framework for neural response generation that simulates dialogues between two agents. Despite using simple heuristics, it produces more diverse, interactive responses and sustains conversations longer.
- The framework combines neural SEQ2SEQ generation with reinforcement learning by simulating dialogues between two agents.It preserves compositional semantic modeling while optimizing future reward across conversations.
- The model captures global properties of good conversations by generating utterances that optimize future reward.
- Despite relying on simple operational heuristics, the framework generates more diverse and interactive responses that foster more sustained conversations.