Source-linked AI summary
Adversarial Learning for Neural Dialogue Generation
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky
TL;DR
Neural dialogue systems trained by maximum likelihood often produce dull, generic, repetitive, and short-sighted responses, motivating better objectives and evaluation. The paper uses adversarial reinforcement learning, with discriminator feedback rewarding human-like responses, and reports improved quality across multiple metrics, while identifying evaluator dependence and uncertain transfer beyond dialogue.
Problem
Maximum-likelihood training can produce dull, generic, repetitive, and short-sighted dialogue, while manually defined rewards may fail to cover crucial aspects of conversation quality.
Method
The paper jointly trains a sequence generator and discriminator in a reinforcement-learning framework, using discriminator outputs as rewards, and applies adversarial evaluation during testing.
Results
The adversarially trained system produces more interactive, interesting, and non-repetitive responses than standard MLE-trained SEQ2SEQ models, with clear improvements across multiple metrics.
Takeaways & Limitations
Adversarial training offers a human-dialogue-based objective for response generation, while adversarial evaluation can assess whether generated text is distinguishable from human text.
Takeaways & Limitations
Adversarial evaluation is model-dependent, and the approach did not show a clear performance boost for machine translation.
Abstract
from arXiv · showhide
In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator---analagous to the human evaluator in the Turing test--- to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, pushing the system to generate dialogues that mostly resemble human dialogues. In addition to adversarial training we describe a model for adversarial {\em evaluation} that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines.
1 Introduction
Open-domain dialogue systems trained with maximum likelihood often produce dull, generic, repetitive, and short-sighted responses. The paper proposes adversarial reinforcement learning and adversarial evaluation to make responses more human-like and assess their quality.
- Maximum-likelihood dialogue models often generate dull, generic, repetitive, and short-sighted responses.
- Manually defined rewards may omit crucial aspects of dialogue quality and produce suboptimal utterances.
- Adversarial training jointly optimizes a sequence generator and discriminator modeled on the Turing-test evaluator.
- The discriminator’s judgments become reinforcement-learning rewards that encourage generated responses to resemble human dialogues.
- Adversarial evaluation applies the Turing-test idea during testing by measuring whether an evaluator can distinguish machine-generated from human-generated text.
- Experiments report more interactive, interesting, and non-repetitive responses than standard maximum-likelihood SEQ2SEQ models.
2 Related Work
Related work frames dialogue generation as source-to-target transduction, increasingly combines neural sequence models with reinforcement learning, and explores more flexible evaluation. The paper also extends adversarial-generation ideas from vision while addressing the difficulty of discrete text.
- Earlier dialogue systems used phrase-based machine translation or neural sequence-to-sequence models to map dialogue history to the next utterance.
- Prior reinforcement-learning dialogue work rewarded informativity, coherence, and ease of answering, while this work uses more general rewards.
- Reinforcement-learning dialogue models have been trained with simulated virtual agents and human users.
- Dialogue evaluation has expanded beyond BLEU and METEOR toward human-rating prediction and next-utterance classification.
- Generative adversarial networks train generators to fool discriminators and have achieved substantial success in image generation.
- Applying adversarial training to text is difficult because discrete outputs prevent straightforward backpropagation of discriminator errors.
- Policy-gradient and actor-critic methods have also formalized sequence generation as reinforcement-learning action selection.
3 Adversarial Training for Dialogue Generation
The proposed model jointly trains a SEQ2SEQ generator and discriminator, using discriminator scores as reinforcement-learning rewards to make generated responses resemble human dialogue. It addresses sequence-level reward limitations with intermediate-step rewards and teacher forcing.
- Adversarial REINFORCE: The model jointly trains a generator policy and a binary discriminator that distinguishes human-generated from machine-generated dialogue.The generator uses an encoder-decoder architecture, while the discriminator encodes dialogue context and response before predicting the source class.
- Adversarial REINFORCE: Discriminator scores for human-generated responses serve as rewards, and REINFORCE updates the generator to maximize expected reward.The generated response is sampled from the policy, evaluated by the discriminator, and used in a likelihood-ratio gradient estimate.
- Reward for Every Generation Step (REGS): Vanilla REINFORCE assigns one sampled sequence-level reward to every token, motivating reward estimation for intermediate or partially decoded sequences.The paper identifies this shared reward as inadequate and proposes Monte Carlo search or a discriminator trained on partial sequences.
- Reward for Every Generation Step (REGS): Monte Carlo search averages discriminator scores from five completed continuations sharing a prefix, but repeating sampling for every prefix is time-consuming.The five-sample setting is specified in the proposed intermediate-reward strategy.
- Reward for Every Generation Step (REGS): REGS trains a discriminator on partial sequences so rewards and baselines can differ across tokens within the same response.To reduce overfitting from shared prefixes, the method randomly samples one subsequence from each collection of subsequences.
- Teacher Forcing: Teacher forcing supplements adversarial updates with human responses because adversarial-only training can become unstable and cause perplexity to skyrocket.Human-generated responses provide the generator more direct access to gold-standard targets during model updates.
4 Adversarial Evaluation
The paper treats adversarial evaluation as a Turing-test-like metric for dialogue generation, but emphasizes that evaluator reliability must be checked because discriminator weakness can inflate apparent success.
- Adversarial evaluation trains an evaluator to distinguish human-generated from machine-generated dialogues and tests it on held-out data.If the two sources are indistinguishable, evaluator accuracy should approach 50 percent.
- Adversarial Success (AdverSuc) is 1 minus evaluator accuracy, so higher values indicate more successful evaluator fooling.
- Evaluator weakness can produce high AdverSuc independently of response quality, because random or constant predictions yield AdverSuc of 0.5.
- The paper proposes manually designed scenarios with known gold-standard outcomes to assess evaluator reliability alongside AdverSuc.These scenarios compare an evaluator’s behavior with expected results across human, machine, random, and shifted-response dialogues.
- The evaluator reliability error metric excludes machine-generated responses and therefore cannot assess evaluator behavior on generated outputs.
- Sampling can raise adversarial success to around 40 percent despite responses being incoherent, irrelevant, or ungrammatical.By contrast, greedy or beam decoding produces adversarial success below 10 percent, corresponding to 90 percent evaluator accuracy.
5 Experimental Results
Experiments evaluate adversarial dialogue models using evaluator reliability, adversarial success, machine-vs-random accuracy, and pairwise human judgments. The proposed adversarial models improve human-rated quality and outperform the baselines on adversarial evaluation, while sampling can produce misleadingly high adversarial success.
- Evaluation setup: The experiments measure evaluator reliability with ERE, adversarial success with AdverSuc, machine-vs-random accuracy, and pairwise human judgments.Table 2 reports ERE, Table 3 reports AdverSuc and machine-vs-random accuracy, and Table 4 reports human judgments.
- Adversarial Evaluation: The hierarchical neural evaluator is more reliable than the concatenation-based neural evaluator, while combining neural and lexicalized features yields the most reliable evaluator.The authors nevertheless use the hierarchical neural setting for subsequent results because it is end-to-end.
- Adversarial Evaluation: Sampling achieves significantly higher AdverSuc than the other models, but its significantly lower machine-vs-random accuracy makes that result misleading.Sampled responses are difficult for evaluators to distinguish from both human and randomly sampled responses.
- Adversarial Evaluation: The two proposed adversarial algorithms outperform the baselines on adversarial evaluation, with REGS performing slightly better than vanilla REINFORCE.Among the baselines, MMI+p(t|s) is better than MLE-greedy, which is better than MLE+BS.
- Human Evaluation: Human judgments show significant quality improvements from the proposed adversarial model in both single-turn and multi-turn dialogue generation.The compared reinforcement-learning system with manually designed rewards improves only multi-turn quality, whereas this model improves both settings.
6 Conclusion and Future Work
The paper proposes adversarial reinforcement learning for response generation and reports clear improvements across multiple metrics. Preliminary machine-translation experiments did not show a clear performance boost, motivating further study of when the approach helps.
- Conclusion: The paper trains a generator with discriminator feedback in a reinforcement-learning framework to produce responses indistinguishable from human-generated dialogues.The discriminator supplies the training signal for generated response sequences.
- Conclusion: Adversarial training produces clear performance improvements on multiple metrics for dialogue response generation.The conclusion summarizes the method's gains across the reported evaluation measures.
- Future Work: Preliminary application of the same training paradigm to machine translation did not show a clear performance boost.The authors conjecture that adversarial training may be more beneficial when generated and reference sequence distributions differ substantially or target entropy is high.