Source-linked AI summary
DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan
TL;DR
Open-domain dialogue systems struggle with content or style inconsistency, limited long-term context, and blandness. DialoGPT extends GPT-2 with dialogue-pair pre-training on Reddit-derived data and achieves state-of-the-art automatic and human evaluation results near human response quality. Its released model and pipeline support future research and adaptation, while generated outputs may retain offensive or biased behavior.
Problem
Open-domain neural response generation systems often suffer from content or style inconsistency, limited long-term contextual information, and blandness.
Method
DialoGPT extends GPT-2’s autoregressive transformer by training on large-scale dialogue pairs and sessions extracted from Reddit discussion chains.
Results
DialoGPT achieves state-of-the-art results in automatic and human evaluation, reaching near-human response quality.
Takeaways & Limitations
The released source code, pre-trained model, and training pipeline facilitate future research and adaptation to new dialogue datasets.
Takeaways & Limitations
Generated outputs may trigger offense, reflect gender and other historical biases, or agree with unethical, biased, or offensive propositions.
Abstract
from arXiv · showhide
We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.
1 Introduction
DialoGPT extends GPT-2 with dialogue-specific pre-training to address inconsistent, bland, and context-limited open-domain responses. It achieves near-human quality in automatic and human evaluations while making its model and training pipeline publicly available.
- DialoGPT is a tunable neural conversational response-generation model trained on Reddit data.
- Conversational response generation is challenging because dialogue has diverse possible responses, competing participant goals, and informal or noisy language.
- Open-domain response systems commonly suffer from content or style inconsistency, limited long-term context, and blandness.
- DialoGPT uses GPT-2-style autoregressive transformers trained on dialogue pairs and sessions extracted from Reddit discussion chains.
- DialoGPT achieves state-of-the-art results in automatic and human evaluation, reaching near-human response quality.
- The released source code, pre-trained model, and training pipeline are intended to facilitate future research and adaptation to new dialogue datasets.
2 Dataset
The dataset converts Reddit reply trees into multi-turn dialogue training instances and applies extensive quality, language, length, offensiveness, and blandness filters. The resulting corpus contains 147,116,725 dialogue instances.
- Reddit comment chains spanning 2005 through 2017 are converted into training instances by extracting each root-to-leaf reply path.
- Filtering removes examples containing URLs, repeated words, insufficient common English words, special markers, excessive length, offensive language, or excluded subreddits.
- Bland responses are filtered by removing instances dominated by frequently seen trigrams; these uninformative responses comprise about 1% of the data.
- After filtering, the dataset comprises 147,116,725 dialogue instances.
3 Method
DialoGPT models multi-turn dialogue as autoregressive language modeling with a GPT-2 transformer, while decoding uses sampling and mutual-information reranking to reduce blandness. A reinforcement-learning alternative was unstable and degenerated into source repetition.
- 3.1 Model Architecture: DialoGPT inherits GPT-2’s transformer architecture and frames a multi-turn dialogue session as one long text for language modeling.
- 3.1 Model Architecture: The dialogue history S and target response T define conditional generation through the probability P(T|S).
- 3.1 Model Architecture: For multi-turn sessions, optimizing one conditional objective corresponds to optimizing each response given all preceding turns.
- 3.2 Mutual Information Maximization: Top-K sampling generates hypotheses, which a backward model reranks using P(Source|Hypothesis) to penalize bland responses.
- 3.2 Mutual Information Maximization: Policy-gradient optimization of the mutual-information reward easily converged to a degenerate solution that repeated the source sentence.
4 Result
DialoGPT is evaluated across automatic metrics, multi-reference Reddit data, generation examples, and human judgments. Larger models, beam search, and MMI re-ranking generally improve reported outcomes, while evaluation and deployment include important caveats.
- Experimental Details: Three model sizes—117M, 345M, and 762M parameters—were trained, with training accelerated through lazy loading, asynchronous data processes, and dynamic batching.Training used up to five epochs for small and medium models and three epochs for the large model.
- DSTC-7 Evaluation: The DSTC-7 evaluation used a 5-reference test set of 2,208 examples and standard metrics including BLEU, METEOR, NIST, Entropy, and Dist-n.The test set was constructed from Reddit sessions containing at least six responses, with one human response held out for human-performance scoring.
- DSTC-7 Evaluation: DialoGPT with 345M parameters and beam search achieved the highest automatic score across most metrics, while beam search substantially improved BLEU and DIST scores.The model was fine-tuned on source-target pairs without using DSTC grounding information.
- DSTC-7 Evaluation: DialoGPT’s automatic scores exceeded held-out human-reference scores, which the authors attribute to the one-to-many nature of conversation rather than greater realism.A generated response can lie near the semantic center of several valid responses, making it closer to tested references than another human response.
- Reddit Multi-reference Evaluation: On the 6K Reddit multi-reference evaluation, larger models consistently outperformed smaller ones, and the best 345M beam-search system scored higher on BLEU than humans.GPT-2 fine-tuning provided larger gains for smaller models, while larger scratch-trained models performed comparably to a GPT-2-fine-tuned model.
5 Related work
The paper situates DialoGPT among open-source toolkits for large-scale pre-trained transformer models and dialogue systems.
- Open-source toolkits support large-scale pre-trained transformers and conversational AI research.Examples include repositories for GPT-2-based conversational transfer learning, large transformer dialogue models, and broader NLP frameworks.
6 Limitations and risks
DialoGPT carries risks from offensive, biased, or otherwise inappropriate generated content, while its release places decoder implementation on users. The human-evaluation procedure may also bias judgments toward extreme ratings.
- DialoGPT is released as a model only, leaving decoder implementation to users.
- The model may generate offensive outputs and reflect gender or other historical biases in its training data.It may also agree with unethical, biased, or offensive propositions, or disagree with ethical statements.
- Inappropriate generated content should not be interpreted as representing the authors’ or Microsoft’s views.
7 Conclusion
DIALOGPT is presented as an open-domain pretrained model trained on a massive real-world Reddit dataset and released as an easy-to-deploy, extensible conversational system. The authors identify toxic-output control and reinforcement learning for response relevance as future investigation areas.
- DIALOGPT is an open-domain pretrained model trained on a massive real-world Reddit dataset.
- The package includes a distributed training pipeline and pretrained models that can be fine-tuned on customized datasets within a few hours.
- DIALOGPT is fully open-sourced and easy to deploy for bootstrapping conversational training with various datasets.
- Future work will investigate toxic-output detection and control, plus reinforcement learning to improve relevance and prevent egregious responses.
A Additional Details of Human Evaluation
Human-evaluation significance testing used 10K bootstrap iterations with α = 0.05. The reported differences between the two DIALOGPT model sizes, and between DIALOGPT 345M and human responses, were not statistically significant.
- 10K bootstrap iterations were used to test differences in means, with p-values computed at α = 0.05.
- The significance-test results are provided in Table 8, whose bold entries indicate differences that are not statistically significant.
- Differences between DIALOGPT 345M and DIALOGPT 762M were not statistically significant.
- Differences between DIALOGPT 345M and human responses were also not statistically significant.