Source-linked AI summary
Recipes for building an open-domain chatbot
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston
TL;DR
Open-domain chatbots require more than larger models and datasets: they must blend engagingness, knowledge, empathy, personality, and consistent conversation. The paper proposes training-data and decoding recipes, finding that these ingredients improve human evaluations, while larger scale alone does not guarantee better results and important failure cases remain.
Problem
Open-domain chatbot quality depends on conversational skills and ingredients beyond model and dataset scaling.
Method
The paper combines large-scale pre-training with skill-focused fine-tuning, Transformer retrieval and generation architectures, and varied decoding strategies.
Results
The 2.7B model makes large gains over the 90M model, while the 9.4B model has no clear human-evaluation win over the 2.7B model despite lower perplexity.
Takeaways & Limitations
Training-data choice and decoding algorithms can substantially affect chatbot quality beyond perplexity and parameter count.
Takeaways & Limitations
The models remain limited by shallow knowledge, simpler language, repeated phrases, and a 128 BPE-token history limit.
Abstract
from arXiv · showhide
Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models.
1 Introduction
The paper develops open-domain chatbot recipes beyond scaling, emphasizing fine-tuning data that teaches desirable conversational skills. Blended Skill Talk targets personality, engagingness, knowledge, and empathy, while also reducing undesirable traits such as toxicity.
- Large improvements can come from fine-tuning data that emphasizes desirable conversational skills.
- Blended Skill Talk trains models to blend personality, engagingness, knowledge, and empathy using conversational data and initial personas and topics.
- Small models using Blended Skill Talk can match or outperform larger models that do not use it.
- Blended Skill Talk fine-tuning can minimize undesirable traits learned from large corpora, such as toxicity.
2. Generation Strategies
Generation quality depends strongly on decoding choices, not just model perplexity. The paper finds that response length and minimum beam-length constraints shape human judgments and the dull-to-spicy trade-off, while evaluation setup and model limitations remain important caveats.
- The choice of decoding algorithm is critical: models with the same perplexity can produce vastly different results.
- Response length affects perceived quality: short replies seem dull or uninterested, while long replies can appear to waffle and not listen.
- Minimum beam-length constraints control the dull-versus-spicy spectrum and can make beam search perform strongly.
- In engagingness, the best model beats Meena 75% to 25%, and in humanness 65% to 35%, with both differences statistically significant.
- The models still show limited in-depth knowledge, simpler language, and repeated phrases, with proposed fixes remaining inconclusive.
2 Model architectures
The paper compares retrieval, generative, and retrieve-and-refine Transformer architectures. Retrieval scores candidate responses, generation produces responses directly, and retrieve-and-refine conditions generation on retrieved dialogue or knowledge.
- The paper considers retrieval, generative, and retrieve-and-refine models, all based on Transformers.
- Retrieval systems score candidate responses given dialogue history and output the highest-scoring response.
- The poly-encoder retriever uses multiple context representations attended to by candidate responses, with 256M and 622M parameter variants using 64 codes.
- Generative models use a standard Seq2Seq Transformer to generate responses rather than selecting from a fixed response set.
- The paper uses 90M, 2.7B, and 9.4B parameter generative models.
- Retrieve-and-refine models append a retrieved dialogue response to the generator input, while knowledge retrieval conditions generation on a selected knowledge sentence.
3 Training Objectives
The paper trains retrieval and generative chatbots with likelihood-based objectives, adding mechanisms to address retrieval-conditioning failures and undesirable generation patterns.
- Cross-entropy training scores the correct retrieval response against sampled negative candidates.Other responses in the batch are reused as negatives, enabling batches of 512 elements.
- Generative models use standard Maximum Likelihood Estimation to predict each token of a gold next utterance from its context.The objective minimizes negative log-likelihood over gold input contexts and responses.
- Retrieve-and-refine training replaces retrieved responses with gold responses α% of the time so the generator learns to use retrieval input.The hyperparameter α controls the transition between retrieval-conditioned and generator-only training.
- Unlikelihood training mixes likelihood with penalties on negative candidate tokens to reduce repetitions and overrepresented vocabulary.Candidates are selected from generated n-grams whose counts exceed the corresponding human-distribution counts.
4 Decoding
The paper compares deterministic search and sampling, then controls response length and repetition because decoding choices strongly affect conversational quality.
- Greedy search selects the highest-probability token at each step, whereas beam search retains a fixed-size set of partial hypotheses.Beam sizes are varied experimentally.
- Sampling restricts generation to a vocabulary subset and is evaluated with top-k sampling and sample-and-rank.Sample-and-rank draws S samples and returns the one with highest probability.
- Beam decoding tends to produce responses shorter than human utterances, while longer high-quality responses can be more engaging.The paper also notes that excessive length can make a bot waffle and fail to listen.
- Minimum-length decoding blocks the end token until a specified sequence length is reached.This is one of two methods used to control response length.
- Predictive-length decoding uses a 4-class classifier to estimate the next-turn length and set a corresponding minimum constraint.Unlike a fixed minimum, it permits naturally variable-length turns.
- The paper applies n-gram blocking with n = 3 to prevent repeated generated phrases and repetitions of input utterances.Both repetitions within the output and copies of previous speaker utterances are blocked.
5 Training Details
Training uses Fairseq for pre-training, large generative models with model parallelism, and ParlAI with sharded execution for dialogue fine-tuning.
- Pre-training uses Fairseq for both ranking and generative models.The 622M ranking model uses a masked-language-model objective, while the 2.7B and 9.4B generative models use Adam.
- The 2.7B and 9.4B generative models use Megatron-LM-style vertical slicing of feed-forward and multihead-attention layers across GPUs.This parallelism minimizes communication across GPUs.
- Fine-tuning uses ParlAI and GPipe-style parallelism, sharding full layers and splitting minibatches into micro-batches.The setup is intended to maximize throughput.
6 Training Data
The paper combines large, broad Reddit pre-training data with smaller focused dialogue tasks, culminating in Blended Skill Talk for personality, empathy, and knowledge.
- The Reddit corpus contains 1.5B training examples spanning diverse topics and is filtered with heuristic quality rules.Filtering removes bot, non-English, deleted, malformed, overly long, very short, URL-containing, and excessively deep-thread comments.
- The final Reddit dataset contains 1.50B comments, 56.8B label BPE tokens, and 88.8B context tokens.Examples are truncated to 128 BPE tokens for both context and response.
- Reddit provides broad content but includes group discussions and substantial noise, motivating smaller cleaner task-focused datasets.
- ConvAI2 trains engaging, friendly conversation conditioned on persona and dialogue history.Its persona conditioning was previously associated with more engaging and consistent bot dialogue.
- Empathetic Dialogues trains models to play a listener who responds empathetically to emotionally grounded situations.
- Wizard of Wikipedia trains topic-focused dialogue that combines engagement with expert knowledge grounded in Wikipedia retrieval.Its dataset contains 194k utterances across 1250 topics.
- Blended Skill Talk combines ConvAI2 personality, Empathetic Dialogues empathy, and Wizard of Wikipedia knowledge in 76k utterances.Training additionally conditions dialogues on a two-sentence persona and, in one third of conversations, a WoW topic.
7 Safety Characteristics
The paper identifies toxic or biased language as a safety risk inherited from human-human pre-training data. It notes that existing mitigation approaches help but do not fully eliminate these issues.
- Toxicity and bias: Human-human training data can transmit toxic or biased language into chatbot responses.The fine-tuning tasks are generally safer because crowdworkers were instructed not to use such language, but the pre-training Reddit data remains riskier.
- Toxicity and bias: Test-time toxic-language classifiers can filter responses, but they are not infallible.The paper evaluates how often these classifiers flag generated responses.
- Gender bias: Gender-bias mitigation through conditional generation showed preliminary success but was not included in the described system.The approach controls the amount of gendered language to make responses more neutral.
8 Evaluation Methods
The paper evaluates dialogue systems primarily through pairwise human comparisons of complete multi-turn conversations. ACUTE-Eval measures engagingness and humanness, while self-chat reduces evaluation costs during development.
- ACUTE-Eval: ACUTE-Eval asks evaluators to compare complete multi-turn dialogues pairwise across systems.Pairwise comparison reduces annotator-score bias and sequential effects relative to Likert evaluations.
- ACUTE-Eval: Previously collected conversations can be compared directly with new systems, reducing resources needed for evaluation and improving consistency across studies.The setup enables comparisons with logs from other systems without recollecting all evaluation data.
- ACUTE-Eval: The evaluation asks which speaker is preferred for a long conversation and which sounds more human.The question phrasings were optimized in prior work to maximize agreement, and engagingness and humanness are treated as distinct measures.
- Self-chat: Self-chat uses models on both sides of a conversation and is used for some modeling and hyperparameter choices when full human evaluation is too costly.The final stage uses full human-bot chat evaluation; self-chats provide personas, topics, and previous utterances in the BST setting.
9 Related Work
Related work describes rapid progress from large pre-trained Transformer models in open-domain dialogue. The paper emphasizes evaluation practices intended to make comparisons more direct and reproducible.
- Open-domain dialogue: Large pre-trained Transformers had produced strong open-domain dialogue results, including leading perplexity and F1 scores.The cited progression includes the ConvAI2 competition and BooksCorpus-based pre-training.
- Retrieval evaluation: Hits@1/K evaluates whether the gold response is ranked first among K candidates for fine-tuned poly-encoder models on BST validation sets.The table reports recall@1 with K −1 random alternative candidates.
- Evaluation comparisons: The paper uses ACUTE-Eval with unbiased crowdworkers to ask humanness directly rather than relying on a proxy.It also reports engagingness as a main metric and uses reproducible experiments to improve comparisons.
10 Results & Analysis
Results show that training data, model scale, and decoding choices materially affect chatbot quality, while automatic metrics do not always predict human judgments. The best BST generative models outperform prior approaches in human evaluations but retain important conversational limitations.
- Model and data effects: Retrieval initially outperforms pure generation, but improved decoding makes generative and RetNRef models outperform retrieval.Without the minimum beam-length constraint, RetNRef achieves a statistically significant improvement over generation (p < 0.001).
- Decoding: A minimum beam length of 20 BPE tokens significantly improves both decoding methods over standard beam search without length control.The authors use a minimum beam length of 20 in the remaining experiments.
- Model scale: Larger models improve performance, but the 9.4B BST model is less engaging than the 2.7B model despite lower perplexity.In direct ACUTE-Eval comparisons, the smaller model wins 56% of engagingness judgments.
- Model and data effects: Fine-tuning on BST improves models’ focus on personality, knowledge, and empathy, while also reducing unsafe responses relative to Reddit-only training.ConvAI2 contains fewer unsafe human utterances than pushshift.io Reddit, which the authors link to the safety difference.
- Human evaluation: BST Generative 2.7B wins 75% of pairwise engagingness judgments and 65% of humanness judgments against Meena.These are author-reported human-evaluation outcomes for the best models.
- Example conversations: Successful examples show the BST 2.7B model sustaining detailed conversations across everyday topics and attempting nuanced comparisons and creative tasks.The examples include cooking, music, movies, pets, a Bach–Justin Bieber comparison, and song writing.
- Limitations: Short human evaluations can miss repetitive or noncommittal behavior, including overused phrases and the “I don’t know” problem.Unlikelihood training reduced phrase overexpression but had a very small or negative effect on engagingness ACUTE-Evals.
11 Released code and models
The authors release their 90M, 2.7B, and 9.4B parameter generative models, along with code, datasets, safety-filtered interaction tools, and evaluation software.
- Fine-tuning code and the datasets are available through ParlAI.
- The release includes a script for interacting with the bot that has safety filtering built in.
- Code for evaluating models with ACUTE-Eval is also provided.
12 Discussion
The discussion reports improved human-evaluated engagingness and humanness but emphasizes that open-domain dialogue remains unsolved, with persistent quality, memory, safety, and evaluation challenges.
- Improved engagingness and humanness in human evaluations do not amount to a solution to open-domain dialogue.
- The models sometimes contradict or repeat themselves, repeat phrases across conversations, and hallucinate knowledge.
- Short 14-turn evaluations likely understate these problems, while Transformer models have limited dialogue history for longer conversations.
- Deployment still faces well-behavedness challenges because bots have less understanding of what they say than humans, despite expectations of greater integrity.
- Model release is presented as a way for the community to understand and fix these issues.
- Training data and decoding algorithms both matter beyond perplexity, and the 9.4B model has no clear human-evaluation win over the 2.7B model despite lower perplexity.