Source-linked AI summary
Towards a Human-like Open-Domain Chatbot
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le
TL;DR
Open-domain chatbots still struggle with nonsensical and vague responses, raising whether scaling an end-to-end model can support high-quality multi-turn conversation. Meena combines large-scale end-to-end training with SSA human evaluation, finding strong perplexity–SSA alignment and 79% SSA for its full version. The evaluation remains bounded by restricted static contexts and by SSA's limited coverage of human-like abilities.
Problem
Open-domain chatbots often produce responses that do not make sense or are vague, leaving open whether scaling end-to-end models can achieve high-quality multi-turn conversation.
Method
Meena is a 2.6B-parameter end-to-end generative chatbot trained on 40B words from filtered public-domain social-media conversations and evaluated with human-rated SSA in static and interactive settings.
Results
79% SSA is achieved by the full Meena system, while SSA strongly correlates with perplexity in static and interactive evaluation.
Takeaways & Limitations
Perplexity on public-domain social-media conversations may serve as an automatic proxy for human judgment of sensibleness and specificity.
Takeaways & Limitations
The static evaluation uses restricted, predominantly one-to-three-turn contexts and does not cover abilities such as deeper question answering, basic math, or broader human-like attributes.
Abstract
from arXiv · showhide
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.
1 Introduction
Meena addresses weaknesses in open-domain chatbots with a large end-to-end model and a human evaluation metric focused on sensible, specific responses. Its results show strong alignment between perplexity and SSA, while evaluation remains limited by the restricted static dataset.
- Open-domain chatbots often produce nonsensical or vague replies, motivating more generally useful conversational systems.
- Meena is a 2.6B-parameter generative chatbot trained end-to-end on 40B words from mined and filtered public-domain social-media conversations.It uses a seq2seq Evolved Transformer and achieves test perplexity of 10.2 with an 8K BPE-subword vocabulary.
- Sensibleness and Specificity Average (SSA) evaluates whether chatbot responses make sense in context and are specific, using human judgments.The metric combines sensibleness, a basic requirement, with specificity to avoid vague or boring responses.
- SSA strongly correlates with Meena's perplexity in both static and interactive human evaluations.The static evaluation used 1,477 multi-turn conversations, while interactive evaluation allowed humans to chat freely.
- 79% SSA is achieved by the full Meena system, compared with 72% for the best end-to-end model and 86% for an average human.The full version adds a filtering mechanism and tuned decoding; the reported human comparison remains below the human score.
- The static evaluation dataset is too restricted to capture all aspects of human conversation, limiting the scope of the conclusions.The paper nevertheless reports that higher SSA and its correlation with perplexity suggest human-like sensibleness and specificity may be within reach with better perplexity.
2 Evaluating chatbots
The paper evaluates chatbot human-likeness with SSA, static and interactive human judgments, and automatic perplexity, while using the MTB as a fixed benchmark. SSA combines sensibleness and specificity to penalize generic responses, and its scores correlate with direct human-likeness judgments and perplexity.
- Evaluation setups: The paper compares static and interactive human evaluations with automatic perplexity, which measures how accurately a model predicts test-set response tokens.Perplexity is lower-is-better and can be directly optimized with standard cross-entropy loss.
- SSA metric: The evaluation sequence asks first whether a response makes sense in context and then whether it is specific to the conversation.Sensibleness covers common sense, logical coherence, and consistency; specificity distinguishes context-related replies from generic ones.
- SSA metric: SSA averages the percentages of responses judged sensible and specific, penalizing chatbots that consistently produce generic replies.GenericBot scores 35% SSA and DialoGPT scores 51%, providing stronger separation than sensibleness alone.
- Validation: SSA aligns strongly with direct human-likeness judgments, with the Figure 2 regression reporting R2 = 0.96.Both evaluations use static mode, but direct human-likeness judgments ask whether a response resembles something a human might say in context.
- Evaluation setups: The Mini-Turing Benchmark contains 1,477 fixed conversational contexts spanning 315 single-turn, 500 two-turn, and 662 three-turn contexts.Static evaluation sends resulting context-response pairs to crowd workers for sensibleness and specificity judgments.
- Validation: Perplexity correlates with human judgments of sensibleness and specificity, supporting its use for rapid development and end-to-end optimization.The paper reports this relationship in both static and interactive evaluation.
3 Meena chatbot
Meena explores whether scaling a simple end-to-end dialog model can produce high-quality open-domain conversation. It combines large-scale training data, an Evolved Transformer, and sample-and-rank decoding to generate more specific responses.
- 3 Meena chatbot: An open research question is whether increasing an end-to-end model’s data and parameter count can achieve high-quality multi-turn conversation without additional components.Meena is presented as the largest end-to-end model in the field at that point.
- 3.1 Training Data: Meena’s training data are mined and filtered from public-domain social media message trees, producing context-response pairs from paths with up to 7 preceding turns.Removing a message also removes all subtrees rooted under it.
- 3.1 Training Data: 867M context-response pairs remain after filtering, and the final dataset contains 341GB of text comprising 40B words.The text is tokenized with an 8K BPE-subword vocabulary.
- 3.2 Model: The best Meena model is a 2.6B-parameter Evolved Transformer seq2seq model, whose perplexity is 10.2 versus 10.7 for a vanilla Transformer at the same training steps.The Evolved Transformer uses 1 encoder block and 13 decoder blocks.
- 3.4 Decoding: Sample-and-rank samples N candidate responses at temperature T and selects the candidate with the highest probability, producing diverse and content-rich responses compared with beam search.The reported evaluations use N = 20 and T = 0.88; lower test-set perplexity is associated with improved sensibleness and specificity under this fixed decoding strategy.
- 3.4 Decoding: Temperature controls the next-token probability distribution: larger values favor rarer tokens, while smaller values favor common but safer and less specific words.T = 1 leaves the distribution unmodified.
4 Results
Meena’s evaluation shows that lower perplexity is strongly associated with higher sensibleness and specificity in both static and interactive settings. The strongest models outperform the tested chatbots, although human-level SSA remains higher.
- SSA-perplexity correlation: R2 = 0.94 for static SSA vs perplexity, with similarly strong correlations for static sensibleness and specificity.Static sensibleness had R2 = 0.93, while static specificity had R2 = 0.94; the trend was close to linear.
- SSA-perplexity correlation: Interactive evaluation showed similarly strong perplexity correlations and similar sensibleness and specificity values to static evaluation.This suggests the static correlation was not caused by dataset bias.
- Evaluation consistency: The lowest-perplexity model’s SSA varied by 2% in static evaluation and 1% in interactive evaluation across seven repeated evaluations.These standard deviations indicate consistency for the evaluation metrics in this setting.
- Human and chatbot comparisons: Mitsuku scored 56% interactive SSA, while DialoGPT scored 48% ± 3% and Cleverbot scored 56%.DialoGPT’s interactive scores were 57% ± 2% sensibleness and 39% ± 4% specificity; Cleverbot’s static SSA was 44%.
- Qualitative comparison: Meena (base) produced sensible and specific responses across the sampled contexts, including longer three-turn contexts where Cleverbot’s responses were not sensible.DialoGPT was more sensible and specific than Cleverbot in longer contexts but less so than Meena (base).
5 Further Advancing SSA
Further improvements raised interactive SSA by refining decoding and removing cross-turn repetitions. The full system reached 79% ± 1% interactive SSA, while its evaluation included serving-time filtering for sensitive or toxic candidates.
- Further Advancing SSA: 72% ± 1% interactive SSA for Meena (base) increased to 79% ± 1% for Meena (full).The improvement combined tuned decoding with a rule detecting cross-turn repetitions.
- Advancing Decoding: Top-k sampling with k = 40 and T = 1.0 increased SSA from 72% to 74% in both interactive and static evaluation.The setting was chosen to mitigate negative effects from the distribution tail.
- Advancing Decoding: N = 20 sample-and-rank improved SSA by approximately 10% over N = 1, whereas N = 400 worsened sensibleness and produced diminishing specificity returns.The improvement motivated future exploration of alternate ranking functions and parameter tuning.
- Reducing Repetition: About one third of Meena (base) interactive conversations contained cross-turn repetitions toward the end.Cross-turn repetition means one turn somewhat repeats an earlier turn.
- Reducing Repetition: A rule removing candidates with long common subsequences raised interactive SSA from 74% ± 1% to 79% ± 1%.The rule was reported to address most cross-turn repetition.
- Filtering: The full Meena evaluation used an additional serving-time classifier to filter potentially sensitive or toxic response candidates.This classifier was part of the filtering mechanism used for the full version.
6 Related Work
Related work frames Meena’s contribution around practical human evaluation and the use of perplexity as an accessible automatic metric. The paper focuses on sensibleness and specificity while acknowledging broader human-likeness dimensions.
- Automatic metrics: BLEU, ROUGE, and related metrics have been found unsuitable for dialogue or language-generation evaluation.This motivates seeking alternatives to common translation and summarization metrics.
- Automatic metrics: Prior learnable metrics require human labels or separate training, whereas perplexity is readily available for neural seq2seq models.The paper reports a strong correlation between perplexity and human evaluation.
- Automatic metrics: Other proposed metrics combine human evaluation with automatic metrics or model likelihood but require human judgments and separately trained estimators or discriminators.The paper characterizes these approaches as too complex for practical use.
- Human evaluation dimensions: Prior work has emphasized diversity or evaluated attributes including repetition, specificity, relatedness, question-asking, engagingness, interestingness, and human likeness.Meena instead concentrates on sensibleness and specificity.
- Human evaluation dimensions: The paper states that its metric does not cover aspects such as empathy.This marks a scope boundary for the human-likeness attributes evaluated.
- Evaluation settings: Unlike recent DialoGPT evaluation limited to single-turn dialogs, Meena evaluates up to 3 turns statically and 14 turns interactively.The paper uses both fixed-context and free-chat evaluation setups.
7 Discussion
The results suggest perplexity may proxy sensibleness and specificity, but evaluation coverage is limited and broader human-like attributes remain unmeasured.
- Perplexity on public-domain social-media conversations may serve as an automatic proxy for sensibleness and specificity.
- Static evaluation is limited because its dataset uses one- to three-turn contexts and is biased by its source conversations.
- Interactive evaluation addresses some static-evaluation bias and scope limitations but prevents granular response comparisons and may overemphasize conversational openings and endings.
- Future evaluations may need attributes beyond sensibleness and specificity, including humor, empathy, deep reasoning, question answering, and knowledge discussion.
A Additional Sample Conversations
The appendix presents randomly sampled interactive conversations collected with Meena, Mitsuku, and XiaoIce after removing sensitive material and redacting personally identifiable information.
- About 100 conversations with Mitsuku, XiaoIce, and Meena full were collected using standard interactive-evaluation instructions.
- The presented samples are random rather than cherry-picked, with potentially sensitive conversations removed from the original sets.
- Potential personally identifiable information was redacted and marked with “REDACTED.”
A.1 Meena
The Meena samples show responses that sustain ordinary social exchanges by asking follow-up questions, handling corrections, and discussing varied everyday topics.
- One sample reports the Meena full system at 79%±1% interactive SSA.
- The appendix identifies the examples as samples from Meena full conversations, including random collected samples after sensitive-content removal.
- The samples cover topics including parties, dancing, fishing, movies, travel, surfing, music, holidays, and health.
- Meena frequently extends conversations with specific follow-up questions about activities, preferences, and personal experiences.
- Meena can acknowledge and repair a misunderstanding during a discussion about fishing equipment and fish.
A.2 Mitsuku
The Mitsuku samples illustrate uneven conversational coherence: the system can maintain simple exchanges but also produces misunderstandings, repetition, and out-of-context replies.
- The examples contrast occasional topical engagement with responses that are confusing, illogical, or out of context.
- Mitsuku repeats responses and fails to resolve some conversational misunderstandings, including a discussion about being late for a meeting.
- Mitsuku can provide direct replies and sustain familiar topics such as games, football, consciousness, and friendship.
- Mitsuku sometimes responds to questions with unrelated or confused statements, as in exchanges about Rome, locations, and personal references.
A.3 XiaoIce
The XiaoIce conversations show frequent topic shifts, contradictions, evasive replies, and occasional attempts at playful or personable interaction. Across the examples, responses often fail to maintain context or answer the user’s question directly.
- Personality: Some conversations show attempts at personality and playful interaction, including self-descriptions, humor, and emotionally framed replies.Examples include calling itself “宇宙无敌美少女”, joking about being made of water, and saying that it feels heartache because it has truly loved.
- Specificity: XiaoIce often uses generic, evasive, or repetitive replies instead of providing specific information.The examples include “不知道”, “你说什么？没听清”, repeated “拜拜”, and responses that redirect questions back to the user.
- Conversational coherence: XiaoIce frequently gives responses that do not address the user’s question or maintain the conversational topic.Examples include answering a question about New York hot-pot restaurants with a preference for a restaurant and responding to questions about time with “没这功能”.
- Conversational coherence: Several exchanges contain contradictions or inconsistent answers that undermine conversational coherence.XiaoIce contradicts itself about liking chocolate and gives inconsistent responses about whether it can draw or who drew its picture.
A.4 Human
The human-human conversations provide varied examples of context-aware, specific, and sometimes imperfect multi-turn dialogue. They include reciprocal questions, concrete personal details, emotional responses, topic development, and occasional grammatical or factual imperfections.
- Topic diversity: The conversations cover both mundane planning and more unusual personal reflections, demonstrating broad variation in human conversational content.Topics range from weekend work and hiking to meeting strangers, sixth-sense experiences, vacation preferences, and favorite films.
- Specificity: Human speakers provide specific details that make their replies closely tied to the current conversation.Examples include naming Diablo as a hiking destination, discussing Thanksgiving in New York, and describing a particular encounter with a stranger in a coffee shop.
- Conversation quality: Human conversations commonly respond directly to prior turns while developing shared topics such as travel, illness, movies, weather, and hobbies.The exchanges build on earlier statements through follow-up questions, acknowledgments, and related personal experiences.
- Social responsiveness: Human dialogue includes reciprocal social responses, such as expressing concern, encouragement, agreement, and shared interests.Speakers respond to illness, anxiety, a missing cat, surfing difficulties, and plans for weekend activities with relevant emotional or practical reactions.
- Imperfection: Human responses are not uniformly polished, including typos, grammatical errors, awkward phrasing, and occasional generic advice.The examples contain errors such as “yuo”, “can walk too far”, and “do some push up”, alongside ordinary conversational imperfections.
B Additional Sample Responses
The additional sample-response section presents contexts and example replies for Meena, Cleverbot, and DialoGPT, alongside GenericBot examples used to illustrate sensibleness and specificity labels. The materials are organized as comparative response samples rather than a new quantitative analysis.
- Additional samples: Table 7 contains additional sample responses from Meena, Cleverbot, and DialoGPT.The examples use author-written contexts that are not part of the MTB dataset.
- Comparative evaluation: The examples support comparison of how different chatbots respond to the same authored conversational contexts.The section places multiple systems’ responses alongside shared prompts rather than presenting independent prompts for each model.
- Additional samples: The sample contexts include questions about favorite colors and opinions about whether blue is popular.Candidate replies vary from agreeing that blue is popular to expressing a personal preference for blue.
- GenericBot examples: Table 8 illustrates GenericBot’s response behavior using fixed contexts and binary labels for sensibleness and specificity.GenericBot replies “I don’t know” to questions and “ok” to statements; the contexts were constructed before response generation and are not in the MTB dataset.
C Additional Figures
The additional figures examine how sampling affects static sensibleness and specificity and relate both measures to human likeness. The reported regression fits indicate especially strong associations between the component measures and human likeness.
- Sampling analysis: Figure 7 compares static sensibleness across the number of sampled responses for top-k decoding and temperature-based sampling.The figure focuses on how the reported sensibleness measure varies with the number of sampled responses under the two decoding approaches.
- Sampling analysis: Figure 8 compares static specificity across the number of sampled responses for top-k decoding and temperature-based sampling.The figure focuses on how the reported specificity measure varies with the number of sampled responses under the two decoding approaches.
- Human-likeness correlations: R2 = 0.99 for sensibleness versus human likeness, indicating a strong correlation in the plotted chatbot comparisons.Each point represents a different chatbot, with the top-right point representing a human, and a regression line is shown.
- Human-likeness correlations: R2 = 0.89 for specificity versus human likeness, also indicating a strong correlation in the plotted chatbot comparisons.Each point represents a different chatbot, with the top-right point representing a human, and a regression line is shown.