Source-linked AI summary
The Second Conversational Intelligence Challenge (ConvAI2)
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, Jason Weston
TL;DR
ConvAI2 addresses the need for better benchmarks and evaluation of open-domain, multi-turn chatbot conversations. It compares systems using automatic and human-centered evaluations and finds that pretrained Transformer variants perform best, while single-word metrics miss important conversational qualities. The results motivate metrics that capture repetition, consistency, and dialogue-act balance.
Problem
Open-domain chatbot research lacks evaluation criteria that adequately measure dialogue quality across multi-turn conversations.
Method
The competition compares chatbot systems with automatic metrics, Mechanical Turk and live volunteer conversations, and analyzes repetition, consistency, and dialogue-act balance.
Results
Pretrained generative Transformer variants were the best-performing models, while automatic metrics missed important aspects of human-rated multi-turn conversation.
Takeaways & Limitations
Future evaluation should move beyond single-word metrics toward sequence-level measures of repetition, consistency, and question-asking versus question-answering balance.
Takeaways & Limitations
F1 can be outperformed by a fixed frequent-word response, indicating that word-overlap metrics may not reflect dialogue quality.
Abstract
from arXiv · showhide
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (i) pretrained Transformer variants are currently the best performing models on this task, (ii) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations) -- in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).
1 Overview of the competition
ConvAI2 established an open-domain chit-chat competition designed to make chatbot systems directly comparable through shared data, evaluation procedures, baselines, and released models. It compared systems with automatic metrics, paid human evaluations, and live volunteer conversations, revealing differences between metric-based and human judgments.
- Motivation: ConvAI2 targeted meaningful open-domain conversation, addressing the lack of established benchmarks and reliable automatic evaluation criteria for chatbots.The competition aimed to provide datasets, evaluation code, baselines, and the winning model as reusable standards.
- Evaluation: Models were evaluated with hidden-test automatic metrics, Amazon Mechanical Turk conversations, and live volunteer conversations, with the grand prize based on human evaluation.Automatic tracks had separate winners, while the overall prize recognized human-evaluation performance.
- Results: Hugging Face won the automatic tracks by a significant margin, while Lost in Conversation won the human-evaluation grand prize among 23 entrants.Hugging Face placed second in the human evaluations.
- Findings: The competition’s analysis found that automatic evaluations showed some correlation with human judgments but missed multi-turn properties such as dialogue-act balance.The cited example is the balance between questions asked and questions answered.
- Task formulation: The competition shifted from the first edition’s Wikipedia-based conversations to engaging chit-chat about personal interests.The organizers reported that the earlier task was not engaging for all participants, whereas ConvAI2 focused on getting to know another person.
2 Competition description and set-up
ConvAI2 used Persona-Chat to test agents that meet, engage, and maintain a consistent persona, combining standardized data, automatic metrics, and human conversations. Participants submitted reproducible systems, and the evaluation pipeline progressed from hidden-test scoring to human and live-chat assessment.
- Data and task: The ConvAI2 dataset assigns speaker pairs personas and asks them to get to know each other through engaging conversation about interests and common ground.Training uses 1155 possible personas, while 100 personas are held out for validation.
- Data and task: Additional rewritten persona sets used rephrases, generalizations, and specializations to reduce reliance on trivial word overlap.The rewrites make the task more challenging while preserving the same train and test personas.
- Evaluation: Human evaluation used Mechanical Turk conversations that measured enjoyment and persona recognition, alongside live volunteer chats through Facebook Messenger and Telegram.The live setup was intended to resemble real-world chatbot applications more closely than the Mechanical Turk setup.
- Reproducibility: Competitors had to provide source code and identify training data and ensembling, while winners were required to release training code for reproducibility.Additional training data was allowed when publicly released or already public.
3 Results and Analysis
The competition found Hugging Face strongest on automatic metrics, while Lost in Conversation won human evaluation. Further analysis exposed weaknesses in automatic metrics and highlighted conversation-level issues such as repetition, persona reliance, and excessive questioning.
- Automatic Metrics: Hugging Face won the automatic evaluation by performing best on F1, Hits@1, and Perplexity.The ranking used the minimum rank across the three metrics, with more than 23 teams evaluated on a hidden test set.
- Human Evaluation: Six teams participated in human evaluation after the top seven advanced and High Five declined to participate.The competition’s grand prize was ultimately determined by Mechanical Turk performance because wild-evaluation results were discounted.
- Additional Automatic Metrics: Revised personas tested reliance on persona word overlap, with Hugging Face performing best and Little Baby close behind.The revised personas were rewritten to have little or no overlap with the originals, making the ranking task more challenging.
- Additional Automatic Metrics: Adding the last partner message as a distractor harmed most ranking models, suggesting excessive reliance on word overlap with the last utterance.A robust model should rarely parrot its partner, so Hits@1 should remain similar with and without this distractor.
- F1 Metric Toy Baseline: A fixed frequent-word reply achieved F1 scores of 19.6 on test and 20.5 on validation, exceeding Hugging Face’s 19.5 and 19.1 respectively.This result indicates that word-overlap-based F1 can reward nonsensical responses.
- Human Evaluation: Lost in Conversation won the competition with an engagingness score of 3.11 out of 4, and Bayesian calibration preserved the score ordering.The final leaderboard reported postcalibration scores.
- Further Analysis of Results: Human-oriented analysis found weakly interpretable links between evaluation scores and rare-word use, repetition proxies, and utterance uniqueness.Humans used more rare words; Hugging Face had 97% utterance uniqueness, while Lost in Conversation had 86%, and repetition proxies missed paraphrased repetition.
- Further Analysis of Results: Hugging Face began responses with question words 107 times in 100 conversations, compared with 12 times for humans, potentially making dialogue feel disjointed.The authors tentatively linked excessive questioning to lower human evaluation and called for conversation-level rather than only utterance-level evaluation.
4 Participating Models
The paper provides brief descriptions of model types used by some top competitors, with further details directed to separate papers and slides.
- Brief model descriptions for some top competitors are provided in Table 10.The table is presented as a short summary rather than a comprehensive model account.
- Detailed papers were planned for some competitors’ models, and the winner’s code was publicly available.
- The ADAPT Centre model is a bi-directional attentive LSTM pretrained with GloVe embeddings, Switchboard, and Open Subtitles.
5 Conclusions and Future Work
The competition found generative Transformer variants performed best, while current automatic metrics miss important multi-turn qualities such as repetition, consistency, and dialogue-act balance. Future work should improve multi-turn evaluation and extend tasks beyond shallow meet-and-greet conversations.
- Models: Generative Transformer variants were the strongest-performing models, but required pretraining or multitasking with larger datasets beyond Persona-Chat.Language-modeling data not explicitly designed for dialogue could also support pretraining.
- Models: Retrieval models scored slightly below generative models in human evaluations, although the comparison was uncertain because no very strong retrieval model was submitted.A retrieve-and-refine approach was identified as an unexplored possibility.
- Automatic vs. Human Evaluation: Current automatic metrics fail to capture multi-turn repetition, consistency, and dialogue-act balance across conversations.Per-word perplexity and Hits@1 evaluate narrower units and do not fully represent conversational flow.
- Towards Multi-turn Evaluation: Models sometimes produced self-inconsistent continuations or repeated questions that humans had already answered, which could harm human evaluations.Examples include contradicting stated interests and asking what someone does after they have already answered.
- Towards Multi-turn Evaluation: Dialogue natural language inference was proposed to evaluate and potentially improve consistency by classifying utterance pairs as entailment, neutrality, or contradiction.The proposed task could support both training and evaluation of conversational consistency.
- Towards more complex tasks: Even a solved Persona-Chat task would cover only short, shallow meet-and-greet exchanges and would not evaluate long-term memory, deep knowledge, or deeper reasoning.The authors identify more complex tasks as necessary for evaluating broader intelligent-agent abilities.
A Example Dialogues
The paper provides example Mechanical Turk conversations for several participating systems. Figures 6–11 show examples from Lost in Conversation, Hugging Face, Little Baby, Mohd Shadab Alam, Happy Minions, and ADAPT Centre.
- Example Dialogues: Figure 6 presents an example Mechanical Turk conversation from Lost in Conversation.
- Example Dialogues: Figure 7 presents an example Mechanical Turk conversation from Hugging Face.
- Example Dialogues: Figure 8 presents an example Mechanical Turk conversation from Little Baby.
- Example Dialogues: Figure 9 presents an example Mechanical Turk conversation from Mohd Shadab Alam.
- Example Dialogues: Figure 10 presents an example Mechanical Turk conversation from Happy Minions.
- Example Dialogues: Figure 11 presents an example Mechanical Turk conversation from ADAPT Centre.