Source-linked AI summary
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents
Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, Jason Weston
TL;DR
Because no single task measures the many skills expected of an open-domain conversational agent, the paper introduces dodecaDialogue, a collection of 12 subtasks, and trains a unified model across them. The model combines dialogue history with external text and images; the authors report state-of-the-art results on 10 subtasks with comparable prior results, while noting that substantial skills remain outside the collection.
Problem
No single task trains or measures an agent across the range of conversational abilities, including engaging dialogue, knowledge use, and image-grounded interaction.
Source: §1 · p. 1Method
Results
The model achieves state-of-the-art results on 10 of the 12 subtasks with comparable prior results, and dialogue datasets are reported as a strong alternative to general-text pre-training.
Source: §1 · p. 1Takeaways & Limitations
The collection provides a baseline for evaluating conversational agents across multiple skills, including image and text tasks.
Source: §6 · p. 8The collection omits skills such as longer conversations involving memory and mixing open-domain conversation with task-oriented goals.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We introduce dodecaDialogue: a set of 12 tasks that measures if a conversational agent can communicate engagingly with personality and empathy, ask questions, answer questions by utilizing knowledge resources, discuss topics and situations, and perceive and converse about images. By multi-tasking on such a broad large-scale set of data, we hope to both move towards and measure progress in producing a single unified agent that can perceive, reason and converse with humans in an open-domain setting. We show that such multi-tasking improves over a BERT pre-trained baseline, largely due to multi-tasking with very large dialogue datasets in a similar domain, and that the multi-tasking in general provides gains to both text and image-based tasks using several metrics in both the fine-tune and task transfer settings. We obtain state-of-the-art results on many of the tasks, providing a strong baseline for this challenge.
1 Introduction
The paper assembles a broad set of conversational skills into one challenge because no single task trains or measures them all. It trains a unified model across 12 subtasks and reports state-of-the-art results on 10 with comparable prior results.
1 Introduction
- The authors introduce dodecaDialogue, a 12-subtask challenge with training, validation, and test data intended to support and measure one agent across conversational skills.The subtasks include very large datasets, such as one with 2.2 billion utterances.Source: §1 · p. 1
- 10 of the 12 subtasks with comparable prior results achieve state-of-the-art results after pre-training on the largest subtasks and multi-tasking across all sources.The model takes an image, external textual information, and dialogue history as input to generate a response.Source: §1 · p. 1
- Dialogue datasets, being more closely linked to the agent’s goals than general text corpora, are presented as a strong alternative for pre-training.The paper also reports that multi-tasking typically incurs small losses against fine-tuning the same system, while zero-shot transfer remains demanding.Source: §1 · p. 1
2 The dodecaDialogue Task
dodecaDialogue brings together existing English datasets that cover complementary conversational abilities, including persona-based interaction, everyday discussion, knowledge-grounded responses, empathy, and image dialogue. Its evaluation examines multi-task performance, single-task fine-tuning, and transfer to tasks withheld during training.
2 The dodecaDialogue Task
- The collection targets a single engaging conversational agent by combining subtasks that cover different goals, from getting to know a partner to discussing topics, showing empathy, and talking about images.All provided subtasks are in English.Source: §2 · p. 2
- ConvAI2 and DailyDialog cover persona-based introductions and everyday conversations, while Cornell Movie and Twitter provide varied discussions and emotional states.DailyDialog spans ten categories; the Twitter variant is text-only and is the collection’s second-largest dataset.Source: §2 · pp. 2–3
- Wizard of Wikipedia and ELI5 target knowledge-grounded discussion and long-form answers, while Ubuntu provides in-depth discussion of a focused technical topic.Wizard of Wikipedia uses gold knowledge, and ELI5 answers are grounded in multiple retrieved documents.Source: §2 · pp. 2–3
- LIGHT grounds dialogue in a dynamic game world, while Empathetic Dialogues asks a listener to respond to a speaker’s personal situation with empathy.LIGHT includes actions and emotes; the Empathetic Dialogues emotion labels are not used here.Source: §2 · pp. 2–3
- The pushshift.io Reddit variant conditions comment generation on the full preceding thread and supplies 2.2 billion training examples, making it the collection’s largest dataset.It does not overlap with ELI5.Source: §2 · p. 3
- Image Chat trains engaging conversations about images with a given personality, whereas IGC focuses on image-related questions and responses.Because IGC only had validation and test sets, most of its validation set was converted into a small training set.Source: §2 · p. 3
2.1 Evaluation
- Evaluation reports common generation metrics and each task’s literature-standard score, assessing all-task multi-tasking, multi-tasking followed by single-task fine-tuning, and leave-one-task-out zero-shot transfer.The zero-shot setting trains on all tasks except the one being evaluated.Source: §2.1 · p. 3
3 Related Work
Prior work shows that large pre-trained transformers perform strongly on several dialogue tasks, while existing task collections generally target broader NLP or question answering. dodecaDialogue distinguishes itself by assembling tasks intended to support an engaging conversational agent.
3.1 Existing Models and Results
- Large pre-trained transformers provide strong prior results on ConvAI2, ELI5, DailyDialog, and Cornell Movie, including results from pre-training on BooksCorpus or CC-NEWS.A BERT-based generative model is also trained as an additional baseline.Source: §3.1 · p. 4
3.2 Related Tasks and Collections
- For feasibility, dodecaDialogue omits additional knowledge, image-based, and question-answering datasets, selecting tasks intended to yield engaging dialogue.The authors suggest that future work could add tasks while retaining the original twelve for evaluation.Source: §3.2 · p. 4
- Unlike collections such as the NLP decathlon, MULTIQA, and Raffel et al.’s broader multi-task work, this collection focuses on dialogue tasks that group naturally into a conversational agent.The component tasks already existed before this collection was assembled.Source: §3.2 · p. 4
4 Models
The paper uses a generative BERT baseline for text-only tasks and an Image+Seq2Seq transformer for image inputs. The latter adds image features to the encoder and is trained with an autoregressive sequence loss.
4 Models
- The BERT baseline adapts BERT to generation with an autoregressive loss over concatenated dialogue context and current generation, but does not handle image-input tasks.BERT is fine-tuned separately for each task, using sentence embeddings to distinguish roles.Source: §4 · p. 4
- Image+Seq2Seq modifies a transformer encoder-decoder by adding projected image features as an extra encoder-output token for the decoder.The model has eight encoder and eight decoder layers; image features are fixed during fine-tuning except for their linear projection.Source: §4 · pp. 4–5
- Table 2 reports validation perplexity for the dodecaDialogue tasks across multiple settings.The image features come from a pre-trained ResNeXt model, and the text transformer is trained with standard autoregressive negative log-likelihood.Source: §4 · pp. 4–5, Table 2 · p. 5
5 Experiments
Experiments compare pre-training, multi-tasking, transfer, grounding, decoding, and human-evaluation settings. The final systems outperform available prior results, while multi-tasking generally remains slightly behind task-specific fine-tuning.
Image and Knowledge Grounding
- Grounding lowers validation perplexity from 16.8 to 8.7 on Wizard of Wikipedia and from 19.5 to 18.3 on Image Chat.The table also reports ELI5 at 21.3 without versus 21.2 with grounding, and IGC at 10.1 in both conditions.Source: §5 · p. 6, Table 4 · p. 6
Transfer Performance between Tasks
- Across Reddit, ConvAI2, Wizard of Wikipedia, and Empathetic Dialogues, training on all four gives the best average transfer, but not per-task performance versus fine-tuning.Reddit-only training transfers to other tasks but is less effective than fine-tuning on the evaluated task.Source: §5 · p. 6
Pre-training
- Average perplexity falls from 49.5 for training from scratch to 17.1 with Reddit pre-training; FastText and Twitter reach 45.7 and 35.6, respectively.The reported BERT pre-training results remain worse than Reddit pre-training.Source: §5 · p. 6
Comparison to Existing Systems
- The multi-task model outperforms all available prior results, and the multi-task-plus-fine-tuned model also does so; two tasks lack prior results.Table 7 reports test performance across task-specific metrics.Source: Table 7 · p. 7, §5 · p. 8
Multi-Task Results
- The All Tasks MT model achieves a dodecaScore of 19.1, beating all pre-train-plus-fine-tune approaches except Reddit pre-training followed by fine-tuning, and a single Reddit model.Many task results are slightly worse than the corresponding best fine-tuned models.Source: §5 · p. 7
Multi-Task followed by Fine-Tuning
- Multi-tasking followed by fine-tuning yields a validation dodecaScore of 16.8, slightly below the 17.1 for fine-tuning alone.Relative up-weighting of the target task during multi-task training generally improves the final fine-tuned model; differences are generally small.Source: Table 5 · p. 6, §5 · p. 7
Human Evaluation
- All three human-evaluation matchups favor the All Tasks MT agent over existing state-of-the-art models, with statistically significant results (binomial test, p < .05).The comparisons cover Image Chat and Wizard of Wikipedia seen and unseen topics.Source: §5 · p. 8, Figure 1 · p. 8
Leave-One-Out Zero-Shot Performance
- Zero-shot multi-task models outperform a Reddit-only model on every held-out task except Reddit, but remain behind multi-task and fine-tuned performance.The study also reports that multi-tasking is successful across decoding-based metrics, while improving further remains a challenge.Source: §5 · p. 8
6 Discussion
The discussion presents dodecaDialogue as a broad conversational challenge with strong multimodal baseline results, while noting that important open-domain skills remain outside its 12 tasks.
6 Discussion
- Multimodal Image+Seq2Seq transformers trained across the tasks provide strong baseline results for dodecaDialogue.The collection is intended to motivate building and evaluating conversational agents with multiple skills.Source: §6 · p. 8
- The challenge targets a single agent with multiple skills, a core goal of AI, and the authors say systems are closer to that goal but still have a long way to go.Source: §6 · p. 9
- The 12-task collection omits skills such as longer conversations involving memory and combining open-domain conversation with task-oriented goals.The authors suggest adding these tasks while continuing to improve models.Source: §6 · p. 9
A Additional Results
The appendix provides additional test and validation comparisons across dodecaDialogue tasks, plus results organized by decoding settings.
A Additional Results
- Table 8 reports test performance across various metrics, comparing multi-task and multi-task-plus-fine-tuned methods.Source: Table 8 · p. 12
- Table 9 reports validation performance across various metrics for multi-task and multi-task-plus-fine-tuned methods.Source: Table 9 · p. 12
- Table 10 gives All Tasks MT validation results across metrics using one decoding-parameter setting.The setting uses beam size 3, minimum response length 10, and repeated tri-gram blocking.Source: Table 10 · p. 12
- Table 11 lists the best decoding parameters for each task based on metric, using the best task-specific multi-task-plus-fine-tuned validation model.It specifies minimum and maximum decoding lengths in tokens.Source: Table 11 · p. 13
B Human Evaluation Further Details
The appendix details human evaluations for Image Chat and Wizard of Wikipedia, including decoding comparisons and a supplementary Likert-rating comparison.
B Human Evaluation Further Details
- Wizard of Wikipedia Likert ratings are reported as mean and standard deviation, and the findings are similar to the main paper’s pairwise ACUTE-Eval results.Humans rated model responses from 1 to 5 at the end of each conversation.Source: B Human Evaluation Further Details · p. 13, Table 15 · p. 14
- Image Chat human evaluations compare decoding schemes for the all-task Image+Seq2Seq model with human outputs.The table marks statistically significant scores using a binomial test at p < .05.Source: Table 12 · p. 13, B Human Evaluation Further Details · p. 13
- Wizard of Wikipedia seen and unseen test-set evaluations compare decoding schemes with human outputs using ACUTE-Eval.The appendix reports all scores as statistically significant by binomial test at p < .05.Source: Table 13 · p. 13, Table 14 · p. 14
C All Tasks Multi-Task Model Examples
The examples illustrate model responses across persona-based, knowledge-grounded, empathetic, image-based, and other conversational settings.
C All Tasks Multi-Task Model Examples
- A persona example shows the model echoing details such as being strong for its age, while another dialogue includes discussion of wine and dancing.Source: C All Tasks Multi-Task Model Examples · p. 15
- Wizard of Wikipedia examples show responses using supplied topic knowledge, including facts about dogs and the literary origins of orcs.In one example, the model responds to a dog conversation with a domestication fact; another uses knowledge about Tolkien’s conception of orcs.Source: C All Tasks Multi-Task Model Examples · p. 15
- The examples pair situations and labeled emotions—excited, disappointed, and lonely—with corresponding Model responses, including concert excitement, a higher-than-advertised TV price, and friends living abroad.Source: C All Tasks Multi-Task Model Examples · p. 16
- The image-chat example includes a model asking whether an image depicts a lake, while other examples draw on Reddit and Wizard of Wikipedia contexts.Source: C All Tasks Multi-Task Model Examples · pp. 15, 17, 18