Source-linked AI summary
MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić
TL;DR
Task-oriented dialogue research lacked sufficiently large and varied annotated data, particularly for multi-domain settings. The paper introduces MultiWOZ, a crowdsourced, fully labeled human-human corpus with dialogue-state and system-act annotations, and reports benchmark baselines across dialogue tasks. Its scale and richness position it as training data and a testbed for modular and end-to-end dialogue modeling, while the collection setting imposes documented boundaries on interaction style.
Problem
Existing dialogue datasets were constrained by limited linguistic variability, missing multi-domain use cases, or insufficient annotations and scale.
Method
The paper constructs MultiWOZ through a crowdsourced Wizard-of-Oz pipeline and annotates dialogues with dialogue states and system dialogue acts.
Results
MultiWOZ contains 10,438 dialogues, including 7,032 multi-domain dialogues, and supports benchmark evaluations of state tracking and generation tasks.
Takeaways & Limitations
MultiWOZ provides large-scale, linguistically rich training data and a challenging testbed for modular and end-to-end dialogue modeling.
Takeaways & Limitations
Human-human typed interactions have significantly longer turns than spoken interactions, limiting direct correspondence to spoken dialogue length.
Abstract
from arXiv · showhide
Even though machine learning has become the major scene in dialogue research community, the real breakthrough has been blocked by the scale of data available. To address this fundamental obstacle, we introduce the Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. At a size of $10$k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora. The contribution of this work apart from the open-sourced dataset labelled with dialogue belief states and dialogue actions is two-fold: firstly, a detailed description of the data collection procedure along with a summary of data structure and analysis is provided. The proposed data-collection pipeline is entirely based on crowd-sourcing without the need of hiring professional annotators; secondly, a set of benchmark results of belief tracking, dialogue act and response generation is reported, which shows the usability of the data and sets a baseline for future studies.
1 Introduction
MultiWOZ addresses limitations in existing task-oriented dialogue corpora by providing a large, multi-domain, fully annotated human-human dataset. It also reports collection details and benchmark baselines for core dialogue components.
- Existing dialogue corpora were constrained by missing annotations, limited capacity, insufficient multi-domain coverage, or low linguistic variability.
- MultiWOZ contains dialogues spanning several domains and topics, with each dialogue annotated by dialogue states and corresponding system dialogue acts.
- Around 10k dialogues make MultiWOZ at least one order of magnitude larger than previously available structured corpora.
- The work describes the data-collection approach, data structure, and statistical analyses.
- Benchmark baselines cover belief tracking, natural language generation, and end-to-end response generation.
2 Related Work
Prior task-oriented dialogue datasets used machine-to-machine, human-to-machine, or human-to-human collection strategies, each with distinct coverage and naturalness trade-offs. MultiWOZ builds on Wizard-of-Oz collection while targeting broader, richer conversations through crowdsourcing.
- Machine-to-machine collection offers template coverage and outcome diversity but risks mismatch with real interactions because dialogue naturalness depends on engineered agents.
- Human-to-machine corpora generally depend on an existing working dialogue system, limiting their use for developing systems in entirely new domains.
- Initial system capabilities can bias human-to-machine data, prompting users to produce simpler inputs that the system can understand.
- Wizard-of-Oz collection records human-human conversations for future conversational-system development.
- Crowdsourced Wizard-of-Oz extensions demonstrated that this framework could collect high-quality typed conversations at scale.
3 Data Collection Set-up
MultiWOZ uses a crowd-sourced Wizard-of-Oz pipeline to collect natural multi-domain conversations and annotate them with dialogue states and acts. The process combines structured task goals, wizard database interaction, quality control, and improved annotator agreement.
- Collection process: The Wizard-of-Oz setup supports large-scale collection of natural human-human conversations at relatively low cost and time.The pipeline was designed around an easy-to-operate wizard interface and easy-to-follow user goals.
- Task design: Task templates define domain ontologies, including informable constraints and requestable information, while allowing alternative goals when no database match exists.Goal descriptions are gradually revealed, and sub-goals such as booking are shown when required.
- Wizard interaction: The wizard enters user-provided information through a persistent web form that queries the back-end database, implicitly producing the belief state.This lets the wizard focus on providing the requested information while maintaining context across turns.
- Collection process: Longer turns than spoken interaction are a limitation of the collection setting.The paper explicitly notes that turn lengths are significantly longer than in spoken interaction.
- Annotation quality: Three trial annotations initially produced weighted κ=0.704, while a restricted annotator-selection procedure improved agreement to κ=0.884.The final agreement was computed over 291 turns despite the substantially larger action set.
- Annotation quality: Dialogue collection preceded annotation, enabling annotators to report errors and allowing subsequent checks that dialogue information matched predefined goals.Reported errors included task deviations and confused utterances.
4 MultiWOZ Dialogue Corpus
MultiWOZ contains tourist-information dialogues across seven domains, with varied single- and multi-domain scenarios. Its statistics show substantial dialogue length, rich system utterances, and frequent multiple dialogue acts per turn.
- Corpus scope: The corpus covers 7 domains—Attraction, Hospital, Police, Hotel, Restaurant, Taxi, and Train—with dialogues spanning 1 to 5 domains.Hotel, Restaurant, Taxi, and Train include booking subtasks.
- Data statistics: 10,438 dialogues contain 115,434 turns, with average lengths of 8.93 turns for single-domain and 15.39 for multi-domain dialogues.Around 70% of dialogues have more than 10 turns.
- Dialogue acts: Almost 60% of dialogue turns contain more than one dialogue act, indicating rich system utterances and concurrent-action challenges for reinforcement-learning models.Figure 3 summarizes dialogue-act frequencies and the number of acts per turn.
- Corpus structure: The corpus includes 3,406 single-domain dialogues and 7,032 multi-domain dialogues, with 1k examples each in the test and development sets.The validation and test sets contain only fully successful dialogues.
- Corpus structure: Each dialogue includes a goal, user and system utterances, a belief state, dialogue acts with slots per turn, and a natural-language task description.The corpus is compared with several prior task-oriented datasets on important statistics.
5 MultiWOZ as a New Benchmark
MultiWOZ is presented as a challenging benchmark for dialogue state tracking, response generation, and dialogue-act-to-text generation. Its richer, longer, multi-domain conversations expose substantial difficulty for existing models.
- Benchmark tasks: MultiWOZ benchmarks dialogue state tracking, dialogue-act-to-text generation, and dialogue-context-to-text generation as separate dialogue-modelling tasks.The benchmark is intended to reveal challenges introduced by the dataset across multiple dialogue components.
- Dialogue state tracking: The model’s performance is poorer on MultiWOZ than WOZ2.0, reflecting the richer and much longer conversations in the new dataset.The comparison uses the same model trained on both corpora, although the results are not directly comparable.
- Dialogue-context-to-text generation: The response-generation system augments a sequence-to-sequence model with an oracle belief tracker and a discrete database-access component.These additional features inform decoder word decisions and make dialogue management and response generation independent of belief-tracking performance.
- Dialogue-context-to-text generation: Baseline models obtain almost 30% lower Inform scores on MultiWOZ than on Cam676, even with a perfect dialogue-state signal.Adding attention improves the MultiWOZ Success score by less than 1%.
- Dialogue-act-to-text generation: MultiWOZ has significantly lower BLEU and slot error metrics than the SFX restaurant dataset, indicating greater difficulty for language generation.More than 60% of dialogue turns contain at least two system acts, which the authors say greatly harms existing-model performance.
6 Conclusions
The paper addresses limitations in existing dialogue datasets by using crowdsourcing to collect a large, linguistically rich corpus of human-human conversations. MultiWOZ is positioned as training data and a challenging testbed for modular and end-to-end dialogue modelling.
- Motivation: Existing dialogue datasets were commonly constrained in linguistic variability or lacked multi-domain use cases.The conclusion identifies these constraints as the motivation for the new collection pipeline.
- Collection: The data-collection pipeline uses crowdsourcing to gather a large-scale corpus of linguistically rich human-human conversations.The paper presents this as the central construction approach for MultiWOZ.
- Use: MultiWOZ is intended to provide training data and a challenging testbed for modular approaches spanning belief tracking and dialogue-act generation.The authors also expect its scale to support research in end-to-end dialogue modelling.
A MTurk Website Set-up
The MTurk setup uses separate interfaces for users, wizards, and system-act annotation. These interfaces support task execution, wizard access across domains, and annotation of system acts.
- User interface: The user interface requires workers to respond to a task description and dialogue history.This is the user-side interaction environment shown in Figure A1.
- Wizard interface: The wizard interface provides a GUI covering all domains for the wizard’s interaction with the back-end.This setup is shown in Figure A2.
- Annotation interface: The annotation interface supports labeling system acts, with the Restaurant domain enabled in the illustrated setup.This configuration is shown in Figure A3.