Source-linked AI summary
Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, Kyu-Young Kim
TL;DR
Taskmaster-1 addresses the shortage of high-quality, goal-oriented conversational data by introducing a 13,215-dialog corpus collected through spoken Wizard of Oz and written self-dialog procedures. The dataset reports richer, more diverse, and more realistic language than MultiWOZ, while providing API-oriented annotations, baseline models, and human evaluations.
Problem
High-quality, goal-oriented dialog data is scarce, hindering progress in statistical dialog understanding and generation.
Method
The paper constructs Taskmaster-1 with spoken Wizard of Oz and written self-dialog collection, using simple API-oriented annotations and evaluating seq2seq baselines with automatic metrics and human judgments.
Results
Taskmaster-1 reports richer, more diverse, and more realistic language than MultiWOZ.
Takeaways & Limitations
The corpus supports research on task-oriented dialog modeling, written versus spoken language, discourse patterns, and error handling.
Takeaways & Limitations
The collection assumes users already have an account with the ticket service, so credit-card information is unnecessary.
Abstract
from arXiv · showhide
A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken "Wizard of Oz" (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is "self-dialog" in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.
1 Introduction
Taskmaster-1 addresses the shortage of high-quality, goal-oriented dialog data with a large, diverse corpus collected through spoken and written procedures. The authors report richer, more realistic language than MultiWOZ and provide model baselines and human evaluations.
- Motivation: High-quality, goal-oriented dialog data remains a major hindrance to progress in statistical dialog understanding and generation.Natural language is context dependent, ambiguous, and rich in discourse and pragmatic cues.
- Dataset and contribution: 13,215 task-based dialogs span six domains and include 5,507 spoken and 7,708 written conversations.The spoken conversations use two-person Wizard of Oz collection, while workers write the complete written self-dialogs.
- Findings: Taskmaster-1 is reported to have richer and more diverse language than MultiWOZ.The dataset contains more unique words and is more difficult for language models to fit.
- Findings: The collection is reported to be more realistic than MultiWOZ because two-person dialogs include more real-world entities without a small knowledge-base restriction.The authors also present neural seq2seq baselines and qualitative human evaluations, with automatic metrics correlating well with human judgments.
2 Related work
Related work contrasts human-machine and human-human dialog data, motivating Wizard of Oz collection as a compromise between natural expression and machine-oriented interaction. Taskmaster-1 combines spoken two-person dialogs with written self-dialogs and API-argument annotations.
- Human-machine vs. human-human dialog: Existing task-based datasets often derive from deployed systems, which may bias conversations toward those systems’ limitations.Purely human-human corpora avoid that bias but can have unsuitable error distributions and turn-taking behavior for dialog systems.
- Wizard of Oz: Wizard of Oz lets users express intent naturally while a human controls responses behind an apparently automated assistant.The machine-oriented interface and slower turn-taking cadence keep interactions from becoming fully complex human discourse.
- Comparison with MultiWOZ: Taskmaster-1 spans roughly 13,000 dialogs across six domains and uses API arguments, whereas MultiWOZ is an entirely written corpus with dialog-state and dialog-act annotations.Taskmaster-1 uses crowdsourced users and trained agents for its two-person dialogs, plus crowdsourced self-dialogs.
- Examples: A sample Taskmaster-1 two-person dialog illustrates the spoken collection format.The cited material identifies the example but does not provide enough detail to summarize its conversational content.
3.1 Overview
Taskmaster-1 combines spoken and written task-oriented dialogs across six domains, balancing conversational fidelity, collection cost, and speaker diversity. Its corpus uses simple API-based annotations and averages about 23 utterances per dialog.
- Collection methods: Spoken dialogs more closely reflect conversational language, while written dialogs are cheaper and easier to gather and increase corpus size and speaker diversity.The two modalities provide complementary collection advantages.
- Overview: Six goal-oriented tasks define the corpus: pizza, auto repair, rides, movie tickets, coffee, and restaurant reservations.The tasks are designed around completing concrete service interactions.
- Collection instructions: The task instructions require workers to cover relevant booking details across several turns and confirm the final ticket information.The supplied instructions include film, people, city, theater, time, ticket cost, and delivery confirmation.
- Overview: About 23 utterances per dialog support context-rich language behaviors.The collection therefore contains multiple-turn interactions rather than isolated requests.
- Annotation: API-based annotation provides grounding while remaining easy for workers to apply consistently.The annotation approach is presented as a simple schema for the dataset.
- Overview: 13,215 dialogs make the corpus comparable in size to recently released datasets such as MultiWOZ.The corpus contains both spoken and written collection formats.
3.2 Two-person, spoken dataset
The spoken two-person dataset uses a Wizard of Oz platform pairing trained assistant agents with crowdsourced users in task-based scenarios. Audio transcription, quality control, and flexible instructions support realistic, varied interactions.
- Platform: The WOz platform pairs assistant agents with crowdsourced users and reproduces a two-participant automated-assistant experience.An example dialog from the dataset is presented in Figure 1.
- Platform: Assistant agents type responses that are played through text-to-speech, while users speak through a laptop and microphone over WebRTC.This setup creates a digital-assistant-like communication style.
- Transcription: Users’ audio is transcribed by additional workers, merged with assistant text, and checked for transcription errors and typos.The resulting full-text conversations are then annotated.
- Instructions: Agents and users receive written instructions that can be modified to broaden coverage of likely user-assistant scenarios.Examples include changing an order after selecting an initial item and handling unavailable options.
- Training and quality control: Agents receive two hours of training on the interface and scenarios including uncooperative users and technical glitches.Problematic users are detected during or after sessions and removed from future requests.
- Self-dialog comparison: Self-dialog instructions ask workers to write a fresh conversation of at least 10 turns while imagining a phone interaction with a personal assistant.The instructions require workers to research details such as theaters, showtimes, prices, and options.
3.3 Self-dialogs (one-person written dataset)
The self-dialog approach uses crowdsourced workers to write complete task conversations, providing an efficient alternative to two-person spoken collection. It produces realistic interactions, including errors and linguistic repairs, while omitting some spoken disfluencies.
- Task scenarios and instructions: Self-dialogs have crowdsourced workers write the entire conversation while imagining a phone-based assistant completing one of the six tasks.The approach targets the same six tasks as the two-person collection procedure.
- Pros and cons of self-dialogs: Self-dialogs avoid misunderstandings between interlocutors while still representing understanding errors, unavailable choices, and other dialog glitches.Workers can construct both successful and problematic conversational paths without frustration caused by two-person interaction.
- Pros and cons of self-dialogs: Crowdsourced workers recreate both error-free interactions and conversations containing linguistic repair, including ultimately unsuccessful ticket-booking scenarios.Figure 5 presents a written dialog involving ticket-availability problems.
- Pros and cons of self-dialogs: Thousands of self-dialogs were collected in days without transcription or trained agents, at roughly six times lower cost per dialog.These benefits make self-dialog collection substantially more efficient and cost effective than the two-person approach.
- Pros and cons of self-dialogs: Self-dialogs cannot reproduce the disfluencies and more complex error patterns found in two-person spoken dialogs.The missing spoken phenomena are described as important for model accuracy and coverage.
3.4 Annotation
Taskmaster-1 uses a simplified annotation scheme centered on API arguments and transaction status rather than detailed dialog-state labels. Arguments are marked according to the variables needed to execute each transaction.
- Annotation strategy: The annotation scheme labels only API arguments, namely the variables required to execute each conversation’s transaction.Examples include ride locations and car type, or movie name, theater, time, and ticket count.
- Transaction status: Each executable transaction label receives an “accept” or “reject” suffix indicating whether the transaction succeeded.The convention handles repeated argument types when users change their minds or requested items become unavailable.
- Transaction status: In Figure 6, the people-count and time variables in the assistant utterance receive the “.accept” label because the transaction was completed successfully.The example illustrates how accepted arguments encode successful completion.
- Transaction status: When a transaction utterance omits its variables, the entire sentence is labeled with the dialog type, such as reservation.accept.The example “The table has been booked for you” is labeled reservation.accept.
4 Dataset Analysis
Dataset analyses compare self-dialogs with two-person conversations and evaluate response-generation baselines using automatic metrics and human judgments. The results characterize diversity, conversational length, model performance, and agreement between evaluation methods.
- Dataset comparison: Compared with MultiWOZ, Taskmaster-1 contains more unique words, nearly twice as many utterances per dialog, higher Transformer perplexity, and lower BLEU scores.The comparison uses self-dialogs and attributes the pattern to less restrictive scripts and knowledge-base constraints.
- Dataset comparison: Self-dialogs have almost three times higher perplexity than two-person conversations, indicating more diverse and less conventional conversational flows.Self-dialogs also contain more unique words, whereas two-person conversations are longer.
- Response generation: The response-generation task models the next utterance as a conditional probability given the preceding dialog history, with each utterance factorized autoregressively over words.The model parameterization may use recurrent, convolutional, or Transformer-based seq2seq architectures.
- Response generation: Transformer is the best-performing model on automatic evaluation metrics among the evaluated seq2seq architectures.The study evaluates perplexity and BLEU alongside ranking and 1–5 Likert ratings for top-performing models.
- Human evaluation: BLEU scores correlate strongly with human ranking judgments, while inter-annotator reliability is higher for ranking than for 1–5 Likert rating.The reported Krippendorff’s alpha values are 0.29 for ranking and 0.21 for rating.
5 Conclusion
Taskmaster-1 addresses the shortage of quality dialog corpora with richer, more diverse unrestricted task-oriented conversations and two collection methodologies. It also provides API-oriented annotations, baseline models, and human-aligned evaluations.
- Taskmaster-1 targets the lack of quality corpora for data-driven dialog research and development.
- Its unrestricted task-oriented conversations provide richer and more diverse language with more real-world entities than current benchmarks.
- The paper presents spoken and written collection methodologies designed to ensure speaker diversity and conversational accuracy.
- Its API-oriented annotation technique is easier for annotators to learn and simpler to apply.
- Baseline neural seq2seq models and qualitative human evaluations show that automatic metrics correlate well with human judgments.