Source-linked AI summary
A Survey of Available Corpora for Building Data-Driven Dialogue Systems
Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, Joelle Pineau
TL;DR
Dialogue research has lagged behind other language applications in adopting data-driven methods, motivating a survey of publicly available dialogue corpora. The paper characterizes datasets and discusses transfer learning, external knowledge, and evaluation, concluding that these resources support research on scalable data-driven and end-to-end systems.
Problem
Dialogue systems have not adopted data-driven approaches as obviously as other speech and language applications, while practical systems still rely substantially on engineering and expert knowledge.
Method
The paper surveys publicly available dialogue datasets, their characteristics and uses, and related issues including transfer learning, external knowledge, and evaluation metrics.
Results
The survey identifies datasets suitable for research, development, and evaluation of data-driven dialogue systems and highlights directions toward scalable end-to-end conversational agents.
Takeaways & Limitations
Transfer learning and external knowledge are highlighted as potential directions for scalable solutions to end-to-end conversational-agent training.
Takeaways & Limitations
The Twitter corpus can require external knowledge to interpret recent public events referenced in conversations.
Abstract
from arXiv · showhide
During the past decade, several areas of speech and language understanding have witnessed substantial breakthroughs from the use of data-driven models. In the area of dialogue systems, the trend is less obvious, and most practical systems are still built through significant engineering and expert knowledge. Nevertheless, several recent results suggest that data-driven approaches are feasible and quite promising. To facilitate research in this area, we have carried out a wide survey of publicly available datasets suitable for data-driven learning of dialogue systems. We discuss important characteristics of these datasets, how they can be used to learn diverse dialogue strategies, and their other potential uses. We also examine methods for transfer learning between datasets and the use of external knowledge. Finally, we discuss appropriate choice of evaluation metrics for the learning objective.
1. Introduction
Dialogue systems have broad applications, but data-driven methods remain less established than in related speech and language tasks. This survey examines available corpora and related approaches to support data-driven dialogue research.
- Dialogue systems support applications including technical support, language learning, and entertainment.
- Data-driven methods infer knowledge and strategies from recorded data and have advanced speech and language understanding.
- Corpus-based learning is one of several training approaches; online human interaction, user simulation, and reinforcement learning are outside this survey’s scope.
- The survey characterizes dialogue interactions, corpus modalities, and datasets for learning dialogue systems.
2. Characteristics of Data-Driven Dialogue Systems
The survey frames data-driven dialogue systems through their architectures, tasks, components, datasets, and learning methods. It covers both component-based systems and end-to-end models while noting important domain and evaluation constraints.
- System architecture: The standard architecture includes speech recognition, language interpretation, state tracking, response generation, natural language generation, and speech synthesis.For text-based dialogue, speech recognition and speech synthesis can be omitted.
- Data-driven systems: Corpus-based data-driven systems optimize most component parameters using statistics observed in real-world dialogue corpora.
- Tasks and objectives: Goal-driven systems typically have explicit task-completion objectives, whereas non-goal-driven systems include applications such as language learning and computer game characters.
- Tasks and objectives: Commercial dialogue systems remain highly domain-specific and heavily dependent on hand-crafted rules and features, with many datasets constrained to narrow domains.
- Learning dialogue system components: Discriminative models predict dialogue-relevant labels or annotations through supervised classification or regression.Examples include user-intent classification, dialogue-state tracking, response selection, and learned natural-language response generation.
- Learning dialogue system components: With proper annotations, discriminative models can be evaluated automatically and integrated into deployed dialogue systems.
- End-to-end dialogue systems: End-to-end dialogue systems map dialogue text to responses without intermediate subcomponents and optimize parameters against a single objective.Information-retrieval and ranking-based response systems are described as one end-to-end category.
3. Dialogue Interaction Types & Aspects
Dialogue corpora differ by modality, interlocutor type, naturalness, and size, and these differences affect their suitability for data-driven dialogue learning.
- Modality: Written, spoken, and multimodal dialogues differ substantially in linguistic properties and socially situated information.Spoken and face-to-face dialogue are generally less formal and more interactive than written language.
- Interlocutors: Human-human and human-machine conversations have different response distributions, understanding errors, and turn-taking behavior.These differences constrain how well one corpus type represents another, especially for spoken systems.
- Learning setting: For goal-driven spoken systems, online learning with real users appears most effective, while sufficiently broad stochastic-policy corpora can support policy bootstrapping.Bootstrapping learns a policy intended to outperform the stochastic policy that generated the corpus.
- Learning setting: Human-human corpora may be more suitable for open-domain systems because they reflect natural, unconstrained, and unscripted interactions.The survey contrasts this with the particular mismatch between human-human data and goal-driven spoken systems.
- Naturalness: Corpus generation influences learned systems because artificial experiments or informed participants may not reflect the true user population.Wizard-of-Oz studies can produce more system-like interactions, but they are expensive and time-consuming.
- Corpus size: Dataset size matters because larger datasets permit less structured models, while linguistic diversity and topic breadth increase the examples needed to model corpus patterns.Large corpora may also help approximate ambiguities and missing modalities through latent stochastic variables.
4. Available Dialogue Datasets
The survey organizes publicly available dialogue datasets by interaction type, domain, modality, and corpus construction, while documenting their key statistics and uses.
- Scope and organization: The survey restricts coverage to datasets already used for dialogue research or system building, plus very large interaction corpora that could be leveraged.This scope excludes the infeasible task of covering all available human-communication data.
- Scope and organization: Tables 1–5 record dataset properties including dialogue counts, average lengths, word counts, interlocutor types, and written or spoken form.The surrounding discussion provides qualitative descriptions, while the tables provide statistics.
- Human-machine datasets: Human-machine corpora are grouped into restaurant and travel information, open-domain knowledge retrieval, and other specialized systems.Some human-human corpora are included when one participant plays the machine role.
- Human-machine datasets: DSTC datasets support structured dialogue prediction, particularly dialogue-state tracking that estimates user intentions throughout a dialogue.The listed examples cover bus information, restaurant booking, and tourist information domains.
- Human-machine datasets: The corpus collection includes travel-booking datasets such as the Carnegie Mellon Communicator Corpus, ATIS Pilot Corpus, and Maluuba Frames Corpus.These datasets vary in system setting, scale, task design, and annotation, including Wizard-of-Oz interaction and semantic frames.
- Open-domain QA: The survey briefly covers recorded human-system QA corpora, while excluding datasets that collect responses without recording dialogues.Ritel is identified as a small Wizard-of-Oz open-domain information-retrieval corpus; WikiQA and MS MARCO are excluded from further discussion.
5. Discussion
The survey discusses challenges and opportunities in using large dialogue corpora, including dataset-specific limitations, transfer learning, external knowledge, and evaluation. These issues shape how data-driven dialogue strategies can be learned and assessed.
- The Twitter Corpus: The Twitter Corpus is large and general-purpose, but its micro-blogging origins introduce drawbacks that may be acceptable for Twitter-specific agents.Tweets contain typos, slang, abbreviations, short compressed utterances, and hashtags that may not reflect conversation elsewhere.
- The Twitter Corpus: Twitter role-playing and references to recent public events create distinctive modeling challenges and motivate external knowledge use.Models can produce accurate responses to role-playing contexts, but event-related conversations may require inferring news through an external knowledge base.
- The Ubuntu Dialogue Corpus: The Ubuntu Dialogue Corpus offers large-scale technical-support data, but noisy disentanglement from multi-party IRC conversations can introduce extraction errors.A common error occurs when an utterance from the original chat is not identified during dialogue extraction.
- The Ubuntu Dialogue Corpus: Ubuntu lacks reward labels, making it difficult to model the corpus within a goal-driven dialogue framework focused on solving users’ problems.The discussion suggests that future work could construct automatic reward labels.
- The Ubuntu Dialogue Corpus: Rare technical vocabulary and changing Ubuntu-specific information make external or up-to-date knowledge important for effective dialogue modeling.Out-of-vocabulary commands, packages, websites, and evolving operating-system solutions are difficult to learn directly from the corpus.
- Future directions: Transfer learning from related datasets may bootstrap learning when large application-specific corpora are unavailable.The survey also highlights unsupervised learning followed by fine-tuning as a possible strategy for open-domain systems trained on multiple data sources.
- Future directions: External knowledge could improve both goal-driven systems and entertaining open-domain agents by informing responses about technical domains, current events, or reviews.The discussion treats knowledge integration as relevant beyond task completion.
- Evaluation metrics: Dialogue evaluation remains difficult because human feedback is costly and no definitive framework provides informative, consistent criteria for generated responses.Suggested evaluation dimensions include human-like output and reproducing the variety of user behavior in the corpus.
6. Conclusion
The conclusion argues that dialogue research is moving toward large-scale, data-driven, potentially end-to-end systems. It presents the survey as a resource for datasets and discusses transfer learning and external knowledge as directions toward scalable conversational-agent training.
- 6. Conclusion: The survey catalogs publicly available datasets suitable for research, development, and evaluation of large-scale data-driven dialogue systems.The authors place this work within an expected shift toward end-to-end trainable dialogue models.
- 6. Conclusion: It also examines dataset-use issues and highlights transfer learning and external knowledge as potential directions for scalable end-to-end conversational-agent training.These directions are presented alongside the survey of datasets rather than as evaluated experimental results.
Appendix A. Learning from Dialogue Corpora
The appendix reviews computational architectures for learning dialogue strategies directly from data, using selected prior work to illustrate how the surveyed datasets have been used.
- Appendix A. Learning from Dialogue Corpora: The appendix surveys a small set of existing computational architectures that learn dialogue strategies from data.Its purpose is illustrative rather than a comprehensive technical treatment of available methods.
A.1 Data Pre-processing
Pre-processing standardizes dialogue corpora with minimal information loss, potentially reducing data scarcity and easing model learning. The survey emphasizes that choices such as anomaly removal, tokenization, and spelling correction can materially affect results.
- A.1 Data Pre-processing: Pre-processing standardizes dialogue datasets with minimal information loss, which can reduce data scarcity and make models easier to train.The survey notes that pre-processing can significantly affect natural-language-processing results.
- A.1 Data Pre-processing: Text-corpus cleaning may remove acronyms, slang, misspellings, phonemicized forms, and other anomalies before modeling.Tokenization is especially important for generative dialogue models because it defines the smallest input unit.
- A.1 Data Pre-processing: Automatic spelling correction is expected to reduce sparsity, but it must be inspected because it can introduce more errors than it fixes.For movie scripts, replacing likely errors using commonly misspelled Wikipedia words outperformed an automatic spelling corrector.
- A.1 Data Pre-processing: Researchers may remove transcriptional non-words from spoken-language corpora when those forms do not provide useful information to the dialogue system.Examples include forms such as “uh” and “oh”.
A.2 Segmenting Speakers and Conversations
Dialogue corpora may lack reliable speaker and conversation boundaries, so researchers use automatic or iterative segmentation methods to recover usable conversational structure.
- Movie-subtitle and chat corpora may lack explicit speaker labels or conversation boundaries, complicating models of speakers and conversations.Chat channels can also contain multiple simultaneous conversations without separators.
- Audio-based speaker diarisation can achieve classification error rates as low as 5%.
- Embedded training iteratively expands a small gold corpus with confident segmentations and retrains the segmentation model.
- Researchers estimate conversation segmentations automatically, including time-based thresholds for separating chat-forum exchanges.
A.3 Discriminative Model Architectures
Discriminative models predict labels or annotations associated with dialogue portions, providing supervised learning approaches for dialogue-related tasks.
- Discriminative models predict labels or annotations manually associated with a portion of a dialogue.Examples include user intent, topic, and specific information.
- The section reviews and contrasts dialogue-related discriminative models from a data-driven learning perspective.
A.3.1 DIALOGUE ACT CLASSIFICATION AND DIALOGUE TOPIC SPOTTING
Dialogue act classification assigns an utterance to one of K dialogue acts, while later approaches address scarce annotations by leveraging larger external text corpora.
- Dialogue act classification maps an isolated user utterance to one of K dialogue acts by estimating P(A | U).A is the dialogue-act variable and U is the user utterance.
- Early approaches trained separate n-gram language models for each dialogue act and used Bayes’ rule to classify new utterances.
- Annotated dialogue-act datasets are scarce, motivating methods that leverage word embeddings learned from substantially larger text corpora.
- Alternative data sources can improve accuracy, and limited feature hand-crafting makes dialogue act classification especially attractive for data-driven approaches.
A.3.2 DIALOGUE STATE TRACKING
Dialogue state tracking estimates the evolving state of a conversation from tracker inputs and external knowledge using statistical and discriminative architectures. These models can outperform baselines, but large state, observation, and action spaces make exact learning and inference intractable.
- DSTC state tracking outputs a probability distribution over predefined state hypotheses and REST from tracker inputs and external knowledge at each time step.
- Open datasets with accurate labels enable rigorous comparative evaluation of dialogue-system classification models.
- DSTC includes hand-crafted systems that aggregate marginal confidence scores through manually specified belief-state updates.For example, a score of 0.9 for today’s bus information updates the belief distribution over hypotheses.
- Dynamic Bayesian and POMDP approaches model latent dialogue states, observed tracker outputs, actions, and rewards while representing uncertainty in variables of interest.
- State tracking becomes intractable for large spaces because temporal dependencies resemble hidden Markov models, with some systems reaching 10^10 configurations.Simplifying assumptions and approximate inference are therefore necessary.
- Statistical state-tracking models can perform well against baseline systems under appropriate structural assumptions and approximations.
- Discriminative alternatives include feed-forward neural networks and conditional random fields, with CRFs modeling temporal dependencies among dialogue states.CRFs enforce coherence across predicted states, unlike feed-forward approaches.
- Discriminative models often require difficult-to-acquire labeled data, and isolated component evaluations cannot guarantee complete dialogue-system performance.
A.4 Response Generation Models
Response generation models either re-rank existing candidate answers or generate responses directly, enabling training from large unannotated dialogue corpora. Generative approaches can represent dialogue concepts beyond fixed annotation schemes.
- Response generation research includes re-ranking candidate answers and fully generative response models.
- Generative response models can learn semantically more powerful dialogue representations because their concepts are limited by dataset content rather than annotation schemes.
A.4.1 RE-RANKING RESPONSE MODELS
Re-ranking models select the most probable response from candidate answers, allowing systems to train on completely unannotated dialogue datasets. The approach has been applied to movie scripts, subtitles, and Ubuntu conversations.
- Re-ranking models output the highest-ranked candidate answer and require only dialogue sequences rather than utterance-level expert annotations.
- Movie-script and subtitle systems retrieve dialogues similar to the current conversation and emit the response from the closest match.
- The Ubuntu Dialogue Corpus model scores context-response pairs using separate context-specific and response-specific LSTMs.
- The affinity model maximizes the likelihood of correct context-response pairs and uses stochastic gradient descent for training.
- Given a context, the trained model selects an appropriate answer from potential responses while assuming corpus responses are appropriate.
A.4.2 FULL GENERATIVE RESPONSE MODELS
Full generative models compose dialogue responses word by word from dialogue history, using maximum-likelihood training and recurrent architectures. They can produce novel utterances, but generic responses remain a major limitation in some settings.
- Generative response strategies model user and system utterances as sequences of tokens conditioned on preceding dialogue history.
- The dialogue probability factorizes across utterances and words, with training based on maximum log-likelihood and stochastic gradient descent.
- Sordoni et al. trained a recurrent neural network on Twitter messages and combined it with machine translation, outperforming a state-of-the-art machine translation system.
- Serban et al. adapted hierarchical recurrent networks for movie subtitles and scripts, adding speech and dialogue acts to the vocabulary.
- Standalone neural generative models often produce generic responses because dialogue data is dominated by pronouns, punctuation, and common verbs.
- Wen et al. conditioned recurrent response generation on dialogue acts and used convolutional re-ranking to generate diverse utterances representing those acts.
- Unlike answer re-rankers, word-by-word generators can create utterances never seen in training and can scale response generation irrespective of dataset size.
A.5 User Simulation Models
User simulation models generate varied, consistent user behavior from fixed corpora and support reinforcement-learning dialogue development. They range from n-gram models to graphical and feature-based models, but limited history can distort user goals.
- User simulation models generate natural, varied, and consistent interactions from a fixed corpus for reinforcement-learning dialogue systems.
- Unlike full generative response models, user simulators model only user utterances conditioned on prior user and system behavior, usually at the dialogue-act level.
- Early n-gram simulators use tensor parameters learned from observation likelihoods or smoothed objectives to model user behavior.
- Because n must remain small despite smoothing, n-gram models cannot preserve user history and goals across many utterances.
- Goal-conditioned n-gram models require hand-crafted slot-value pairs and expert-defined goal distributions during simulation.
- Feature-based and graphical models summarize prior interactions, represent changing goals and knowledge, and generate utterances through ancestral sampling.
- User simulation models and full generative models have developed independently, leaving opportunities to transfer and merge ideas between them.