Source-linked AI summary
DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, Shuzi Niu
TL;DR
Dialogue research lacks high-quality datasets that capture realistic daily conversations rather than noisy, dispersed, or task-specific interactions. The paper develops and manually annotates DailyDialog, then evaluates retrieval- and generation-based approaches; the dataset covers ten topics, realistic dialog flows, and rich emotion, although its emotion labeling follows an acknowledged limitation of the BigSix framework.
Problem
Existing dialogue datasets are often limited by small task-specific collections, dispersed long conversations, or noisy social-media exchanges unlike real conversations.
Method
The paper develops DailyDialog, a high-quality multi-turn daily-life dataset with manual communication-intention and emotion labels, and evaluates retrieval- and generation-based approaches.
Results
DailyDialog covers ten topics, realistic dialog-flow patterns, and rich emotion, while evaluation results are described as initial but indicative.
Takeaways & Limitations
DailyDialog provides a manually labeled resource reflecting realistic everyday communication for dialogue-system research.
Takeaways & Limitations
The authors note that imbalanced emotion categories may make the BigSix-based emotion labeling inappropriate, but retain it for consistency with previous work.
Abstract
from arXiv · showhide
We develop a high-quality multi-turn dialog dataset, DailyDialog, which is intriguing in several aspects. The language is human-written and less noisy. The dialogues in the dataset reflect our daily communication way and cover various topics about our daily life. We also manually label the developed dataset with communication intention and emotion information. Then, we evaluate existing approaches on DailyDialog dataset and hope it benefit the research field of dialog systems.
1 Introduction
DailyDialog addresses limitations in existing dialogue datasets by providing high-quality, human-written multi-turn conversations about daily life, manually labeled for communication purposes and emotion. The paper motivates the dataset through realistic conversational flow and emotional expression, then evaluates established dialogue approaches on it.
- Dataset gap: Existing dialogue datasets are often small and task-specific, overly dispersed across long conversations, or noisy and unlike real conversations.The paper identifies these limitations in traditional domain-specific datasets, Switchboard, OpenSubtitles, Twitter Dialog Corpus, and Chinese Weibo.
- Dataset contribution: DailyDialog is a high-quality multi-turn dataset containing conversations about everyday life.
- Communication patterns: Daily conversations commonly exchange information through context-dependent responses, questions, and suggestions rather than isolated question-answer turns.The paper uses this interaction pattern to motivate studying realistic dialog flow.
- Communication patterns: Daily conversations are rich in emotion because emotional expressions communicate respect, empathy, and understanding between speakers.
- Paper scope: The paper describes dataset construction and annotation, analyzes its characteristics, and evaluates retrieval-based and generation-based approaches.
2 Dataset Construction
DailyDialog is constructed from English-learning websites and refined into human-written, focused, reasonably sized conversations that represent everyday communication. The dataset is manually annotated with dialog acts and emotions, using expert annotation and agreement checking.
- Data collection: DailyDialog is crawled from websites for practicing English dialogue in daily life and is designed to preserve human-written conversational properties.
- Data characteristics: DailyDialog conversations focus on a particular topic and physical context, including service interactions and personal discussions.
- Data characteristics: Approximately 8 speaker turns per conversation make DailyDialog more suitable for training compact conversational models than datasets with 150+ or 1,000+ turns.
- Data processing: 13,118 multi-turn dialogues remain after deduplication, filtering out conversations with more than two speakers, and automatic spelling correction.The dataset averages roughly 8 speaker turns and about 15 tokens per utterance.
- Annotation: The dataset is manually labeled to represent communication intentions and emotions underlying information exchange and social bonding.
- Annotation: Three experts annotated the data after training and discussion, achieving 78.9% inter-annotator agreement across the whole dataset.Disagreements were handled through majority voting or reannotation to find a common annotation.
3 Characteristics
DailyDialog is designed to reflect everyday communication through broad topics, natural dialog flows, manually labeled intentions and emotions, and distinctive multi-turn patterns. Its statistics document topic coverage, dialog-act structure, emotional content, and a limitation of the adopted emotion-labeling scheme.
- 3.4 Rich Emotion: DailyDialog contains rich emotions that are manually labeled to improve annotation accuracy and distinguish it from most existing dialogue datasets.The paper links emotional content to social bonding and summarizes the resulting emotion statistics in Table 3.
- 3.1 Daily Topics: DailyDialog covers ten daily-life topic categories, led by Relationship (33.33%), Ordinary Life (28.26%), and Work (14.49%).The dataset includes scenarios such as holidays, tourism, shops, restaurants, and other everyday activities.
- 3.2 Bi-turn Dialog Flow: Its dialog-act flows include common Questions-Inform and Directives-Commissives exchanges that model information transfer, suggestions, and commitments.The dataset contains more than ten thousand examples of approximately 8-turn dialog-act flows.
- 3.3 Certain Communication Pattern: Two distinctive multi-turn patterns occur in 2,398 (18.3%) and 1,203 (9.2%) dialogues, respectively.The first pattern combines answering with initiating a new question; the second involves successive proposals followed by a commitment-like response.
- 3.4 Rich Emotion: The authors note that imbalanced emotion categories may make labeling according to the BigSix Theory improper, but retain the scheme for consistency with prior work.Developing a novel emotion theory is outside the scope of this work.
- 3.4 Rich Emotion: The dataset includes 3,675 (28.0%) dialogues ending with happy emotions and 113 (0.8%) that shift from negative to positive emotions.The authors present these statistics as motivating future research on conversational agents that regulate conversations toward happy endings.
4 Evaluating Existing Approaches
The paper evaluates retrieval- and generation-based dialogue approaches on DailyDialog, including variants using intention, emotion, attention, hierarchical context, labels, and pretraining. Results indicate that labels and contextual modeling help, while out-of-domain pretraining lowers BLEU despite improving perplexity.
- Compared Approaches: The evaluation compares embedding-, feature-, and reranking-based retrieval methods alongside neural response-generation models.Retrieval variants include intention and emotion reranking; generation variants include Seq2Seq, attention, HRED, label-enhanced, and pretrained models.
- Compared Approaches: “Equivalence” percentages show only subtle improvements with labels and are not considered a reliable indicator of response coherence or suitability.The authors caution that higher label equivalence does not necessarily mean a better retrieved response.
- Intention And Emotion Matters: Intention and emotion reranking produces more suitable retrieved responses than feature-based retrieval without reranking in the reported case studies.The intention-matched response follows the test dialog-act flow, while intention-and-emotion reranking selects a response aligned with the emotion history.
- Compared Approaches: HRED achieves the highest BLEU scores among the compared generation approaches, while attention-based models generally outperform vanilla Seq2Seq.The authors attribute HRED’s advantage to its use of dialogue history.
- Compared Approaches: Label information improves generation performance even when incorporated through the paper’s simplest label-enhanced design.The finding is reported as consistent with previous work.
- Compared Approaches: OpenSubtitle pretraining lowers perplexity but produces poorer BLEU scores and can generate context-irrelevant responses on DailyDialog.The paper conjectures that domain and conversation-length differences explain the weaker target-domain performance.
5 Related Work
Prior dialogue research uses task-specific, social-media, movie, forum, and open-domain datasets with retrieval- or sequence-to-sequence generation methods. These resources differ substantially in domain, conversational structure, and noise, motivating broader dataset development.
- Traditional and Task-Oriented Systems: Traditional dialogue systems primarily rely on template-based or retrieval-based approaches trained for specific domains and tasks.Examples include TRAINS, DBOX, bAbI, Movie Dialog, and Ubuntu datasets.
- Neural Dialogue Generation: More recent systems concentrate on sequence-to-sequence encoder-decoder generation trained with large-scale open-domain datasets.Such datasets include social-network conversations, Reddit posts, and movie subtitles.
- Neural Dialogue Generation: Social-network dialogue datasets are constructed from posts and replies, while movie-subtitle corpora are widely used for pretraining conversational models.The related work discusses Sina Weibo, Twitter, Reddit, OpenSubtitle, and related movie datasets.
6 Conclusions and Future Work
DailyDialog is presented as a high-quality, manually labeled, multi-turn dataset covering realistic topics, dialog flows, and emotions. The authors view its evaluation results as initial but indicative and identify future work on dialog-flow mechanisms and topic-based transfer.
- Conclusions: DailyDialog covers ten topics, common dialog flows, unique multi-turn patterns, and rich emotional content.These properties are presented as reflecting realistic everyday communication.
- Conclusions: The evaluation results are described as initial but indicative of the dataset’s research value.The paper does not present the evaluation as definitive.
- Future Work: Future work will explore advanced mechanisms for unique multi-turn flows and use topic information through domain adaptation and transfer learning.The dataset is made available for future research.