Source-linked AI summary
DialogSum: A Real-Life Scenario Dialogue Summarization Dataset
Yulong Chen, Yang Liu, Liang Chen, Yue Zhang
TL;DR
Dialogue summarization has been comparatively understudied because suitable datasets are limited, despite its relevance to real-life conversations. The paper introduces DIALOGSUM, a large-scale spoken-dialogue summarization dataset, and empirically evaluates neural summarizers. Results show that DIALOGSUM is highly abstractive and relatively challenging, with difficulties involving discourse structure and coreference-related bookkeeping.
Problem
Dialogue summarization has received less attention partly because suitable summarization datasets built on dialogue texts are scarce.
Method
The paper constructs DIALOGSUM from multiple spoken-dialogue sources, preprocesses the data into a unified format, obtains observer-perspective summaries, labels topics, and evaluates neural summarization models.
Results
DIALOGSUM is highly abstractive and relatively more challenging than existing summarization datasets, with difficulties arising from multi-turn discourse structures and bookkeeping of entities and events.
Takeaways & Limitations
Dialogue summarization requires attention to discourse relations, coreferences, ellipsis, interlocutor intents, and other characteristics of spoken dialogue.
Takeaways & Limitations
Model performance decreases as the number of dialogue turns grows.
Abstract
from arXiv · showhide
Proposal of large-scale datasets has facilitated research on deep neural models for news summarization. Deep learning can also be potentially useful for spoken dialogue summarization, which can benefit a range of real-life scenarios including customer service management and medication tracking. To this end, we propose DialogSum, a large-scale labeled dialogue summarization dataset. We conduct empirical analysis on DialogSum using state-of-the-art neural summarizers. Experimental results show unique challenges in dialogue summarization, such as spoken terms, special discourse structures, coreferences and ellipsis, pragmatics and social common sense, which require specific representation learning technologies to better deal with.
1 Introduction
Dialogue summarization has lagged behind monologic summarization because suitable datasets are scarce, while real-life conversations differ from online chats in style, topics, and discourse demands. DIALOGSUM addresses this gap with a large-scale dataset of spoken dialogues and shows that dialogue summarization is highly abstractive and comparatively challenging.
- Dialogue summarization has received less research attention partly because suitable dialogue summarization datasets are scarce.
- SAMSum contains short online chats, whereas real-life dialogues are more formal and cover more diverse task-oriented scenarios such as business negotiation.SAMSum conversations average around 94 tokens and focus largely on leisure and social chats.
- DIALOGSUM supports applications such as customer-needs analysis, personal record management, and training large neural dialogue summarization models.The dataset contains substantially more dialogues than AMI, and its dialogues are 39.8% longer on average than SAMSum dialogues.
- DIALOGSUM is a large-scale dialogue summarization dataset built from spoken dialogues spanning daily-life topics including schooling, work, medication, shopping, and travel.The data come from three public dialogue corpora and an English speaking practice website; annotators summarize dialogues from an observer perspective and label topics manually.
- Experiments find that DIALOGSUM is more amenable to abstractive summarizers but relatively more challenging than existing summarization datasets.The main difficulties involve multi-turn discourse structures and tracking entities and events across utterances.
2 The DIALOGSUM Dataset
DIALOGSUM combines diverse spoken-dialogue sources with structured preprocessing, annotation criteria, and quality checks. Its analyses characterize the dataset as longer and more abstractive than online-chat dialogue, while pretrained models achieve the strongest results.
- Data Collection: DIALOGSUM draws on DailyDialog, DREAM, MuTual, and an additional English-speaking-practice website, covering varied daily-life dialogue sources.The source datasets include 13k, 6k, and 9k dialogues or transcripts, respectively.
- Dialogue Characteristics: The dialogues are multi-turn, task-oriented, and longer than chitchats, containing more events and discourse relations that complicate summarization.Reported average dialogue lengths are 118.8 tokens for DailyDialog, 124.6 for DREAM, and 136.1 for MuTual.
- Data Cleaning and Pre-Processing: Preprocessing removes non-English characters, corrects errors, deduplicates examples, and converts different source formats into biturn dialogue flows.Continuous turns by the same speaker are merged during preprocessing.
- Annotation: Annotators produce brief, salient, observer-perspective summaries that preserve named entities, with additional guidance on tense, discourse relations, emotion, and speaker intent.Summaries should be no longer than 20% of the conversation length and use formal language.
- Annotation: DIALOGSUM uses speaker tags when real names are unavailable and assigns each dialogue a short topic label.The topic is approximately three tokens long.
- Quality Control: Three independently written test summaries support agreement analysis: ROUGE-2 is relatively low while ROUGE-L is high, indicating variable wording but similar content and ordering.Summaries are cross-checked, and 10% of summaries are manually sampled for further quality control.
3 Experiments
Experiments compare abstractive and extractive summarization models across dialogue and news datasets to characterize DIALOGSUM. Results show that pretrained models perform best, while DIALOGSUM is more abstractive, challenging, and distinct from SAMSum.
- Non-pretrained abstractive models outperform LEAD, while pretrained models achieve the best results despite pretraining on monologic texts.
- Spoken vs Written: All three models perform better on dialogue datasets than on XSum, while pretrained-model gains are smallest on DIALOGSUM: BARTLARGE improves R1 by 11.37% there.The authors relate the smaller gain to DIALOGSUM's more spoken style and to differences between dialogue and monologue.
- DIALOGSUM vs SAMSum: DIALOGSUM is more abstractive and challenging than existing dialogue datasets, with lower extractive and abstractive-model performance than SAMSum.Its longer inputs, lower compression rate, and spoken, open-domain character are identified as possible reasons for lower performance.
- DIALOGSUM vs SAMSum: Cross-dataset transfer causes substantial performance drops in both directions, while training on DIALOGSUM yields better performance and a smaller drop, suggesting greater generalization ability.
4 Human Evaluation
Human evaluations show that UNILMV2BASE generally outperforms Transformer on dialogue-specific capabilities and summary quality, but both remain below human performance. Models particularly struggle with discourse relations, coreference, speaker intents, and factual consistency.
- Fluency, Consistency, Relevance and Coherence: Human-annotated summaries receive the best scores across fluency, consistency, relevance, and coherence, while UNILMV2BASE consistently exceeds Transformer but remains lower than human performance.Model-generated summaries score highest on fluency and lowest on consistency, indicating grammatical output can still contain factual errors.
- Discourse Relation: Long-distance relations between events make discourse understanding difficult, and model performance decreases as dialogue turns increase.The evaluation highlights that important relations may be dispersed across multiple turns and require explicit connective choices in summaries.
- Discourse Relation: UNILMV2 achieves a better discourse-relation average score of −0.23 than Transformer’s −0.77, yet its −1 category remains dominant and performance is far from 1.On the discourse-relation subset, ROUGE scores decrease by 1.56 ∼ 3.26% for R1, 1.73 ∼ 3.22% for R2, and 2.37 ∼ 4.07% for RL relative to the full test set.
- Coreference Information: UNILMV2BASE preserves interlocutor identities and coreference better than Transformer, although its summaries remain inconsistent and far from human annotations.Transformer scores are mostly −1, whereas UNILMV2BASE has more 0 scores and an average closer to 0.
- Intent Identification: UNILMV2BASE identifies speakers’ intents more accurately than Transformer, but still performs below human-annotated summaries.Most Transformer summaries are scored −1 for intent identification, while UNILMV2BASE performs much better but remains below human performance.
5 Challenges in DIALOGSUM
DIALOGSUM exposes challenges specific to spoken, multi-turn dialogue: dispersed discourse relations, frequent coreference and ellipsis, interruptions, and pragmatics that require understanding speakers’ actions and intents.
- Error Analysis: Incorrect coreference is the most frequent error type for both models, while UNILMV2BASE reduces unfactual-information errors by 52% relative to Transformer.UNILMV2BASE performs better overall, but its error rates remain high.
- Discourse Structure: Spoken dialogue distributes important information across distinctive discourse structures, including closely related utterances separated by many turns.This structure differs from monologic text and appears in negotiations, procedures, medical consultations, and police reports.
- Spoken Dialogue Phenomena: LEAD outperforms LONGEST by over 3% on DIALOGSUM because greetings and farewells can express essential intents, consequences, and future plans.Interruptions can make utterances incomplete, add redundancy, and damage coherence.
- Coreference and Ellipsis: Frequent coreference and ellipsis require models to resolve omitted or indirect references, such as “my bill” referring to “#Person 2#’s bill”.These communication patterns save words and avoid repetition for humans but challenge neural models.
- Pragmatics and Social Common Sense: Pragmatics and social common sense require summaries to capture what speakers are doing, not only what they explicitly say.Expressions such as “Here you are” and “Goodbye” can imply payment and completion of checkout, respectively.
6 Conclusion
The paper presents DIALOGSUM as a large-scale real-life dialogue summarization dataset and empirically characterizes its abstractive nature and dialogue-specific challenges. It concludes that discourse and complex coreference remain important problems for future modeling.
- DIALOGSUM is a large-scale dialogue summarization dataset whose characteristics and challenges are investigated empirically.
- Experiments with typical models show that DIALOGSUM is highly abstractive and poses unique challenges in discourse and complex coreferences.
- The paper discusses the uniqueness of spoken dialogue summarization and lists key problems for future modeling.
7 Ethics Consideration
The dataset uses dialogue sources that are publicly accessible for academic use, and annotation involved trained linguistics-qualified annotators with compensation procedures.
- DIALOGSUM draws on DailyDialog, DREAM, MuTual, and an English speaking practice website whose dialogue sources are publicly accessible for academic use.
- Annotators with English Linguistics or Applied Linguistics degrees received training and were paid during training, with final compensation based on measured annotation time.
A Dialogue Topics
DialogSum dialogue topics are clustered using GloVe representations and k-means, with twenty clusters whose proportions are presented in Figure 3.
- The dialogue topics are clustered with k-means using GloVe embeddings, setting k = 20.
- Figure 3 presents the proportions of the resulting topic clusters.
- Table 12 lists the cluster topics and corresponding human-assigned identifiers.
B Dialogue Turns
The study evaluates model performance across dialogue-turn counts because longer dialogues involve more complicated information flow and discourse structure. Transformer and UNILMV2BASE performance decreases as the number of turns grows, indicating a challenge from longer interactions.
- The test data are split by dialogue-turn count using a step size of 3 to evaluate model performance across dialogue lengths.
- Transformer and UNILMV2BASE performance decreases as the number of dialogue turns grows.Figure 4 reports model performance against the number of dialogue turns; T denotes Transformer and UNI denotes UNILMV2.
- More interlocutor interactions and complicated discourse structures bring challenges for neural dialogue summarization models.The same phenomenon was observed for SAMSum by Chen and Yang (2020).
- Figure 3 presents dialogue-topic proportions, while Table 12 presents the corresponding topic-cluster information.