Source-linked AI summary
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, Aleksander Wawer
TL;DR
Abstractive dialogue summarization lacked a comprehensive dataset and evaluation approach suited to multi-speaker chat. The paper constructs and evaluates the SAMSum Corpus, finding that human judgments diverge from ROUGE-based comparisons and motivating dialogue-specific models and metrics.
Problem
Dialogue summarization lacks a comprehensive, high-quality corpus, while standard single-speaker summarization tools and metrics may not suit multi-speaker conversations.
Method
The paper introduces SAMSum, manually annotated chat dialogues, and evaluates general-purpose summarization models using automated ROUGE and human evaluation.
Results
Human evaluators rate dialogue summaries worse than news summaries despite dialogue models achieving higher ROUGE scores than news models.
Takeaways & Limitations
Dialogue summarization requires dedicated models that account for speaker assignments and a new metric beyond ROUGE.
Takeaways & Limitations
ROUGE is more indicative of summary quality for news than for dialogues, so a new dialogue-summary metric is needed.
Abstract
from arXiv · showhide
This paper introduces the SAMSum Corpus, a new dataset with abstractive dialogue summaries. We investigate the challenges it poses for automated summarization by testing several models and comparing their results with those obtained on a corpus of news articles. We show that model-generated summaries of dialogues achieve higher ROUGE scores than the model-generated summaries of news -- in contrast with human evaluators' judgement. This suggests that a challenging task of abstractive dialogue summarization requires dedicated models and non-standard quality measures. To our knowledge, our study is the first attempt to introduce a high-quality chat-dialogues corpus, manually annotated with abstractive summarizations, which can be used by the research community for further studies.
1 Introduction and related work
Abstractive dialogue summarization lacked a comprehensive, high-quality dialogue corpus comparable to news datasets. The paper addresses this gap by introducing SAMSum, a freely available corpus of over 16k chat dialogues with manually annotated summaries.
- Single-speaker documents such as news and scientific publications have dominated summarization research because large annotated news datasets are available.
- Existing dialogue resources were limited, including AMI’s 141 dialogues and the Argumentative Dialogue Summary Corpus’s 45 dialogues.
- SAMSum contains over 16k chat dialogues with manually annotated summaries and is freely available to the research community.
2 SAMSum Corpus
SAMSum was constructed as a large, messenger-like dialogue corpus rather than adapted from existing datasets. Linguists created and summarized the conversations, which were then validated and cleaned.
- Existing chatbot, SMS, IRC, movie, meeting, and other dialogue datasets were rejected because they differed from typical messenger conversations.
- Linguists created natural messenger-like conversations spanning informal, semi-formal, and formal styles, including slang, emoticons, typos, and diverse topics.
- Language experts annotated each dialogue with one short, third-person summary that extracts important information and includes interlocutors’ names.
- Two linguists validated 50 conversations for messenger plausibility and summarizability, while semiautomatic cleaning corrected structural deviations and flagged name typos for review.
- The corpus contains 16,369 conversations grouped by utterance count, with about 75% involving two interlocutors.
3 Dialogues baselines
The paper evaluates simple dialogue baselines adapted from Lead-n summarization. Lead-3 performs better than MIDDLE in this corpus, although the best dialogue baseline is not identified in the supplied passage.
- Lead-3 summarizes a document using its three leading sentences, assuming that the beginning contains the most significant information.
- The proposed simple baselines select middle utterances, longest utterances, utterances exceeding a length threshold, or utterances from the most active speaker.
- Lead-3 was expected to perform poorly because conversations often begin with greetings, but greetings frequently contain questions or information in this dataset.
- Lead-3 outperformed the MIDDLE baseline in the reported evaluation, while the supplied passage only states that another dialogue baseline performed best.
4 Experimental setup
The experiments train and evaluate several summarization architectures on dialogue data, with an additional joint-training strategy using CNN/Daily Mail. They also test speaker separators and use ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.
- 4.1 Data preparation: Each architecture is trained and evaluated on dialogue data, or trained jointly on CNN/Daily Mail and dialogue training data before dialogue-test evaluation.
- 4.1 Data preparation: A dialogue variant inserts special separator tokens between utterances to mark their boundaries.
- Models use beam size 5, while inputs are truncated to 400 tokens and summaries to 100 tokens.
- 4.2 Models: The evaluated architectures include Pointer Generator, Transformer, Fast Abs RL, Fast Abs RL Enhanced, LightConv, and DynamicConv.
- 4.2 Models: Fast Abs RL Enhanced appends the names of other interlocutors after each utterance to provide speaker information during generation.
- Evaluation reports stemming-based F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L.
5 Results
Across dialogue and news summarization, model performance varies with architecture, training data, pretrained embeddings, and utterance separation. DynamicConv with GPT-2 embeddings trained on joined news and dialogue data with separation tokens achieves the best ROUGE performance for dialogues.
- ROUGE-1 exceeds 39, ROUGE-2 exceeds 17, and ROUGE-L exceeds 36 for the best models in both domains.
- All tested dialogue models outperform the LONGEST-3 baseline, whereas only one model surpasses the Lead-3 news baseline on all three metrics.
- Transformer-based architectures benefit from joint news-and-dialogue training, unlike the Pointer Generator and Fast Abs RL models.
- Adding utterance separation tokens improves most models, with the largest gains occurring under joint training.
- Enhancing Fast Abs RL utterances with the other interlocutors’ names produces higher ROUGE values.
- GPT-2 pretrained embeddings yield the largest performance improvements for LightConv and DynamicConv.
- Dialogue-only models reach ROUGE levels similar to or better than models trained on the much larger CNN/Daily Mail corpus.
- DynamicConv with GPT-2 embeddings, joint training, and utterance separation is the best-performing model according to ROUGE.
6 Linguistic verification of summaries
The paper evaluates ROUGE against human judgments because n-gram overlap may be inadequate for abstractive summaries. Human evaluation shows that high ROUGE scores for dialogue summaries can coincide with poor perceived quality, and ROUGE aligns better with human ratings for news.
- ROUGE evaluates generated summaries by comparing them with reference summaries using n-gram overlap.
- Two linguists rated 150 news summaries and 100 dialogue summaries on a scale from −1 to 1.
- Annotator agreement was 0.371 for news and 0.506 for dialogues using linear weighted Cohen’s kappa.
- Dialogue summaries can receive high ROUGE scores while human annotators mark their outputs as poor.
- ROUGE corresponds with human-assessed quality better for news summaries than for dialogue summaries.
7 Difficulties in dialogue summarization
Dialogue summarization is difficult because relevant information is distributed across speakers and utterances, while chats also contain pronouns, slang, typos, and conversational structure. Models consequently struggle with speaker roles, context, and selecting the most important information.
- Important information is scattered across discussions, questions, and greetings among different speakers, unlike the clearer flow of structured news articles.
- Pronouns, shortened words, slang, and typos further complicate the interpretation of messenger conversations.
- Models frequently confuse interlocutors’ names or reverse who performs and receives an action.For example, one model assigned the earplug purchase to Randolph instead of Maya.
- Fast Abs RL often mishandles arrangements because it selects important utterances before summarizing them separately, narrowing the available context.
- Models also disagree about which dialogue information deserves emphasis, sometimes selecting a different detail from the reference or overemphasizing an unimportant phrase.
8 Discussion
The paper introduces a dialogue-summarization dataset and compares general-purpose models on dialogue and news summarization using ROUGE and human evaluation. Results show that ROUGE favors dialogue summaries, while human evaluators prefer news summaries, motivating dialogue-specific metrics and models.
- The paper introduces a dataset for abstractive dialogue summarization and compares it with news summarization using automated ROUGE and human evaluation.
- Existing summarization tools and quality metrics were developed primarily for single-speaker documents and may not suit multi-speaker conversations.
- General-purpose models receive worse human evaluations on dialogue summaries than on news summaries because dialogue information is distributed across complex, noisy interactions.
- ROUGE scores suggest that dialogue summarization models outperform news models, but independent manual analysis shows that this conclusion is false.
- News summaries receive higher human scores despite lower ROUGE values, indicating that ROUGE is more informative for news than dialogue and that a new dialogue metric is needed.
9 Conclusions
The paper addresses the lack of a proper dataset for abstractive dialogue summarization by creating a comprehensive resource, while identifying challenges requiring dedicated tools and non-standard quality measures.
- The study addresses the lack of a proper dataset for abstractive dialogue summarization and presents a comprehensive resource for future research.The authors describe creating the first comprehensive resource of this type, while noting that longer, multi-topic dialogues could make future datasets more challenging.
- Future work could create datasets with longer dialogues spanning multiple topics rather than covering only one topic.
- Dialogue summarization may require dedicated tools and non-standard measures to capture abstractive summary quality relevantly.