Source-linked AI summary
MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization
Chenguang Zhu, Yang Liu, Jie Mei, Michael Zeng
TL;DR
Public dialogue summarization datasets are limited in size and domain, motivating a larger resource based on media interviews. The paper constructs MediaSum from NPR and CNN transcripts with associated descriptions, analyzes interview positional bias, and shows transfer-learning gains on other dialogue summarization tasks.
Problem
Public dialogue summarization datasets are few, small, narrow-domain, or unavailable, limiting the available large-scale public evidence for dialogue summarization.
Method
The paper collects NPR and CNN interview transcripts with overviews or topic descriptions, segments CNN interviews to align topics, and analyzes dataset statistics and positional word patterns.
Results
MediaSum contains 463.6K transcripts and summaries, is an order of magnitude larger than existing corpora, exhibits interview-specific positional bias, and improves performance on other dialogue summarization tasks after fine-tuning.
Takeaways & Limitations
MediaSum provides a large, multi-party, multi-domain resource that can support dialogue summarization and transfer learning across related tasks.
Takeaways & Limitations
The transcripts and summaries may contain the biased views of the media and guests and should be viewed with discretion.
Abstract
from arXiv · showhide
MediaSum, a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries. To create this dataset, we collect interview transcripts from NPR and CNN and employ the overview and topic descriptions as summaries. Compared with existing public corpora for dialogue summarization, our dataset is an order of magnitude larger and contains complex multi-party conversations from multiple domains. We conduct statistical analysis to demonstrate the unique positional bias exhibited in the transcripts of televised and radioed interviews. We also show that MediaSum can be used in transfer learning to improve a model's performance on other dialogue summarization tasks.
1 Introduction
Dialogue summarization needs larger, more diverse public datasets than existing small or narrow-domain corpora provide. MediaSum addresses this gap with NPR and CNN interviews and supports transfer learning across dialogue summarization tasks.
- Existing public dialogue summarization datasets are few, small, narrow-domain, or unavailable because dialogue labeling and privacy create substantial barriers.AMI and ICSI contain only 137 and 59 meetings, respectively.
- Media interviews provide varied, colloquial, multi-party conversations with accompanying overviews or topic descriptions that can serve as summaries.
- MediaSum combines NPR overviews and CNN topic descriptions into a large-scale dialogue summarization dataset.The collection includes 49.4K NPR transcripts and 414.2K CNN transcript segments.
- 463.6K transcripts with summaries comprise MediaSum, which contains more speakers, longer conversations, and an order-of-magnitude greater scale than existing public datasets.
- Interview summaries exhibit positional bias: televised interviews emphasize summary keywords near the beginning, whereas radio interviews emphasize them near both ends.
- Fine-tuning benchmark summarization models on MediaSum improves performance on AMI, ICSI, and SAMSum dialogue summarization tasks.
2 Related Work
Dialogue summarization research has comparatively few datasets with summaries, and available public resources are often small, artificial, or domain-specific.
- Many dialogue datasets lack corresponding summaries because dialogue structure and language differ substantially from written articles.
- AMI and ICSI provide only 137 and 59 meeting transcripts with abstractive summaries, respectively.
- Existing resources span task-oriented, scripted, artificial, entertainment, customer-support, and medical domains, limiting public breadth.
3 Media Interview Dataset: MEDIASUM
MediaSum combines NPR and CNN interview transcripts with associated descriptions, using segmentation to align CNN topics with transcript segments. Its statistics show broad, multi-party conversations and distinctive positional patterns in televised and radio interviews.
- Data collection: The dataset construction uses NPR transcript overviews and CNN topic descriptions as summaries, with CNN interviews segmented at commercial-break boundaries for topic matching.
- Data collection: 414.2K CNN transcript segments and 463.6K combined NPR-CNN transcript-summary pairs remain after collection and filtering.Summaries shorter than five words are filtered out.
- Data collection: NPR summaries are much longer and more abstractive than CNN summaries despite similar dialogue lengths and speaker counts.
- Data statistics: The dataset’s top topic clusters are politics (26.3%), international news (13.3%), crime (12.7%), economy (12.5%), and US news (11.7%).
- Data statistics: MEDIASUM dialogues average 30.0 turns, 6.5 speakers, and 1,553.7 words, while summaries average 14.4 words.
- Data statistics: MediaSum contains longer, more multi-party, and broader-domain conversations than several public datasets, including radioed and televised interviews.
- Positional bias: Summary words concentrate near transcript beginnings in CNN and NPR, while NPR also shows many summary words near the end.
4 Experiments
The experiments evaluate benchmark summarization models on MEDIASUM and test whether MEDIASUM supports transfer to other dialogue summarization datasets.
- 4.1 Results on MediaSum: MEDIASUM is compared with other dialogue summarization datasets using average dialogue words, summary words, turns, and speakers.
- 4.1 Results on MediaSum: Benchmark models are applied to MEDIASUM using concatenated speaker-labeled transcript turns, with LEAD-3 included as a baseline.
- 4.1 Results on MediaSum: 10K instances are reserved for validation and another 10K for testing, with hyperparameters selected using validation ROUGE-L.
- 4.1 Results on MediaSum: LEAD-3 performs relatively weakly, while pretrained BART and UniLM outperform non-pretrained PTGen on MEDIASUM.
- 4.2 Transfer Learning: UniLM is fine-tuned on MEDIASUM and then trained on AMI, ICSI, and SAMSum for transfer evaluation.
- 4.2 Transfer Learning: Training on MEDIASUM improves performance on all three target datasets, demonstrating its potential as a transfer-learning dataset.
5 Conclusion
The conclusion presents MEDIASUM as a large-scale media interview dataset for dialogue summarization. It emphasizes its scale, multi-party and multi-domain conversations, and use for transfer learning.
- MEDIASUM contains 463.6K transcripts and summaries from NPR and CNN, with transcript segmentation used to align CNN topic descriptions to segments.
- The dataset is an order of magnitude larger than existing corpora and contains complex multi-party conversations from multiple domains.
- Table 5 illustrates a MEDIASUM dialogue and summary, with equal numbers of strings in the utterance and speaker fields.
- MEDIASUM can be used for transfer learning to improve model performance on other dialogue summarization tasks.
Ethics
The ethics discussion notes that the dataset uses publicly available media transcripts under research-only guidelines and may preserve the biases of media and guests.
- The dataset uses only publicly available transcripts from media sources and follows their research-only-use guideline.
- Transcripts and summaries may contain biased views because they reflect the perspectives of media and guests.
A Data statistics
MEDIASUM dialogues are generally substantial in length and commonly involve multiple speakers.
- Most MEDIASUM dialogues contain more than 500 words and involve 2 to 5 speakers.
B Topic analysis
MEDIASUM’s topic analysis uses LDA-derived word clusters and reveals a strong concentration in news-related themes, especially politics.
- Topic analysis: LDA identifies the main dialogue topics by clustering words and manually naming clusters from their top 10 words.The analysis uses scikit-learn’s LDA implementation.
D Implementation Details
The experiments specify model-specific optimization settings, dataset splits, and training procedures for reproducible evaluation.
- Model training: BART is trained for 10 epochs with learning rate 2 × 10^-5 and batch size 24.Beam search uses width 3, with generated summaries limited to 3–80 tokens.
- Model training: PTGen uses a 50,000-word vocabulary and a 512-dimensional LSTM encoder-decoder trained with Adagrad.Its learning rate is 0.1 and training lasts 10 epochs.
- Model training: UniLM is trained with Adam for 100,000 steps, including 2,000 warmup steps, at learning rate 1.5 × 10^-5.All experiments truncate inputs after 1,024 tokens and use 8 V100 GPUs.
- Dataset splits: The AMI and ICSI splits use 100/17/20 and 43/10/6 train/dev/test partitions, respectively.SAMSum follows the split reported by Gliwa et al. (2019).
E Results on partitions
The partition results compare CNN and NPR test performance, while the accompanying dataset analyses describe MEDIASUM’s dialogue distributions and topic organization.
- Partition results: Training on MEDIASUM improves NPR-partition ROUGE-L by 0.6% compared with training on NPR alone.The reported comparison concerns the NPR partition.
- Partition results: Results on the NPR partition are better than those on the CNN partition.
- Positional analysis: Figure 3 tracks non-stop topic-word frequency across normalized dialogue positions in original CNN transcripts with one topic.Most topic keywords occur at the beginning, with a slight increase near the end.
- Partition results: Table 7 reports ROUGE-1, ROUGE-2, and ROUGE-L F1 scores for CNN and NPR test partitions.Models use their corresponding partition training data, except UniLMCom, trained on all MEDIASUM.