Source-linked AI summary
Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization
Yen-Ju Lu, Kunxiao Gao, Mingrui Liang, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba
TL;DR
Spoken dialogue summarization lacks corpora linking audio, summaries, and paralinguistic cues. Spoken DialogSum constructs such a corpus from rewritten DialogSum scripts and synthesized speech, and raw-speech Audio-LLMs improve emotional-summary ROUGE-L over a cascaded baseline.
Problem
Spoken dialogue summarization lacks corpora linking audio, summaries, and paralinguistic cues, while existing benchmarks often separate tasks and omit semantic-acoustic interaction.
Source: §1 · p. 1Method
Spoken DialogSum rewrites DialogSum scripts with conversational features, assigns emotion and prosodic labels, and synthesizes speech paired with factual and emotion-rich summaries.
Source: §1 · p. 1Results
28%: raw-speech Audio-LLMs improve emotional-summary ROUGE-L over a cascaded ASR+LLM pipeline.
Source: §7 · p. 9Takeaways & Limitations
The benchmark probes dialogue understanding through text-only factual summaries, emotion-rich summaries combining lexical and acoustic cues, and acoustic-only speaker-attribute prediction.
Source: §7 · p. 8The dataset is intended solely for academic speech and language research, and the authors warn against demographic profiling or surveillance uses of paralinguistic classifiers.
Source: Ethics Statement · p. 9
Abstract
from arXiv · showhide
Recent audio language models can follow long conversations. However, research on emotion-aware or spoken dialogue summarization is constrained by the lack of data that links speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels for speaker age, gender, and emotion. The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary. We release an online demo at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/, with plans to release the full dataset in the near future. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end speech modeling.
1. Introduction
Spoken DialogSum addresses the shortage of spoken-dialogue summarization data linking audio, summaries, and paralinguistic cues. It converts DialogSum scripts into annotated speech dialogues and benchmarks factual summarization, emotion-rich summarization, and speaker-attribute classification.
1. Introduction
- Existing benchmarks often target a single task, while spoken-dialogue summarization lacks data linking speech, summaries, and paralinguistic information.Text-based dialogue datasets rely on written transcripts, and the cited DialogSum scripts lack backchannels and disfluencies.Source: §1 · p. 1
- The pipeline rewrites DialogSum scripts with natural disfluencies and backchannels, then assigns emotion and synthesizes emotion-rich speech.It also generates an emotion-focused summary alongside the primary summary.Source: §1 · p. 1
- Spoken DialogSum pairs raw multi-speaker audio with factual and emotion-rich summaries and utterance-level emotion, gender, and age labels.Its three benchmark tasks cover text-only factual summarization, cross-modal emotion-rich summarization, and acoustic-only attribute classification.Source: §1 · p. 1
- 29%: the Audio-LLM improves emotion-rich summarization ROUGE-L over the cascaded ASR→LLM baseline, supporting joint semantic and acoustic modeling.The evaluation uses emotion-rich references derived from speech emotion labels.Source: §1 · p. 2
2. Related Work
Prior work includes text-based dialogue summarization, spoken corpora annotated for prosody, and methods for synthesizing more interactive dialogue. These lines of work establish distinct dataset and synthesis capabilities.
2.1. Text-based Dialogue Summarization
- Text-based dialogue benchmarks include SAMSum and DialogSum, while MediaSum and SummScreen extend the scale and settings of transcript-based summarization.SAMSum contains 16K messenger-style dialogues; MediaSum contains 463K media-interview transcripts.Source: §2.1 · p. 2
2.2. Spoken Dialogue Corpora with Prosodic Information
- Spoken corpora support prosodic analysis through annotations such as intonation, disfluencies, dialogue acts, pauses, emphasis, overlap, and tone-unit markings.AMI is a small-scale meeting-summarization benchmark with fewer than 300 noisy, overlapping recordings.Source: §2.2 · p. 2
2.3. Conversational Dialogue Synthesis
- Dialogue-synthesis work adds spontaneous phenomena and listener responses through style-transfer TTS, backchannel prediction, and synthetic datasets such as Behavior-SD.The cited approaches also model response timing and type, and some incorporate speaker personality and topic.Source: §2.3 · p. 2
3. Realistic Spoken Dialogue Data Generation
Spoken DialogSum is generated through a three-stage conversion of dialogue text: style transfer, backchannel insertion, and emotion assignment.
3.1. Rich Text Dialogue Generation
Rich dialogue generation preserves alignment with the original script while adding Switchboard-style conversational features and overlapping backchannels. Model-based evaluation reports improved conversational realism alongside a slight coherence reduction.
3.1.1. Style Transfer
- Style transfer uses Switchboard samples to insert similar fillers and hesitations into scripted DialogSum dialogues, making them more natural and interactive while keeping them aligned with their original summaries.The style transfer uses a pre-trained instructed LLAMA3.3 70B model.Source: §3.1.1 · p. 3
3.1.2. Backchannel Insertion
- Backchannels are inserted as overlapping mid-turn interruptions, with Switchboard examples guiding varied and natural responses.The overlap is intended to make the dialogues more realistic and also increases the difficulty for models to understand them.Source: §3.1.2 · p. 3
3.1.3. Dialogue Evaluation
- Spoken DialogSum scores 4.81 in Oral Naturalness versus 3.86 for DialogSum and 4.25 for Switchboard; average scores are 4.48, 4.19, and 4.02, respectively.Natural backchannels are attributed with making dialogues sound more interactive and human-like; coherence and focus decrease slightly from 4.59 to 4.49 versus DialogSum.Source: Table 2 · p. 3, §3.1.3 · pp. 3–4
3.2. Spoken Dialogue Generation
The pipeline builds expressive multi-speaker dialogue audio through speaker-bank construction, utterance-level style annotation, conditional TTS, and timing-driven overlap placement.
3.2.1. Speaker bank construction
- The speaker bank uses GigaSpeech annotations for age, gender, pitch, expressiveness, and speaking rate.Pitch level is based on speaker-level mean pitch relative to gender; utterance-level pitch standard deviation serves as a proxy for monotone or animated delivery.Source: §3.2.1 · p. 4
3.2.2. Emotion Assignment
- Emotion labels, pitch variation, and speaking rate are assigned per utterance and used to condition the TTS model.GPT-4o-mini assigns one of eight emotion labels and estimates pitch standard deviation and speaking rate, discretized into three levels.Source: §3.2.2 · p. 4
3.2.3. Conditional TTS model
- Zonos-hybrid synthesizes speech conditioned on speakers randomly selected from a 20,385-voice bank, with filtered prompts and adjusted prosodic inputs.Prompts are longer than five seconds and drawn from audiobook or podcast recordings; the pipeline also slows and pads brief backchannels.Source: §3.2.3 · pp. 4–5
3.2.4. Timing-driven utterance placement
- The pipeline places interruptions before the host turn ends and backchannels at the next speaker’s turn, using CANDOR timing patterns and silence padding.Interruptions are placed 1.5 seconds before the host turn ends; backchannels are placed at the start of the following speaker’s turn.Source: §3.2.4 · p. 5
4. The Spoken DialogSum Datasets
Spoken DialogSum pairs multi-speaker audio with factual and emotion-rich summaries, and reports human ratings and corpus comparisons.
Dataset statistics
- 13,460 dialogues and 251,575 utterances span roughly 160 hours, with both concise and emotion-rich summaries for each dialogue.The reported emotion labels include 45.72% Neutral and 32.3% Happiness; over 40% of utterances are labeled Happiness or Sadness.Source: §4 · p. 5
Human evaluation
- 3.78 was Spoken DialogSum’s overall human-evaluation average, second to MELD’s 4.12, while its sound-quality score exceeded Switchboard and MELD.Sound quality was 3.89 for Spoken DialogSum, 2.88 for Switchboard, and 3.58 for MELD.Source: §4 · p. 6, Table 5 · p. 6
Dataset comparison
- Spoken DialogSum offers about 160 hours of audio, far more than MELD’s 12 and DailyTalk’s 20, plus per-utterance pitch-standard-deviation and speaking-rate labels.The paper describes it as the first large-scale spoken-dialogue corpus with emotion-rich summaries; MELD has no available summaries.Source: §4 · p. 6
Dataset contribution
- The authors present Spoken DialogSum as suited to emotion summarization and related large-scale spoken-dialogue tasks.The corpus combines large scale, style annotations, and dedicated emotion-focused summaries.Source: §4 · p. 6
5. Experimental Setup
The setup tests semantic summarization, acoustic attribute prediction, and summarization that combines lexical meaning with vocal affect.
5. Experimental Setup
- The three tasks cover transcript-based factual summarization, audio-only speaker-attribute classification, and emotion-rich summarization integrating speech meaning and vocal affect.The authors characterize them as a continuum from text-only reasoning to multimodal fusion and audio-only interpretation.Source: §5 · p. 6
5.1. Baseline Models
- Baselines include transcript-only LLaMA-2, cascaded Whisper plus LLaMA-2, end-to-end Audio-LLMs, and a wav2vec 2.0-based age-and-gender classifier.The end-to-end systems include WavLLM, Qwen-Audio-Chat, Audio-Flamingo3, LTU-AS, and SALMONN.Source: §5.1 · pp. 6–7
5.2. Evaluation Framework
- Factual summaries are scored with ROUGE and BERTScore against three references; emotion-rich summaries describe each speaker’s affect in one sentence.Attribute prediction covers age group, gender, and conversation-level emotion, reporting accuracy and weighted F1; text-only and cascaded models are excluded.Source: §5.2 · p. 7
6. Results
Results distinguish semantic-only summarization from emotion-rich summarization and acoustic attribute prediction; rankings vary across those tasks.
6.1. Dialogue and Emotion-Rich Summarization Results
- SALMONN-13B scores best overall on emotion-rich summarization, with WavLLM close behind, while transcript-only LLaMA-2 and its cascaded variant lead on factual summarization.Text-only baselines slump on emotion-rich summarization; cross-modal models show mixed gains.Source: §6.1 · p. 8
6.2. Paralinguistic Attribute Prediction
- 66.3 Acc and 65.2 F1 for age, and 95.4 Acc/F1 for gender, were Wav2Vec2’s scores; LTU-AS led WavLLM in emotion accuracy on both benchmarks.For emotion accuracy, LTU-AS scored 49.1 versus WavLLM’s 42.5 on IEMOCAP, and 47.8 versus 45.8 on EmoSum.Source: §6.2 · p. 8, Table 9 · p. 8
7. Conclusion
Spoken DialogSum benchmarks dialogue understanding across factual, emotion-rich, and acoustic-only tasks using 13,460 synthesized conversations. Baselines show gains for raw-speech Audio-LLMs on emotional summaries and Wav2Vec 2.0 classifiers on age and gender prediction.
7. Conclusion
- The benchmark covers text-only factual summarization, emotion-rich summarization from lexical and acoustic cues, and acoustic-only prediction of speaker emotion, gender, and age.It contains 13,460 dialogues (∼165 h) built from DialogSum scripts with Switchboard-style back-channels and expressive conditional-TTS audio.Source: §7 · pp. 8–9
- 28% higher ROUGE-L for emotional summaries with raw-speech Audio-LLMs than with a cascaded ASR+LLM pipeline.Human evaluations also found higher naturalness and emotion consistency than prior synthetic dialogue corpora.Source: §7 · p. 9
- Wav2Vec 2.0–based classifiers show strong gains in utterance-level age and gender prediction.The conclusion reports this result without giving numerical values.Source: §7 · p. 9