Source-linked AI summary
DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, Michael Zeng
TL;DR
Long, multi-person dialogues are difficult for existing NLP systems to understand and summarize. DialogLM addresses this with window-based dialogue denoising and hybrid attention, and it surpasses previous state-of-the-art models across five benchmarks and three long-dialogue tasks.
Problem
Long multi-person dialogues are difficult for current NLP systems to process because they combine lengthy inputs with dialogic structure.
Method
DialogLM corrupts a multi-turn dialogue window with dialogue-inspired noise, reconstructs it from the remaining conversation, and combines sparse with global attention for longer inputs.
Results
DialogLM surpasses previous state-of-the-art models across five benchmarks covering dialogue summarization, abstractive question answering, and topic segmentation.
Takeaways & Limitations
The results support DialogLM as an effective pre-trained model for long-dialogue understanding and summarization across multiple tasks.
Abstract
from arXiv · showhide
Dialogue is an essential part of human communication and cooperation. Existing research mainly focuses on short dialogue scenarios in a one-on-one fashion. However, multi-person interactions in the real world, such as meetings or interviews, are frequently over a few thousand words. There is still a lack of corresponding research and powerful tools to understand and process such long dialogues. Therefore, in this work, we present a pre-training framework for long dialogue understanding and summarization. Considering the nature of long conversations, we propose a window-based denoising approach for generative pre-training. For a dialogue, it corrupts a window of text with dialogue-inspired noise, and guides the model to reconstruct this window based on the content of the remaining conversation. Furthermore, to process longer input, we augment the model with sparse attention which is combined with conventional attention in a hybrid manner. We conduct extensive experiments on five datasets of long dialogues, covering tasks of dialogue summarization, abstractive question answering and topic segmentation. Experimentally, we show that our pre-trained model DialogLM significantly surpasses the state-of-the-art models across datasets and tasks. Source code and all the pre-trained models are available on our GitHub repository (https://github.com/microsoft/DialogLM).
Introduction
DialogLM addresses the difficulty of understanding and summarizing lengthy, multi-person dialogues, which existing systems struggle to process. It combines dialogue-specific window denoising with hybrid attention and achieves strong results across long-dialogue tasks.
- Motivation: Long conversations from meetings, interviews, and debates are difficult for current NLP systems because they combine lengthy inputs with dialogic structure.The paper motivates tools that can help users quickly understand the gist and extract relevant information from these transcripts.
- Approach: The model pre-trains by corrupting a multi-turn dialogue window with dialogue-inspired noise and reconstructing it from the noisy window and the remaining conversation.This window contains more coherent and informative content than sentence-level masking, while requiring fewer resources than full-text denoising for long sequences.
- Approach: Five dialogue-specific noise types disrupt speaker information, turn structure, text spans, and turn order so the model learns dialogue format and complete-conversation content.The noises are Speaker Mask, Turn Splitting, Turn Merging, Text Infilling, and Turn Permutation.
- Approach: Hybrid attention combines sparse attention in most layers with global self-attention in the remaining layers to process longer inputs while retaining full-dialogue information.This design targets the long-sequence limitations of general-purpose pre-trained models.
- Results: DialogLM surpasses previous models by a large margin across long-dialogue understanding and summarization tasks, including summarization, abstractive question answering, and segmentation.It outperforms BART and Longformer on five datasets spanning meeting and screenplay domains, while also outperforming strong baselines for dialogue segmentation.
Related Work
Prior work addresses dialogue processing through task-specific dialogue models and long-sequence attention mechanisms. DialogLM combines dialogue-oriented pre-training with sparse and global attention to target long conversational inputs.
- Dialogue Pre-training: Dialogue-related pre-trained models generally target specific applications such as response generation, response selection, or multiparty conversation understanding.They either continue pre-training on conversational data or train task-specific downstream systems.
- Long-Sequence Processing: Transformer self-attention has quadratic complexity in sequence length, motivating methods such as compression, bucket-based local attention, and sliding-window attention.These approaches reduce computational cost by restricting or compressing token interactions.
- Dialogue Pre-training: Table 1 presents dialogue-related masking operations for generating noisy windows, distinguishing masked speaker names from masked utterance spans.The figure uses blue [MASK] tokens for speaker names and red [MASK] tokens for text spans.
- Long-Sequence Processing: DialogLM adopts Sinkhorn attention together with global attention to reduce computational cost while retaining access to global dialogue information.This hybrid strategy combines local processing with full-attention layers.
Method
DialogLM uses window-based denoising to reconstruct corrupted multi-turn regions of long dialogues and hybrid attention to scale Transformer processing. Its noise operations target dialogue structure, while sparse and global attention balance efficiency with full-context modeling.
- Window-based Denoising: DialogLM selects a contiguous multi-turn window, injects dialogue-related noise, and trains its decoder to reconstruct the original window from the noisy input and remaining dialogue.The objective models the conditional distribution of the original window given the corrupted full conversation.
- Window-based Denoising: Window-based denoising is designed for long conversations because full-text denoising is computationally expensive, whereas individual turns may be too uninformative to represent core dialogue content.Multiple successive turns are more likely to contain coherent and meaningful information.
- Dialogue-Inspired Noise: The five dialogue-inspired noises are Speaker Mask, Turn Splitting, Turn Merging, Text Infilling, and Turn Permutation.Together, they disrupt speaker identities, turn boundaries, text spans, and ordering so reconstruction requires understanding dialogue structure and context.
- Hybrid Attention: Hybrid attention equips most Transformer layers with sparse Sinkhorn attention while retaining global self-attention in other layers.The design captures local information efficiently while preserving full-dialogue semantics for applications such as summarization.
- Model Architecture: The model addresses limitations of BART and UNILM by adding dialogue-format pre-training data and tasks, while extending their short pre-training input lengths.The architecture uses Transformer as its backbone and applies hybrid attention for long dialogue processing.
Experiments
Experiments evaluate DIALOGLM on five long-dialogue benchmarks spanning summarization, abstractive question answering, and topic segmentation in meeting and screenplay domains. The model generally outperforms prior systems, while sparse hybrid attention is most beneficial for longer inputs.
- Experimental Results: DIALOGLM achieves state-of-the-art results on most metrics across meeting and screenplay datasets, outperforming strong baselines across long-dialogue tasks.The evaluation covers five datasets and includes dialogue summarization, abstractive question answering, and topic segmentation.
- Experimental Results: DIALOGLM improves ROUGE-1 over UNILM-base by more than 2.5 points on AMI and ICSI and about 5.0 points on QMSum.The reported changes are 51.92→54.49 on AMI, 46.75→49.25 on ICSI, and 29.14→34.02 on QMSum.
- Experimental Results: DIALOGLM-sparse surpasses DIALOGLM on ICSI and QMSum by about 0.2 points in ROUGE-2 and ROUGE-L when meetings exceed 5,000 words.It obtains state-of-the-art ROUGE-2 and ROUGE-L scores on these datasets, whereas hybrid attention has a slightly negative impact on the shorter AMI meetings.
- Experimental Results: Dialogue-oriented pre-training improves topic segmentation, and hybrid attention further improves performance on AMI and QMSum.Segmentation is evaluated with Pk and WinDiff, for which lower scores indicate predictions closer to the ground truth.
- Experimental Results: On screenplay datasets, DIALOGLM-sparse outperforms BART-Large by 1.93 ROUGE-1 points on ForeverDreaming and 2.04 points on TVMegaSite.The reported comparisons are 33.82→35.75 and 43.54→45.58, respectively.
- Experimental Results: Ablations show that removing pre-training causes substantial degradation on QMSum and TVMegaSite, while all five dialogue-inspired noises contribute to performance.Human evaluation also finds DIALOGLM more informative and reliable than BART, although neural models remain below human summaries in fluency.
DialogLM with LED
DialogLED applies the paper’s window-based denoising framework to the Longformer-Encoder-Decoder architecture, improving results across long-dialogue datasets and tasks.
- DialogLM with LED: DialogLED is created by initializing LED with its original weights and further training it on long-dialogue data using window-based denoising.
- DialogLM with LED: Tables 8 and 9 report DialogLED results on meeting-style and screenplay-style datasets, respectively.
- DialogLM with LED: DialogLED improves the original LED on all five datasets and achieves new state-of-the-art results on four datasets except TVMegaSite.The authors attribute the remaining comparison to DialogLM on TVMegaSite while noting DialogLED has more parameters.
Conclusion
The paper introduces window-based denoising and hybrid attention for long dialogue understanding and summarization, and reports state-of-the-art performance across five benchmarks and three tasks.
- Conclusion: DIALOGLM corrupts a multi-turn dialogue window with five dialogue-inspired noises and reconstructs it from the remaining conversation.
- Conclusion: Hybrid attention adapts the model to longer dialogue scenarios.
- Conclusion: DIALOGLM outperforms previous state-of-the-art models on five benchmarks spanning three long-dialogue understanding and summarization tasks.