PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization

Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, Ting Liu

arXiv:2105.12544v2cs.CL

TL;DR

Dialogue annotations often rely on dialog-agnostic tools or labor-intensive human labels. This paper uses DialoGPT as an unsupervised annotator to add three types of labels to dialogues. The method improves results on SAMSum and AMI and reaches new state-of-the-art performance on SAMSum.

  • Problem

    Dialogue annotations often come from dialog-agnostic open-domain tools or require labor-intensive human annotation.

    Source: §1 · p. 1

  • Method

    The paper uses DialoGPT as an unsupervised annotator for keyword extraction, redundancy detection, and topic segmentation, then supplies the labels to BART and PGN summarizers.

    Source: §1 · pp. 1–2

  • Results

    The method consistently improves BART and PGN on SAMSum and AMI, and the combined annotations achieve new state-of-the-art performance on SAMSum.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The reported improvements span both pretrained BART and non-pretrained PGN on the two evaluated dialogue summarization datasets.

    Source: §1 · p. 2

    Topic segmentation rests on the assumption that a response difficult to predict from context may signal a new topic.

    Source: §3.3.3 · p. 4

Abstract

from arXiv · show

Current dialogue summarization systems usually encode the text with a number of general semantic features (e.g., keywords and topics) to gain more powerful dialogue modeling capabilities. However, these features are obtained via open-domain toolkits that are dialog-agnostic or heavily relied on human annotations. In this paper, we show how DialoGPT, a pre-trained model for conversational response generation, can be developed as an unsupervised dialogue annotator, which takes advantage of dialogue background knowledge encoded in DialoGPT. We apply DialoGPT to label three types of features on two dialogue summarization datasets, SAMSum and AMI, and employ pre-trained and non pre-trained models as our summarizes. Experimental results show that our proposed method can obtain remarkable improvements on both datasets and achieves new state-of-the-art performance on the SAMSum dataset.

1 Introduction

The paper uses DialoGPT as an unsupervised annotator to label dialogue features, addressing the limits of dialog-agnostic tools and manual annotation. These labels improve summarization results on SAMSum and AMI, with a new state of the art on SAMSum.

1 Introduction

  • A good summary is framed in terms of informativeness, redundancy, and relevance.The paper relates its three annotation tasks to these summary qualities.Source: §1 · p. 1
  • Existing dialogue annotations often come from dialog-agnostic open-domain toolkits or labor-intensive human annotation.Prior work incorporated features intended to improve informativeness, reduce redundancy, and improve relevance.Source: §1 · p. 1
  • The proposed unsupervised DialoGPT annotator extracts keywords, detects redundant utterances, and segments dialogue into topics.These tasks are intended to measure informativeness, redundancy, and relevance, respectively.Source: §1 · p. 1
  • Keywords are unpredictable words, redundant utterances are useless for context representation, and topic boundaries may precede unpredictable utterances.The topic-boundary criterion assumes a hard-to-infer utterance may belong to a new topic.Source: §1 · pp. 1–2
  • The method consistently improves BART and PGN on SAMSum and AMI, and combining all three annotations yields new state-of-the-art performance on SAMSum.The paper also reports remarkable improvements over strong baselines on both datasets.Source: §1 · p. 2

2 Preliminaries

The preliminaries define dialogue summarization as producing a condensed summary from a sequence of speaker-labeled utterances. They introduce DialoGPT as a conversational response model and outline the annotator's preprocessing, model pass, and labeling stages.

2.1 Task Definition

  • Dialogue summarization maps a speaker-labeled sequence of utterances to a condensed summary.Each utterance contains words and an end-of-utterance marker; the dialogue is serialized with speaker identities.Source: §2.1 · p. 2

2.2 DialoGPT

  • DialoGPT is a GPT-2-derived conversational response model trained on 147M conversation-like exchanges from Reddit comment chains.It models a response conditioned on the preceding dialogue context.Source: §2.2 · p. 2

2 Preliminaries

  • The annotator preprocesses dialogue as context-response pairs and a full dialogue sequence, then uses DialoGPT losses and context representations to produce three kinds of labels.The labels cover keyword extraction, redundancy detection, and topic segmentation.Source: Figure 2 · p. 3

3 Method

The method uses a three-step DialoGPT annotator workflow—dialogue preprocessing, forward passing, and annotation—and describes BART and PGN as summarizers.

3.4 Summarizer

  • The method describes BART and PGN as summarizers and trains model parameters to maximize conditional likelihood on dialogue-summary pairs.Table 1 reports dataset statistics using pair counts and average turns, dialogue tokens, and summary tokens.Source: §3 · p. 3, §3.4 · p. 5, Table 1 · p. 5

3.1 Dialogue Preprocessing

  • Dialogue preprocessing serializes utterances or forms adjacent context-response pairs, omitting speaker information because DialoGPT was trained without it.The paper notes that DialoGPT can simulate dialogue patterns involving the same or different speakers despite this omission.Source: §3.1 · p. 3

3.2 DialoGPT Forward Passing

  • For each context-response pair, DialoGPT computes response losses; a dialogue pass also extracts EOS representations for each dialogue prefix.Losses are not computed for the first utterance, which serves only as context.Source: §3.2 · p. 3

3.3.1 Keywords Extraction: DialoGPTKE

  • Unpredictable response words are selected as keywords, speaker names are added, and the resulting keywords are appended to the dialogue.The selection rate is tuned using a heuristic based on summary and dialogue lengths; the first word of each utterance is ignored.Source: §3.3.1 · p. 4

3.3.2 Redundancy Detection: DialoGPTRD

  • Redundancy detection scans backward through adjacent context representations, marks an utterance when cosine similarity exceeds tRD, and inserts an [RD] tag before it.Figure 3 illustrates u5 being marked when hEOS4 and hEOS5 exceed tRD=0.99.Source: §3.3.2 · p. 4, Figure 3 · p. 4

3.3.3 Topic Segmentation: DialoGPTTS

  • Utterances with the highest predicted losses are selected as topic boundaries, with a [TS] tag inserted before each selected utterance.The selection rate rTS is tuned using a heuristic based on the average number of summary sentences divided by dialogue utterances in training.Source: §3.3.3 · p. 4

4 Experiments

Experiments evaluate DialoGPT annotations on SAMSum and AMI using multiple summarizers and automatic and human assessments. The combined annotations improve benchmark performance, while analyses show different benefits for keywords, redundancy detection, and topic segmentation.

4.1 Datasets

  • SAMSum covers human-generated dialogues from varied real-life scenes, while AMI contains four-participant meetings about a remote-control design project.The experiments use these two datasets.Source: §4.1 · p. 5

4.2 Implementation Details

  • DialoGPT uses dataset-specific keyword and topic ratios and redundancy thresholds; BART and PGN use separately specified model and decoding settings.SAMSum uses keyword and topic ratios of 15 and a redundancy threshold of 0.99; AMI uses 4, 5, and 0.95 respectively.Source: §4.2 · p. 5

4.3 Baselines and Metrics

  • Evaluation compares established SAMSum and AMI summarization baselines using ROUGE and BERTScore.SAMSum baselines include LONGEST-3, TextRank, Transformer, D-HGN, TGDGA, DialoGPT, and MV-BART; AMI baselines include SummaRunner, UNS, TopicSeg, and HMNet.Source: §4.3 · p. 6

4.4 Automatic Evaluation

  • BART and PGN improve with annotations on SAMSum and AMI, and BART(DALL) achieves state-of-the-art performance on SAMSum.BART(DALL) and PGN(DALL) also outperform baseline models on BERTScore.Source: §4.4 · p. 6, Table 2 · p. 6, Table 4 · p. 6

4.6 Analysis

  • On SAMSum, keyword annotations outperform redundancy and topic annotations; on AMI, PGN(DRD) contributes most, while HMNet and TopicSeg score higher.The authors associate keywords with retaining essential information in shorter dialogues and redundancy detection with verbose meeting transcripts; HMNet requires news-summary pretraining, while TopicSeg uses a complex attention mechanism.Source: §4.4 · p. 6

4.5 Human Evaluation

  • Human ratings favor the method across three metrics, with DRD strongest in conciseness and DTS stronger in coverage; HMNet leads informativeness and coverage.The AMI ratings remain below golden summaries, which the authors say indicates greater difficulty.Source: §4.5 · p. 7, Table 5 · p. 7

4.6 Analysis

  • Keyword extraction achieves the best F1, chiefly through higher recall, while entities yield the best precision and TextRank and entities have poor recall.Against a rule-based redundancy annotator, DialoGPTRD performs better, especially on long, verbose AMI transcripts.Source: §4.6 · pp. 7–8, Table 6 · p. 7, Table 7 · p. 7

4.6 Analysis; 4.7 Case Study

  • DialoGPT topic segmentation is comparable to C99 with DialoGPT embeddings; on AMI, PGN with gold topic annotations achieves the best result.In the SAMSum example, BART(DALL) produces a better summary with higher ROUGE than other models, yet some redundancy remains; the authors attribute this to insufficient training on the small dataset.Source: §4.6 · p. 8, Table 9 · p. 8, §4.7 · p. 8

5 Related Work

Prior dialogue-summarization work adds auxiliary features such as keywords, redundancy cues, and topics, while pre-trained models have also been used unsupervisedly in other tasks. This paper places DialoGPT in that unsupervised paradigm as an annotator for three dialogue features.

  • Prior dialogue summarization uses keywords, redundancy detection, topics, dialogue acts, discourse, and commonsense as auxiliary information.The paper combines keywords, redundant utterances, and topics.Source: §5 · p. 8
  • A SAMSum example contrasts BART’s longer, redundant summaries and MV-BART’s topic coverage with BART(DALL)’s higher ROUGE and better summary.The example also notes that BART(DALL) can retain some redundant information.Source: Figure 4 · p. 9
  • Unsupervised uses of pre-trained models include domain labeling, data augmentation, and knowledge-graph construction; this paper proposes DialoGPT for three unsupervised annotation tasks.The tasks are keyword extraction, redundancy detection, and topic segmentation.Source: §5 · p. 9

6 Conclusion

The paper explores DialoGPT as an unsupervised annotator for dialogue summarization, using keyword extraction, redundancy detection, and topic segmentation. Across SAMSum and AMI, annotations improve pre-trained BART and non-pre-trained PGN; combining all three achieves new state-of-the-art performance on SAMSum.

6 Conclusion

  • The annotator applies keyword extraction, redundancy detection, and topic segmentation to dialogue summarization.The paper investigates DialoGPT as an unsupervised annotator.Source: §6 · p. 9
  • Combining keyword extraction, redundancy detection, and topic segmentation achieves new state-of-the-art performance on SAMSum.The paper reports consistent improvements over pre-trained BART and non-pre-trained PGN on both SAMSum and AMI.Source: §6 · p. 9
  • The annotations are evaluated on SAMSum and AMI with pre-trained BART and non-pre-trained PGN summarizers.The paper reports consistent improvements on both datasets.Source: §6 · p. 9

A Evaluation Details

Evaluation uses ROUGE and BERTScore, with Py-rouge following Gliwa et al. (2019) and the official BERTScore implementation.

A Evaluation Details

B Ablation Studies for Annotations

Ablations compare annotation combinations on SAMSum and AMI. Two annotations improve results over the original setting, and combining all three gives the best ROUGE scores.

B Ablation Studies for Annotations

  • On both datasets, summarizers trained with two annotations can improve over their original results.The experiments compare combinations of the three annotation types.Source: B Ablation Studies for Annotations · p. 12
  • Combining all three annotations yields the best results in every ROUGE score for both summarizers.The ablation results also report improvements from training with two of the three annotations.Source: B Ablation Studies for Annotations · p. 12
  • Two-annotation datasets can outperform corresponding single-annotation datasets, including BART(DKE+RD) over BART(DKE) and BART(DRD).The reported comparison illustrates the pairwise result on SAMSum.Source: B Ablation Studies for Annotations · p. 12
  • DKE+TS yields relatively small improvements versus DRD+TS and DKE+RD, while indicating non-overlapping information in DialoGPTKE and DialoGPTTS.Source: B Ablation Studies for Annotations · p. 12

C Hyper-parameter Search Results

Hyper-parameter searches vary keyword extraction ratios, redundancy similarity thresholds, and topic segmentation ratios. The selected settings differ between SAMSum and AMI.

C Hyper-parameter Search Results

Loading 2105.12544v2…