PaperlayerView exact revision on arXiv

Source-linked AI summary

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

arXiv:2112.08713v2cs.CL

TL;DR

Pre-trained dialogue summarizers can produce unsupported content, and reducing factual errors while maintaining summary quality remains challenging. The paper introduces an error typology and contrastive fine-tuning with a dialogue-specific loss. Evaluations report improved faithfulness and baseline performance across dialogue and meeting summarization datasets.

  • Problem

    Pre-trained dialogue summarizers can generate hallucinated content unsupported by the source, and improving summary quality while reducing factual inconsistency remains challenging.

    Source: §1 · p. 1

  • Method

    The paper introduces a factual-error typology and trains with contrastive examples and a dialogue-specific loss to target errors and capture information flow between speakers.

    Source: §1 · pp. 1–2

  • Results

    CONFIT significantly reduces factual errors and improves faithfulness, and outperforms baseline pre-trained models on SAMSum and AMI in automatic metrics and human evaluation.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The reported generalization from SAMSum dialogue summarization to AMI meeting summarization supports applying CONFIT across these two settings.

    Source: §1 · p. 2

    Error-pattern comparisons may be affected by sample size: the analysis uses 20 AMI dialogues versus 100 SAMSum dialogues.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained models, substantial amounts of hallucinated content are found during the human evaluation. Pre-trained models are most commonly fine-tuned with cross-entropy loss for text summarization, which may not be an optimal strategy. In this work, we provide a typology of factual errors with annotation data to highlight the types of errors and move away from a binary understanding of factuality. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called ConFiT. Based on our linguistically-informed typology of errors, we design different modular objectives that each target a specific type. Specifically, we utilize hard negative samples with errors to reduce the generation of factual inconsistency. In order to capture the key information between speakers, we also design a dialogue-specific loss. Using human evaluation and automatic faithfulness metrics, we show that our model significantly reduces all kinds of factual errors on the dialogue summarization, SAMSum corpus. Moreover, our model could be generalized to the meeting summarization, AMI corpus, and it produces significantly higher scores than most of the baselines on both datasets regarding word-overlap metrics.

1 Introduction

Pre-trained dialogue summarizers can hallucinate, and improving summary quality while reducing factual inconsistency remains challenging. CONFIT combines a linguistically informed error typology with contrastive fine-tuning and a dialogue-specific loss; evaluations report stronger baseline performance and fewer factual errors.

1 Introduction

  • Pre-trained models can generate content unsupported by their dialogue sources, while improving summary quality and reducing hallucination remains challenging.Source: §1 · p. 1
  • The authors introduce a typology and annotation protocol to examine different kinds of factual errors rather than labeling summaries only as faithful or unfaithful.Their analysis identifies dialogue flow, informal speaker interactions, and complex coreference as dialogue-summarization challenges.Source: §1 · pp. 1–2
  • CONFIT uses contrastive fine-tuning to distinguish factual summaries from erroneous ones and a dialogue-specific loss to capture information flow between speakers.Its objectives target errors identified by the proposed linguistically informed typology.Source: §1 · pp. 1–2
  • CONFIT outperforms baseline pre-trained models on SAMSum and AMI in automatic metrics including ROUGE and BARTScore, as well as human evaluation.The paper reports that CONFIT significantly reduces factual errors and improves factual consistency.Source: §1 · p. 2

2 New Taxonomy of Factuality Errors for Abstractive Dialogue Summarization

The paper develops an eight-category factuality-error taxonomy from annotations of generated SAMSum summaries and examines error patterns across models. In a preliminary human evaluation, missing information and wrong references together account for approximately 45% of errors.

2 New Taxonomy of Factuality Errors for Abstractive Dialogue Summarization

  • The taxonomy is based on manual annotation of errors in summaries from representative SAMSum baseline models, followed by analysis of error statistics.The authors use the observed generation behavior to guide their model design.Source: §2 · p. 2, §2.1 · p. 3
  • The taxonomy defines missing information, redundant information, circumstantial errors, and object errors as distinct factuality-error categories.Circumstantial errors concern predicate details such as date, time, or location; object errors concern incorrect direct or indirect objects for non-personal entities.Source: §2 · p. 3
  • Wrong-reference, negation, tense, and modality errors capture distinct mismatches in entities, polarity, grammatical tense, and modal meaning.Source: §2 · p. 3

2.1 Annotation and Analysis

  • Approximately 45% of errors in the evaluated model summaries are missing-information or wrong-reference errors.The preliminary human evaluation covered four models and 19 SAMSum dialogues; these patterns informed CONFIT's targeting of frequent errors.Source: §2.1 · pp. 3–4

3 CONFIT Model

ConFiT augments cross-entropy fine-tuning with contrastive and self-supervised losses to improve factual consistency. Its objectives target summary errors and help capture who is speaking in dialogue.

3 CONFIT Model

  • ConFiT adds contrastive and self-supervised losses to cross-entropy training, with coefficients controlling their contributions to the overall objective.The approach is motivated by shortcomings of cross-entropy loss, including sub-optimal generalization and instability.Source: §3 · p. 4

3.1 Contrastive Loss

  • Contrastive training distinguishes reference summaries from incorrect negatives, using targeted perturbations to address factual-error categories.The negatives include noun and verb swaps, masked numbers or years, deleted dialogue sentences, and coreferent entities masked and filled with BART.Source: §3.1 · pp. 4–5

3.2 Self-supervised Loss

  • A self-supervised loss predicts whether sampled dialogue tokens or utterances belong to the same speaker, helping track dynamic roles and first-person references.The paper says this supplementary loss improves faithfulness for dialogues containing several first-person references.Source: §3.2 · p. 5

4 Experiments

The experiments assess ConFiT on SAMSum chat dialogues and AMI meeting transcripts using pretrained models with dataset-specific training schedules. Evaluation combines overlap and faithfulness metrics with human ratings.

4.1 Dataset

  • SAMSum contains 16,369 English chat dialogues, while AMI contains 137 multiparty meeting transcripts, allowing evaluation on conversation- and meeting-style inputs.Both datasets provide abstractive summaries; the paper describes SAMSum summaries as multi-sentence and AMI summaries as generic.Source: §4.1 · p. 5

4.2 Experiment Settings

  • BART, Pegasus, and T5 were trained with dataset-specific epoch or step schedules and learning rates on SAMSum and AMI.On SAMSum, training used 3 BART epochs, 20 Pegasus epochs, and 20 T5 epochs; on AMI, it used 6,000 BART steps, 24,000 Pegasus steps, and 20,000 T5 steps. Learning rates ranged from 1e −05 to 1e −04 across the reported model and dataset settings.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • Evaluation combines ROUGE overlap, BARTScore faithfulness scoring, and blinded human ratings of 100 SAMSum and 20 AMI dialogues.Human raters scored faithfulness from 1 to 10 and marked taxonomy error types; outputs were shuffled within each dialogue before assignment.Source: §4.3 · pp. 5–6

5 Results

Across SAMSum and AMI, CONFIT improves ROUGE-1, ROUGE-L, and human faithfulness scores over baselines, although BARTScore falls on AMI. Error analysis and a selected dialogue case illustrate both broad reductions and remaining limitations.

5 Results

  • Across SAMSum and AMI, all three CONFIT models significantly beat baselines on ROUGE-1, ROUGE-L, and human faithfulness, while BARTScore falls on AMI.The authors note that human evaluators nevertheless rated all three CONFIT models as more faithful than their corresponding baselines on both datasets.Source: §5 · p. 6, Table 5 · p. 7, Table 6 · p. 7

5.1 Error Analysis

  • On SAMSum, CONFIT reduces missing, redundant, wrong-reference, and circumstance errors across models; wrong-reference reductions are 20%, 7%, and 33% for BART, Pegasus, and T5.The error percentages are based on human evaluation of summaries from 100 SAMSum dialogues, and one summary can contain multiple error types.Source: §5.1 · p. 6, Table 3 · p. 7
  • On AMI, reductions across error types are less consistent across models; the authors suggest the 20-dialogue sample, versus 100 SAMSum dialogues, may contribute.The error-type percentages are summarized for outputs from 20 AMI dialogues.Source: §5.1 · p. 7, Table 4 · p. 7

5.2 Case Study

  • In the selected SAMSum case, CONFIT corrects BART’s circumstance error and coreference errors in T5 and Pegasus; all three CONFIT summaries score higher than their baselines.The examples show corrections to claims about whose car was involved and, for T5, a missing subject.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

6 Related Work

Prior dialogue-summarization work addresses conversational complexity through varied linguistic and structural features, but the approaches have not converged on a simple, practical solution. The paper presents its taxonomy and annotations as a way to gain insight into factual inconsistency.

6 Related Work

  • Multi-party dialogue poses challenges including pauses, false starts, reconfirmations, hesitations, and speaker interruptions.Prior approaches have drawn on semantic features, named entities, coreference, conversational structure, utterance flow, syntax, and granularity control.Source: §6 · p. 8
  • Dialogue-summarization approaches span semantic and structural features, but have not converged on a simple, practical solution.Examples in the passage include keywords, domain terminology, topics, entailment and background knowledge, named entities, coreference, conversational structure, and utterance flow.Source: §6 · p. 8
  • The proposed factual-error taxonomy and annotations are presented as a way to gain deeper insight into causes of factual inconsistency in abstractive dialogue summaries.Source: §6 · p. 8

7 Conclusion

The paper concludes that CONFIT combines contrastive and self-supervised dialogue-specific fine-tuning to improve faithfulness and reduce factual errors. Experiments on SAMSum and AMI report better performance than cross-entropy-fine-tuned baselines under automatic and human evaluation.

7 Conclusion

  • CONFIT combines contrastive learning from positive and factual-error examples with a self-supervised loss for information flow between interlocutors.The conclusion describes both losses as additions to the fine-tuning objective.Source: §7 · p. 8
  • On SAMSum and AMI, CONFIT outperforms traditional cross-entropy-fine-tuned baselines on automatic metrics and human evaluation.Source: §7 · p. 8
  • The authors present carefully designed fine-tuning objectives that capture important structures and features of the text as an approach to improving summarization faithfulness.Source: §7 · p. 8

8 Ethics Statement

The human evaluation used volunteer English speakers and assessed faithfulness with a 1-to-10 scale. The paper reports no use of personally identifiable information and describes the datasets as non-sensitive and unlikely to harm annotators.

8 Ethics Statement

  • Seven volunteer English-speaking participants were recruited for the human error annotation.Source: §8 · p. 9
  • Annotators assessed faithfulness on a 1-to-10 scale, with summaries from each dialogue grouped together and randomly shuffled to prevent guessing model identity from ordering.The annotation spreadsheet omitted model names, while a separate key allowed them to be recovered after evaluation.Source: §8 · p. 9
  • The experiments used no personally identifiable information; the existing datasets were described as non-sensitive and unlikely to harm annotators.Source: §8 · p. 9

Related papers

Loading 2112.08713v2…