Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

1 Introduction

• Across model categories, factuality-metric performance varies substantially, and no metric is best for both CNN/DM and XSum FTSOTA summaries.Category-agnostic gains are concentrated on OLD or EXFORMER summaries, which the authors describe as less important to detect.Source: §1 · p. 1
• AGGREFACT aggregates nine annotated datasets and stratifies them by the underlying summarization model to enable finer-grained factuality-metric comparisons.The benchmark supports comparisons by model category, dataset, and error type.Source: §1 · pp. 1–2
• Metrics claiming state-of-the-art performance still differ significantly in identifying the same error types across CNN/DM and XSum.The paper also reports that no metric is superior across all settings or error types.Source: §1 · p. 2
• The authors recommend evaluating on state-of-the-art summaries, choosing metrics for the downstream task, and applying error-type annotations consistently with prior work.They also call for extending analysis to domains such as dialogue and medical evidence summarization, with suitable annotations.Source: §1 · p. 2

2.1 Benchmark Standardization

• AGGREFACT targets the older-model skew in existing evaluations by incorporating recent annotations on state-of-the-art Transformer models and separating FTSOTA, EXFORMER, and OLD categories.FTSOTA includes fine-tuned models such as BART, PEGASUS, and T5; EXFORMER covers early Transformer models, while OLD includes models such as Pointer-Generator and BottomUp.Source: §2.1 · pp. 2–3

2.2 Benchmark Datasets

• The benchmark aggregates nine publicly available annotated datasets and provides separate AGGREFACT-CNN and AGGREFACT-XSUM subsets, with human evaluations supplying binary factuality labels.It excludes CoGen-Summ because its task ranks summary pairs rather than classifying individual summaries, and modifies Polytope labels for addition, omission, and duplication errors.Source: §2.1 · p. 3, §2.2 · p. 3
• The benchmark removes duplicate instances, manually corrects 100 conflicting labels, and evaluates metrics with weighted average balanced accuracy across datasets.Metric thresholds are selected on validation data for each model category and dataset before converting scores to binary predictions.Source: §2.2 · p. 3, §2.3 · pp. 3–4

3 Comparison of Factuality Metrics

• The evaluation includes trained metrics and ChatGPT-based metrics, with some prompting models to output binary decisions and others to score factuality on a scale.Thresholding converts metric scores into binary labels; the paper distinguishes threshold-per-dataset from single-threshold settings.Source: §3 · p. 4
• For every factuality metric, average thresholds are higher on AGGREFACT-CNN than AGGREFACT-XSUM across model categories.The authors say this makes a single threshold across both datasets difficult to balance and potentially poor for at least one dataset.Source: §3 · p. 4, Figure 1 · p. 4
• Higher CNN/DM thresholds are linked to XSum's more abstractive summaries and greater error frequency, which lower metric scores even for factual cases.For CNN/DM, smaller deviations from the source may indicate non-factuality.Source: §3 · p. 4
• On AGGREFACT-CNN, trained and ChatGPT-based factuality metrics both perform best on OLD-model summaries.The authors attribute this pattern to obvious errors, such as repetition, being easier to detect than nuanced errors from newer models.Source: §3 · p. 4

AGGREFACT-CNN AGGREFACT-XSUM

• About a 10% balanced accuracy decrease on average occurs from OLD to FTSOTA on CNN/DM; OLD summaries contain more easily detected errors.The majority of annotated summaries come from OLD models, so category-agnostic evaluations weight those results more heavily.Source: AGGREFACT-CNN AGGREFACT-XSUM · p. 5, Table 3 · p. 5
• On AGGREFACT-XSUM, trained metrics perform best on FTSOTA, while ChatGPT-based metrics perform best on EXFORMER; ChatGPT-ZS and ChatGPT-Star set new EXFORMER state-of-the-art results.The authors favor category-wise comparisons to better inform metric choice for each model category.Source: AGGREFACT-CNN AGGREFACT-XSUM · p. 5
• On AGGREFACT-CNN-FTSOTA, QuestEval performs best, without significant improvement over other trained metrics but with significant improvement over ChatGPT-based metrics; on XSUM-FTSOTA, DAE significantly outperforms all others.These comparisons use a single threshold; metric rankings differ slightly from the threshold-per-dataset setting.Source: Table 4 · p. 5, AGGREFACT-CNN AGGREFACT-XSUM · p. 5
• The separate FTSOTA benchmark uses validation and test splits from the FTSOTA subsets of AGGREFACT-CNN and AGGREFACT-XSUM, enabling comparisons with a single threshold.Table 4 reports balanced accuracy and 95% confidence intervals for this single-threshold setting.Source: AGGREFACT-CNN AGGREFACT-XSUM · p. 5, Table 4 · p. 5
• Metrics promoted as improvements on category-agnostic SummaC did not show similar gains on FTSOTA summaries, motivating comparisons focused on state-of-the-art outputs.The authors note that ChatGPT-metric performance can vary substantially with prompts; they use the original papers’ prompts.Source: AGGREFACT-CNN AGGREFACT-XSUM · p. 5

4 Finer-grained Error Analysis

• Error types are consolidated into a shared taxonomy so that factuality metrics can be compared across datasets and analyzed by error type.The section’s finer-grained analysis depends on mapping each dataset’s annotations into the shared categories.Source: §4 · p. 6, §4.1 · p. 6

4.1 A Taxonomy of Error Types

• The unified taxonomy groups most errors as intrinsic or extrinsic noun-phrase or predicate errors, adding whole-sentence categories for errors annotated at sentence level.The analysis maps four datasets with fine-grained annotations; summaries may receive multiple error types, assigned when more than one annotator marks them.Source: §4.1 · p. 6

4.2 Error Mapping

• Manual inspection found over 90% accuracy for the error-type mapping, but FRANK was least accurate and some disagreements reflected the original annotations.For example, FRANK’s entity errors could be intrinsic or extrinsic; event errors in Goyal’21 could map to noun-phrase or predicate categories.Source: §4.2 · p. 7

4.3 Distribution Shift of Error Types

• BART’s mapped error distributions differ between datasets, and XSumFaith and FRANK show dramatically different distributions even for the same summaries.The authors attribute the latter discrepancy partly to FRANK’s taxonomy conversion and the difficulty of mapping its “out of article” errors to a single unified type.Source: §4.3 · p. 7
• Across the combined datasets, extrinsic errors make up approximately 50% of CNN/DM errors and 70% of XSum errors.The CNN/DM proportion decreases slightly from OLD to more recent models, while the XSum proportion remains unchanged.Source: Figure 4 · p. 8, §4.3 · p. 8

4.4 Error Detection by Type

• Recent metrics, including SummaC-Conv, QAFactEval, and ChatGPT-based metrics, achieve higher recall for most error types, but recall varies by dataset.For extrinsic-noun phrase errors, recall falls 10–30% for trained metrics on XSum versus CNN/DM; ChatGPT metrics report 20–30% higher recall on CNN/DM than XSum.Source: §4.4 · pp. 8–9, Table 5 · p. 8
• DAE performs less well on AGGREFACT-CNN-ERROR despite XSumFaith supervision, and no metric is uniformly effective across error types and datasets.The authors suggest their conclusion also applies to FTSOTA summaries, which contain less obvious errors.Source: §4.4 · p. 9

5 Recommendations

• Evaluate factuality metrics on FTSOTA summaries, since results differ from those on EXFORMER or OLD summaries.This helps test whether apparent gains reflect progress on current systems or fixes to errors in outdated systems.Source: §5 · p. 9
• Build living factuality benchmarks that add annotated summaries from LLMs as more powerful summarization systems emerge.The authors report prior findings that existing metrics cannot reliably detect errors in GPT-3 summaries.Source: §5 · p. 9
• Choose a factuality metric for the downstream task because no evaluated metric wins across all settings or error types.The authors also note that current ChatGPT-based metrics perform poorly on modern summarization systems, though they may be a starting point for LLM-based evaluation.Source: §5 · p. 9
• Use more consistent error taxonomies to make comparisons across datasets possible, prioritizing annotations and evaluation on SOTA summaries.The authors report that datasets with finer-than-sentence-level annotations are easier and more accurate to map to their unified taxonomy.Source: §5 · p. 9

Annotate and evaluate on non-news datasets

• Extend annotation and evaluation beyond news, including dialogue and medical evidence summarization, where error distributions may differ.The authors encourage development of metrics that perform well across domains.Source: Annotate and evaluate on non-news… · p. 9

6 Conclusion

• The paper analyzes factuality metrics across a large meta-benchmark assembled from existing datasets.Source: §6 · p. 9
• State-of-the-art fine-tuned summarization models still present challenges for detecting factual errors, and reliance on older datasets often overestimates detector performance.Source: §6 · p. 9
• A unified taxonomy highlights differences between datasets and summarization models, as well as the difficulty of unifying concepts in this problem space.Source: §6 · p. 9

Limitations

• The evaluation is limited to English newswire, excluding other domains where error types may differ, such as dialogue summarization.The authors note that wrong-reference errors in dialogue are not fully evaluated by current metrics.Source: Limitations · p. 10
• Error-type mapping inherits limitations in the quality and annotator agreement of prior datasets because the authors did not conduct large-scale reannotation.Source: Limitations · p. 10

• FTSOTA includes pretrained Transformer models BART, T5, and PEGASUS, fine-tuned on summarization datasets.They are pretrained on massive text corpora.Source: A Model Categories · p. 13
• EXFORMER includes BERTSumExt, BERTSumAbs, GPT-2, TransS2S, and BERTS2S.Source: A Model Categories · p. 13
• OLD includes FastAbsRl, TConvS2S, BottomUp, PGNet, NeuSUM, BanditSum, SummaRuNNer, TextRank, CBDec, RNES, ROUGESal, ImproveAbs, MultiTask, and UnifiedExtAbs.Source: A Model Categories · p. 13

• DAE evaluates each summary dependency arc for entailment against the article and averages the arc scores; its default model is trained on XSumFaith.Source: B Factuality Metrics · p. 13
• QuestEval aggregates answer-overlap scores from article-derived questions answered by the summary and summary-derived questions answered by the article, using the unweighted version.Source: B Factuality Metrics · pp. 13–14
• SummaC-ZS scores each summary sentence against input sentences with an NLI model, takes the maximum entailment score, then averages across summary sentences.Its default model and hyperparameters may return a negative score.Source: B Factuality Metrics · p. 14
• SummaC-Conv bins entailment scores for each summary sentence against input sentences, then uses a learned one-dimensional convolution to score it.The convolution parameters are learned on synthetic FactCC data.Source: B Factuality Metrics · p. 14
• QAFactEval is a QA-based metric with optimized question answering, question generation, and answer-overlap components.Source: B Factuality Metrics · p. 14
• ChatGPT-ZS and ChatGPT-CoT prompt for binary factuality labels, while ChatGPT-DA and ChatGPT-Star assign scores on 0–100 and one-to-five scales, respectively.CoT adds step-by-step reasoning to its zero-shot prompt; DA uses direct assessment.Source: B Factuality Metrics · p. 14

• The taxonomy also includes negation [Zhang et al., 2020; Kryscinski et al., 2020; Huang et al., 2020; Zeng et al., 2021], adjective [Zhang et al., 2020], coreference [Zhang et al., 2020; Kryscinski et al., 2020; Pagnoni et al., 2021; Nan et al., 2021b], and number errors [Kryscinski et al., 2020; Nan et al., 2021b; Chen et al., 2021; Cao et al., 2020].Source: C Surveyed Error Types · pp. 14–15
• Further types are entity errors [Kryscinski et al., 2020; Pagnoni et al., 2021; Zeng et al., 2021; Wang et al., 2020; Nan et al., 2021b,a; Chen et al., 2021; Cao et al., 2020], attribute errors [Pagnoni et al., 2021], pronoun errors [Kryscinski et al., 2020; Cao et al., 2020], and commonsense errors [Kryscinski et al., 2020].Source: C Surveyed Error Types · p. 15
• Surveyed errors include temporal [Kryscinski et al., 2020; Cao et al., 2020], predicate [Pagnoni et al., 2021], discourse-link [Pagnoni et al., 2021], and relation errors [Nan et al., 2021a,b].Source: C Surveyed Error Types · p. 15
• Other surveyed types are quantity errors [Zhao et al., 2020], event errors [Goyal and Durrett, 2021], noun-phrase errors [Wang et al., 2020; Goyal and Durrett, 2021], and circumstance errors [Pagnoni et al., 2021].Source: C Surveyed Error Types · p. 15