Source-linked AI summary
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Artidoro Pagnoni, Vidhisha Balachandran, Yulia Tsvetkov
TL;DR
Abstractive summaries often contain factual inconsistencies, but factuality metrics lack common benchmarks and mainly reduce errors to binary labels. The paper introduces a linguistically grounded typology and uses it to build FRANK from annotated CNN/DM and XSum summaries. FRANK supports fine-grained error analysis and reveals that metric correlations with human factuality judgments are generally low, while dataset and model differences constrain interpretation.
Problem
About ∼30% of summaries contain factual inconsistencies, while factuality metrics lack common benchmarks and binary labels provide limited insight into error types.
Method
The paper develops a linguistically grounded factual-error typology, collects sentence-level human annotations for 2250 summaries from CNN/DM and XSum, and benchmarks factuality metrics.
Results
FRANK enables fine-grained benchmarking, while all evaluated metrics show low overall correlations with human factuality judgments; FactCC reaches 0.20 Pearson and 0.30 Spearman correlation.
Takeaways & Limitations
FRANK moves factuality evaluation beyond summary-level binary labels by measuring the quantity and categories of factual violations and the errors metrics capture.
Takeaways & Limitations
Summaries across datasets come from different models because publicly available outputs differ, and sentence-level judgments were chosen after full-summary Likert ratings produced lower agreement.
Abstract
from arXiv · showhide
Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights into the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations, we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgment as well as their specific strengths and weaknesses.
1 Introduction
Factual inconsistencies remain common in abstractive summaries, while existing factuality metrics lack shared benchmarks and fine-grained error analysis. FRANK addresses these gaps with a linguistically grounded typology, human annotations, and metric benchmarking.
- ∼30% of summaries have factual inconsistencies, motivating reliable factuality evaluation for information consumption.
- Existing n-gram metrics are insufficient for factual correctness, while newer factuality metrics cannot be compared without common benchmarks.
- Binary factuality labels obscure subjectivity, error importance, and the differing impact of multiple errors.
- The proposed typology makes binary decisions more objective, achieving near-perfect crowd–expert agreement (κ = 0.86).
- The typology categorizes factual errors by quantity and type, enabling deeper analysis than factual-versus-hallucinated labels.
- FRANK provides human judgments for model-generated summaries from CNN/DM and XSum to assess systems and benchmark factuality metrics.
2 Typology of Factual Errors
The typology organizes factual errors across semantic frames, discourse relations, and content verifiability. It distinguishes specific error categories so summaries can be analyzed beyond a binary factuality label.
- Factual errors arise at semantic-frame, discourse, and content-verifiability levels.
- Semantic Frame Errors: Predicate Error marks a summary predicate or frame that is inconsistent with the source text.
- Semantic Frame Errors: Entity Error marks wrong primary arguments, attributes, or directionality despite the original relation being expressed.
- Semantic Frame Errors: Circumstance Error marks incorrect non-core attributes such as location, time, manner, direction, or modality.
- Discourse Errors: Coreference Error covers incorrect or ambiguous references, while Discourse Link Error covers incorrect temporal or rhetorical links between statements.
- Content Verifiability Errors: Out of Article Error covers information not deducible from the source, and Grammatical Error covers incomprehensible or ambiguous malformed statements.
- Others captures factual errors outside the listed categories, while Not an Error identifies statements without errors.
3 Dataset Creation
The study builds a fine-grained factuality dataset from model-generated summaries on CNN/DM and XSum, using sentence-level human annotations based on the proposed error typology. Its protocol combines annotator training, qualification, and expert validation to support reliable judgments.
- Dataset construction: Annotations cover five abstractive summarization models on CNN/DM and four on XSum, targeting diverse fine-grained factuality errors.The dataset uses publicly available model outputs from both datasets.
- Annotation collection: Three independent annotators label summaries from 250 articles per dataset, producing 1,250 CNN/DM and 1,000 XSum model outputs.Each sentence is judged in the context of the entire summary, allowing discourse errors spanning multiple sentences to be identified.
- Annotation protocol: The two-step protocol first judges sentence factuality, then assigns one or more factual-error categories to sentences marked non-factual.Multiple categories are allowed when a sentence contains multiple errors.
- Annotation quality: Annotators receive clear instructions, tutorials, qualification testing above 85%, and continuous evaluation through an intuitive MTurk interface.The interface was designed to support high-quality crowdsourced annotations.
- Agreement: Sentence-level factuality judgments achieve κ = 0.58 with 91% majority agreement, while eight-category labels achieve κ = 0.39 with 73.9% agreement.The category annotation is reported as moderate agreement when all three annotators agree that a sentence is not factual.
- Agreement: Agreement between crowd majority and an expert reaches κ = 0.86 on 201 datapoints, indicating nearly perfect agreement.The authors report this as higher than prior crowd–expert agreement.
4 Summarization Model Analysis
The model analysis finds substantial factual unreliability, especially on XSum, with error rates and error categories varying across datasets and model families. Pretraining improves factuality, but discourse-level errors remain difficult on CNN/DM and factuality remains challenging on XSum.
- Error categories: CNN/DM errors are dominated by Entity Error and Coreference Error, whereas XSum errors are dominated by Out of Article Error and Entity Error.XSum has no discourse errors because its summaries contain only one sentence.
- Model comparisons: PGN produces 26% incorrect CNN/DM summaries, compared with 74% for S2S and 62% for BUS, but exceeds 96% incorrect summaries on XSum.The authors relate PGN’s CNN/DM behavior to extractive copying and contrast it with XSum’s more abstractive setting.
- Model comparisons: Large-scale pretrained models improve factuality on both datasets, with larger gains on CNN/DM; on XSum, BertS2S improves over non-pretrained models by only about 10%.On CNN/DM, BERTSum and BART show half the error rate of BUS.
- Error categories: Pretrained transformers reduce CNN/DM frame-level errors, but discourse errors remain challenging and form a larger share for BERTSum and BART.LSTM models have more grammatical errors than transformer and CNN-based models.
5 Factuality Metric Evaluation
FRANK benchmarks factuality metrics with diverse, fine-grained annotations and controls for dataset biases. Metrics correlate weakly overall, with substantial variation across datasets, model types, and factual-error categories.
- Benchmark: FRANK combines 2250 summaries from different systems and datasets with sentence-level factual-error categories for fine-grained metric evaluation.Its protocol controls for dataset biases and analyzes which error types each metric captures.
- Overall Results: FactCC and BERTScore P achieve the strongest overall correlations, at 0.20 Pearson and 0.30 Spearman for FactCC, and 0.27 Pearson and 0.35 Spearman for BERTScore P.All metrics show low correlations with human factuality judgments overall.
- Overall Results: General metrics Rouge-1 and METEOR reach ρ = 0.14, close to FactCC’s ρ = 0.20, while FEQA is not statistically significant.QAGS has a low but significant correlation of ρ = 0.06.
- Dataset Effects: Entailment metrics have significantly higher partial Pearson correlation on CNN/DM than XSum, where their correlation decreases by a factor of four.QAGS and the OpenIE baseline show similar dataset-dependent behavior, likely reflecting XSum’s high Out of Article error rate.
- Model Effects: On CNN/DM, entailment metrics and QAGS perform significantly better on non-pretrained than pretrained models.The authors relate this pattern to the similarity between training-time artificial errors and non-pretrained-model mistakes.
- Error Analysis: ROUGE-L correlates best with content-verifiability errors, OpenIE with semantic-frame errors, and DAE most strongly with discourse errors.FactCC is nearly uncorrelated with discourse errors, whereas QAGS is best correlated with semantic-frame errors and QGA metrics correlate negatively with discourse errors.
6 Related Work
Prior factuality evaluation uses n-gram, entailment, and question-generation/answering metrics, but existing annotations and benchmarks differ in coverage and scale. FRANK addresses this by covering both CNN/DM and XSum with broader annotations.
- Existing Metrics: BLEU, ROUGE, and METEOR are insufficient for factual correctness and show weak correlation with human factuality judgments.These n-gram-overlap metrics compare lexical similarity rather than factual consistency.
- Existing Metrics: Entailment metrics verify summary units against the source article but require supervision on factual and hallucinated data.FactCC is trained on CNN/DM augmented with artificial factual errors.
- Existing Metrics: FEQA and QAGS generate questions from the summary and answer them separately using the article and summary.These methods reduce factuality evaluation to question generation and answering.
- Prior Annotations: FRANK covers both CNN/DM and XSum to increase data diversity, whereas earlier annotation efforts differed in dataset coverage and scale.Earlier studies annotated 200 or 503 sentences and 470 summaries, according to the cited comparison.
7 Conclusion
FRANK introduces a linguistically grounded factual-error typology and a 2250-summary benchmark with fine-grained annotations. It supports moving beyond binary summary-level factuality toward analysis of error types and metric capabilities.
- Conclusion: FRANK contains human annotations for 2250 summaries from CNN/DM and XSum, covering diverse systems and factual-error categories.The benchmark supports both system factuality assessment and factuality-metric evaluation.
- Conclusion: FRANK’s fine-grained annotations move factuality analysis beyond binary summary-level labels toward identifying specific error types.The benchmark is designed to reveal which factual errors different metrics can capture.
8 Ethical Considerations
The study collected crowd annotations through Amazon Mechanical Turk and used compensation and bonuses intended to support annotation quality.
- Ethical Considerations: Mechanical Turk workers were paid 50% more than the average American minimum wage and offered additional bonuses.Worker information was not released, and worker IDs were anonymized.
A.1 Model details
The paper introduces the models used in the human evaluation task that constructs FRANK.
- The human evaluation task uses model-generated summaries to construct FRANK.
- The model details are presented as part of the FRANK construction process.
- This section concerns the models used for human evaluation rather than the factuality metrics themselves.
A.1.1 CNN/DM datset
This section describes the datasets, models, factuality metrics, annotation procedures, and analyses used to build and evaluate FRANK.
- Datasets and models: The study uses five models on CNN/DM and four models on XSum, whose outputs provide diverse generated summaries for evaluation.
- Metrics: The benchmark compares BERTScore, OpenIE, FEQA, QAGS, DAE, and FactCC using embedding, extraction, question-answering, and entailment-based approaches.
- Metric caveat: DAE results involve article truncation because the model was trained with a maximum concatenated article-summary length of 128.
- Analysis: The analysis examines annotator disagreement through a majority–minority confusion matrix and evaluates metric correlations using partial correlations and Hotelling-Williams tests.