Source-linked AI summary
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, Jianfeng Gao
TL;DR
Common text-quality metrics can miss factual errors in neural generation, motivating better evaluation of factuality metrics. GO FIGURE tests metrics with five conditions across summarization tasks; its analysis finds that reference-summary tests are efficient, but metric behavior can differ between controlled and generated data.
Problem
Common generation metrics may miss structural features such as negation and correlate poorly with human judgments, while neural summaries can contain factual inconsistencies.
Source: §1 · p. 1Method
Results
GO FIGURE evaluates metric sensitivity and validity efficiently using reference summaries; ROUGE-1 and ROUGE-L fail across domains, and some metrics can reverse direction on generated data.
Source: §6.1 · pp. 4–5Takeaways & Limitations
The authors will release the framework and diagnostic datasets to aid development of summarization factuality metrics.
Source: §6.1 · p. 5Factuality is assessed against the source because references may omit salient facts or be unavailable; boundedness tests focus on realistic summarization cases.
Source: §2 · p. 1, §2.2 · p. 2
Abstract
from arXiv · showhide
While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question. This major discrepancy between the surface-level fluency and the content-level correctness of neural generation has motivated a new line of research that seeks automatic metrics for evaluating the factuality of machine text. In this paper, we introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. We propose five necessary and intuitive conditions to evaluate factuality metrics on diagnostic factuality data across three different summarization tasks. Our benchmark analysis on ten factuality metrics reveals that our meta-evaluation framework provides a robust and efficient evaluation that is extensible to multiple types of factual consistency and standard generation metrics, including QA metrics. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
1 Introduction
Neural generation can be fluent yet factually inconsistent, and common text-quality metrics may miss important factual errors. GO FIGURE addresses the need to evaluate factuality metrics across domains.
1 Introduction
- Summarization models have been reported to produce up to 30% factual inconsistencies.These errors can involve distorted or fabricated facts about the source.Source: §1 · p. 1
- Common generation-quality metrics may miss structural features such as negation and correlate poorly with human judgments.These limitations have prompted a search for factuality-driven summarization metrics.Source: §1 · p. 1
- GO FIGURE evaluates factuality metrics across extreme, multi-sentence news, and dialogue summarization.Its contributions include diagnostic conditions, a controlled dataset of context-summary pairs, and annotated model-generated summaries.Source: §1 · p. 1
2 Factuality Metric Meta Evaluation
GO FIGURE evaluates factuality metrics using five conditions and diagnostic tests. Its analysis treats factuality as grounding in the source document and scopes boundedness tests to realistic summarization cases.
2 Factuality Metric Meta Evaluation
- Five conditions are defined for evaluating whether a metric M(D, Si) measures summary factuality with respect to source document D.Reference summaries may omit salient source facts or be unavailable, so factuality is framed in terms of grounding in the source.Source: §2 · p. 1
2.1 Testing Factuality Metric Validity
- The validity tests set empirical bounds, measure sensitivity to injected errors, check robustness and generality across domains and error types, and compare scores with human assessments.The lower bound uses a randomly sampled corpus summary, while the reference ground-truth summary serves as the upper bound.Source: §2.1 · pp. 1–2
2.2 Theoretical Cases
- A hypothetical case expects M(D, D) ≈ 1 and M(D, ∅) ≈ 0 for a metric M valued in [0, 1], a document D, and an empty-string summary ∅.The controlled analysis uses the same common factual error types as the sampled generated summaries, which account for 96.37% of all errors.Source: §2.2 · p. 2, Figure 1 · p. 2
3 Evaluation Datasets
The evaluation covers three summarization domains and uses both controlled error transformations and model-generated summaries. The datasets support testing metric behavior under controlled and real-data conditions.
3 Evaluation Datasets
- The evaluation uses XSUM one-sentence BBC summaries, multi-sentence CNN/DailyMail summaries, and SAMSUM dialogue summaries.SAMSUM consists of English conversations written by linguists and aligned multi-sentence summaries.Source: §3 · pp. 2–3
3.1 Diagnostic Datasets
- Two diagnostic datasets pair source documents with transformed references containing simulated factual errors or summaries generated by transformer models.The authors sample 500 source-summary pairs for each domain.Source: §3.1 · p. 3
3.1.1 Model-Generated Datasets
- For the model-generated dataset, fine-tuned T5 models use beam search or sample-based decoding, and summaries receive fine-grained factual-error annotations.The T5 models were pretrained on news summarization data.Source: §3.1.1 · p. 3
4 Factuality Metrics for Evaluation
The evaluation includes factuality metrics based on question answering and cloze prediction, alongside BERTScore and standard summarization metrics.
4 Factuality Metrics for Evaluation
- The evaluated question-answering metrics are SummaQA, which uses source-derived QA pairs, and FEQA, which uses summary-derived QA pairs.The study also includes BLANC-Help and BLANC-Tune for the masked-token cloze task.Source: §4 · p. 3
- The evaluation also measures BERTScore and standard summarization metrics such as ROUGE.BERTScore is described as aimed primarily at coherency rather than factual consistency.Source: §4 · p. 3
5 Meta-Analysis of Factuality Metrics
On controlled data, QA metrics, ROUGE-(2/3), and BERTScore generally evaluate factuality well, while performance varies by metric and error type. Controlled-data diagnostics generally identify stronger metrics on generated summaries, though FEQA is an exception on XSUM and SAMSUM.
5.1 Controlled Data Experiments
- QA metrics, ROUGE-(2/3), and BERTScore generally evaluate factuality well on controlled data, while ROUGE-(1/L) are frequently invalid and Cloze performance varies across domains.On news domains, performance is generally lower for non-entity errors, except for QA metrics, ROUGE-(2/3), and BERTScore.Source: §5.1 · p. 3
5.2 Comparison with Human Evaluation of Model Generations
- Metrics that assign higher scores to more inconsistent XSUM summaries on controlled data also show invalid behavior on model generations.This supports controlled-data analysis as a diagnostic for weak factuality metrics.Source: §5.2 · p. 3
- Controlled analysis generally identifies SummaQA, ROUGE-(2/3), and BERTScore as better-performing XSUM metrics, but FEQA is an exception.FEQA performs well on controlled XSUM analysis but only approaches this performance on SAMSUM in human evaluation; ROUGE-3 is more consistently correlated with factuality than other ROUGE variations across domains.Source: §5.2 · p. 3
6 Related Work
Prior evaluation research largely focused on general output quality rather than factuality, while work on QA and cloze approaches had examined specific domains. The discussion argues that GO FIGURE can assess metric sensitivity and validity efficiently, while emphasizing the need for human-annotated testing and careful question generation.
6 Related Work
- Earlier evaluation research focused mainly on general output quality, coherence, and fluency rather than factuality, while QA and cloze approaches had been studied on specific domains.The paper aims to evaluate these metrics more broadly, including dialogue summarization.Source: §6 · p. 4
6.1 Discussion of Meta Evaluation and Conclusion
- GO FIGURE evaluates factuality metrics’ sensitivity and validity using reference summaries, without computationally intensive testing across summarization model variants.The authors contrast this with prior work concentrated on one domain and dataset.Source: §6.1 · p. 4
- ROUGE-1 and ROUGE-L fail to measure factual inconsistency accurately across domains in the controlled analysis.A change from “decreased” to “increased” can dramatically change meaning without affecting ROUGE-L; the authors note that human-annotated analysis remains necessary as an upper bound on meta-evaluation quality.Source: §6.1 · p. 5
- QA metrics can perform well across domains, but their effectiveness depends on question generation; asking questions from the source is most robust across domains.The paper notes that both evaluated QA metrics use SQuAD-based systems.Source: §6.1 · p. 5
- The authors plan to release GO FIGURE and its diagnostic datasets, and suggest simulated-data meta-metric results as potential rewards for reinforcement-learning approaches.Source: §6.1 · p. 5
7 Ethics and Broader Impact Statement
The broader-impact discussion presents the framework as a way to aid development of more robust factuality metrics and evaluate metrics for identifying human-written errors. It also notes that human evaluation involved potentially distressing content and describes the review status of crowd-sourced trials.
7 Ethics and Broader Impact Statement
- Human evaluations included news and dialogue content that could reference distressing events or abnormal social behavior; annotators participated voluntarily with prior knowledge of the content type.Crowd-sourced trials were conducted under an IRB exemption.Source: §7 · p. 5
- The framework could aid development of more robust factuality metrics and evaluate metrics for identifying human-written errors.The authors connect this goal to the risk of fluent generations containing factual inaccuracies.Source: §7 · p. 5
A.1 Additional Details of Datasets
The appendix states that dataset statistics are provided for each domain in Table 6.
A.1 Additional Details of Datasets
- Table 6 provides dataset statistics for each domain.Source: §A.1 · p. 8
A.2 Evaluation Metric Details
The section describes four metric families: question answering, masked-token prediction, semantic similarity, and lexical overlap. Their scores assess candidate summaries through answer accuracy, token prediction, embedding overlap, or n-gram overlap.
A.2 Evaluation Metric Details
- SummaQA generates questions from the source and answers them using the candidate summary, while FEQA generates questions from the summary and answers them using the source.Their scores typically aggregate answer F1; SummaQA also generally includes model-confidence probabilities.Source: §A.2 · p. 8
- BLANC-Help and BLANC-Tune score masked-token prediction in the source using summary information, supplied directly for Help or through fine-tuning for Tune.Both aim to capture fluency, informativeness, and factual correctness.Source: §A.2 · p. 8
- BERTScore measures overlap between contextual embeddings of the source or reference document and the candidate summary.The cited work reports stronger correlation with human coherence judgments than standard summarization metrics and similarity to n-gram metrics on CNNDM factual consistency (Wang et al., 2020).Source: §A.2 · p. 8
- ROUGE measures n-gram overlap, with ROUGE-L using longest common subsequence overlap; the evaluation tests ROUGE-1, ROUGE-2, and ROUGE-L.The paper follows prior factual-consistency evaluations (Wang et al., 2020) and notes that ROUGE can underweight good summaries (Novikova et al., 2017).Source: §A.2 · p. 8
A.3 Simulated Data Transformations
The appendix describes simulated factual-error transformations, metric and model settings, and the human-annotation procedure. It also notes that error injection can affect corpora differently depending on summary length.
A.3 Simulated Data Transformations
- Entity extraction supports substitutions: intrinsic errors swap entities with another source entity of the same label, while extrinsic errors draw from corpus entities.The transformations extract entities, verbs, and adjectives from summaries using spaCy; extrinsic errors also replace adjectives with WordNet antonyms.Source: §A.3 · p. 8
- Pronoun errors swap randomly extracted pronoun sets, while verb negation uses rules based on verb tense predicted from suffixes and preceding words.Table 7 is identified as a table of possible factual errors.Source: §A.3 · p. 8, Table 7 · p. 9
- Injected-error effects vary with average summary length, and future work may explore length-controlled injection based on evaluation objectives.The same methodology is used across corpora to maintain consistency.Source: §A.3 · p. 9
A.4 Metric Implementation Details
- BLANC variants receive at most 400 document-input tokens because of BERT context limits; BERTScore uses Roberta-large.Source: §A.4 · p. 9
A.5 T5 Training
- The T5-base model has 220M parameters and is trained on news summaries for each domain using AdaFactor, a 0.001 learning rate, and batch size 8.The learning rate was tuned on a development set using ROUGE, testing values from 0.01 to 0.0001; the best models used one random seed on NVIDIA V100 GPUs.Source: §A.5 · p. 9
A.5.1 Human Annotation Layout
- Annotators assess source, reference, and candidate summaries with factuality choices; they can mark error types, consult knowledge bases for difficult cases, and treat incoherent summaries as maximally inconsistent.Ambiguous imprecision is labeled factually inaccurate under the paper’s strict binary notion.Source: §A.5.1 · p. 9
- Agreement on 30 XSUM summaries was Krippendorff’s α = 0.32 with three annotators and α = 0.44 with two; remaining annotations used one in-house expert.Table 8 summarizes five randomized transformation runs across L1–L3, with average error counts and transformed-summary percentages split by entity versus non-entity errors.Source: §A.5.1 · p. 9, Table 8 · p. 10