Source-linked AI summary
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong
TL;DR
Factual-consistency evaluation lacks consistent conclusions about entailment- versus QA-based metrics across differing experimental setups. The paper systematically analyzes metric components, proposes QAFactEval, and finds improved QA performance plus complementary signals from combining QA and entailment metrics.
Problem
Prior evaluations of entailment- and QA-based factual-consistency metrics use differing setups and reach contrasting conclusions about which paradigm performs best.
Method
The paper extensively compares metric components and optimizes a QA-based pipeline, especially question generation and answerability classification, to create QAFactEval.
Results
14% average improvement over previous QA-based metrics is achieved on SummaC, QAFactEval outperforms the best entailment-based metric, and combining QA with entailment further improves performance.
Takeaways & Limitations
Question generation and answerability detection are key components for improving QA-based factual-consistency metrics, while QA and entailment provide complementary signals.
Takeaways & Limitations
Current metrics fail to detect all factual inconsistencies, and their English-trained models and datasets may propagate cultural, political, or gender biases.
Abstract
from arXiv · showhide
Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
1 Introduction
Factual consistency is essential because summarization models can produce fluent summaries that are inconsistent with their source documents. This paper compares entailment- and QA-based metrics, analyzes QA pipeline components, and proposes QAFactEval.
- Factual consistency remains essential because state-of-the-art summarization models are not always consistent with their source documents.
- Entailment-based metrics assess whether summary content is entailed by the source, whereas QA-based metrics score answers to questions generated from the summary.
- Prior comparisons reached differing conclusions about the stronger paradigm because they varied baseline models, input granularity, and datasets.
- The authors analyze answer selection, question generation, question answering, and answer-overlap evaluation as four QA-metric components.
- 14% average improvement over prior QA-based metrics is achieved on the six-dataset SummaC benchmark, while QAFactEval also outperforms entailment-based metrics.
- QA-based and NLI-based metrics provide complementary signals that can be combined through a learned network for further performance improvement.
2 Related Work
Related work develops entailment- and QA-based approaches for evaluating factual consistency and increasingly emphasizes comparisons across datasets. SummaC unifies entailment-metric analysis, while this work extends comprehensive evaluation toward QA metrics and correlation analysis.
- Entailment-based work has examined ranking, classification, contradiction extraction, input granularity, adversarial training, and entailment backbones.
- Laban et al. unify entailment-metric comparisons by studying granularity, base models, and hyperparameter choices, but prior comparisons often used isolated datasets.
- QA-based metrics have been applied to summary quality dimensions beyond factual consistency and can better measure information-unit overlap for relevance than embedding-based metrics.
- Summarization benchmarking has documented model biases and failure modes, collected multidimensional annotations, and developed error typologies and factual-consistency meta-evaluation.
- This work uses the SummaC benchmark, emphasizes QA-based factual-consistency metrics, and adds correlation analysis for broader metric understanding.
3 Evaluation Metrics
The evaluation metrics comprise entailment, QA-based, and learned approaches. QA metrics decompose factual consistency into answer selection, question generation, question answering, filtering, and overlap scoring, while learned metrics aggregate fine-grained or sentence-level signals.
- Entailment-based Metrics: Entailment metrics produce scores for summary sentences and average them, with models differing in granularity, training data, and aggregation.
- Entailment-based Metrics: MNLI uses RoBERTa-large trained on MNLI and scores each summary sentence by its maximum entailment score over source sentences.
- QA Metric Components: QAFACTEVAL combines answer selection, question generation, source-document question answering, answer-overlap evaluation, and question filtering.
- QA Metric Components: Question generation conditions on selected answers and summary context, while question answering uses the source document to produce extractive or abstractive answers.
- QA Metric Components: Unanswerable questions can receive zero overlap through the Answerability Penalty, and the final score averages remaining question scores after filtering.
- Learned Metrics: SCConv aggregates sentence-level entailment scores, whereas QAFACTEVAL-NLI combines SCConv with finer-grained answer-overlap scores.
- Additional Metrics: QuestEval is a prior state-of-the-art QA metric that generates questions from the source and answers them using the summary, potentially confounding factual consistency with relevance.
4 Methodology
The study evaluates factual-consistency metrics on SummaC datasets and reports ablation, final, and correlation analyses. It uses established benchmark datasets, modified Polytope labeling, and synthetic or supervised training settings.
- The evaluation includes binary classification and correlation analyses, with separate settings for ablations and final results.
- SummaC labels examples positive when they contain no factual inconsistencies or receive the highest Likert score, and negative otherwise.
- The benchmark covers CGS, XSF, Polytope, FactCC, SummEval, and FRANK, while QAGs is used only for correlation analysis.
- Polytope examples labeled as Omission or Addition were treated as factually consistent because they were largely extractive and factually consistent.
- Metrics were implemented from original repositories or SacreRouge, with learned metrics using released Laban et al. code and PyTorch or Transformers.
- Binary-classification thresholds are selected on validation data and applied to test data, while ablations split validation data to preserve the test set.
- Synthetic learned-metric training uses 50k FactCC points, while supervised training fine-tunes on each dataset’s SummaC validation set before test evaluation.
5 Results
The results show that QAFACTEVAL’s performance depends strongly on component choices, especially answer selection, question generation, answerability handling, and overlap scoring. On SummaC, it substantially improves over prior QA metrics and entailment-based metrics, while QA and NLI signals provide complementary gains when combined.
- Ablation Results: NP Chunk answer selection performs best, whereas named-entity selection causes a large performance decrease because it extracts only three entities on average versus over 10 answers for other approaches.Max NP and selecting all answers produce only slight decreases relative to NP Chunks.
- Ablation Results: QAFACTEVAL’s ablations show that answer selection, question generation, answerability handling, and answer-overlap scoring materially affect factual consistency performance.The best configuration uses NP Chunk answer selection, BART-large (QA2D) question generation, answerability filtering and penalty, and a learned overlap metric.
- Ablation Results: BART-large (QA2D) produces the strongest question-generation results, with longer, more extractive questions that are easier for the QA model to answer than more abstractive alternatives.Its questions average about 17 tokens and 20% novel unigrams, compared with about 10 tokens from other models and about 47% novel unigrams for T5-base (SQuAD).
- Ablation Results: Answerability filtering and the Answerability Penalty are important because answer overlap can incorrectly score contextually inconsistent answers as correct.Removing both components causes a drastic performance decrease, particularly for intrinsic errors where the same answer appears in different contexts.
- Overall Results: QAFACTEVAL substantially improves over QuestEval and outperforms all other entailment-based metrics on the six-dataset SummaC benchmark.QAFACTEVAL-NLI provides additional improvements, including a statistically significant overall benchmark gain after supervised fine-tuning.
- Overall Results: QAFACTEVAL and SCConv provide complementary signals that improve performance when combined, whereas individually fine-tuning either learned metric does not improve over the corresponding non-supervised metric.The learned combination is dataset dependent, and training on combined validation sets does not improve results.
- Correlation Analysis: QAFACTEVAL performs well across most correlation datasets, but FactCC synthetic-data fine-tuning does not capture XSF’s aggregated, word-level factuality scores.The authors leave supervised fine-tuning with a regression loss for future work.
6 Conclusion
The conclusion identifies question generation and answerability detection as key components for improving QA-based factual consistency metrics. It also finds that combining QA-based and entailment-based metrics yields state-of-the-art performance on the benchmark.
- Question generation and answerability detection are identified as key components for improving QA-based factual consistency metrics.
- QA-based metrics with properly optimized components outperform entailment-based metrics on a comprehensive factual consistency benchmark.
- A combined metric using entailment and QA-based signals achieves state-of-the-art performance on the benchmark.
- The work offers a basis for future QA-based factual consistency research through fairer comparisons across datasets and settings.
7 Ethical Considerations
The paper highlights potential dataset and model biases, incomplete detection of factual inconsistencies, and environmental costs associated with its experiments. These considerations constrain how the metrics should be applied and interpreted.
- Dataset Biases: The metrics are based on English-trained models and may mainly represent English-speaking culture.
- Dataset Biases: Political or gender biases may exist in the datasets and models and may subsequently be propagated by the metrics.
- Dataset Biases: The authors did not stress test the metrics for potential biases and advise users to remain aware of these issues.
- Misuse Potential and Failure Mode: The current metrics fail to detect all factual inconsistencies, limiting their use as filters for downstream applications.
- Environmental Cost: Experiments primarily used A100 GPUs, while training learned answer-overlap components can take a couple of hours.
- Environmental Cost: Future work may examine whether distilled backbone models affect factual consistency evaluation.
A Additional Data and Model Details
This section provides additional information about the paper’s statistical testing, benchmark statistics, and QA-based experiments.
- The section provides details regarding statistical testing.
- It also provides benchmark statistics.
- Miscellaneous details regarding the QA-based experiments are included.
A.1 Statistical Testing
The authors use bootstrap resampling and corrected confidence-interval comparisons to assess whether SummaC improvements are statistically significant. Significance appears for the overall benchmark and FRANK subset, but not the other datasets.
- Bootstrap resampling is used to test whether improvements on the SummaC benchmark are statistically significant.
- Confidence intervals are compared at significance levels of 0.05 and 0.01 with Bonferroni correction.
- Statistically significant differences at the 0.01 level occur for QAFACTEVAL-NLI (supervised) on FRANK and for the overall benchmark.
- No statistically significant differences are observed on the other benchmark datasets.
- The overall significance result indicates consistent good performance across datasets, despite other metrics sometimes performing comparably or better on individual datasets.
A.2 Benchmark Statistics
The benchmark section documents dataset-statistics conventions, licensing information, correlation exclusions, and supplementary ablation details for the evaluation setup.
- The Polytope replication contains more positive examples because Omission and Addition are excluded as factual-consistency errors.
- XSF is restricted to source-related factual-consistency labels, producing fewer examples than the original SummaC benchmark.
- Dataset and annotation licenses were reviewed, and their intended uses were considered compatible with research use.
- Summary-level correlations are excluded for QAGS because the dataset lacks annotations for multiple models.
- Table 6 reports validation and test-set sizes and validation-set positive-example percentages for six SummaC datasets.
- The ablation setup keeps the best component constant, while replacing NP Chunks with a second-best answer-selection component reduced all scores.
B Additional Correlation Results
Additional tables report instance-level and summary-level correlations across factual-consistency datasets using Spearman, Kendall, and Pearson coefficients.
- Table 7 reports instance-level Spearman correlation coefficients across factual-consistency evaluation datasets.
- The correlation tables divide metrics into entailment-based, QA-based, and learned metrics fine-tuned on synthetic or supervised data.
- The tables identify the two highest-correlated metrics for each dataset in bold.
- Table 8 reports instance-level Kendall correlation coefficients across factual-consistency evaluation datasets.
- Table 9 reports summary-level Pearson correlation coefficients across factual-consistency evaluation datasets.
- Tables 10 and 11 report summary-level Spearman and Kendall correlation coefficients, respectively.