Source-linked AI summary
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Alex Wang, Kyunghyun Cho, Mike Lewis
TL;DR
Abstractive summarization models frequently produce factually inconsistent summaries, while standard metrics often miss these errors. The paper proposes QAGS, which generates questions from summaries, answers them using both summaries and sources, and compares the answers. QAGS correlates substantially better with human factuality judgments than other automatic metrics and provides interpretable inconsistency cues.
Problem
Generated summaries often contain factual inconsistencies, but existing automatic metrics are largely insensitive to them.
Method
QAGS generates summary-based questions, answers them using both the source and summary, and compares corresponding answers to score factual consistency.
Results
QAGS correlates substantially better with human factuality judgments than other automatic metrics, including 54.52 versus 17.72 Pearson correlation for ROUGE-2 on CNN/DailyMail.
Takeaways & Limitations
Questions and answers generated during QAGS computation indicate which summary tokens are inconsistent and why.
Takeaways & Limitations
QAGS detects factual inconsistencies relative to the source but does not measure fluency, readability, or factual recall.
Abstract
from arXiv · showhide
Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.
1 Introduction
Abstractive summarization models often produce fluent summaries with factual inconsistencies, while standard evaluation metrics poorly detect such errors. The paper introduces QAGS, which uses questions, answers, and answer similarity to evaluate factual consistency, and reports stronger human-judgment correlations than existing metrics.
- Model-generated summaries frequently contain factual inconsistencies despite being fluent and topical.
- Standard n-gram metrics are insensitive to semantic errors, leaving human evaluation slow, costly, and difficult.
- QAGS generates questions about a summary, answers them using the input and summary, and scores similarity between corresponding answers.
- QAGS requires only a question-answering dataset and can apply wherever question-answering models are available.
- 54.52 Pearson correlation for QAGS versus 17.72 for ROUGE-2 on CNN/DailyMail demonstrates substantially stronger agreement with human factuality judgments.
- QAGS remains more strongly correlated with human judgments than other automatic metrics under the worst ablation settings.
- The paper contributes QAGS, human factuality judgments for two datasets, robustness analyses, interpretability analyses, and released models and code.
2 Background: Automatically Evaluating Machine Generated Text
Common evaluation metrics rely on reference-text n-gram overlap, but this approach is inadequate for factual-consistency checking. References can be costly and sparse, while small semantic changes may preserve high overlap and reverse meaning.
- N-gram metrics compare generated summaries with reference texts using overlap-based precision, recall, or F1 measures.
- ROUGE is the de facto standard summarization metric, with ROUGE-n measuring n-gram F1 and ROUGE-L measuring longest common subsequence.
- N-gram metrics require references, whose collection is expensive and challenging, while many datasets provide only one reference.
- Single-reference comparison is especially inadequate for summarization because many outputs can be acceptable.
- Minor changes can reverse meaning while leaving most n-grams unchanged, making factual inconsistencies difficult to detect.
3 A Framework for Automatically Evaluating Factual Consistency
The proposed framework evaluates whether a summary preserves source information by generating questions from the summary and comparing source- and summary-derived answers. It avoids reference texts and focuses evaluation on semantically relevant content.
- The framework models questions from the summary and answer distributions conditioned on questions and either the source or summary.
- Its consistency objective is maximized when the summary yields the same answers as the source for questions generated from the summary.
- QAGS generates questions from the summary, answers them with both source and summary, compares corresponding answers, and averages similarities.
- Unlike n-gram methods, the framework needs no reference text and emphasizes semantically relevant portions through questions.
- Highly probable questions, such as beam-search outputs, reduce the number of questions needed but may introduce bias.
4 QAGS
QAGS instantiates the framework with question-generation, extractive question-answering, and token-level answer-similarity components. These choices provide a practical factual-consistency score while assuming relevant facts appear as text spans.
- QAGS specifies question, answer, and answer-similarity distributions before applying the framework to summarization.
- Question Generation: Question generation uses answer-conditional neural sequence-to-sequence models with named entities and noun phrases as answer candidates.
- Question Generation: Candidate questions are filtered using heuristics and by removing questions for which the QA model predicts no answer.
- Question Answering: Extractive QA models instantiate the answer distributions because the framework assumes facts appear as text spans in the article and summary.
- Answer Similarity: Token-level F1 compares the source- and summary-derived answers as the answer-similarity function.
5 Experiments
Experiments evaluate QAGS against automatic metrics and human factual-consistency judgments on CNN/DM and XSUM, including robustness to model quality, domain shift, and question count.
- Results: On CNN/DM, QAGS nearly doubles the next-best automatic metric’s correlation, while QA detects errors caused by sentence fusion.The QA model produces different answers for the source and summary when two source sentences are fused into an incorrect statement.
- Results: On XSUM, all metrics correlate worse with human judgments than on CNN/DM, but QAGS still outperforms the next-best automatic metric.The lower correlations reflect XSUM’s more abstractive summaries.
- Ablations: QAGS remains robust across QA and QG model quality, although weaker or mismatched models can reduce correlations.QA-model correlations are stable; increasing QG perplexity decreases CNN/DM correlation somewhat, with no clear XSUM trend.
- Ablations: Domain shift lowers QAGS correlations from 54.53 to 51.53 on CNN/DM and from 17.49 to 15.28 on XSUM, yet QAGS still outperforms other metrics.The shift uses a SQuAD-trained QG model instead of one trained on NewsQA.
- Ablations: Increasing the number of questions improves correlation, with a large gain from 10 to 20 and decreasing marginal benefit beyond 50.Even five questions substantially outperform other automatic metrics.
- Results: Using exact match instead of F1 yields Pearson correlations of 45.97 and 18.10, versus 54.53 and 17.49 with F1 on CNN/DM and XSUM.F1 is less restrictive than exact match for comparing QA answers.
6 Re-ranking with QAGS
QAGS is compared with NLI-based factual-consistency methods on a sentence-ranking experiment, where it outperforms the alternatives without task-specific supervision.
- The evaluation uses two summary sentences per source sentence, with one factually consistent and the other inconsistent.
- QAGS outperforms BERT NLI, ESIM, and FactCC on the sentence-ranking experiment from Falke et al. (2019).The experiment contains 373 CNN/DailyMail source-sentence triplets paired with consistent and inconsistent generated summary sentences.
- QAGS requires no special supervision for the factual-consistency checking task.
7 Qualitative Analysis
Qualitative analyses show that QAGS produces interpretable questions and answers that expose summary errors, while also revealing answer-model and similarity-based failure modes.
- Interpretability: QAGS questions and answers indicate which summary tokens are factually consistent or inconsistent.The paper presents example questions and answers generated during QAGS computation.
- Interpretability: QAGS detects errors in attackers’ names, attack locations, and weapons because its questions focus on these details.Named entities and noun phrases make QAGS particularly effective for this error type.
- Error Analysis: 8.75% of generated questions are nonsensical, while 3.00% are well-formed but unanswerable from the conditioned summary.The authors report that most generated questions are understandable and on-topic.
- Error Analysis: 8.25% of questions are well-formed but unanswerable from the source, usually because the summary contains a hallucinated fact.The question-generation model can turn a hallucinated summary fact into a question.
- Error Analysis: The QA model incorrectly answers 32.50% of potentially answerable article questions, compared with 1.75% for summary questions.This indicates limited transfer ability for the QA model from summary-conditioned questions to article answers.
- Failure Modes: F1 misses similarity in 8.00% of examples when article and summary answers are correct but have high lexical variation.The authors suggest exploring similarity metrics beyond n-gram approaches.
8 Related Work
The paper situates QAGS among n-gram, information-extraction, NLI, and QA-based approaches, differing from prior QA metrics through model-generated questions conditioned on generated summaries.
- ROUGE remains the de facto standard for summarization evaluation, despite limited ability to assess summary quality.
- Information-extraction methods measure factual overlap but restrict facts to predefined schemas, while NLI methods have been found brittle.
- Prior QA-based metrics use Cloze-style questions formed by masking entities in source documents or reference summaries.
- QAGS instead generates a broader range of questions conditioned on the generated summary rather than the reference summary or source article.
9 Conclusion
The paper introduces QAGS as an interpretable factual-consistency metric for abstractive summarization and reports stronger alignment with human judgments than standard and NLI-based metrics.
- QAGS automatically detects factual inconsistencies in conditionally generated text and measures inconsistencies in abstractive summaries.
- QAGS correlates with human factuality judgments significantly better than standard automatic summarization metrics.
- QAGS outperforms related NLI-based approaches to factual-consistency checking.
- The questions and answers generated during QAGS computation indicate which summary tokens are inconsistent and why.
- Error analysis indicates that improved QA models are a direction for future work.
A Human Evaluation Task Design
The factual-consistency annotation task used screened workers, quality checks, explanations, and dataset-specific instructions for CNN/DM and XSUM.
- Workers were US-based, had at least 1,000 approved HITs, and maintained a 98% acceptance rate.
- Workers received $0.15 initially and a $0.85 bonus for each correct annotation.The total reward was $1.00 per correct annotation.
- The task applied time, attention, and explanation checks to validate annotation quality.Workers completing the task under 30 seconds failed; attention checks used copied and corrupted article sentences, while each summary sentence required an explanation.
- CNN/DM and XSUM used slightly different instructions to accommodate dataset-specific quirks.For XSUM, the reference summary was prepended to the source because workers otherwise struggled to identify factual inconsistencies.
B Model and Generation Details
QAGS generates questions from summaries, answers them using source articles, and filters the resulting questions before scoring factual consistency.
- Question Generation: Question generation fine-tunes BART on answer-conditional NewsQA data, predicting questions from an answer and its source article.At test time, answer candidates are drawn from named entities and noun phrases using spaCy.
- Question Generation: Beam search uses size 10, length penalty 1.0, trigram repetition blocking, and question lengths from 8 to 60 tokens.Top-k and top-p sampling produced diverse but noisy questions, so the reported decoding setup uses beam search.
- Question Filtering: Question filtering removes malformed or short questions, then discards questions that the QA model judges unanswerable.The system removes text after the first question mark, rejects questions shorter than three tokens, and retains up to 20 probable questions.
- Question Answering: Question answering fine-tunes BERT using the original work’s procedure and AdamW optimization.
- Question Answering: SQuAD2.0 supplies unanswerable questions for filtering and for detecting summary facts that cannot be answered from the source article.The question and answer are appended to the source article with intervening special marker tokens.