Source-linked AI summary
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization
Esin Durmus, He He, Mona Diab
TL;DR
Neural abstractive summarizers can produce information inconsistent with their sources, while existing metrics inadequately assess faithfulness. This paper studies the abstractiveness–faithfulness trade-off and proposes FEQA, a QA-based metric that correlates better with human judgments, especially for highly abstractive summaries.
Problem
Neural abstractive summarizers may generate unfaithful information, and existing automatic metrics do not effectively evaluate faithfulness against source documents.
Method
The paper collects human faithfulness annotations across two datasets and proposes FEQA, which generates summary questions and checks whether a QA model can answer them from the source document.
Results
FEQA has significantly higher correlation with human faithfulness scores than metrics based on n-gram overlap, embeddings, and language understanding, including on highly abstractive XSum summaries.
Takeaways & Limitations
Faithfulness declines as summaries become more abstractive, and QA-based evaluation is useful for model development but does not replace human or human-in-the-loop evaluation.
Takeaways & Limitations
FEQA is limited by the quality of its QA model, while faithfulness annotation also has relatively low inter-annotator agreement.
Abstract
from arXiv · showhide
Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.
1 Introduction
The paper studies faithfulness evaluation for abstractive summaries, where generated content can conflict with the source. It finds that greater abstractiveness increases unfaithfulness and proposes QA-based evaluation that correlates better with human judgments, especially for highly abstractive summaries.
- Around 30% of generated summaries contain unfaithful information, especially when sentences combine content from multiple source sentences.
- Human annotations across CNN/DM and XSum show that unfaithful sentences increase as summaries become more abstractive and overlap less with the source.
- FEQA generates question-answer pairs from the summary and validates the answers against the source document using reading-comprehension models.
- FEQA has significantly higher correlation with human faithfulness scores than n-gram, embedding, and language-understanding metrics, and is the only metric correlating on highly abstractive XSum summaries.
2 The Abstractiveness-Faithfulness Tradeoff
The paper characterizes abstractiveness by how much summary sentences copy or recombine source text, then examines how dataset and model choices affect generated summaries. More abstractive outputs are associated with lower faithfulness, especially on XSum, while human annotation also becomes harder.
- 2.1 Characterizing Abstractiveness of a Summary: Abstractiveness is categorized by increasing levels of source copying: sentence extraction, span extraction, word extraction, and perfect fusion.Each sentence receives the first qualified category in that order; novel n-gram percentage is also computed.
- 2.2 Is abstractiveness from the model or the data?: Models are evaluated on CNN/DM and XSum using abstractiveness measures for generated and reference summaries.CNN/DM contains news highlights, whereas XSum contains highly abstractive single-sentence BBC summaries.
- 2.2 Is abstractiveness from the model or the data?: CNN/DM is more extractive than XSum: CNN/DM commonly uses sentence compression, while XSum mainly paraphrases content with novel n-grams.No XSum reference sentence is formed by copying from a single source sentence.
- 2.2 Is abstractiveness from the model or the data?: Training data has a larger influence on model-output abstractiveness than the model architecture alone.Models trained on CNN/DM are near-extractive, whereas the same models trained on XSum are significantly more abstractive.
- 2.2 Is abstractiveness from the model or the data?: Model extraction strategies differ: PGC mostly copies complete source sentences, while FASTRL, BOTTOMUP, and BERTSUM more often compress by deletion; BOTTOMUP also uses more fusion than PGC.These distinctions reflect differences in how copying and extraction are implemented across architectures.
- 2.3.1 Human Annotation Results: Faithfulness declines with greater abstractiveness: near-extractive CNN/DM outputs are more faithful than highly abstractive XSum outputs.PGC and TCONV produce faithfulness errors in more than half of their XSum sentences; conflicting information is more common on CNN/DM, while hallucination is more common on XSum.
3 FEQA: Faithfulness Evaluation with Question Answering
FEQA evaluates summary faithfulness by translating important summary spans into questions and checking whether a document QA model recovers matching answers. This addresses the challenge of verifying highly abstractive sentences where surface similarity may fail.
- Motivation: FEQA targets highly abstractive sentences, where surface similarity matching would fail.The desired representation abstracts away surface form while remaining domain-general and interpretable for error analysis.
- FEQA overview: The pipeline masks important summary spans, generates questions using each span as the gold answer, and runs QA on the source document.Masked spans include noun phrases and named entities; the final faithfulness score is the QA answer performance against the summary answers.
- FEQA overview: FEQA represents summary meaning with question-answer pairs and verifies the answers against the source document.More matched answers imply a more faithful summary because the queried information is consistent across the summary and source.
- Question generation: A learned BART model generates natural-language questions from declarative sentences containing masked answer spans.The question generator is trained by adapting the QA2D dataset.
- Answer verification: The verification step uses off-the-shelf QA models and averages F1 against the summary’s gold answers as the sentence-level faithfulness score.The method does not constrain the choice of QA model.
4 Experiments
The experiments compare automatic faithfulness metrics against human annotations on CNN/DM and XSum. QA-based evaluation correlates better with human faithfulness than competing metrics, while highly abstractive settings remain difficult and content selection should be measured separately.
- Metric comparison: QA-based evaluation has higher correlation with human faithfulness scores than other metrics on both CNN/DM and XSum.The comparison covers 748 CNN/DM and 286 XSum document-summary sentence pairs using Pearson and Spearman correlations.
- Metric comparison: Word-overlap metrics correlate with faithfulness in more extractive CNN/DM settings but not in more abstractive XSum settings.All metrics have significantly lower correlation with human scores for XSum, where deeper source-summary understanding is necessary.
- Metric comparison: Entailment metrics usually lack significant faithfulness correlation, while BERTScore can reward overlapping concepts even when the content differs.The paper attributes entailment failures to heuristics such as lexical overlap in models trained on current entailment datasets.
- Content selection and faithfulness: Content selection and faithfulness should be measured separately rather than combined into one score.High content-selection scores do not necessarily imply faithfulness, especially in XSum.
- QA analysis and limitations: Manual analysis found that 78% of QA questions received the correct faithful-versus-unfaithful behavior.About 94% of generated questions were mostly grammatical and correct given their masked spans.
- QA analysis and limitations: FEQA’s QA system makes more mistakes on long articles and can penalize correct answers when source and gold answers do not exactly match.The latter issue is especially relevant in more abstractive settings.
5 Related Work
Prior work uses automated evaluation, QA, entailment, and related approaches to assess generated text. This paper introduces automated question generation for faithfulness evaluation and extends QA-based comparison beyond faithfulness.
- Earlier summarization work used QA as an extrinsic evaluation, and later human evaluations asked readers to answer questions based on summaries.
- This work is the first to apply automated question generation and makes its QA-based metric applicable to semantic comparison between any two texts.
- Automated NLG evaluation is challenging because it often requires deep understanding of the text.
- Word-overlap metrics are commonly used but are known to correlate poorly with human judgments.
6 Conclusion
The paper identifies a trade-off between abstractiveness and faithfulness and proposes FEQA for evaluating summary faithfulness. FEQA correlates better with human judgments, but final evaluation still requires human annotation or human-in-the-loop methods.
- More abstractive sentences tend to concatenate unrelated spans and hallucinate details, reflecting a trade-off between abstractiveness and faithfulness.
- FEQA is a QA-based metric for evaluating the faithfulness of generated summaries.
- FEQA correlates better with human judgment and is useful for model development.
- FEQA is limited by QA-model quality, so final evaluation should still rely on human annotation or human-in-the-loop methods.
A Summarization Datasets
The experiments use CNN/DM and XSum, alongside several summarization models and structured human annotation procedures. The supplied material also documents model architectures, faithfulness guidelines, and examples of meaningful but unfaithful outputs.
- Summarization Datasets: The experiments are conducted on the CNN/DM and XSum summarization datasets.
- Summarization Models: The evaluated systems include copy-based, extract-then-condense, constrained-copying, topic-aware convolutional, and BERT-based summarization models.
- Human Annotation: Workers assess grammaticality using hierarchical judgments that distinguish nonsensical sentences, grammatical issues, and sentences that make sense.
- Human Annotation: Faithfulness annotation compares each output sentence with the source and distinguishes information absent from the source from information conflicting with it.
- Unfaithful Outputs: The examples include meaningful but unfaithful sentences and cases with high content overlap despite unfaithfulness.
D.1 Examples for nonsensical sentences
The examples illustrate nonsensical generations produced by several summarization systems on CNN/DM and XSum. They include repetition, contradictory entities, malformed phrases, and implausible relations.
- PGC produces malformed fragments such as “Sandals, 34, office.co.uk, luluguinness.com.”
- FASTRL outputs include an incomplete phrase about Easter triduum and a contradictory scoreline in which Chelsea defeats Chelsea 5 −3.
- BOTTOMUP generates a malformed sentence combining an actress's name with the phrase “oily vegetables.”
- PGC, TCONV, and BERTSUM produce repeated or self-referential entities involving judges, Stoke City, and Johnny Depp’s management group.
D.2 Examples for meaningful but unfaithful sentences
Meaningful summaries can still be unfaithful, with errors differing across CNN/DM and XSum. Content-overlap metrics may miss these factual problems.
- CNN/DM: CNN/DM faithfulness errors are mostly caused by incorrect concatenation of source sentences.Models fuse two source sentences into a new sentence inconsistent with the source context.
- XSum: XSum faithfulness mistakes are mostly hallucinations involving entities, events, or dates absent from the source.
- Content overlap: Similar content-overlap scores can accompany faithful and unfaithful summaries.Unfaithful outputs may use similar words and discuss similar topics while containing hallucinated or inaccurate information.
D.4 Limitations of the datasets
The CNN/DM and XSum datasets contain noise because they were automatically crawled, and references may include information absent from their source documents.
- Dataset noise: Automatic crawling introduces noise into the CNN/DM and XSum datasets.Source documents can contain phrases such as “click here for the latest news”.
- Reference reliability: Some references contain additional world knowledge not present in the source document.One example adds Ms. Wood’s first name and her role as Plaid Cymru leader.