Source-linked AI summary
QuestEval: Summarization Asks for Fact-based Evaluation
Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang
TL;DR
Summarization metrics often correlate poorly with human judgments, particularly when summaries must select salient information and remain factually consistent. QuestEval unifies QA-based precision and recall with query weighting in a reference-less framework. It correlates better with human judgments across four evaluation dimensions while requiring no gold reference.
Problem
Current summarization metrics correlate poorly with human judgments and fail to adequately capture salient information selection and factual consistency.
Method
QuestEval unifies QA-based precision and recall, adds question weighting and negative sampling, and uses question generation and answering without gold references.
Results
QuestEval correlates dramatically better with human judgments across consistency, coherence, fluency, and relevance than existing metrics.
Takeaways & Limitations
QuestEval enables more accurate comparison between summarization systems and evaluation on datasets without gold references.
Takeaways & Limitations
Using larger beam sizes increases inference time linearly, while the improvement from K = 1 to K = 20 is small.
Abstract
from arXiv · showhide
Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.
1 Introduction
Summarization evaluation is difficult because reference-based and n-gram metrics correlate poorly with human judgments and miss factual consistency. QuestEval addresses these limitations as a reference-less metric by unifying QA-based precision and recall, weighting generated queries, and improving correlation across evaluation dimensions.
- Motivation: Summarization has many possible correct outputs, making evaluation from a single reference especially difficult.Information selection is critical because summaries must be shorter than their source documents.
- Motivation: N-gram metrics such as ROUGE poorly reflect human preferences and do not measure factual consistency.Fluent summaries can still contain false or unsupported information.
- QuestEval: QuestEval unifies precision- and recall-based QA metrics to obtain a more robust summarization metric.The framework is designed to evaluate summaries without requiring a ground-truth reference.
- QuestEval: QuestEval learns query saliency so that evaluation can account for information selection.The weighting component extends prior QA-based approaches with improved recall.
- Evaluation: QuestEval achieves state-of-the-art correlation with human judgments across all evaluated dimensions on CNN/Daily Mail and XSUM annotations.The paper highlights its effectiveness for measuring factual consistency.
2 Related Work
Prior summarization metrics rely mainly on n-gram or semantic overlap, while QA-based metrics assess information recall or precision. QuestEval builds on these approaches to address their complementary limitations.
- N-gram and semantic metrics: ROUGE, CIDEr, and METEOR are widely used overlap-based metrics, but SummEval found none correlated better with humans than ROUGE.ROUGE measures recall of reference n-grams in the evaluated summary.
- Recent alternatives: BERTScore has performance similar to ROUGE, while NLI-based factuality metrics have produced mixed results relative to QA-based methods.These approaches use pretrained language models or entailment judgments rather than question answering alone.
- QA-based metrics: SummaQA generates questions from the source to test recalled information, whereas QAGS generates questions from the summary to test precision.SummaQA weights questions equally and therefore lacks a mechanism to select the most important information.
3 A Question-Answering based Framework
QuestEval evaluates summaries with question generation and answering, using both source documents and generated summaries to assess factual consistency and relevance without human references. Its components generate answer-conditioned questions and use a pretrained QA model to answer them, including an unanswerable output.
- Framework: QuestEval combines question generation and question answering to evaluate factual consistency and relevance without human references.The framework consists of QG and QA components.
- Question Answering: The QA component uses a pretrained T5 model to extract answers from a source document given a question.QA(r|T, q) denotes the probability of answer r for question q on text T, while QA(T, q) is the greedily generated answer.
- Question Answering: The QA model can output an unanswerable token because an evaluated summary may not contain the answer to a question.This explicitly represents questions that cannot be answered from the summary.
- Question Generation: The QG component uses a T5 model fine-tuned to generate human-like questions conditioned on an answer and its source document.At test time, selected answers from the text condition question generation.
- Question Generation: QuestEval selects named entities and nouns as candidate answers, generates questions by beam search, and retains pairs whose answers the QA model predicts correctly.QG(T) is the set of question-answer pairs satisfying QA(T, q) = r.
4 The QuestEval metric
QuestEval evaluates summaries through factual consistency and coverage of important source information. It combines precision and recall using question-answering signals, including answer overlap, answerability, and query importance.
- 4.1 Precision: Precision measures factual consistency by comparing answers generated from the source document and evaluated summary for summary-generated questions.The comparison uses question-answering answer overlap to detect whether the summary and source support the same answers.
- 4.2 Recall: Recall measures whether the summary contains important information from the source, using weighted answerability for source-generated questions.The query weighter assigns importance probabilities to questions and is applied to recall, not precision.
- 4.2 Recall: QuestEval addresses answer-expression variation in recall while retaining F1 for precision because factual consistency requires detecting incorrect answers.Answerability can recognize differently expressed correct answers, whereas precision must distinguish answerable but incorrect summary claims.
- QuestEval score: The final QuestEval score is the harmonic mean of precision and recall and lies in the same numerical range as ROUGE or BLEU.The method therefore combines factual consistency and source-information coverage into one comparable score.
5 Experiments
QuestEval is evaluated against human judgments across SummEval and QAGS-XSUM, with analyses of reference efficiency, query weighting, explainability, negative sampling, and computational cost. It achieves stronger correlations while avoiding gold references, though added beam search increases inference time.
- Evaluation datasets: QuestEval is evaluated on SummEval and QAGS-XSUM, including consistency, coherence, fluency, relevance, and correctness judgments.SummEval contains 12,800 summary-level annotations from 16 systems; QAGS-XSUM contains 239 BART outputs assessed for correctness.
- Main results: 18%: unifying precision and recall improves average correlation from 28.4 to 33.5, combining complementary strengths of SummaQA and QAGS.SummaQA correlates more with Relevance, whereas QAGS correlates more with Consistency.
- Query weighting: +4%, from 37.5 to 39.2: learned question weighting most benefits Relevance by identifying questions targeting important information.Important-and-answered questions correlate positively with Relevance, while important-but-unanswered questions correlate negatively; answered-but-unimportant questions do not correlate.
- Main results: 33.5 versus 11.8 for BERTScore: QuestEval achieves a substantially higher average Pearson correlation while requiring no references.The comparison is reported for SummEval, where QA-based metrics are reference-free and QuestEval unifies precision- and recall-oriented components.
- Model training: Negative sampling lowers performance from 33.3 to 32.4 on SummEval and from 30.4 to 28.5 on QAGS-XSUM when omitted.The authors relate this to better separation of negatively sampled questions by QA models trained with synthetic unanswerable examples.
- Computational complexity: 1.2 points: increasing QG beam size from K = 1 to K = 20 improves correlation only from 34.4 to 35.6, while inference time grows linearly.The experiments use K = 20 beams because retaining question variants improves correlations, but the computational trade-off limits efficiency.
6 Conclusion
QuestEval is a reference-less framework that unifies and extends QA-based summarization evaluation with question weighting and negative sampling. It correlates better with human judgments while supporting evaluation without gold references, and is being adapted to other NLG tasks.
- QuestEval is a new reference-less framework for evaluating summarization models.
- It unifies and extends prior QA-based approaches with question weighting and negative sampling.
- QuestEval accounts for factual consistency, relevance, and information selection in summary evaluation.
- QuestEval correlates dramatically better with human judgments than existing metrics without requiring any gold reference.
- The authors are adapting QuestEval to machine translation and text simplification, and plan a multilingual version.