PaperlayerView exact revision on arXiv

Source-linked AI summary

Shortcomings of Question Answering Based Factuality Frameworks for Error Localization

Ryo Kamoi, Tanya Goyal, Greg Durrett

arXiv:2210.06748v2cs.CL

TL;DR

QA-based factuality metrics are widely used and are intended to localize errors in summaries, but their span-level performance had not been systematically evaluated. This paper analyzes their error localization and question-generation failures, finding poor localization and persistent problems even with human-generated questions.

  • Problem

    QA-based factuality metrics are widely used and motivated by error localization, but their ability to identify erroneous summary spans had not been systematically evaluated.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates QA metrics’ span-level predictions and analyzes question-generation failures, including a human question-generation study.

    Source: §1 · p. 2

  • Results

    QA-based metrics show very poor span-level localization, and inherited errors in generated questions can misclassify factual spans; human question generation does not improve localization.

    Source: §1 · p. 2, §9 · p. 9

  • Takeaways & Limitations

    The observed localization problems indicate fundamental issues in the QA framework that stronger QA and question-generation models alone cannot fix.

    Source: §1 · p. 2, §9 · p. 9

    The evaluation covers a limited set of summarization model outputs.

    Source: §10 · p. 9

Abstract

from arXiv · show

Despite recent progress in abstractive summarization, models often generate summaries with factual errors. Numerous approaches to detect these errors have been proposed, the most popular of which are question answering (QA)-based factuality metrics. These have been shown to work well at predicting summary-level factuality and have potential to localize errors within summaries, but this latter capability has not been systematically evaluated in past research. In this paper, we conduct the first such analysis and find that, contrary to our expectations, QA-based frameworks fail to correctly identify error spans in generated summaries and are outperformed by trivial exact match baselines. Our analysis reveals a major reason for such poor localization: questions generated by the QG module often inherit errors from non-factual summaries which are then propagated further into downstream modules. Moreover, even human-in-the-loop question generation cannot easily offset these problems. Our experiments conclusively show that there exist fundamental issues with localization using the QA framework which cannot be fixed solely by stronger QA and QG models.

1 Introduction

QA-based factuality metrics are widely used and motivated by span-level error localization, but their actual localization performance had not been systematically evaluated. This paper finds that QA metrics localize errors poorly and investigates why.

1 Introduction

  • Although QA-based metrics are widely believed to localize errors, their actual span-level performance had not been systematically evaluated.The paper evaluates error-span identification independently of each metric’s summary-level judgment.Source: §1 · pp. 1–2
  • QA-based metrics show poor span-level error detection, and trivial exact-match baselines outperform them at error localization.Good summary-level performance is rarely accompanied by correct span-level detection.Source: §1 · p. 2
  • Questions for non-factual summaries can inherit errors, causing factual spans to be classified as non-factual during question answering.Shortening questions can reduce inherited errors but leave them under-specified, and human question generation does not improve localization.Source: §1 · p. 2
  • The paper studies QA-based localization, analyzes question-generation failure factors, and uses a human study to test whether human-generated questions address the problems.The study examines inherited errors in long questions and the trade-off with short, under-specified questions.Source: §1 · p. 2

2 QA-Based Factuality Metrics

QA-based factuality metrics turn summary spans into questions, answer them using the source document, and compare predicted answers with the original spans. The paper describes this shared framework and the two metrics it evaluates.

2.1 Basic Framework

  • The framework scores summary spans by comparing source-document answers with candidate spans, then averages span scores into a summary-level factuality score.It generates questions for noun phrases and named entities in the summary.Source: §2.1 · p. 2, Figure 2 · p. 3
  • Span-level similarity scores are intended to reflect factuality: factual spans should receive high similarity and non-factual spans low similarity.The paper uses these scores to evaluate localization, beyond the sentence-level performance assessed in prior work.Source: §2.1 · p. 3

2.2 QA Metrics compared

  • The study focuses on QuestEval and QAFactEval, two of the best-performing QA-based metrics from prior work.QuestEval uses T5-Large models and averages lexical overlap, BERTScore, and answerability; QAFactEval uses an ELECTRA-based QA model, a BART-based question generator, and LERC scoring.Source: §2.2 · p. 3

3 Experimental Setup

The experiments evaluate summary- and span-level factuality predictions against human annotations, using two annotated datasets and several comparison baselines. The setup restricts span evaluation to noun phrases and named entities.

3.1 Task Definition

  • Summary-level predictions are thresholded using validation-set thresholds selected to maximize F1, while span-level predictions use the best-F1 threshold and macro-averaged scores.Evaluation is limited to named entities and noun phrases, a generous setting because errors outside those spans are not penalized.Source: §3.1 · pp. 3–4

3.2 Datasets

  • CLIFF and GD21 provide token-level human factuality annotations for summaries generated by BART and PEGASUS or BART, respectively.Gold summary and span labels are factual only when all tokens in the corresponding unit are factual; about 26% of non-factual tokens fall outside evaluated noun phrases and named entities.Source: §3.2 · p. 4

3.3 Baselines for Comparison

  • The exact-match baseline labels selected summary tokens factual when they occur in the source, while DAE supplies dependency-arc factuality judgments converted to span-level predictions.The baselines’ token-level judgments are converted to span-level judgments to match the evaluation framework.Source: §3.3 · p. 4

4 Summary vs. Span Level Performance

QA metrics’ summary-level performance does not reliably predict their span-level localization performance. Across datasets, exact match outperforms QA metrics at error localization.

4 Summary vs. Span Level Performance

  • Exact match consistently outperforms QA metrics at error localization across all datasets, despite QA metrics’ stronger or comparable summary-level performance on some datasets.The mismatch is substantial for GD21 and CLIFF (CNN/DM).Source: §4 · pp. 4–5, Figure 3 · p. 5
  • No system has a clear advantage over other systems at summary level across all datasets, while DAE’s localization performance is more consistent with its summary-level performance.Source: §4 · p. 5
  • Span-level ROC diagonals arise when many spans receive identical factuality scores, including scores assigned to filtered spans and QAFE unanswerable questions.Source: §4 · p. 5

5 Why do QA metrics fail at span-level error localization?

Generated questions can copy non-factual summary content, leading QA models to misclassify factual spans. The analysis also identifies a trade-off between inherited errors in long questions and under-specification in short ones.

5 Why do QA metrics fail at span-level error localization?

  • Questions for factual spans can inherit non-factual summary content, prompting presupposition failures that classify those factual spans as non-factual.For example, a question can include a false claim about how long an item has been collected.Source: §5 · p. 5, Figure 4 · p. 6
  • Inherited summary errors differ from errors introduced by imperfect QG models and persist independently of improvements to QA or QG models.Source: §5 · p. 6
  • Inherited errors are more common in QAFactEval than QuestEval, which the analysis attributes to QAFactEval’s longer questions.The reported percentages cover non-factual summaries only.Source: §5 · p. 6, Table 2 · p. 6
  • Across settings, factual spans with inherited errors in their questions are more likely to be classified as non-factual, with extrinsic errors tending to harm localization more than intrinsic errors.Source: §5 · p. 6, Table 3 · p. 6
  • Short questions can be under-specified while long questions can inherit errors, creating a specificity trade-off whose effects differ between span- and summary-level performance.Very short questions may permit multiple source-document answers, while inherited errors can hurt span-level performance without affecting summary-level performance.Source: §5 · pp. 6–7

6 Can Human QG Improve Localization?

The authors replace QAFactEval’s automatic question generation with human-written questions of varying lengths to test whether question specificity affects error localization. Human questions generally do not improve localization, while oracle selection does, but selecting the best question requires prior knowledge of summary errors.

6.1 Experiment Design

  • The experiment replaces QAFactEval’s question generation for summary-span pairs with human questions of varying lengths to isolate question length and specificity from model differences.Annotators wrote shortest, longest, and intermediate questions; question length configurations also included an oracle setting.Source: §6.1 · pp. 7–8
  • The study used 150 CLIFF summaries and 2,186 human-written questions, selecting only non-factual CNN/DM summaries because that subset contains few non-factual spans.Half of the summaries were used as validation data.Source: §6.1 · p. 7

6.2 Results

  • Longer questions inherit more errors from non-factual summaries, whereas shorter questions reduce inherited errors but may be underspecified.The Figure 6 analysis considers non-factual summaries because questions generated for factual summaries do not inherit errors.Source: §6.2 · p. 8, Figure 6 · p. 8
  • Human-written questions generally perform similarly to QAFactEval, while oracle questions significantly improve localization without revealing a reliable way to select question length in evaluation.The oracle setting chooses the highest-scoring question for factual spans and the lowest-scoring question for non-factual spans.Source: Figure 7 · p. 8, §6.2 · p. 8

7 Discussion

The discussion argues that QA-based factuality metrics have fundamental problems for producing interpretable, localizable error judgments. It suggests entailment-based approaches because QA’s answer-comparison step can introduce difficulties, even when long questions resemble entailment evaluation.

7 Discussion

  • Long questions can improve summary-level performance by avoiding underspecification, but answering them still introduces comparison difficulties and may yield incorrect evaluations.Questions containing nearly all sentence information can function as a weak form of entailment evaluation.Source: §7 · p. 8
  • The authors characterize QA-based metrics’ localization problems as fundamental and difficult to address in future work.They state that these metrics lack a structural advantage over entailment-based metrics for localizable judgments.Source: §7 · pp. 8–9
  • Because QA-based metrics do not produce interpretable, localizable error judgments, the authors argue they have no structural advantage over entailment-based metrics.The discussion recommends that future system designers favor entailment-based approaches.Source: §7 · pp. 8–9

8 Related Work

Related work spans factuality evaluation, error localization, and applications for localized judgments. The authors distinguish the dominant focus on summary-level metrics from work representing factuality through dependency arcs or semantic graphs.

8 Related Work

9 Conclusion

The conclusion reports that QA-based factuality metrics perform poorly at error localization despite being motivated by that task. The authors analyze factors that hinder localization and use a human study to examine whether human-level question generation resolves them.

9 Conclusion

  • QA-based factuality metrics exhibit extremely poor error-localization capabilities despite their motivation to localize errors.Source: §9 · p. 9
  • The paper analyzes issues in current QA-based metrics that hinder localization performance.Source: §9 · p. 9
  • A human study finds that fundamental issues with the QA framework cannot be fixed simply by stronger models.Source: §9 · p. 9

10 Limitations

The evaluation is constrained by a limited set of outputs and English, mainly newswire summaries; other domains may involve different factuality errors. The appendices also document metric settings, span-score conversion, ignored tokens, and generated-question statistics.

10 Limitations

  • No large-scale span-level factuality dataset exists, and the evaluation covers a limited set of outputs, English, and mainly newswire summaries.The authors note that changing summarization models could quickly make such a dataset outdated, while dialogue and narrative may involve different factuality errors.Source: §10 · p. 9

A.1 QuestEval

  • QuestEval uses implementation version 0.2.4 with the recommended summarization task and do_weighter=False settings.Source: §A.1 · p. 12

A.2 DAE

  • The DAE baseline uses an author-provided model trained on an XSum subset distinct from CLIFF and GD21 examples.Source: §A.2 · p. 12

A.3 Converting Tokens-level factuality scores to span-level scores

  • A span is labeled non-factual when it contains at least one non-factual token in the EM and DAE baselines.Token-level scores are converted to span-level scores to align with QA metrics.Source: §A.3 · p. 12

A.4 Statistics for Ignored non-NP/NE tokens

  • QA metrics omit tokens outside named-entity and noun-phrase boundaries, including some tokens annotated as non-factual.Tables 4 and 5 provide statistics and examples of these ignored tokens.Source: §A.4 · p. 12, Table 4 · p. 12, Table 5 · p. 12

B Statistics of Generated Questions in QA-Based Metrics

C Performance Loss due to Span Filtering

Filtering removes some candidate spans, but QA localization remains poor even on retained spans, and QAFactEval’s subset gains are associated with questions containing fewer factual errors. Additional evaluation covers human question annotation and performance across thresholds.

C Performance Loss due to Span Filtering

D Additional Details about Human QG Annotation

  • Human annotators discarded invalid candidate spans and wrote questions of varying lengths and specificity, sometimes revising automatic questions.Half of the annotated summaries were reserved for validation.Source: D Additional Details about Human QG… · p. 13

E Summary and Span Level Evaluation

Loading 2210.06748v2…