PaperlayerView exact revision on arXiv

Source-linked AI summary

Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking

Mubashara Akhtar, Michael Schlichtkrull, Andreas Vlachos

arXiv:2411.05375v2cs.CLcs.AIcs.IRcs.LG

TL;DR

Automated fact-checking evidence metrics can rely on verdicts, closed-source exact matches, or surface matching, each with limitations for evaluating evidence. Ev2R combines reference alignment with verdict-level proxy scoring; the paper reports stronger human-rating correlation and adversarial robustness than existing approaches. Its findings are limited to English-language datasets.

  • Problem

    AFC evidence evaluation can overlook valid evidence or rely on closed knowledge sources and surface matching that do not capture alternative evidence paths.

    Source: §1 · p. 1

  • Method

    Ev2R combines reference-based evidence comparison with verdict-level proxy scoring to assess alignment with references and the annotated verdict.

    Source: §1 · pp. 1–2

  • Results

    Ev2R outperforms traditional reference-based baselines across VitaminC, FEVER, and AVeriTeC, with greater adversarial robustness than baselines.

    Source: §5 · pp. 10–11

  • Takeaways & Limitations

    Reference-based scoring correlates more strongly with human assessments for reference coverage and retrieved-evidence relevance, while proxy-reference scoring provides the strongest reported correlations across datasets.

    Source: §5 · p. 10

    The evaluated datasets are restricted to English, limiting the findings' generalisability to multilingual fact-checking contexts.

    Source: §7 · p. 12

Abstract

from arXiv · show

Current automated fact-checking (AFC) approaches typically evaluate evidence either implicitly via the predicted verdicts or through exact matches with predefined closed knowledge sources, such as Wikipedia. However, these methods are limited due to their reliance on evaluation metrics originally designed for other purposes and constraints from closed knowledge sources. In this work, we introduce \textbf{\textcolor{skyblue}{Ev\textsuperscript{2}}\textcolor{orangebrown}{R}} which combines the strengths of reference-based evaluation and verdict-level proxy scoring. Ev\textsuperscript{2}R jointly assesses how well the evidence aligns with the gold references and how reliably it supports the verdict, addressing the shortcomings of prior methods. We evaluate Ev\textsuperscript{2}R against three types of evidence evaluation approaches: reference-based, proxy-reference, and reference-less baselines. Assessments against human ratings and adversarial tests demonstrate that Ev\textsuperscript{2}R consistently outperforms existing scoring approaches in accuracy and robustness. It achieves stronger correlation with human judgments and greater robustness to adversarial perturbations, establishing it as a reliable metric for evidence evaluation in AFC.\footnote{Code is available at \href{https://github.com/mubasharaak/fc-evidence-evaluation}{https://github.com/mubasharaak/fc-evidence-evaluation}.}

1 Introduction

Prior evidence metrics can overlook evidence quality, reject valid alternatives, or rely on closed sources and surface matching. Ev2R combines reference alignment with verdict-level scoring, and the paper reports stronger human-rating correlation and adversarial robustness than existing approaches.

  • Prior approaches rely on verdicts, exact matches in closed knowledge sources, or token matching, which can miss evidence quality, valid alternatives, or open-web evidence.Token-matching metrics are sensitive to surface-form differences and may fail to recognize alternative evidence paths.Source: §1 · p. 1
  • Ev2R combines reference-based evidence comparison with a proxy-reference component that uses the annotated verdict label to assess retrieved evidence.Reference-based scoring reflects precision and recall against annotations, while verdict alignment provides a potentially noisy signal for alternative reasoning paths.Source: §1 · pp. 1–2
  • Evaluation spans AVeriTeC, FEVER, and VitaminC against reference-based, proxy-reference, and reference-less baselines.AVeriTeC uses real-world claims and web evidence; FEVER and VitaminC include multiple evidence sets per claim, sometimes with conflicting verdicts.Source: §1 · p. 2
  • Ev2R outperforms baselines in correlation with human ratings across all three datasets and is more robust to redundancy and noise perturbations.The authors conclude that combining reference- and verdict-level scoring offers a more reliable approach to evidence evaluation in AFC.Source: §1 · p. 2

2 Related Work

Evidence-based AFC faces evaluation challenges when evidence is open-ended, multiple reasoning paths are valid, or annotations omit later evidence. NLG evaluation offers reference-based, proxy-reference, and reference-less approaches that inform ways to assess generated text.

2.1 Evidence-based AFC

  • Closed-source AFC datasets enable simple conditional evidence scoring but restrict candidate evidence to Wikipedia, a limitation described as unrealistic.Recent internet-evidence datasets complicate evaluation because document or paragraph exact matches no longer suffice.Source: §2.1 · pp. 2–3
  • Open-web claims may have multiple valid evidence sets, and correct evidence can be absent from annotations if it emerged after dataset creation.Different reasoning chains can verify or contradict a claim, so a single annotated set may not capture every valid route.Source: §2.1 · p. 3

2.2 Evaluation Approaches for Natural Language Generation

  • NLG evaluation has moved from heuristic string-matching metrics toward learned language-model metrics and LLM-based conditional generation.These approaches draw on the open-ended nature of generation tasks, where multiple outputs may be valid against one reference.Source: §2.2 · p. 3
  • NLG evaluation distinguishes reference-based comparison, proxy-reference scoring, and reference-less scoring, including factuality checks over atomic facts.Examples include contextualized-similarity metrics, proxy checks using questions or entailment labels, and FactScore's ratio of supported facts.Source: §2.2 · pp. 3–4

3 Ev2R: A Weighted Scorer for Evidence Evaluation

Ev2R combines reference-based factual alignment with verdict-label proxy scoring. Its components assess evidence precision and recall, verdict alignment, and a final equally weighted score.

3 Ev2R: A Weighted Scorer for Evidence Evaluation

  • Ev2R combines reference-based precision and recall with proxy-based alignment between predicted and reference verdict labels.The verdict label acts as a proxy for evaluating evidence accuracy.Source: §3 · p. 4

3.1 Reference-based

  • The reference-based scorer decomposes retrieved and reference evidence into atomic facts, then checks their factual support against one another.The scorer relies on the provided texts rather than external sources or background knowledge.Source: §3.1 · p. 4
  • Reference-based precision measures the share of retrieved facts supported by the reference, while recall measures how many reference facts appear in retrieved evidence.The scorer also provides an aggregated F1 score; reference-only scoring may underestimate correct evidence that differs from the reference set.Source: §3.1 · pp. 4–5

3.2 Proxy-reference

  • The proxy-reference component uses a fine-tuned DeBERTa model to estimate confidence in the reference verdict for a claim-evidence pair.The model was initialized from a checkpoint pretrained on diverse NLI datasets and further fine-tuned on AFC datasets.Source: §3.2 · p. 5

3.3 Final Weighted Scorer

  • The final Ev2R score equally weights the reference-based F1 score and proxy score using α = 0.5.This combination reflects evidence correctness and completeness relative to the reference, together with alignment to the verdict.Source: §3.3 · p. 5

4 Evaluation of Ev2R

The evaluation combines adversarial tests, human ratings, and evidence pairs from AFC benchmarks to assess Ev2R and compare it with baseline scorers. The section describes the data sources, rating dimensions, and baseline approaches used.

4 Evaluation of Ev2R

  • The evaluation draws on perturbed evidence, AVeriTeC shared-task submissions, and evidence pairs from FEVER and VitaminC, then compares Ev2R with baseline scorers.The section outlines these three data sources before describing the evaluation methods and scorers.Source: §4 · pp. 5–6

4.1 Evaluation with Adversarial Tests

  • Adversarial tests perturb claims or evidence with semantics-preserving changes, such as typos, and meaning-changing changes, such as removing significant evidence.The tests assess whether scores stay stable under semantically equivalent changes and drop when evidence meaning changes.Source: §4.1 · p. 6

4.2 Evaluation with Human Ratings

  • Human evaluation uses AVeriTeC evidence samples and measures coverage, relevance, and verdict agreement; categorical verdict annotations show Krippendorff’s Alpha 0.732 and Fleiss’ Kappa 0.727.The final set contains 278 evidence annotations after filtering teams’ annotations against gold-labeled samples; continuous ratings have standard deviations from 0.804 to 1.318.Source: §4.2 · pp. 6–7

4.3 Evaluation with additional benchmarks

  • FEVER and VitaminC provide benchmark evidence pairs; for each claim, evidence sets with matching verdicts receive agreement 1 and those with differing verdicts receive 0.FEVER contains claims derived from Wikipedia sentences, while VitaminC uses near-identical claim-evidence pairs differing in subtle factual details.Source: §4.3 · p. 7

4.4 Baselines

  • Baselines span reference-based BLEURT and NLG metrics, verdict-proxy LLM scoring, and reference-less atomic-fact checks against the claim and retrieved evidence.The reference-less scorer decomposes claims into atomic facts; the LLM proxy uses the log probability of the correct veracity label.Source: §4.4 · pp. 7–8

5 Results and Discussion

Ev2R scorers generally correlate strongly with human judgments and outperform traditional reference-based metrics, while adversarial and qualitative analyses show how combining reference and verdict signals addresses distinct evidence-scoring challenges.

5 Results and Discussion

  • Across datasets, Ev2R outperforms traditional reference-based metrics; the trained proxy-reference scorer has the highest overall correlation except on AVeriTeC, where GPT4o surpasses it.Ev2R scorers rank among the top two across almost all datasets and rating categories.Source: §5 · pp. 8–9
  • Proxy-reference correlations with human ratings are ρ = 0.375, r = 0.393 for FEVER; ρ = 0.250, r = 0.273 for VitaminC; and ρ = 0.475, r = 0.450 for AVeriTeC (all p-values < 0.001).Reference-based Ev2R also correlates more highly with human assessment on Coverage and Relevance dimensions.Source: §5 · p. 10, Table 4 · p. 10
  • Backbone choice has limited overall effect, with Gemini variants stronger on coverage and relevance, GPT-4o stronger on verdict agreement, and no model best across all dimensions and datasets.The GPT-4o and proxy-reference combination correlates strongly on AVeriTeC and is competitive overall.Source: §5 · p. 10
  • Ev2R components show robustness in adversarial tests, while traditional metrics including BLEURT are highly sensitive to semantics-altering tests.The reference-based and proxy-reference components, alongside trained reference-based baselines, show small performance drops on semantics-preserving tests.Source: §5 · pp. 10–11, Table 5 · p. 11

5.1 Qualitative analysis of evidence evaluation

  • Reference-based precision and recall expose different evidence qualities, but can score alternative reasoning paths lower when they reach the same correct verdict.More retrieved detail than the reference may reduce precision while leaving recall high; proxy-reference scoring is incorporated to stabilize scoring in these cases.Source: §5.1 · p. 11
  • Proxy-reference scoring can give a low score to related evidence when predicted and reference evidence support diverging verdict labels.Precision and recall from reference-based scoring can add interpretable detail in these cases.Source: §5.1 · pp. 11–12

5.2 How Ev2R addresses key challenges of evidence evaluation for AFC

  • Ev2R evaluates reference alignment alongside verdict agreement, addressing the risk that a correct verdict may be predicted without genuinely relying on retrieved evidence.Across AVeriTeC, FEVER, and VitaminC, results show high correlation with human judgments on verdict agreement, relevance, and reference-evidence coverage.Source: §5.2 · p. 12

Assessment relying on closed knowledge sources & surface-level matching of evidence

  • Ev2R decomposes evidence into atomic facts to assess alignment beyond exact matches, supporting evaluation of unannotated web evidence and alternative valid reasoning paths.Across FEVER, VitaminC, and AVeriTeC, the paper reports higher human-judgment correlations and stable performance under adversarial testing.Source: Assessment relying on closed knowledge… · p. 12

6 Conclusion

Ev2R jointly evaluates evidence against gold references and the verdict it supports. The paper reports strong human-judgment correlation and robustness compared with baseline approaches.

  • Ev2R combines reference-based and verdict-level proxy components to evaluate evidence alignment and verdict support.Source: §6 · p. 12
  • The evaluation compares Ev2R with three baseline approaches for evidence evaluation.Source: §6 · p. 12
  • Ev2R correlates strongly with human judgments on relevant evaluation criteria and is reported as robust for evidence evaluation in AFC.Source: §6 · p. 12

7 Limitations & Ethics statement

The study’s datasets are English-only, and its support/refute labels describe claim-evidence relationships rather than real-world correctness.

  • The English-only datasets limit generalizability to global fact-checking contexts where multilingual capabilities are essential.Source: §7 · p. 12
  • The authors note that English-only benchmarks are common in the field and may distort perceptions of AFC progress.Source: §7 · p. 12
  • Support and refute labels represent relationships between claims and evidence, not real-world correctness of claims.Source: §7 · p. 12
  • Participants were informed about data collection and its purpose, and could withdraw or provide feedback.Source: §7 · p. 12

A Evaluation with adversarial perturbations

Table 6 provides descriptions of categories of adversarial tests.

  • Table 6 presents categories of adversarial tests alongside descriptions.Source: Table 6 · p. 18

B Additional Evaluation Dimensions

The study collected human ratings for AVeriTeC test instances and reports additional evaluation dimensions concerning evidence presentation and consistency.

  • Two AFC-familiar graduate-student or postdoctoral annotators rated 100 randomly selected AVeriTeC test instances, yielding 200 annotations.Per-sample, per-dimension ratings were averaged; verdict disagreements went to a third annotator and majority vote.Source: B Additional Evaluation Dimensions · p. 18
  • Coherence measures whether evidence sentences connect sensibly and make sense as a whole.The passage links coherent evidence with presenting the verification process and rationale comprehensibly.Source: B Additional Evaluation Dimensions · p. 18
  • Repetition measures repeated content, while consistency assesses whether retrieved evidence is logically consistent.The passage connects avoiding repetition with reader focus and consistency with process reliability and trust.Source: B Additional Evaluation Dimensions · p. 18

E Human evaluation

The supplied evaluation materials center on correlations between human-rated AVeriTeC submissions and proposed scorers, alongside examples and prompt descriptions for evidence-scoring variants.

E Human evaluation

  • Tables 7 and 8 report correlations between human-rated AVeriTeC submissions and proposed scorers across five quality dimensions.Blue and brown mark the highest and second-highest scores per column; reference-based precision and recall inputs to F1 are gray.Source: Table 7 · p. 19, Table 8 · p. 20
  • One example shows reference and predicted evidence contradicting the claim while following alternative evidence paths.Source: Table 9 · p. 21
  • Table 10 provides an example of predicted evidence containing repetitions.Source: Table 10 · p. 21
  • Table 11 gives an example from manual evaluation of proxy-reference evaluation on an AVeriTeC shared-task system.Source: Table 11 · p. 22
  • The prompt examples decompose claims into facts for reference-less scoring, compare verdict labels for proxy scoring, and decompose reference and retrieved evidence into atomic facts for reference-based scoring.Source: Figure 2 · p. 22, Table 12 · p. 23, Figure 3 · p. 24
  • Table 20 covers correlations between reference and predicted scores for verdict agreement on FEVER, VitaminC, and AVeriTeC.Source: Table 13 · p. 24

Related papers

Loading 2411.05375v2…