PaperlayerView exact revision on arXiv

Source-linked AI summary

From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification

Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

arXiv:2310.11675v3cs.IR

TL;DR

Existing FV retrievers often rank evidence by relevance, which may not match what a claim verifier finds useful. FER trains retrieval with verifier feedback based on utility divergence from ground-truth evidence, and reports a 23.7% F1 gain over the SOTA baseline on FEVER.

  • Problem

    Off-the-shelf probability-ranking retrieval may return sentences whose relevance rankings do not align with the evidence judgments of the claim verifier.

    Source: §1 · p. 1

  • Method

    FER uses a coarse-to-fine retriever trained with evidence classification and verifier feedback measuring utility divergence between retrieved and ground-truth evidence.

    Source: §1 · p. 1, §3.2 · pp. 2–3

  • Results

    23.7% F1 performance gain over the SOTA baseline was reported on FEVER.

    Source: §1 · p. 1

  • Takeaways & Limitations

    On the FEVER test set, all tested claim verification models using FER-retrieved evidence outperform their respective original versions based on PRP.

    Source: §4.3 · p. 4

    The retrieval and verification models are optimized independently rather than through conditional or joint end-to-end optimization.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Retrieval-enhanced methods have become a primary approach in fact verification (FV); it requires reasoning over multiple retrieved pieces of evidence to verify the integrity of a claim. To retrieve evidence, existing work often employs off-the-shelf retrieval models whose design is based on the probability ranking principle. We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence. We introduce the feedback-based evidence retriever(FER) that optimizes the evidence retrieval process by incorporating feedback from the claim verifier. As a feedback signal we use the divergence in utility between how effectively the verifier utilizes the retrieved evidence and the ground-truth evidence to produce the final claim label. Empirical studies demonstrate the superiority of FER over prevailing baselines.

1 Introduction

Fact verification retrieves evidence for a claim verifier, but common PRP-based methods rank sentences by relevance even though those rankings may not match the verifier’s judgments. FER instead uses verifier-derived utility feedback, and on FEVER it reports a 23.7% F1 gain over the SOTA baseline.

1 Introduction

  • Existing fact-verification systems commonly retrieve relevant sentences with off-the-shelf models based on the probability ranking principle, but their rankings may not match what claim verifiers judge to be evidence.The retrieval models are designed around relevance, while retrieved sentences are consumed by a claim verifier.Source: §1 · p. 1
  • FER shifts evidence retrieval from relevance toward verifier utility, using divergence between verifier judgments on retrieved and ground-truth evidence as feedback.Its coarse-to-fine strategy first gathers candidate sentences, then re-evaluates them with verifier feedback.Source: §1 · p. 1
  • 23.7% F1 performance gain over the SOTA baseline was reported for FER on FEVER.The paper reports this result on the large-scale FEVER dataset.Source: §1 · p. 1

2 Background

FEVER systems predict one of three veracity labels for each claim using supporting evidence from Wikipedia. Their pipeline separates document retrieval, evidence retrieval, and claim verification; this paper focuses on evidence retrieval.

2 Background

  • FEVER systems assign each human-generated claim one of three labels—SUPPORTS, REFUTES, or NOT ENOUGH INFO—based on supporting evidence from Wikipedia.The dataset is identified as the FEVER task.Source: §2 · p. 2
  • The FV pipeline retrieves related documents, extracts evidence from their sentences, and aggregates retrieved evidence to predict a claim label; this work focuses on evidence retrieval.The related-work discussion is in Appendix A.1.Source: §2 · p. 2

3 Our Approach

FER uses a coarse-to-fine retrieval architecture: it first forms a PRP-based candidate set, then trains a fine-grained retriever with evidence labels and feedback from a claim verifier. The verifier feedback measures utility divergence between predictions based on retrieved and ground-truth evidence.

3 Our Approach

  • FER has two stages: PRP-based coarse retrieval recalls candidate sentences from related documents, and fine-grained retrieval selects evidence from that candidate set using verifier feedback.Figure 1 presents the overall architecture.Source: §3 · p. 2, Figure 1 · p. 2

3.1 Coarse retrieval

  • For coarse retrieval, a BERT ranking model scores claim–sentence pairs with a linear layer and tanh, trains with pairwise loss, and returns the top-K sentences as candidates.The claim and sentence pair are represented using BERT’s [CLS] hidden state.Source: §3.1 · p. 2

3.2 Fine-grained retrieval with feedback

  • The fine-grained retriever receives the claim and candidate sentences as one concatenated input and is trained with evidence-classification and utility-divergence objectives.Evidence classification uses ground-truth sentence labels to encourage selection of plausible evidence.Source: §3.2 · p. 2
  • Utility feedback is the divergence between verifier judgments based on retrieved sentences and ground-truth evidence; the verifier’s probability distributions provide this signal to train the retriever.The verifier is fixed after training, and the paper notes that other advanced verifiers can provide feedback.Source: §3.2 · pp. 2–3
  • At inference, the optimized retriever supplies evidence that can be passed to advanced verification models to verify a test claim.Source: §3.2 · p. 3

4 Experiments

The experiments evaluate FER on FEVER for evidence retrieval and claim verification, including how retrieval performance varies with candidate-set size and verifier feedback. FER improves evidence-retrieval precision and F1 over state-of-the-art methods, and evidence retrieved by FER improves claim-verification performance.

4.1 Experimental settings

  • The experiments use FEVER’s SUPPORTS, REFUTES, and NOT ENOUGH INFO labels and evaluate evidence retrieval with precision, recall, and F1.Claim verification is evaluated with FEVER score and label accuracy; evidence-retrieval F1 is prioritized.Source: §4.1 · p. 3

4.2 Results on evidence retrieval

  • FER significantly outperforms state-of-the-art methods in evidence-retrieval precision and F1, while its higher precision comes with lower recall than baselines.The authors attribute this trade-off to FER retrieving smaller, more precise evidence sets.Source: §4.2 · p. 3
  • FER without L_cla achieves higher precision than FER without L_uti, while removing L_cla yields poorer recall and F1 than removing L_uti.The authors interpret the precision comparison as evidence that verifier feedback helps identify precise evidence for verification.Source: §4.2 · p. 4

4.3 Results on claim verification

  • All tested claim-verification models perform better with evidence retrieved by FER than with evidence from their original PRP-based systems.The analysis reports that relevant evidence from the original papers may not always be useful for verification or may contain conflicting pieces.Source: §4.3 · p. 4

4.4 Quality analysis

  • KGAT and GAT are described as promising feedback providers, whereas BERT Pair cannot provide the required distribution over final claim labels.The authors identify exploring alternative feedback formulations as future work.Source: §4.4 · p. 4
  • Recall remains high across candidate-set sizes K; lower K hurts precision, while increasing K supplies more information and may cause slight recall fluctuations.The authors suggest recall stays high because FEVER claims average 1.86 relevant sentences and K exceeds 5.Source: Figure 2 · p. 4

5 Conclusion

The conclusion presents FER as a verifier-feedback framework that shifts evidence retrieval beyond relevance alone. On FEVER, the authors report that it identifies evidence that is both relevant and useful for claim verification.

5 Conclusion

  • FER is a feedback-based evidence retrieval framework for fact verification that incorporates feedback from the claim verifier.The conclusion contrasts its utility emphasis with traditional probability ranking principle approaches based on relevance.Source: §5 · p. 4
  • FER emphasizes evidence utility for verification rather than relying on relevance alone.Source: §5 · p. 4
  • On FEVER, the authors report that FER identifies evidence that is both relevant and useful for claim verification.They describe the experimental results as demonstrating FER’s effectiveness.Source: §5 · p. 5

Limitations

The authors identify limits in optimization, feedback formulation, and evaluation scope. Their experiments cover only FEVER and evidence retrieval.

Limitations

  • FER’s retrieval and verification models are optimized independently, without conditional or joint end-to-end optimization.The authors suggest future work could optimize both models with a single objective function.Source: Limitations · p. 5
  • The study computes utility divergence from the verifier’s probability distribution, while other divergence measures and feedback forms remain unexplored.The authors mention gradients of verification loss as one alternative.Source: Limitations · p. 5
  • FER’s effectiveness is demonstrated only on the FEVER dataset and for evidence retrieval, not across other datasets or document retrieval.Source: Limitations · p. 5

A.1 Related work

The section briefly reviews fact verification and retrieval-enhanced machine learning.

A.1 Related work

  • The related-work section identifies fact verification and retrieval-enhanced machine learning as its two review areas.Source: §A.1 · p. 8

A.1.1 Fact Verification

Fact verification retrieves evidence from reliable corpora and uses it to assess claim truthfulness. Since verification accuracy depends on that evidence, precise retrieval matters.

A.1.1 Fact Verification

  • Fact verification retrieves evidential sentences from reliable corpora such as Wikipedia to assess claims through document retrieval, evidence retrieval, and claim verification.The final stage aggregates retrieved evidence to predict a claim label.Source: §A.1.1 · p. 8
  • Evidence retrieval commonly ranks sentences by relevance under the probability ranking principle, while generative approaches produce document titles and evidence identifiers.The paper describes entity-linking, keyword-matching, and feature-based approaches for document retrieval.Source: §A.1.1 · p. 8
  • Claim verification uses entailment-style inference or graph reasoning, and its accuracy depends heavily on the retrieved evidence.The paper states that precise evidence can significantly affect verification outcomes.Source: §A.1.1 · p. 8

A.1.2 Retrieval-enhanced machine learning

Retrieval-enhanced machine learning includes retrieval-only systems and systems that train the retriever using prediction-model feedback. The paper notes a cost trade-off for scaling models and limitations in one feedback approach.

A.1.2 Retrieval-enhanced machine learning

  • Increasing model parameter size can improve accuracy but raises model-design and training costs.The paper places retrieval enhancement among approaches used to alleviate model memory.Source: §A.1.2 · p. 8
  • Retrieval-only systems supply query-related responses to a predictor, whereas retrieval-with-feedback systems further train the retriever using predictor feedback.The paper says most current fact-verification models primarily use retrieval-only methods.Source: §A.1.2 · p. 8
  • A prior method uses predictor loss to optimize retrieval, but may not suit large datasets and may struggle to capture sentence utility without ground-truth evidence.The paper links the large-dataset concern to computing loss over all provided sentences.Source: §A.1.2 · p. 8

A.2 The training process of the fine-grained retriever

The appendix details fine-grained retriever training, experimental settings and evaluation, comparison systems, development-set analyses, and illustrative FEVER cases. The cases show that fine-grained retrieval can recover useful lower-ranked evidence, while some ground-truth evidence remains missed.

A.2 The training process of the fine-grained retriever

  • The fine-grained retriever classifies candidate sentences, passes selected sentences to the verifier, and receives gradients from utility divergence between retrieved and ground-truth evidence judgments.Gumbel Softmax enables gradients to flow through the selected sentence vectors back to the retriever.Source: §A.2 · pp. 8–9

A.3 Reproducibility

  • The authors state that source code and trained models will be made publicly available upon publication to support reproducibility.Source: §A.3 · p. 9

A.3.1 Experimental settings

  • Document retrieval averages four entity-linked pages per claim, while coarse and fine retrieval use BERT-based sentence ranking and claim-plus-candidate classification.Online evaluation scores the first five predicted evidence sentences; the fine-grained retriever has a 512-token maximum input length.Source: §A.3.1 · p. 9

A.3.2 Evaluation metric

  • FEVER evaluation uses evidence Precision, Recall, and F1 at five sentences, plus FEVER score and label accuracy, which does not account for evidence validity.Source: §A.3.2 · pp. 9–10

A.3.3 Evidence retrieval baselines

  • Evidence-retrieval comparisons include sparse, ranking, graph, neural, deep-Q-learning, generative, and sentence-classification baselines.The listed systems include TF-IDF, KGAT, DREAM, DQN, GERE, and Stammbach.Source: §A.3.3 · p. 10

A.4.1 Performance on the development set.

  • Development-set analyses report verifier performance using original-paper versus FER evidence, retrieval performance under different verifier feedback, and retrieval performance across K values.The supplied passages identify the comparisons but do not provide their numerical outcomes.Source: Table 4 · p. 10, Table 5 · p. 10, Figure 3 · p. 11

A.3.4 Veracity prediction

  • The verifier comparisons include BERT-pair and BERT-concat models, GEAR's graphical attention network, and KGAT's graph attention network.Source: §A.3.4 · p. 10

A.4.2 Case study

  • Fine-grained retrieval recovers lower-ranked ground-truth sentences in two examples, but misses one gold sentence in the second case, where the omission does not significantly affect claim judgment.The missed evidence illustrates the reported low-recall issue that the authors say needs further resolution.Source: §A.4.2 · p. 11, Table 6 · p. 12

Related papers

Loading 2310.11675v3…