PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers

Wooseok Seo, Seungju Han, Jaehun Jung, Benjamin Newman, Seungwon Lim, Seungbeen Lee, Ximing Lu, Yejin Choi, Youngjae Yu

arXiv:2506.13342v2cs.AIcs.CLcs.LG

TL;DR

Fact verifiers are central to factuality evaluation, yet comprehensive study of these models remains underexplored. They evaluate 12 pre-trained LLMs and one specialized fact-verifier, use LLM-as-a-judge to identify ambiguous or mislabeled examples, and test synthetic multi-hop training for fact verification. Refinement changes model rankings, few-shot frontier LLMs are top-performing, and synthetic multi-hop training improves performance on complex reasoning benchmarks without harming other datasets.

  • Problem

    Fact verifiers are central to factuality evaluation, yet comprehensive study of these models remains underexplored.

    Source: §1 · p. 1

  • Method

    They evaluate 12 pre-trained LLMs and one specialized fact-verifier, use LLM-as-a-judge to identify ambiguous or mislabeled examples, and test synthetic multi-hop training for fact verification.

    Source: §1 · pp. 2–3

  • Results

    Refinement changes model rankings, few-shot frontier LLMs are top-performing, and synthetic multi-hop training improves performance on complex reasoning benchmarks without harming other datasets.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Future comparisons should address ambiguous and mislabeled examples and include few-shot LLM baselines; synthetic multi-hop data is a promising route for small verifiers.

    Source: §1 · pp. 2–3

    The finding that 16% of the initial collection was mislabeled or ambiguous should not be generalized to the original full datasets, given prior filtering and sampling.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

Fact verification is essential for ensuring the reliability of LLM applications. In this study, we evaluate 12 pre-trained LLMs and one specialized fact-verifier, including frontier LLMs and open-weight reasoning LLMs, using a collection of examples from 14 fact-checking benchmarks. We share three findings intended to guide future development of more robust fact verifiers. First, we highlight the importance of addressing annotation errors and ambiguity in datasets, demonstrating that approximately 16\% of ambiguous or incorrectly labeled data substantially influences model rankings. Neglecting this issue may result in misleading conclusions during comparative evaluations, and we suggest using a systematic pipeline utilizing LLM-as-a-judge to help identify these issues at scale. Second, we discover that frontier LLMs with few-shot in-context examples, often overlooked in previous works, achieve top-tier performance. We therefore recommend that future studies include comparisons with these simple yet highly effective baselines. Lastly, despite their effectiveness, frontier LLMs incur substantial costs, motivating the development of small, fine-tuned fact verifiers. We show that these small models still have room for improvement, particularly on instances that require complex reasoning. Encouragingly, we demonstrate that augmenting training with synthetic multi-hop reasoning data significantly enhances their capabilities in such instances. We release our code, model, and dataset at https://github.com/just1nseo/verifying-the-verifiers.

1 Introduction

The paper studies fact verifiers because they are central to LLM factuality evaluation, yet comprehensive investigation remains underexplored. Across 14 benchmarks, it finds that data quality affects rankings, few-shot frontier LLMs are strong baselines, and synthetic multi-hop data can improve small verifiers on complex reasoning tasks.

1 Introduction

  • The study evaluates 12 pre-trained LLMs and one specialized verifier on examples from 14 benchmarks, addressing the limited comprehensive study of fact verifiers.The benchmarks are intended to balance data sources and label distributions.Source: §1 · pp. 1–2
  • At least 9.1% of the initial examples were ambiguous and 6.6% were mislabeled, while the LLM-judge pipeline left annotators to inspect less than 20% of the dataset.The pipeline uses verifier outputs, LLM judgments, and manual review to identify ambiguous cases and correct labels where necessary.Source: §1 · p. 2, Figure 1 · p. 2
  • Refinement changes model rankings: MiniCheck initially ranks above o1 and R1-distilled Qwen 32B, but falls lower after refinement.On GRAYFACTS, zero-shot o1 scores 9.4 macro F1, while Llama 3.1 8B scores 13.5; the authors hypothesize nuanced labeling differences contribute to this result.Source: §1 · p. 2
  • Few-shot prompting improves performance for 12 of 13 models, with few-shot o1 achieving the highest overall performance.The authors suspect nuanced fact-verification edge cases make it difficult to write zero-shot instructions that cover the tasks.Source: §1 · p. 2
  • On complex multi-hop reasoning examples, the small fine-tuned verifier substantially lags behind larger models.The paper identifies examples from CoverBench and Hover as driving much of the gap.Source: §1 · pp. 2–3
  • Synthetic multi-hop reasoning data significantly improves small-verifier performance on challenging benchmarks without compromising results on other datasets.The approach targets the complex reasoning instances where the small fine-tuned model trails larger models.Source: §1 · p. 3

2 Task Setup of Fact Verification

The task setup defines fact verification as document-grounded attribution and assembles a diverse, balanced benchmark suite with filtering and macro F1 evaluation.

2 Task Setup of Fact Verification

  • Fact verification is binary classification: determine whether a statement is attributable or not attributable to its document.For three-way datasets, contradictory examples are mapped to not attributable for evaluation.Source: §2 · p. 3
  • The 14 datasets span expert and general domains, with examples sampled to balance both data sources and labels.The collection combines 11 LLM-AggreFact datasets with SciFact, Hover, and CoverBench.Source: §2 · p. 3
  • The collection filters unverifiable or ambiguous statements and removes trivial verbatim matches before balancing the resulting dataset.The first stage filters approximately 42% of examples, and n-gram overlap filtering removes an additional 3%.Source: §2 · p. 3
  • The final collection contains 1,749 examples after duplicate CoverBench instances are removed.The resulting dataset is balanced to a 1:1 label distribution.Source: §2 · p. 3
  • Macro F1 averages the F1 score for each label to address label imbalance in benchmarks.Three-way labels and model outputs are mapped into the two-label space when needed.Source: §2 · p. 3
  • The Red Devils and United example illustrates how knowledge-level and contextual ambiguity can coexist in a benchmark instance.A model may connect both names to Manchester United, while an annotator unaware of that equivalence may not.Source: Figure 2 · p. 4

3 Ambiguity and Annotation Errors in Fact Verification Benchmarks

The paper develops a pipeline to identify annotation errors and ambiguity in fact-verification benchmarks, then uses human review to construct CLEARFACTS and GRAYFACTS. Its analysis finds substantial mislabeled and ambiguous data, which the two datasets separate for evaluation.

3.1 Systemically Identifying Label Ambiguity and Annotation Errors

  • Four frontier LLMs flag cases when a verdict differs from the original label; three judges retain only cases whose rationales receive unanimous positive evaluations.The judges assess completeness, logical coherence, and faithfulness.Source: §3.1 · p. 4
  • The pipeline yielded 344 candidates from 1,749 examples (19.7%), allowing annotators to inspect fewer than 20% rather than the full dataset.The authors describe the approach as useful where human annotation is costly.Source: §3.1 · pp. 4–5

3.2 Results

  • Table 1 covers 344 inspected examples; among 1,749 examples, 117 (6.7%) were mislabeled and 159 (9.1%) ambiguous.CLEARFACTS contains non-ambiguous and label-corrected examples, while GRAYFACTS contains ambiguous ones.Source: Table 1 · p. 5, §3.2 · p. 5

3.1 Systemically Identifying Label Ambiguity and Annotation Errors

  • Two annotators assess each reasoning trace: agreement that it is correct and un-debatable marks a mislabeled item, agreement that it is incorrect and un-debatable marks a model error, and other cases are ambiguous.Initial inter-annotator agreement was 52.4%; a further round reviewed two types of disagreement.Source: §3.1 · p. 5

3.2 Results

  • GRAYFACTS ambiguity categories include knowledge-level, linguistic, contextual, numerical, and other; when causes coexist, examples receive the primary cause category.The categories distinguish missing background knowledge, multiple or vague meanings, incomplete or indirect context, and precision-sensitive numbers.Source: §3.2 · p. 6

4 Evaluating Fact Verifiers with CLEARFACTS

The evaluation compares 12 pre-trained LLMs and MiniCheck on examples from fact-verification benchmarks. Few-shot prompting performs strongly, refinement changes model rankings, and MiniCheck's remaining gaps are concentrated on complex-reasoning benchmarks.

4.1 Considered Fact Verifiers

  • The study evaluates 12 LLMs—including open-weight, reasoning, and frontier models—alongside MiniCheck; the few-shot setup uses the same nine in-context examples across LLMs.The nine examples are drawn from ANLI and a synthetic multi-hop dataset.Source: §4.1 · p. 6, §4.2 · pp. 7–8

4.2 Findings

  • 12 of 13 models improved with few-shot prompting on CLEARFACTS; few-shot o1 led at macro F1 88.7.The figure reports macro F1 for MiniCheck and 12 LLMs under zero-shot and few-shot settings.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Refinement reversed MiniCheck’s apparent lead over o1 and two R1 models; on GRAYFACTS, Llama3.1 8B outperformed o1 and o1/Claude 3.7-Sonnet agreement fell to 69.2%.The two models’ agreement was 85.3% on CLEARFACTS.Source: §4.2 · p. 7
  • MiniCheck 7B outperformed similarly sized Llama3.1 8B but struggled on CoverBench and Hover; synthetic multi-hop training improved performance on those complex-reasoning benchmarks.MiniCheck performed strongly on AggreFact and SciFact, while Table 5 groups CLEARFACTS examples by four source subsets.Source: Table 5 · p. 8, §4.2 · p. 8

5 CLEARCHECK: Fine-tuning Models for Complex Fact-verification Tasks Requiring Reasoning

CLEARCHECK adds synthetic multi-hop verification data to fine-tuning, substantially improving small-model performance on complex reasoning examples. The data is built from linked Wikipedia documents and supports statements grounded in those documents.

Motivation

  • Small fine-tuned models substantially underperform larger models, particularly on instances requiring complex reasoning.This motivates training smaller models with synthetic multi-hop verification data.Source: §5 · p. 8

Synthetic multi-hop fact verification data

  • The synthetic data is generated by iteratively extracting facts, asking related questions, and answering them using retrieval-augmented generation across Wikipedia documents.Statements are grounded in supporting documents; negative examples are made by removing support or introducing contradictions.Source: §5 · p. 9

Model training

  • CLEARCHECK augments 57K ANLI training examples with 25.2K synthetic multi-hop fact-verification examples.The augmented dataset is compared with a verifier fine-tuned using ANLI alone.Source: §5 · p. 9

Results

  • Synthetic multi-hop training data significantly improves overall performance, with substantial gains on CoverBench and Hover.The comparison is between a model trained only on ANLI and CLEARCHECK, which adds reasoning data.Source: §5 · p. 9
  • CoT reasoning and direct-answer training yield no different evaluation results, but CoT makes verifier outputs legible for error detection.CLEARCHECK is trained to provide either direct answers or reasoning followed by a conclusion.Source: §5 · p. 9

6 Related Works

Related work covers fact verifiers in long-form factuality evaluation, benchmark development, and efforts to address benchmark errors and ambiguity. Existing benchmarks span diverse verification tasks and domains.

Long-form factuality evaluation of LLMs

  • Long-form factuality systems decompose LLM outputs into sentences or atomic claims and use fact verifiers to check them against knowledge sources.VeriScore retains only statements that LLMs determine to be verifiable.Source: §6 · p. 9

Fact verification benchmarks

  • LLM-AggreFact aggregates 11 benchmarks spanning summarization, retrieval-augmented generation, factuality evaluation, and human-written claim checking.Other benchmarks include SciFact for scientific claims, Hover for multi-hop reasoning, and CoverBench for complex verification.Source: §6 · pp. 9–10

Fixing benchmark errors and ambiguity

  • Prior work addresses benchmark errors and ambiguity in reasoning, knowledge-based, QA, NLI, and fact-verification tasks.Platinum-benchmark and MMLU-Redux target annotation errors and ambiguities, while Godbole & Jia (2025) raise reliability concerns without directly addressing these issues.Source: §6 · p. 10

7 Conclusion

The study evaluates 12 pre-trained LLMs and one specialized fact-verifier on examples from 14 benchmarks. It identifies benchmark imperfections, strong few-shot frontier-model baselines, and gains from synthetic multi-hop training for smaller models.

Conclusion

  • Evaluation of 12 pre-trained LLMs and one specialized verifier across 14 benchmarks identifies distortions from dataset imperfections and strong few-shot frontier-model baselines.The authors revise mislabeled instances into CLEARFACTS and categorize ambiguous examples in GRAYFACTS.Source: §7 · p. 10
  • Frontier LLMs using few-shot prompting are strong but often-overlooked baselines, while small specialized models show limited capabilities.The study uses CLEARFACTS and GRAYFACTS to examine verifier behavior and dataset imperfections.Source: §7 · p. 10
  • Synthetic multi-hop reasoning data substantially improves smaller models' complex reasoning capability.The authors present these findings as guidance for developing robust, reliable, and efficient fact-verification models.Source: §7 · p. 10

Ethics Statement

The authors describe careful annotation processes and commit to publicly releasing their code, model, and dataset for reproducibility and transparency.

Ethics Statement

  • The authors say annotation processes were carried out carefully to ensure annotation and study quality.They state this as part of their ethics statement.Source: Ethics Statement · p. 10
  • The authors commit to publicly releasing their code, model, and dataset for reproducibility and transparency.They also frame the release as a contribution to the broader research community.Source: Ethics Statement · p. 10
  • The authors state that they aim to demonstrate honesty and contribute to the research community.Source: Ethics Statement · p. 10

A Limitations

The authors caution that the observed annotation-error and ambiguity rate is specific to their filtered, balanced sample, while noting that ambiguity-focused evaluation and stronger small verifiers remain open directions.

A Limitations

  • The 16% mislabeled-or-ambiguous rate should not be generalized to the original full dataset because the sample was filtered, balanced, and incompletely annotated.The authors also acknowledge that some examples were not manually inspected.Source: A Limitations · p. 15
  • Further research on ambiguous examples is needed because real-world statements may not be self-contained or clearly interpretable.The authors cite Min et al. (2020) and Glockner et al. (2024) as references for ambiguity-focused evaluation.Source: A Limitations · p. 15
  • Multi-hop training substantially improves small fact verifiers, but they still trail the most capable frontier LLMs.The authors identify more effective training algorithms as a direction for closing this gap.Source: A Limitations · p. 15

B.1 Factuality Tuning

  • Smaller fact verifiers are used as reward models in factuality tuning because large models are costly to run across many instances.Examples include FactTune, Flame, and Xie et al.'s reasoning verifier.Source: §B.1 · p. 15

C.1 Judge Implementation Details

The appendix details the evaluation and annotation procedures, then reports dataset-specific patterns, error analyses, and fine-tuning ablations. Few-shot benefits vary across datasets, and the reported ablations support synthetic data and multi-task training.

C.1 Judge Implementation Details

  • The triple-judge framework evaluates distinct dimensions of verifier performance, with each judge assigning a binary score of zero or one.The framework is designed to capture the multifaceted reasoning involved in factual verification.Source: §C.1 · p. 16

C.3 Human Annotation Details

  • Two annotators assess reasoning correctness and debatable points; an additional review round revised 65 annotations among 106 cases.The interface presents each annotator with a document, statement, and model-generated rationale.Source: §C.3 · p. 16

C.4 Model Fine-Tuning Details in §5

  • Multi-task training minimizes the average of two NLL losses: one for direct-answer responses and one for chain-of-thought responses.Both response types are constructed for each training prompt.Source: §C.4 · p. 17

D.1 Additional Results on Two-stage Filtering Process in Section 2

  • The two-stage filtering process targets a clean, verifiable subset, prioritizing filtering precision rather than recall.An additional experiment samples 100 examples per dataset to estimate how many remain after classification; those rates do not represent each dataset's overall verifiability.Source: §D.1 · p. 17

D.2 Examples of Ambiguous Cases in GRAYFACTS datsaet

  • The appendix provides qualitative examples for the GRAYFACTS ambiguity types in Tables 14–17.It also lists the specific model versions used in the experiments.Source: §C.2 · p. 16, §D.2 · p. 18

D.3 Detailed Evaluation Results on CLEARFACTS

  • F1 improvements are greater on AggreFact and CoverBench than on Hover and SciFact; few-shot examples primarily benefit AggreFact and Hover, while complex CoverBench contexts remain difficult.The authors attribute CoverBench's difficulty to its complex contextual structures, which few-shot examples do not fully address.Source: §D.3 · p. 18

D.4 Detailed Evaluation Results on GRAYFACTS

  • Bias in the percentage predicted attributable persists across GRAYFACTS ambiguity categories, while CLEARFACTS error types are tabulated separately.Table 10 reports % Attr.; higher or lower values do not indicate better performance.Source: §D.4 · p. 18, §D.5 · p. 18, Table 7 · p. 18

D.6 Ablation Study for CLEARCHECK

  • CLEARFACTS ablations found the new synthetic multi-hop fact-verification data and multi-task learning helpful, while substituting the smaller InternLM2.5-7B base model showed model choice was not very critical.The three setups omitted the synthetic data, omitted the multi-task training objective, or used InternLM2.5-7B for comparison with MiniCheck 7B.Source: §D.6 · p. 18, Table 8 · p. 19

D.7 Ablation Study for Generalization of Few-shot Examples

The appendix tests whether few-shot results depend on the chosen examples and provides prompt templates for fact verification and judging. The templates specify attribution rules, reasoning instructions, and a conservative test for statement verifiability.

D.7 Ablation Study for Generalization of Few-shot Examples

  • Across four additional few-shot sets, models show little performance variance, and some results exceed those reported in the paper.The sets were tested on two proprietary and two open LLMs.Source: §D.7 · p. 19, Table 9 · p. 19

E Prompt Templates

  • The appendix provides templates for fact verifiers, LLM-as-a-judge prompts, and a verifiability classifier.The listed figures are 6–11.Source: E Prompt Templates · p. 19

FEW SHOT TEMPLATE

  • The few-shot prompt labels statements not supported or contradicted by the document as not attributable, and supported statements as attributable.It asks the model to reason step by step and use only document information.Source: FEW SHOT TEMPLATE · p. 25

SAFE PROMPT TEMPLATE

  • The SAFE-style prompt asks whether a statement is supported by the document, allowing strong implication rather than requiring explicit support.The statement and document are provided as the inputs.Source: SAFE PROMPT TEMPLATE · p. 26

COMPLETENESS BINARY JUDGE TEMPLATE

REASONING BINARY JUDGE TEMPLATE

LOGICAL COHERENCY BINARY JUDGE TEMPLATE

VERIFIABILITY CHECKER TEMPLATE

  • The verifiability checker classifies statements as verifiable, ambiguous, or unverifiable, reserving “verifiable” for fully self-contained, checkable claims.When uncertain, the prompt directs the classifier to choose ambiguous or unverifiable.Source: VERIFIABILITY CHECKER TEMPLATE · p. 30, Figure 11 · p. 30
Loading 2506.13342v2…