PaperlayerView exact revision on arXiv

Source-linked AI summary

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov

arXiv:2311.09000v3cs.CL

TL;DR

Open-domain LLM responses lack gold-standard reference answers, and human factuality review is costly. Factcheck-Bench provides a fine-grained annotation framework and document-level benchmark for evaluating automatic fact-checkers. Experiments show that current verifiers struggle to identify false claims.

  • Problem

    Open-domain LLM responses lack gold-standard reference answers for factual evaluation, while manual assessment demands substantial time and cost.

    Source: §1 · p. 1

  • Method

    Factcheck-Bench annotates factuality across eight detection and correction steps and constructs a document-level claim-based benchmark for evaluating automatic fact-checkers.

    Source: §1 · pp. 1–2

  • Results

    Current verifiers struggle to identify open-domain false claims, with the best F1=0.63 even when using external knowledge.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The benchmark’s fine-grained labels support evaluation and analysis of automatic fact-checkers across component modules.

    Source: §1 · pp. 1–2

    The dataset contains only 94 question-response pairs, though the authors consider it suitable for preliminary evaluation of automatic fact-checking systems.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.

1 Introduction

Open-domain LLM responses lack gold-standard answers for factual evaluation, while human review is costly and automatic checkers may be inaccurate. Factcheck-Bench offers a fine-grained framework and benchmark for evaluating automatic fact-checkers.

1 Introduction

  • Open-domain LLM responses lack gold-standard reference answers, making factual evaluation difficult and manual assessment costly.Automatic fact-checkers offer an alternative, but their verification results are not necessarily accurate.Source: §1 · p. 1
  • Prior evaluations often assess only final true-or-false judgments, limiting error localization, while disparate evaluation methods hinder comparison.The paper identifies guidance for improving fact-checking pipelines as under-explored.Source: §1 · p. 1
  • Factcheck-Bench labels eight steps in automatic factual-error detection and correction, enabling unified evaluation of intermediate component results.The paper also proposes a fine-grained framework for annotating LLM factuality and an end-to-end solution for detecting and correcting factual errors.Source: §1 · pp. 1–2
  • Evaluation of popular checkers, including FacTool and FactScore, finds large headroom for improvement in LLM fact-checking.The work also releases its annotation tool, data, and code.Source: §1 · p. 2

2 Fine-grained Annotation Framework

The framework divides factual-error detection and correction into eight subtasks, with detailed labels intended to support evaluation of component modules. It covers claim preparation, evidence assessment, correction decisions, and response revision.

2 Fine-grained Annotation Framework

  • The framework separates factual-error detection and correction into eight subtasks, enabling evaluation of a wide array of fact-checking modules.Practical systems may merge steps such as decomposition and decontextualisation.Source: §2 · pp. 2–4
  • Responses are split into context-independent atomic statements and claims, with coreferences resolved before independent checking.For example, a pronoun such as “It” should be replaced with its referent; claim granularity can vary, and each claim is intended to contain one property or fact to assess.Source: §2 · p. 2
  • Statements are classified by checkworthiness, accounting for users’ interest in truth and the claim’s importance to the response.The framework distinguishes factual claims, opinions, non-claims, and other statements, and assigns three importance levels to claims.Source: §2 · pp. 2–3
  • Evidence is retrieved and assigned a stance; labels distinguish support, partial support, refutation, and irrelevance.Partial support captures evidence that supports only part of a claim; refutation requires a clear opposing fact, unlike irrelevant evidence.Source: §2 · p. 3
  • When evidence conflicts, correction decisions remain an open question; the authors consider source reliability and retrieve extra information to judge dependability.The framework also distinguishes true, false, and not-enough-evidence factuality labels.Source: §2 · p. 3
  • Claim edits can delete a claim, replace information, or remove information; revised claims are merged with other statements into a fluent response.The revision aims to preserve the original intent and style without unnecessary changes, and removes duplicate content when applicable.Source: §2 · p. 3

3 Dataset Construction

Factcheck-Bench provides a document-level, claim-based benchmark with fine-grained annotations for evaluating LLM fact-checking subtasks and complete pipelines. Its 94 examples capture verifiability and factual errors, while analysis highlights limits in evidence retrieval and the prevalence of correctable false claims.

3.1 Data Collection

  • The 110 selected examples focused on fact-intensive, potentially false responses, drawn from user-reported hallucinations, brainstorming, and Dolly QA data.The selection process prioritized responses likely to contain factual errors because full annotation takes about 15–30 minutes per instance.Source: §3.1 · p. 4

3.2 Annotation

  • The annotation tool supports semi-automatic labeling by incorporating outputs such as decomposed claims and retrieved evidence for annotators to edit.Annotators proceed through three serial stages, with evidence retrieval and stance judgments followed by correction and response revision.Source: §3.2 · p. 5
  • With the tool, labeling a 50–150-word document takes 15–30 minutes; two annotators first annotate independently, then consolidate disagreements toward consensus, with a third rater used during second-step consolidation to catch unintentional mistakes.Cases still unresolved even with third-rater participation are discarded.Source: §3.2 · p. 5

3.3 Data Analysis

  • The final dataset contains 94 ChatGPT prompt-response pairs, 678 atomic claims, and 3,305 claim-evidence-stance triplets.Of the claims, 661 were checkworthy; the annotations also include evidence, factual labels, and revised claims, sentences, and responses.Source: §3.3 · pp. 5–6
  • 159 of 661 checkworthy claims were factually incorrect and needed correction; 30 more remained undetermined despite manual retrieval.The undetermined cases involved expert-level knowledge or private details that were difficult to verify using publicly available sources.Source: §3.3 · p. 6
  • Most original answers remained semantically similar to human revisions despite factual edits, with BERTScore-F1 0.955 and normalized edit distance 0.354.The analysis characterizes the errors as often minor factual mistakes in details.Source: §3.3 · p. 6

4 Unit Test for Fact-checkers

The unit tests compare automatic fact-checking methods with human annotations across decomposition, checkworthiness, evidence stance, claim verification, and response revision; sentence splitting and evidence retrieval quality are excluded from these tests.

4.1 Automatic vs. Manual Decomposition

  • For 211 sentences, human and ChatGPT decompositions matched in claim count across 521 claims, with word overlap of 0.88.Counts differed for 66 of 277 checkworthy sentences: ChatGPT produced more claims for 48 and fewer for 18.Source: §4.1 · p. 7

4.2 Checkworthiness

  • ChatGPT’s checkworthiness accuracy was below majority-guess baselines amid extreme class imbalance, and it missed 46 checkworthy sentences, or 15%.ChatGPT nevertheless had much higher recall than the majority guess.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Verification

  • Zero-shot stance prediction remained challenging, with absolute F1 below 0.5; fine-tuned RoBERTa-large-mnli outperformed zero-shot ChatGPT, mainly on support.LLaMA2 barely predicted partial support, so complete and partial support were merged into one label.Source: §4.3 · p. 7, Table 4 · p. 7
  • F1=0.63 on false claims makes Factcheck-GPT the strongest verifier; false claims were identified less accurately than true claims across approaches.Wikipedia-based performance was inferior to web-article performance, largely because of Wikipedia’s knowledge coverage.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Revision

  • 43 GPT-4 revisions were preferred to 23 ChatGPT revisions, while intrinsic metrics favored ChatGPT, leaving intrinsic evaluation misaligned with human preferences.Human raters also preferred revisions without questions, 38 to 28; evaluating revised-response quality with intrinsic metrics remains an open question.Source: §4.4 · p. 8, Table 6 · p. 8

5 Conclusion

The conclusion presents a fine-grained annotation framework and benchmark for evaluating automatic fact-checkers, then reports where factuality evaluation remains difficult.

5 Conclusion

  • The benchmark contains 678 open-domain claims with annotations across eight subtasks for detecting and correcting factual errors in long documents.The work also provides an annotation tool, benchmark, and code.Source: §5 · p. 8
  • Human annotations indicate that LLMs are prone to factual errors involving expert-level knowledge and exclusive details known by small groups.Source: §5 · p. 8
  • Current verifiers struggle to identify open-domain false claims even with external knowledge, with the best F1=0.63; intrinsic revision metrics misalign with human preferences.Source: §5 · p. 8

Limitations

The authors identify limitations in dataset scale, handling inter-claim dependencies, and evidence quality. These constrain the benchmark’s scope and the fact-checking pipeline’s ability to assess and correct some factual errors.

Limitations

  • Only 94 question-response pairs make the dataset small-scale, although the authors consider it sufficient for preliminary evaluation.They plan to expand it in English, Chinese, and Arabic; they note that evaluating 100 responses costs over 30–50 USD on average, depending on length.Source: Limitations · p. 9
  • Current approaches, including this solution, cannot assess overall logical correctness or whether procedural steps are out of order.They also struggle when invalidating one claim should require deleting the whole text, or when implicit claims depend on context.Source: Limitations · p. 9
  • More than half of automatically retrieved evidence is irrelevant, and the authors identify relevance improvement as critical to fact-checking accuracy.Source: Limitations · p. 9

Ethics and Broader Impact

The paper flags bias and coverage risks for its framework, while describing fact-checking as a staged process and positioning its work within open-domain, end-to-end verification. It also notes that the framework can apply beyond LLM outputs.

Ethics and Broader Impact

  • Automatic claim decomposition and evidence retrieval may introduce biases into annotation results, while incomplete claim coverage may yield inaccurate or unfair assessments in some domains.LLM responses may also inherit biases from the models involved.Source: Ethics and Broader Impact · p. 9
  • The framework is applicable to human-written documents as well as LLM outputs.Source: Ethics and Broader Impact · p. 9

A.1 What is Fact-checking?

  • Fact-checking typically comprises claim detection, evidence retrieval, verdict prediction, and optionally justification prediction.The paper defines it as assessing whether written claims are manipulated or true.Source: §A.1 · p. 12

A.2 Conventional Fact-checking

  • The paper targets open-domain generations without reference answers, unlike prior work focused on task-specific generations with references or specific topics and domains.It aims for a holistic framework that evaluates systems detecting and correcting factual errors end to end.Source: §A.2 · pp. 12–13

A.3 LLM Fact-checking

The paper situates its approach among post-inference fact-checking methods and argues for fine-grained claim-level correction. Its dataset provides factual labels and revisions at claim, sentence, and response levels, with an emphasis on long documents.

A.3 LLM Fact-checking

  • Claim-level editing supports precise corrections and preservation of the original text, whereas whole-response revision can miss false spans and introduce irrelevant descriptions or new hallucinations.RARR and CoVe generate retrieval questions and then edit the whole response; claim-level changes can correct a false number or delete a hallucinated sentence.Source: §A.3 · p. 13
  • FacTool requires users to supply gold atomic claims, and its checking process uses external APIs, complicating use and evaluation where online API calls are disallowed.Source: §A.3 · p. 13
  • The framework decomposes fact-checking into eight subtasks and accepts documents, sentences, or claims as inputs, with check-worthiness selection enabling customized verification.The subtasks cover decomposition, decontextualization, checkworthiness, evidence retrieval, stance detection, correction determination, claim correction, and response revision.Source: §A.3 · p. 13
  • Prior LLM hallucination datasets may be detection-only or task-specific; the paper notes that HaluEval labels hallucination spans, while FELM uses segment-level labels and FactPrompts responses tend to be short.The paper contrasts these resources with its claim-level annotations and revised text.Source: §A.3 · p. 14
  • Factcheck-Bench provides factual labels and revised text at atomic-claim, decontextualized-sentence, and response levels, emphasizing long documents.It is intended as a benchmark for evaluating fact-checkers; unlike FELM, which has sentence-level true/false labels without correction, it annotates and edits factual errors for atomic claims.Source: §A.3 · p. 14

B Dataset

The dataset combines examples drawn from observed ChatGPT failures and brainstorming with selected ChatGPT responses to closed- and open-domain questions. The latter were filtered to retain longer answers with low similarity to human answers.

B.1 Sources

  • The authors collected 23 ChatGPT failure examples from social media, web articles, and related papers, then added 22 brainstormed questions whose responses contained manually verified factual errors.Examples had to contain factual errors rather than reasoning, math, coding, or bias failures, and be independent of a dialogue.Source: §B.1 · p. 15
  • The authors selected closed- and open-question-answering examples from Dolly-15k because they focus on open-domain generations with more factual statements.Source: §B.1 · p. 15
  • From ChatGPT responses to 1,773 closed and 3,700 open QA pairs, filtering yielded 1,136 question-response pairs, including 45 from the first source.The filtering removed unanswerable-without-context questions, unanswered questions, and responses under 200 characters; selected responses had cosine similarity <= 0.5 to human answers using SimCSE embeddings.Source: §B.1 · p. 15

B.2 Data Selection

The section selects fact-intensive, likely false examples for a time-consuming annotation process. It uses FactScore to filter responses, then gathers evidence for atomic claims and annotates 110 examples.

B.2 Data Selection

  • FactScore was used to select fact-intensive, factually false cases because annotation takes about 15–30 minutes per instance.The selection process retained all 45 pairs from the first source and Dolly examples with FactScore below 0.2.Source: §B.2 · pp. 15–16
  • ChatGPT decomposed responses sequentially using three demonstrations; 105 of 1,136 examples required sentence-by-sentence processing after only the first sentence was decomposed.NLTK sentence splits were used for those cases; ChatGPT was preferred where it preserved citation references as a whole.Source: §B.2 · p. 15
  • Sentence-by-sentence processing was chosen to reduce distortion and make annotation easier by limiting each unit to roughly 1–5 claims rather than more than five for long responses.Source: §B.2 · pp. 15–16
  • For each atomic claim, ChatGPT generated search queries, Google supplied pages, and reranked passages yielded the top five evidence items.The reranker combined lexical and semantic similarity, using Sentence-BERT embeddings.Source: §B.2 · p. 16
  • FactScore measures the percentage of a generation’s atomic claims supported by a knowledge source, with an instruction-tuned model verifying claims against gathered evidence.Source: §B.2 · p. 16
  • 110 examples were annotated after filtering; the final set included 45 pairs from one source, 33 closed-question pairs, and 37 open questions.Seven similar or unhelpful questions were removed before the first annotation stage; the authors note that further cases would be annotated later.Source: §B.2 · p. 16

B.3 Annotation

Preliminary manual annotation exposed practical limits in annotating factuality end to end. It was slow, difficult to perform reliably, and especially demanding when gathering evidence.

B.3 Annotation

  • A focused annotator needed more than four hours to annotate a roughly 400-word document with about 20 sentences.Evidence collection took the most time and effort, particularly for unfamiliar topics.Source: §B.3 · p. 16
  • Manual evidence selection from Google results risked missing relevant passages because people could not review all relevant pages within the available time.The authors contrast human judgment with machines’ ability to traverse sources.Source: §B.3 · p. 17
  • The authors report that high annotator agreement was hard to achieve, especially for decomposition, evidence collection, and stance identification.Source: §B.3 · p. 17

B.4 Data Analysis

The analysis shows that automatically retrieved evidence often did not determine claim factuality, while the guidelines define how claims are segmented, checked, evidenced, and corrected. The accompanying prompt examples illustrate sentence-to-claim decomposition.

B.4 Data Analysis

  • After removing 16 examples lacking a standard gold answer, the analysis retained 94 instances; procedure questions, riddles, and non-factual questions remain outside the current fact-checking pipelines’ fact-focused scope.The authors note that causality and global logic across an answer are under-explored.Source: §B.4 · p. 17
  • Responses usually contained 2–5 sentences; 142 sentences were most important, 121 intermediate, and 48 not important, while 278 contained checkworthy statements.Source: §B.4 · p. 17
  • 2,057 of 3,305 evidence items were irrelevant; claims with enough automatic evidence had 690 completely-support and 150 refute stances, versus only 17 strong stances among claims without enough evidence.Source: §B.4 · p. 18, Figure 6 · p. 18

B.5 Prompt to Generate Atomic Claims

  • The prompt examples show sentence-level input being decomposed into atomic claims, with examples demonstrating how a sentence can yield one or multiple claims.Source: Table 9 · p. 19, §B.5 · p. 21

C.1 Decompose

  • Decomposition starts with sentences, splits information-heavy statements without over-fragmenting them, and keeps strongly dependent information together.Source: §C.1 · p. 22

C.2 Decontextualise / C.3 Identify checkworthy claim

  • Decontextualization replaces pronouns with specific entities when needed to preserve meaning; checkworthy claims are objective, non-commonsense facts, while private facts without public evidence are uncheckable.Annotators separately classify statements as factual claims, opinions, non-claims, or other, and decide whether each is worth checking.Source: §C.2 · p. 22, §C.3 · p. 22

C.4 Retrieve and collect evidence / C.5 Identify evidence stance

  • For checkworthy claims, annotators collect relevant web evidence and record its query, source, reliability, and sufficiency; fewer than five items may suffice, while controversial claims may need more.They then label each evidence item as supporting, partially supporting, refuting, or irrelevant.Source: §C.4 · p. 23, §C.5 · p. 23

C.6 Determine correction / C.7 Edit, Merge and Deduplicate

  • Correction weighs source reliability and stance counts when evidence conflicts, edits refuted claim partitions, and otherwise aims for minimal changes before merging and deduplicating statements.Source: §C.6 · p. 23, §C.7 · p. 23

F Example of Subtask 5

The appendix provides a revised-response comparison and screenshots of interfaces supporting the annotation workflow. The supplied captions identify the interfaces’ tasks but do not report their outcomes.

F Example of Subtask 5

  • Table 11 compares revised responses from GPT-3.5-turbo and GPT-4 under different prompts, and the caption prefers Prompt1 with GPT-4.Source: Table 11 · p. 27

G.1 Decomposition, Decontextualization and Check-worthiness detection

  • Figure 11 shows the interface for decomposition, decontextualization, and checkworthiness detection.Source: Figure 11 · p. 28

G.2 Evidence stance identification and Claim correction

  • Figure 12 shows the interface for evidence-stance identification and claim correction.Source: Figure 12 · p. 29

G.3 Claim Merge and Deduplication

  • Figure 13 shows the interface for claim merging and deduplication.Source: Figure 13 · p. 30

Related papers

Loading 2311.09000v3…