PaperlayerView exact revision on arXiv

Source-linked AI summary

FELM: Benchmarking Factuality Evaluation of Large Language Models

Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He

arXiv:2310.00741v2cs.CL

TL;DR

Factuality evaluators need stronger evaluation because LLMs generate errors and existing benchmarks underrepresent diverse long-form settings. FELM addresses this gap with a multi-domain, segment-level benchmark whose annotations include error types and reference links. Retrieval improves detection, but current LLM evaluators remain far from reliably identifying factual errors.

  • Problem

    Existing factuality evaluation mainly covers narrow tasks or domains, motivating a broader benchmark for assessing evaluators of diverse LLM-generated text.

    Source: §1 · p. 2

  • Method

    FELM collects ChatGPT responses across five domains and annotates fine-grained segments with factuality labels, error information, and reference links.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Retrieval-augmented reference documents increase segment-level F1 by 6.4 points for ChatGPT and 5.5 points for GPT-4, while most detectors remain unsatisfactory overall.

    Source: §4.2 · p. 8

  • Takeaways & Limitations

    FELM shows that retrieval benefits factual-error detection, while current LLMs still struggle to faithfully evaluate factuality.

    Source: §4.2 · p. 8

    FELM responses come solely from ChatGPT, so detector performance may differ when evaluating generations from other LLMs.

    Source: §5 · p. 10

Abstract

from arXiv · show

Assessing factuality of text generated by large language models (LLMs) is an emerging yet crucial research area, aimed at alerting users to potential errors and guiding the development of more reliable LLMs. Nonetheless, the evaluators assessing factuality necessitate suitable evaluation themselves to gauge progress and foster advancements. This direction remains under-explored, resulting in substantial impediments to the progress of factuality evaluators. To mitigate this issue, we introduce a benchmark for Factuality Evaluation of large Language Models, referred to as felm. In this benchmark, we collect responses generated from LLMs and annotate factuality labels in a fine-grained manner. Contrary to previous studies that primarily concentrate on the factuality of world knowledge (e.g.~information from Wikipedia), felm focuses on factuality across diverse domains, spanning from world knowledge to math and reasoning. Our annotation is based on text segments, which can help pinpoint specific factual errors. The factuality annotations are further supplemented by predefined error types and reference links that either support or contradict the statement. In our experiments, we investigate the performance of several LLM-based factuality evaluators on felm, including both vanilla LLMs and those augmented with retrieval mechanisms and chain-of-thought processes. Our findings reveal that while retrieval aids factuality evaluation, current LLMs are far from satisfactory to faithfully detect factual errors.

1 Introduction

Factuality evaluators are needed because LLMs can generate falsehoods, while prior evaluation largely targets narrow tasks and domains. FELM broadens the benchmark scope and provides fine-grained annotations for developing and assessing such evaluators.

Motivation

  • LLM-generated falsehoods create risks for practical use, motivating evaluators that can detect errors and explain them with evidence.The envisioned system segments responses, assesses each segment, highlights errors, categorizes error types, and provides reference links.Source: §1 · p. 1

FELM contribution

  • FELM’s annotation scheme follows the desired evaluator output by linking factual-error spans to explanations and supporting or contradicting references.This scheme is intended to drive development of factuality evaluators.Source: Figure 1 · p. 2, §1 · p. 2

Research gap

  • Prior factuality evaluation mainly covers specific tasks, domains, and less capable generators, leaving long-form LLM text in diverse settings under-evaluated.The paper frames meta-evaluation benchmarks as necessary for measuring progress in factuality evaluators.Source: §1 · p. 2

FELM contribution

  • FELM covers world knowledge, science and technology, math, writing and recommendation, and reasoning rather than only world knowledge.Its construction gathers prompts, collects ChatGPT responses, segments them, and obtains human factuality, error-type, error-reason, and reference annotations.Source: §1 · p. 2

2 Related Work

Existing factuality benchmarks commonly focus on summarization, world knowledge, or dialogue with predefined background evidence. FELM instead targets open-domain, long-form responses from LLMs under realistic task settings.

Existing benchmarks

  • Prior benchmarks mainly evaluate summarization or world-knowledge factuality, often by checking whether generated text is entailed by collected evidence.Table 1 compares published benchmarks according to model-generated responses verified against collected evidence.Source: §2 · p. 2, Table 1 · p. 3

FELM distinction

  • Some benchmarks focus on knowledge-based dialogue with pre-injected background knowledge, whereas FELM studies open-domain responses generated without external knowledge sources.FELM examines factual errors in long-form LLM responses across different task scenarios under a zero-shot setting.Source: §2 · p. 3

3 FELM

FELM defines factuality evaluation across five domains and uses fine-grained, segment-level annotations enriched with error information and references. It constructs the benchmark from diverse prompts, ChatGPT responses, heuristic segmentation, and expert annotation, yielding 817 samples and 3948 segments.

3.2 Factuality on Diverse Domains

  • FELM covers world knowledge, science and technology, recommendation and writing, reasoning, and math to represent factuality errors across varied LLM applications.Its scope includes fabricated scientific claims or citations, entity details in open-ended writing and recommendations, reasoning traces, and mathematical solutions.Source: §3.1 · p. 3, §3.2 · p. 4

3.1 Design Principles

  • FELM adopts segment-level annotation to make factuality judgments directly interpretable while allowing claim-based evaluators to map atomic facts back to response segments.The benchmark also records error reasons, predefined error types, and reference links that support or contradict each segment.Source: §3.1 · pp. 3–4

3.3 Overview of FELM

  • 817 samples and 3948 segments form FELM, with at least 100 samples and 500 segments per domain, average response length of 81.6 tokens, and a 31.8% response-level error rate.Table 2 defines the benchmark statistics, including error rate, positive and negative segment counts, average lengths, and annotator agreement.Source: §3.3 · p. 4, Table 2 · p. 5

3.4 Construction Process: Prompt Collection

  • Prompts come from Quora, Twitter, MMLU, TruthfulQA, GSM8K, MATH, online sources, ChatGPT generations, and author-written queries across the five domains.The domain-specific prompt pools combine existing benchmarks, online material, generated questions, and manually drafted prompts.Source: §3.4 · pp. 4–5

3.5 Construction Process: Response Generation & Segmentation

  • ChatGPT generates zero-shot responses, which FELM segments with sentence tokenization for standard prose and ChatGPT prompting for recommendation samples containing lists, numbers, or markdown.The segmentation is heuristic and primarily intended to improve interpretability rather than determine the evaluator’s preferred prediction granularity.Source: §3.5 · pp. 5–6, Table 3 · p. 6

3.6 Construction Process: Human Annotation

  • Six expert annotators label factuality, error reasons, four error types, and supporting or contradictory references, with two annotators agreeing on average 90.7% of segments.Author review, adjudication of disagreements, super-review, and additional checks found the examined references reliable and samples safe.Source: §3.6 · pp. 6–7

4 Experiment

The experiment evaluates LLM-based factuality detectors on FELM using segment- and claim-based methods, multiple prompting variants, and F1-based metrics. Results show that retrieval helps, but detector performance remains limited and varies by model, method, and domain.

4.1 Experimental Setup

  • The evaluators use segment- or claim-based judgments with vanilla, chain-of-thought, reference-link, and reference-document variants across Vicuna-33B, ChatGPT, and GPT-4.Claim-based evaluation labels a segment incorrect when any associated atomic claim is incorrect; math and reasoning omit references and claim-based reporting.Source: §4.1 · pp. 7–8
  • Performance is measured with factual-error F1, precision, recall, and balanced accuracy at both segment and response levels, with F1 emphasized in the main text.Table 4 reports segment- and response-level detector results for the evaluated LLM backbones.Source: §4.1 · p. 8, Table 4 · p. 9

4.2 Experiment Results

  • GPT-4 evaluators exceed 40 average F1 in some settings, while most ChatGPT detectors lack verification ability without external tools and Vicuna-33B has near-random balanced accuracy.FELM is harder than prior summarization benchmarks, where reported balanced accuracy is generally higher.Source: §4.2 · p. 8, Table 4 · p. 9
  • Retrieval-augmented reference documents increase segment-level F1 by 6.4 points for ChatGPT and 5.5 points for GPT-4 over vanilla methods, with retrieval content performing best across tested domains.Both reference links and reference documents improve factual-error detection.Source: §4.2 · p. 8
  • Chain-of-thought improves GPT-4 on nearly all domains but does not help ChatGPT, while self-consistency substantially boosts ChatGPT chain-of-thought performance beyond vanilla prompting.The reported pattern differs by backbone model and prompting strategy.Source: §4.2 · pp. 8–9
  • ChatGPT detectors improve with claim-based segmentation, whereas GPT-4 detectors decline, including a 4.5-point decrease for the vanilla method.The two evaluator backbones respond differently to the choice between segment- and claim-based evaluation.Source: §4.2 · p. 10

5 Conclusion

FELM combines segment-level factuality annotation with broad prompt coverage across five domains. Its prompt collection draws on established datasets, online sources, author-designed prompts, and ChatGPT-generated recommendations.

5 Conclusion

  • FELM evaluates factuality across five domains and uses segment-level annotations to pinpoint factual errors.The benchmark covers world knowledge, science and technology, math, writing and recommendation, and reasoning.Source: §5 · p. 10
  • The benchmark is designed around authentic LLM factual errors, broad factuality coverage beyond world knowledge, and fine-grained segment annotations.These principles define the benchmark’s intended scope and error-localization capability.Source: §5 · p. 10

A Prompt Collection

  • Prompts were collected from various sources, including online platforms, standard benchmarks, ChatGPT generations, and manual author drafting.The supplied collection includes Quora, MMLU, TruthfulQA, online sources, and self-instructed ChatGPT prompts.Source: A Prompt Collection · p. 14
  • Recommendation prompts were generated through self-instruction and cover books, courses, restaurants, and tourist attractions.Writing tasks request articles or essays and are included because autoregressive generation can accumulate errors in longer texts.Source: A Prompt Collection · p. 14
  • Reasoning prompts primarily come from GSM8K, including more than 200 challenging questions where text-Davinci-003 made mistakes.Additional reasoning prompts came from online sources and author designs.Source: A Prompt Collection · p. 14
  • Math prompts mainly come from MATH questions where text-davinci-003 made mistakes, with coverage of algebra, counting, and probability.A small portion came from online sources and the authors.Source: A Prompt Collection · p. 14

B Annotation Page

The annotation tool supports structured factuality labeling and evidence collection. It enables annotators to record both error classifications and explanatory references.

B Annotation Page

  • The HTML/JavaScript annotation tool lets annotators label factuality, identify error types, explain errors, and provide reference links.These functions support structured annotation of each evaluated response.Source: B Annotation Page · p. 14
  • Annotators can provide reasons for identified factuality errors alongside their labels.The tool therefore records explanatory information rather than only a binary factuality judgment.Source: B Annotation Page · p. 14
  • Reference links are included in the annotation interface to document supporting or contradicting evidence.The passage specifies reference-link inclusion as a tool function.Source: B Annotation Page · p. 14

C Additional Human Verification

Additional checks assessed reference reliability and response safety and validity. Both sampled evaluations reported full compliance with their respective criteria.

C Additional Human Verification

  • 100% of reference links in a random sample of 100 FELM examples were judged reliable by the paper authors.Reliability meant that linked content was free from misinformation or rumors.Source: C Additional Human Verification · p. 14
  • The human verification evaluated both reference quality and response quality using separate sampled assessments.Reference reliability was assessed by authors, whereas response safety and validity were assessed by MTurk workers.Source: C Additional Human Verification · pp. 14–15
  • All 100 sampled GPT-generated responses met safety and validity standards in evaluation by three MTurk workers each.Safety excluded harassment, sexual, or violent content, while validity required complete and meaningful responses.Source: C Additional Human Verification · p. 15

D Experimental Details and Prompts

The experiments use fixed decoding, model, token, retrieval, and prompting settings. Supplementary tables provide the exact one-shot prompts for claim extraction and factuality evaluation.

D Experimental Details and Prompts

  • Experiments use greedy decoding, gpt-3.5-turbo-0301 and gpt-4-0314, and token limits of 1500 for extraction and 100 for detection.These settings apply throughout the reported experiments.Source: D Experimental Details and Prompts · p. 15
  • Retrieval-augmented methods divide documents into 512-token chunks and select the most relevant chunk with BM25.When multiple reference links exist, the retrieved chunks are concatenated.Source: D Experimental Details and Prompts · p. 15
  • The same extraction and factuality-determination prompts are used for world knowledge and writing/recommendation domains to support consistent comparison.The prompt tables present exact instructions for the evaluated methods.Source: D Experimental Details and Prompts · p. 15, Table 5 · p. 16, Table 6 · p. 17
  • The study reports spending 0.5 USD on response generation, 22.04 USD on ChatGPT factuality evaluation, and 132.3 USD on GPT-4 evaluation.The costs include multiple iterations of attempting prompts for ChatGPT evaluation.Source: D Experimental Details and Prompts · p. 15
  • Tables 5 through 9 provide one-shot prompts for claim extraction and vanilla, chain-of-thought, and retrieval-augmented evaluation settings.The retrieval variants use either reference links or reference documents.Source: Table 5 · p. 16, Table 6 · p. 17, Table 7 · p. 18

E Additional Results

This section reports factual-error detector performance on FELM at both segment and response levels, with results summarized in Table 10.

  • Table 10 reports balanced accuracy for Vicuna-33B, ChatGPT, and GPT-4 detectors at segment and response levels.The comparison covers all evaluation settings described in §4.Source: Table 10 · p. 21, E Additional Results · p. 21
  • Overall scores for Link and Doc account for math and reasoning domains by substituting vanilla numbers where those methods degenerate.Source: Table 10 · p. 21
  • For claim-based methods, the overall average uses segment-based numbers for math and reasoning because claim extraction degenerates to segment-based evaluation there.Source: Table 10 · p. 21

F Example for four error types

This section provides examples illustrating the four predefined factual-error types used in FELM.

G Results of Self-Consistency on Chain-of-Thought Prompting

The self-consistency experiment evaluates whether majority voting over nine sampled responses improves ChatGPT’s chain-of-thought factuality evaluation.

Self-Consistency Results

Self-Consistency Method

Self-Consistency Results

  • Table 12 reports ChatGPT-CoT results using F1, precision, and recall for segment- and response-level evaluation.Source: Table 12 · p. 22
Loading 2310.00741v2…