PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-form factuality in large language models

Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, Quoc V. Le

arXiv:2403.18802v4cs.CLcs.AIcs.LG

TL;DR

LLMs often produce factual errors on open-ended, fact-seeking prompts, motivating better evaluation of long-form factuality. The paper introduces LongFact, SAFE, and F1@K, and finds that SAFE compares favorably with crowdsourced human annotators while larger models generally achieve better long-form factuality. The authors also identify limitations from SAFE’s reliance on the capabilities of its underlying LLM and on Google Search as a knowledge source.

  • Problem

    LLMs often produce factual errors on open-ended questions, while long-form responses are difficult to evaluate with predetermined references that cannot comprehensively cover their facts.

  • Method

    The paper creates LongFact and uses SAFE to decompose responses into individual facts, assess relevance, and verify relevant facts through iterative Google Search reasoning.

  • Results

    Larger language models generally achieve better long-form factuality, while SAFE agrees with 72% of human annotations and wins 76% of 100 sampled disagreement cases.

  • Takeaways & Limitations

    LongFact and SAFE provide a benchmark and automated evaluation approach for comparing long-form factuality across models and broad topics.

  • Takeaways & Limitations

    SAFE depends on the underlying LLM’s instruction-following and reasoning capabilities and on Google Search, which may be insufficient in some corner cases or expert domains.

Abstract

from arXiv · show

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, we propose extending F1 score as an aggregated metric for long-form factuality. To do so, we balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyperparameter representing a user's preferred response length (recall). Empirically, we demonstrate that LLM agents can outperform crowdsourced human annotators - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times cheaper than human annotators. We also benchmark thirteen language models on LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding that larger language models generally achieve better long-form factuality. LongFact, SAFE, and all experimental code are available at https://github.com/google-deepmind/long-form-factuality.

1 Introduction

LLMs still produce factual errors on open-ended, in-depth questions, limiting their reliability in real-world settings. The paper introduces LongFact, SAFE, and F1@K to benchmark and quantify long-form factuality.

  • LLMs often produce factual errors about established facts such as dates, statistics, and occupations.These errors weaken factuality and reliability when factually accurate responses are expected.
  • LongFact is a GPT-4-generated benchmark of 2,280 fact-seeking prompts spanning 38 manually selected topics.The prompt set targets long-form factuality across a wide variety of domains.
  • SAFE automatically evaluates long-form factuality by decomposing responses into facts and checking them with Google Search.It uses an LLM agent to propose search queries and reason about whether search results support each fact.
  • 72% agreement with human annotations and 76% accuracy on 100 disagreement cases show SAFE can outperform crowdsourced human annotators.SAFE is also reported as 20× cheaper than human annotators.

2 LongFact: Using LLMs to generate a multi-topic benchmark for long-form factuality

Existing factuality benchmarks largely test short answers or cover limited topic ranges, leaving long-form, multi-topic factuality insufficiently evaluated. LongFact addresses this gap with GPT-4-generated prompts requiring detailed responses across 38 topics.

  • Existing benchmarks generally test single-fact short answers rather than several-paragraph responses.Examples include TruthfulQA, HaluEval, FreshQA, HalluQA, and FELM.
  • FActScore may require long-form responses but does not cover a broad range of topics.
  • LongFact uses GPT-4 to generate questions requiring long-form responses with multiple detailed factoids.Questions target a specific concept or object within a given topic.
  • LongFact contains 38 manually selected topics organized into four supercategories and two tasks: Concepts and Objects.It generates 30 unique prompts per topic, totaling 1,140 prompts per task.

3 SAFE: LLM agents as factuality autoraters

Long-form factuality is difficult to evaluate with fixed reference answers because responses can contain many facts that are hard to cover comprehensively and nonredundantly. SAFE instead evaluates individual facts with an LLM agent that iteratively searches Google and reasons over the results.

  • Fixed reference answers are poorly suited to long-form responses because they cannot comprehensively cover all possible facts.Existing automated methods work better when prompts have a definite, singular short-answer response.
  • SAFE evaluates long-form responses at the granularity of individual facts rather than sentences.This isolates each fact for more precise and focused evaluation.
  • SAFE splits responses into self-contained facts, determines relevance, and checks relevant facts against Google Search results.
  • SAFE’s key innovation is using an LLM agent to generate multi-step search queries and reason about whether results support each fact.The process is designed for claims requiring multi-step evaluation.

4 LLM agents can be better factuality annotators than humans

SAFE provides an automated alternative to human factuality annotation by evaluating individual facts in long-form responses. It agrees with human annotations on most facts, is more often correct in sampled disagreements, and costs substantially less.

  • 72.0% of SAFE annotations agree with human annotations across 16,011 individual facts.
  • 76% of SAFE annotations were correct in 100 randomly sampled disagreements, compared with 19% for human annotations.The comparison used researcher annotations with full internet access as ground truth.
  • $0.19 per model response was the SAFE annotation cost, versus $4 using crowdsourced human annotations.SAFE's total cost for the 496 prompt–response pairs was $96.31.
  • SAFE was more than 20× cheaper than crowdsourced human annotators while outperforming them in the reported comparison.

5 F1@K: Extending F1 with recall from human-preferred length

The paper extends factuality evaluation beyond precision by combining supported-fact precision with recall relative to a user-selected preferred response length. F1@K provides a bounded aggregate score without requiring preset ground-truth answers.

  • Factuality quality depends on both precision, the percentage of supported facts, and recall, the percentage of relevant facts provided.
  • F1@K measures both precision and recall without requiring preset ground-truth answers, using counts of facts from each label category.
  • F1@K uses K as the number of supported facts required for full recall, reflecting a user's preferred response length.
  • F1@K combines factual precision and recall using standard F1 and is bounded between 0 and 1.A score of 1 requires no unsupported facts and at least K supported facts.

6 Larger LLMs are more factual

The benchmark evaluates thirteen language models from four families on the same LongFact-Objects prompts using SAFE and F1@K. Larger models generally achieve better long-form factuality, although some within-family comparisons are not fully explained.

  • Thirteen models from Gemini, GPT, Claude, and PaLM-2 were evaluated on 250 LongFact-Objects prompts using SAFE and F1@K.The benchmark used K = 64 and K = 178.
  • Larger language models generally achieve better long-form factuality.
  • GPT-4-Turbo, Gemini-Ultra, and PaLM-2-L-IT-RLHF were the three most-factual models at both selected K values.
  • Claude-3-Sonnet achieved similar long-form factuality to Claude-3-Opus despite being a smaller model.The paper states that the reason for this result is unclear because further model details were unavailable.

7 Related work

The paper addresses limitations in existing factuality benchmarks and evaluation methods for long-form responses. It contributes a broader prompt set, scalable evaluation, and an aggregate metric incorporating recall.

  • Existing benchmarks commonly test isolated factoids, whereas long-form responses contain a broader range of possible factual claims.
  • The benchmark evaluates thirteen models using raw SAFE metrics and aggregated precision, recall, and F1@K on 250 LongFact-Objects prompts.
  • Long-form factuality is difficult to quantify because response quality depends on both factual precision and coverage.
  • F1@K measures precision and recall without requiring preset ground-truth answers.

8 Limitations

The paper identifies limitations arising from LLM dependence, Google Search coverage, and F1@K’s treatment of repeated facts.

  • LLM dependence: SAFE and LongFact quality depends directly on the capabilities of the LLM used, including instruction following and reasoning.Weak models may decompose responses poorly, generate unrelated searches, or misjudge evidential support.
  • Search dependence: Google Search may fail to provide sufficient evidence for corner cases or expert-level domains such as law and medicine.The authors therefore report whether facts are supported by search results rather than claiming global factuality.
  • Search dependence: SAFE can be adapted to restricted or predetermined sources by adding site restrictions or replacing the search API with professional databases.This provides a way to target sources better suited to particular domains.
  • Metric limitations: F1@K assumes responses contain no repeated facts, allowing supported facts to be repeated to game the metric.The authors suggest duplicate removal or alternative metrics such as fluency and usefulness, but leave duplicate handling for future work.

9 Conclusion

The paper introduces LongFact, SAFE, and F1@K to benchmark and evaluate long-form factuality. SAFE agrees with human annotations on 72% of facts, wins 76% of sampled disagreements, and larger models generally perform better, while search-based labels remain a proxy for global factuality.

  • Conclusion: LongFact contains 2,280 prompts across 38 topics, while SAFE decomposes responses into facts and verifies relevant facts with Google Search.F1@K combines precision with recall relative to a hyperparameter K representing preferred response length.
  • Conclusion: SAFE agreed with 72% of human annotations and won 76% of 100 randomly sampled disagreement cases.The study also reports that SAFE was more than 20 times cheaper than crowdsourced human annotators.
  • Conclusion: Benchmarking thirteen models from Gemini, GPT, Claude, and PaLM-2 found that larger language models generally achieved better long-form factuality.The paper makes its experimental code publicly available.
  • Conclusion: The study leaves reliable measurement of long-form hallucination unclear because it evaluates factuality against world knowledge rather than models’ internal knowledge.It identifies improving model factuality and SAFE’s search-enabled agents as directions for future work.
  • Conclusion: Google Search is treated as the best-available proxy for factual accuracy because it contains substantial information, although some accurate statements cannot be found there.The paper distinguishes support by search results from global factuality.

A.3 What are the common causes of failure for SAFE?

SAFE’s errors arise from reasoning, search, and revision components, while human annotation errors often involve confusing irrelevance with lack of support. The F1@K analysis further relates factuality measurement to human model preferences and response-length choices.

  • SAFE error causes: SAFE’s three error causes correspond to language-model reasoning, Google Search, and response revision.Reasoning errors involve relevance or supportedness judgments; search errors reflect missing necessary information; revision errors produce non-self-contained facts.
  • Human annotation errors: Over one-third of presumed human errors involved confusing “irrelevant” with “not-supported”.These facts were often relevant to the prompt but unsupported by the reference Wikipedia page.
  • Factuality measurement: F1@K includes recall relative to K, a hyperparameter representing the number of supported facts a user prefers in a response.The metric balances factual precision with response completeness up to the preferred supported-fact count.
  • Factuality measurement: F1@64 correlated significantly with Chatbot Arena ELO at r = 0.754 (p = 0.031), whereas precision correlated at r = 0.502 (p = 0.205).Including recall therefore produced a measurement more correlated with human model preferences in this comparison.

B.1 Data-generation process

LongFact is generated with GPT-4 from 38 manually selected topics, producing concept- and object-focused prompts for long-form factuality evaluation. SAFE then processes responses into individual facts, filters for relevance, and evaluates relevant facts using search-based supportedness judgments.

  • Data-generation process: LongFact uses 38 manually selected topics to cover long-form factuality across academic and less academic domains.The topics include MMLU subjects plus celebrities, gaming, movies, and music.
  • Data-generation process: GPT-4 generates prompts about niche concepts or objects, with separate LongFact-Concepts and LongFact-Objects tasks.The prompt templates use topic-specific in-context exemplars and are designed to require long-form responses containing multiple detailed factoids.
  • Data-generation process: The generation process creates 60 prompts per topic and task, removes duplicates manually, and randomly retains 30 deduplicated prompts per topic.This yields 1,140 prompts for each task and 2,280 prompts overall.
  • SAFE evaluation process: SAFE splits responses into individual facts, revises them to be self-contained, determines relevance, and searches for evidence supporting relevant facts.Its relevance labels are “supported”, “not-supported”, and “irrelevant”; irrelevant facts are skipped during supportedness evaluation.
  • SAFE evaluation process: SAFE’s fact-splitting output correlates with human annotations at Pearson r = 0.798 and Spearman r = 0.846 across 496 prompt–response pairs.The authors report no errors in manually examined random examples and characterize the splitting quality as reasonably high.
  • SAFE evaluation process: Forcing exactly five search queries can create duplicate queries, especially when objective claims require fewer queries to verify.The authors suggest allowing SAFE to stop once enough evidence has been collected.

C.7 Search scope ablation

The search-scope ablation compares unrestricted web search with Wikipedia-only search and finds that open-web search more often supplies the needed information, although Wikipedia-only search can avoid irrelevant web evidence in some cases.

  • Search scope: SAFE’s default search scope covers the entire Internet through the Serper Google Search API.The ablation compares this unrestricted setup with Wikipedia-only queries.
  • Search scope: Open-web search found the right information when Wikipedia lacked it, while Wikipedia-only search sometimes avoided irrelevant web results.These were the two observed cases in which the search scope changed a claim’s rating.
  • Search scope: The open-web advantage appeared three times more often than the Wikipedia-only advantage.The authors suggest open-web search is especially helpful for open-domain questions where Wikipedia contains less information.

C.8 Hyperparameter ablation

SAFE performs best with five search queries per fact and three results per query; expanding either setting provides no benefit or can degrade annotation quality. In professional domains, SAFE remains useful but errors in fact decomposition and self-containment limit performance.

  • Search-query ablation: Five search queries per individual fact generally achieve SAFE’s highest correlation with human annotations, while additional queries add computational cost without improving correlation.The study fixes three results per query while varying the number of queries.
  • Search-result ablation: Three returned search results per query achieve the highest correlation with human annotations; too few or too many results reduce performance.Too few results may omit necessary evidence, whereas too many may obscure important information or add lower-quality results.
  • Professional domains: SAFE agrees with researchers on 79.7% of law facts and 94.9% of medicine facts in an additional professional-domain study.The evaluation covered 133 law facts and 137 medicine facts.
  • Professional domains: In nuanced domains, SAFE’s main challenges are decomposing long sentences into atomic facts and revising those facts into self-contained statements.Examples include extracting an incomplete claim and preserving the correct subject during revision.
  • Professional domains: The authors suggest stronger-reasoning LLMs could improve self-contained fact generation more than search-engine interaction in nuanced domains.SAFE can still reason from search results to judge whether a well-formed atomic fact is supported.

D Metric details

F1@K balances factual precision with the number of supported facts a user expects, so K controls the response-length preference encoded by the metric. The metric reduces to precision at K = 1 and increasingly emphasizes supported-fact count as K grows.

  • Assumptions: F1@K assumes model responses contain distinct facts because repeated supported facts could otherwise inflate the score without adding user information.The authors report no such repetition in their benchmarked models and suggest fluency or usefulness metrics better capture repetition.
  • Metric interpretation: K represents the number of supported facts required for full recall and encodes a user’s indifference to additional supported facts after that point.Choosing K therefore models a preferred response length for factuality evaluation.
  • Metric behavior: F1@1 ranks responses solely by factual precision, increasing monotonically with precision.At K = 1, the recall component combines with precision so the resulting ranking depends only on Prec(y).
  • Metric behavior: For sufficiently large K, F1@K ranks responses by the number of supported facts.Because F1@K approaches zero as K grows, the analysis considers the rescaled quantity K · F1@K.
  • Precision–recall curves: Longer responses increase factual recall but lower precision, exposing a precision–recall trade-off controlled by response length.Shorter responses have higher precision, while longer responses contain more supported facts alongside a larger proportion of errors.
  • Model scaling: Under the paper’s scaling assumptions, larger models generally achieve better long-form factuality across model families.The reported pattern is strongest for Gemini, GPT, and PaLM-2; Claude shows a larger generational than within-generation scaling effect.
  • Model scaling: At sufficiently large K, model rankings remain relatively constant, whereas K = 1 can substantially change rankings.The comparison uses K values of 1, 48, 64, 84, and 178.
  • Model scaling: RLHF improves PaLM-2-L-IT-RLHF’s supported-fact count to 72.9 facts per response versus 13.2 for PaLM-2-L-IT.The RLHF model also achieves better F1 at every tested K, including K = 1.

E.5 Full model-benchmarking results

The full benchmark evaluates thirteen language models on 250 randomly sampled LongFact-Objects prompts using SAFE-derived raw and F1@K metrics. The supplied passages specify the evaluation design but not the table’s individual model rows.

  • Evaluation design: The benchmark evaluates thirteen language models on the same 250 randomly sampled LongFact-Objects prompts.Responses are evaluated with SAFE, and metrics are averaged across the 250 examples.
  • Reported metrics: The benchmark reports supported, not-supported, and irrelevant fact counts together with precision, recall, and F1@K.Raw metrics come from SAFE evaluation, while aggregated metrics use F1@K.
  • Prompt coverage: The LongFact-Objects benchmark spans prompts covering topics from history, science, technology, finance, law, medicine, and other domains.The listed examples include both concept-oriented and object-oriented factuality prompts across many subject areas.

2. Limitations

The paper identifies several scope boundaries: it includes no theoretical results, human-subjects research, or released models, while reporting detailed experimental reproducibility information.

  • The paper includes no theoretical results, so theorem assumptions and proofs are not applicable to its contributions.
  • The authors state that experiments and implementations are described in detail, with reproducibility steps documented in the appendices.
  • The authors release no models; their dataset was language-model-generated, manually reviewed, and accompanied by a canary string intended to prevent misuse.
  • The study does not involve crowdsourcing or human-subjects research, and therefore reports no related institutional review process.
Loading 2403.18802v4…