PaperlayerView exact revision on arXiv

Source-linked AI summary

The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality

Aileen Cheng, Alon Jacovi, Amir Globerson, Ben Golan, Charles Kwong, Chris Alberti, Connie Tao, Eyal Ben-David, Gaurav Singh Tomar, Lukas Haas, Yonatan Bitton, Adam Bloniarz, Aijun Bai, Andrew Wang, Anfal Siddiqui, Arturo Bajuelos Castillo, Aviel Atias, Chang Liu, Corey Fry, Daniel Balle, Deepanway Ghosal, Doron Kukliansky, Dror Marcus, Elena Gribovskaya, Eran Ofek, Honglei Zhuang, Itay Laish, Jan Ackermann, Lily Wang, Meg Risdal, Megan Barnes, Michael Fink, Mohamed Amin, Moran Ambar, Natan Potikha, Nikita Gupta, Nitzan Katz, Noam Velan, Ofir Roval, Ori Ram, Polina Zablotskaia, Prathamesh Bang, Priyanka Agrawal, Rakesh Ghiya, Sanjay Ganapathy, Simon Baumgartner, Sofia Erell, Sushant Prakash, Thibault Sellam, Vikram Rao, Xuanhui Wang, Yaroslav Akulov, Yulong Yang, Zhen Yang, Zhixin Lai, Zhongru Wu, Anca Dragan, Avinatan Hassidim, Fernando Pereira, Slav Petrov, Srinivasan Venkatachary, Tulsee Doshi, Yossi Matias, Sasha Goldshtein, Dipanjan Das

arXiv:2512.10791v1cs.CLcs.AI

TL;DR

LLMs continue to generate factually inaccurate text, while earlier work often evaluated only specific capabilities. The FACTS Leaderboard combines four benchmarks covering different factuality settings into a holistic score; the top-performing model has average accuracy of 69%, leaving headroom for progress.

  • Problem

    LLMs continue to generate factual errors, and prior benchmarks have focused on specific capabilities rather than evaluating factuality across a wider spectrum of tasks.

    Source: §1 · p. 1

  • Method

    The FACTS Leaderboard aggregates four specialized benchmarks and reports a holistic FACTS Score averaging accuracy across the four tasks.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    The top-performing model on the FACTS suite has average accuracy of 69%, leaving considerable headroom for future progress.

    Source: §7 · p. 16

  • Takeaways & Limitations

    The four subsets target distinct capabilities and can serve to drive research on those fronts.

    Source: §7 · p. 16

    The suite does not cover video understanding or fast-changing information.

    Source: §7 · p. 16

Abstract

from arXiv · show

We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models' world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model's overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts .

1. Introduction

LLMs still produce factually inaccurate text, and prior benchmarks have often assessed only particular factuality capabilities. The FACTS Leaderboard broadens evaluation across four tasks spanning modalities, knowledge sources, and response formats.

1. Introduction

  • Prior benchmarks focused on specific capabilities, while factuality evaluation needs to span contexts, modalities, knowledge sources, and response formats.A model may summarize a document well yet struggle with factoid questions from memory, or vice versa.Source: §1 · pp. 1–2
  • The FACTS Leaderboard aggregates four specialized sub-leaderboards to provide a holistic evaluation suite.The suite aims to measure factual reliability across diverse tasks rather than a narrow set.Source: §1 · p. 2
  • FACTS Multimodal covers image questions, Parametric covers internal factoid knowledge, Search covers search-tool responses, and Grounding v2 covers document grounding with improved judges.The four components address distinct dimensions of factuality.Source: §1 · p. 2

2. The FACTS Leaderboard

The FACTS Leaderboard reports task-specific metrics alongside a holistic FACTS Score, the average accuracy across four tasks. Its public and private sets support an open leaderboard while keeping some prompts unreleased to mitigate overfitting.

2. The FACTS Leaderboard

  • The live leaderboard accepts new model submissions, while only a subset of prompts is public to mitigate overfitting.Kaggle conducts all model evaluations.Source: §2 · p. 3
  • The FACTS Score is the average accuracy across four tasks, with each task's accuracy averaged over public and private sets.Detailed task-specific metrics are also published; separate task analysis yields the most complete insight.Source: §2 · p. 3
  • Table 1 reports FACTS Score and subset accuracy with 95% confidence intervals for proprietary API-based models on the full benchmark.The overview covers both public and private sets.Source: §2 · p. 3
  • The following sections describe each benchmark's dataset construction, tailored metrics, and benchmark-specific outcomes.Source: §2 · p. 3

3. FACTS Multimodal

FACTS Multimodal evaluates factual answers to image-based questions using human-authored rubrics and automated judges. Its results show different model-family strengths in Coverage and No-Contradiction, while its autoraters are validated against human annotations.

3.1. Data

  • The roughly 1,500-question benchmark has 711 public and 811 private items spanning visual description, chart interpretation, object recognition, and reasoning about visual scenes.Figure 1 presents the distribution of image and question categories in the public set.Source: §3.1 · p. 3, Figure 1 · p. 5

3.2. Metrics

  • Human-authored rubrics distinguish essential from non-essential facts, and automated judges assess coverage and contradictions against the rubric, common knowledge, and the image.The two verdicts measure whether essential facts are included and whether the answer makes contradictory claims.Source: §3.2 · pp. 3–4
  • A response counts as accurate only when it covers the essential facts and contains no contradictions.The overall accuracy score is the percentage of responses satisfying both conditions.Source: §3.2 · p. 4

3.3. Autorater Validation.

  • For Coverage, the autorater’s percentage of essential facts supported correlates with human judgments at Spearman’s 0.64; at a 0.5 threshold, macro F1 is 72.3.The human annotation task mirrored the autorater’s assessment of essential facts.Source: §3.3 · p. 5
  • No-Contradiction validation against fine-grained human annotations achieves macro F1 78.2, with annotators reviewing responses sentence by sentence.For this evaluation, the positive class denotes the absence of a contradiction.Source: §3.3 · p. 5, Table 4 · p. 6

3.2. Metrics

  • Gemini models show higher Coverage, while GPT models achieve the highest No-Contradiction scores on the main FACTS Multimodal benchmark.Table 3 reports detailed results for the private and public sets.Source: §3.2 · p. 4, Table 3 · p. 6

4. FACTS Parametric

FACTS Parametric tests closed-book recall of challenging, user-interest factoids whose answers are supported by Wikipedia. It combines adversarial selection and human verification with metrics that distinguish raw accuracy from performance on attempted answers.

4.1. Data

  • The benchmark contains 2,104 QA pairs, split evenly between public and private sets, with questions reflecting user interest and answers confirmed in Wikipedia.Questions were adversarially sampled and human-verified; the public set spans topics and answer types including politics, sports, technology, people, dates, and places.Source: §4.1 · p. 6, §4 · p. 6

4.1.1. FACTS Parametric properties

  • Questions target one unambiguous, concise fact with a specified answer type and a stable or explicitly time-anchored answer.Wikipedia support is required to assess recall of knowledge expected to be available from pretraining.Source: §4.1.1 · p. 7

4.1.2. Data Processing

  • Selection filters user-interest questions with Gemini-2.5-Pro search, retains Wikipedia-supported answers, then keeps questions missed by five open-weight models answering closed-book.Three independent annotators verify factuality and compliance, extract Wikipedia evidence, and correct or discard unsuitable pairs; “correctly” at the preliminary filter means matching the silver answer.Source: §4.1.2 · p. 8

4.2. Metrics

  • GPT-o3 reaches 57.0% accuracy versus GPT-5’s 55.7%, but GPT-5 has higher attempted accuracy (64.3% vs. 58.2%) and F1 (59.7 vs. 57.6).GPT-5 hedges in 13.3% of cases versus 1.9% for GPT-o3.Source: Table 6 · p. 10, §4.2 · p. 10
  • The grader reports correct, incorrect, not-attempted, or unknown; accuracy is primary, with hedging rate, attempted accuracy, and F1 as secondary metrics.Three Gemini-2.5-Pro grades are averaged per response, and the paper reports that this judge preserves the relative trends and rankings of a mixed-model panel.Source: §4.2 · p. 10

5. FACTS Search

FACTS Search evaluates factual responses produced with web search, emphasizing difficult tail-entity and multi-hop questions rather than trying to guarantee that every answer is absent from model training data. Its results describe search behavior and accuracy-related trade-offs across models.

5. FACTS Search

  • The benchmark targets tail entities and multi-hop queries, since training-data absence is difficult to guarantee across models with different training data and cutoff dates.The evaluation set has 1,884 questions: 890 public and 994 private.Source: §5 · p. 11, §5.1 · p. 11

5.3. Metrics

  • Gemini 3 Pro, the highest-performing model, searches less on average than other top models, while Grok searches most; Claude appears to favor attempted-query accuracy over overall accuracy.The paper reports these patterns alongside its main FACTS Search results.Source: Table 8 · p. 12, §5.3 · p. 13

5.1. Data

  • Questions were retained only if all three raters judged them correct, unique, and immutable, then excluded if Gemini 2.5 Flash without search answered correctly, emphasizing need for search.Final subset sizes were 328 Hard Tail, 932 Wiki Two-Hop, 268 Wiki Multi-Doc, and 356 KG Hops.Source: §5.1 · p. 13

5.2. Search Engine

  • All evaluated models use the same Brave Search API and tool description, with search results appended to the model’s context after a tool call.Source: §5.2 · p. 13

5.3. Metrics

  • Gemini 2.0 Flash judges each response against the query and gold response as correct, incorrect, or not attempted.Source: §5.3 · p. 13

6. FACTS Grounding v2

FACTS Grounding v2 retains the v1 prompt set and evaluation approach while updating its judges and prompt. It scores grounding in context while marking responses that fail to address the user request as inaccurate.

6.1. Data

  • FACTS Grounding v2 retains v1’s prompts: long-form Q&A, summarization, and rewriting tasks based on web-sourced documents or reviews, with responses instructed to use only the supplied context.Prompts span document lengths up to 32k tokens and enterprise domains including finance, technology, retail, medical, and legal.Source: §6.1 · p. 14

6.2. Metrics

  • The new judge models combined with the v2 prompt outperform the other tested judge-model and prompt combinations against human ratings on a 320-example heldout set.The unadjusted factuality score averages the percentage of accurate responses from the two judges.Source: Table 10 · p. 15, §6.2 · p. 15
  • Two judges, Gemini 2.5 Flash and GPT-5, label a response inaccurate if any information-bearing claim is not grounded in the prompt’s context.Using two judges is intended to reduce bias from a particular judge model; v1 used a different judge set.Source: §6.2 · p. 15
  • Responses that do not sufficiently answer the request are ineligible and count as inaccurate, preventing short but grounded answers from earning a high factuality score while evading user intent.Source: §6.2 · pp. 15–16, Table 11 · p. 16

7. Conclusion

The FACTS suite is presented as a challenge for current models, with the top model averaging 69% accuracy. Its four benchmarks address distinct factuality capabilities, while video, fast-changing information, and some tool-use challenges remain outside its coverage.

7. Conclusion

  • 69% average accuracy for the top-performing model leaves headroom for progress on the FACTS suite.The authors also call for finer-grained analysis of question difficulty, including entity infrequency and search “tailness.”Source: §7 · p. 16
  • FACTS separates factuality across image understanding, internal knowledge, search tools, and grounding responses in context.The authors describe these as capabilities that can drive research on distinct fronts.Source: §7 · p. 16
  • The suite does not cover video understanding or fast-changing information, and knowledge-base calls introduce further tool-use challenges.Source: §7 · p. 16

8. Contributions and Acknowledgments

The paper credits named leads, benchmark designers, the FACTS Team, and sponsors. Its acknowledgments also recognize model access, leaderboard support, expert annotation, and reviewer feedback.

8. Contributions and Acknowledgments

  • Connie Tao, Dipanjan Das, Lukas Haas, and Sasha Goldshtein are listed as leads.Source: §8 · p. 17
  • The paper separately credits a benchmark-design group and the broader FACTS Team.Source: §8 · p. 17
  • Acknowledgments thank Gemini for model access, Kaggle for leaderboard support, expert annotators for examples, and reviewers for feedback.Source: §8 · p. 17

Related papers

Loading 2512.10791v1…