Source-linked AI summary
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein, Dipanjan Das
TL;DR
Measuring factuality in long-form generation is difficult because responses contain many claims and tasks vary widely. FACTS Grounding addresses this with a document-grounded benchmark and leaderboard, using eligibility filtering and aggregated judge evaluation to assess factuality.
Problem
Long-form factuality measurement is challenging because every response claim requires inspection and existing benchmarks often cover narrower use cases.
Method
The benchmark pairs human-authored user requests with web documents up to 32k tokens and evaluates long-form responses using public and private splits, eligibility filtering, and multiple automated judge models.
Results
Disqualifying ineligible responses reduces final factuality scores by 1%–5%, while aggregating multiple judges addresses self-rating bias averaging +3.23%.
Takeaways & Limitations
FACTS Grounding provides a maintained benchmark for testing whether models generate long-form responses that remain grounded in supplied documents while fulfilling user requests.
Takeaways & Limitations
Because context documents were collected from the internet, some may have appeared in models’ pre-training corpora, creating a possible data-contamination concern.
Abstract
from arXiv · showhide
We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard.
1. Introduction
The paper frames context-grounded factuality as a difficult measurement problem for long-form responses. FACTS Grounding addresses this gap with a benchmark and leaderboard covering varied requests over documents up to 32k tokens, while excluding external-knowledge factuality.
- 1. Introduction: Long-form factuality requires inspecting each response claim against its source, making measurement harder than short-answer evaluation.Existing automatic evaluators also face limited context windows and insufficient reasoning ability for this task.
- 1. Introduction: The benchmark targets factuality with respect to a provided document, requiring models to synthesize derivable information while fulfilling varied user requests.Requests include summarization, fact finding, analysis, and comparison, extending beyond summarization alone.
- 1. Introduction: The work addresses context-grounded factuality rather than factuality against external sources or general world knowledge.The paper explicitly states that scenario (2) is not captured.
- 1. Introduction: The evaluation rigorously tests automatic evaluators on held-out data and aggregates multiple judges to reduce evaluator bias.This design responds to the difficulty of scaling reliable factuality measurement for long-form generation.
- 1. Introduction: FACTS Grounding provides a benchmark and leaderboard for long-form responses grounded in documents up to 32k tokens.The benchmark includes 860 public Open examples and 859 private Blind examples of human-written prompts.
2. Data
The FACTS Grounding dataset combines human-authored, validated prompts with diverse long-form documents and tasks. Quality assurance enforces context-only answering and removes unsuitable examples, while public and private splits support reliable evaluation.
- 2. Data: The dataset uses human-written document–request pairs subjected to validation and filtering, with balanced random sampling for public and private splits.This curation is intended to support benchmark reliability.
- 2.1. Annotation: Human raters created long-form Q&A, summarization, and rewriting tasks from web documents or review sets, with system instructions requiring context-only responses.Each request is designed to be answerable using its paired context.
- 2.1. Annotation: Prompts span documents up to 32k tokens and enterprise domains including finance, technology, retail, medical, and legal.The annotation instructions excluded creative responses, expert-level knowledge, mathematical or logical reasoning, and text metaanalysis.
- 2.1. Annotation: Manual quality assurance removed examples violating context-only instructions, requiring creative writing, demanding specialized knowledge or complex reasoning, or containing unreadable OCR.The final dataset has a mean document length of 2.5k tokens and a maximum of 32k tokens.
- 2.2. Data Quality Assurance: Internet-sourced documents may overlap with model pretraining data, although the requests and system instructions are described as non-contaminated.The authors argue that the benchmark still measures grounding to supplied context rather than use of external knowledge.
3. Metrics
FACTS Grounding separates instruction-following eligibility from factual grounding, then aggregates multiple judges’ accuracy verdicts into an unadjusted factuality score.
- 3.1. Unadjusted Factuality Score: A response is accurate only if every information-bearing claim is grounded in the user request and context document.
- 3.1. Unadjusted Factuality Score: Three judge models are used to reduce bias from any single judge, including favorable assessment of its own outputs.
- 3.1. Unadjusted Factuality Score: The unadjusted factuality score averages the percentages of accurate responses assigned by three judge models.A response is accurate only when its information-bearing claims are grounded in the prompt contents or require no grounding.
- 3.2. Disqualifying Ineligible Responses: Short, evasive responses can score highly on grounding while failing to address the user’s request meaningfully.The benchmark therefore treats instruction-following as a distinct eligibility task rather than relying on factuality alone.
- 3.2. Disqualifying Ineligible Responses: Responses are marked ineligible only when all three judges agree that they do not sufficiently answer the user’s request.Judge prompt templates are selected using Macro-F1 on a private test set, and eligibility classifications are ensembled by consensus.
4. Results
The results show that multiple-judge aggregation is important for evaluation, while disqualifying responses that fail the request lowers scores and can alter rankings.
- 4. Results: The fused rank exactly matches the ranking produced by the final factuality score.The fused rank combines six rankings from each split and judge model using the Condorcet algorithm.
- 4. Results: +3.23% is the mean increase when models judge their own outputs rather than other models’ outputs.The finding motivates aggregating multiple judge models despite the additional computational cost.
- 4. Results: Disqualifying ineligible responses reduces the final factuality score by 1%–5%.These responses are treated as inaccurate in the final score.
- 4. Results: Gemini 1.5 Flash moves from rank 1 to rank 2 after ineligible responses are disqualified.
5. Conclusion
FACTS Grounding is designed to test whether language models maintain factual accuracy in long-form responses grounded in a supplied document and tailored to the user’s request.
- 5. Conclusion: The leaderboard challenges models to generate long-form responses that remain grounded in a provided document while following user requests and instructions.The authors encourage researchers to use the benchmark to advance factual capabilities and factuality-evaluation methods.
6. Contributions and Acknowledgements
The paper credits contributors across experimental design, organization, early experimentation, sponsorship, idea generation, writing, and external support.
- Contributions: Experimental design and reported experiments were conducted by Alon Jacovi, Andrew Wang, Chris Alberti, Jon Lipovetz, and Michelle Liu.
- Contributions: Connie Tao, Dipanjan Das, Kate Olszewska, Lukas Haas, and Nate Keating managed the effort’s overall organization.
- Contributions: A large group of contributors developed ideas, collected data, and ran early experiments.The named contributors include Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, and Sasha Goldshtein.
- Contributions: Sponsors provided technical guidance, while Slav Petrov and Madhu Gurumurthy proposed the leaderboard idea.
- Acknowledgements: All authors wrote parts of the report, and the paper acknowledges support from the Gemini and Kaggle teams, expert annotators, and reviewers.
A. Judge Prompt Templates
The judge prompt asks models to determine whether a response is accurate relative to provided evidence, using only an Accurate or Inaccurate verdict.
- The judge compares the response against the supplied evidence to assess factual accuracy.
- It outputs “Accurate” when the response is verified by the evidence.
- It outputs “Inaccurate” when the response contradicts the evidence or cannot be verified.
JSON (Alt)
The alternative JSON judge decomposes responses sentence by sentence, assigns context-relative labels, and returns structured rationales and excerpts before applying additional span-level checks.
- The judge analyzes each response sentence according to its relationship with the provided context.
- It labels sentences as supported, unsupported, contradictory, or no_rad based on contextual entailment and factual attribution.
- Supported sentences require a context excerpt that fully entails the sentence, while contradictory sentences require a contradicting excerpt.
- The input separates the context from the model-generated response for sentence-level analysis.
- Each output JSON object contains the sentence, label, rationale, and relevant context excerpt, with one object per line.
- The alternative template also checks individual spans for accuracy against the evidence and asks for a rationale.
Implicit span-level
The implicit span-level evaluation combines instruction-following checks with sentence-level accuracy judgments, and rejects responses receiving major instruction-following issues from all judges.
- The accuracy check decomposes the response into sentences and marks the full response Inaccurate if even one sentence is inaccurate.
- Responses are ineligible when all language-model judges identify a Major Issue(s) in instruction following.
- The rubric evaluates the test response for instruction following while calibrating judgments against a baseline response.
- Judges first identify the user-query instructions and assess whether the test and baseline responses meet each requirement.
- The instruction-following verdict uses three categories: No Issues, Minor Issue(s), and Major Issue(s).