Source-linked AI summary
Measuring short-form factuality in large language models
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus
TL;DR
Language-model factuality is difficult to measure, especially when outputs contain many claims and hallucinations limit broader adoption. The paper introduces SimpleQA, a benchmark of short, fact-seeking questions with uniquely gradable answers, and reports that it challenges frontier models while supporting measurement of calibration. Its scope remains limited to short answers with a single verifiable answer.
Problem
Language models sometimes produce false or unsupported outputs, while factuality is difficult to evaluate for arbitrary claims and long completions.
Method
SimpleQA is a benchmark of 4,326 short, fact-seeking questions designed with uniquely gradable answers and collected to challenge frontier models.
Results
SimpleQA is likely challenging for frontier models generally; Claude attempts fewer questions than GPT-4o, producing a similar F-score despite fewer correct answers.
Takeaways & Limitations
SimpleQA provides a targeted way to measure factuality and calibration, including whether stated confidence tracks actual accuracy.
Takeaways & Limitations
SimpleQA measures factuality only for short, fact-seeking queries with a single, verifiable answer, so generalization to lengthy factual responses remains open.
Abstract
from arXiv · showhide
We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.
1 Introduction
SimpleQA targets the difficult measurement problem of factuality by evaluating short, fact-seeking questions with uniquely gradable answers. Its 4,326-question design emphasizes correctness, usability, frontier-model challenge, topic diversity, and reliable measurement.
- Factuality is difficult to measure because arbitrary claims are challenging to evaluate and long model outputs may contain dozens of factual claims.
- SimpleQA contains 4,326 short, fact-seeking questions designed to make factuality measurement more tractable.
- Reference answers are determined by two independent AI trainers, and questions are written so predicted answers are easily gradable.
- SimpleQA is fast and simple to run, with 4,326 questions expected to produce relatively low run-to-run variance.
- SimpleQA is designed to challenge frontier models, with GPT-4o and Claude each scoring below 50%.
- The dataset spans topics including history, science and technology, art, geography, and television shows.
- SimpleQA aims to provide a simple and reliable dataset for measuring frontier-model factuality.
2 Data collection and verification
SimpleQA was built through constrained question writing, independent verification, source checks, and adversarial review to support reliable factuality evaluation. A third-trainer audit estimated an approximately 3% benchmark error rate, while the dataset spans diverse topics and answer types.
- Verification: Questions were independently answered by a second trainer, and only matching answers were retained.Verification also checked whether questions had single answers and timeless references.
- Adversarial difficulty: At least one reviewed frontier-model completion had to be incorrect, making the benchmark challenging rather than saturated.Trainers reviewed four model completions during question creation.
- Verification: Two unique website domains among the trainers’ 2–4 sources were required to improve reference-answer correctness.This source-diversity filter was applied after answer agreement.
- Quality audit: 94.4% third-trainer performance was observed on a random 1,000-example audit, with 4.1% actually incorrect answers after manual review.Some errors reflected incomplete answers or contradictions with cited sources.
- Quality audit: Approximately 3% of the benchmark was estimated to contain real data issues, including ambiguity, contradictory sources, and multiple correct answers.The estimate assumes no false positives from the prompted ChatGPT grader.
- Dataset diversity: Science & Technology, Politics, and Art were the most common topics, while dates, people, numbers, places, and other answers contributed varied coverage.Science & Technology had 858 questions, Politics 709, and Art 550; answer types were 32.8% dates, 24.1% people, 15.3% numbers, 9.9% places, and 18.0% other.
- Grading and metrics: Completions were graded as correct, incorrect, or not attempted, with overall correct and correct given attempted combined into an F-score.The F-score is the harmonic mean of the two metrics, but below 50% performance it can incentivize guessing at 50% confidence.
3 Evaluation of models
SimpleQA evaluation found higher performance for larger models across the tested OpenAI and Anthropic systems. Claude models also tended to attempt fewer questions, so lower correctness could yield a similar F-score to GPT-4o.
- Model performance: Larger models achieved higher SimpleQA performance than smaller counterparts across the evaluated OpenAI and Anthropic models.GPT-4o outperformed GPT-4o-mini, o1-preview outperformed o1-mini, and opus led the Claude 3 series.
- Attempt behavior: Claude-3.5 Sonnet had fewer correct answers than GPT-4o but attempted fewer questions, resulting in a similar F-score.Attempt behavior therefore materially affected the aggregate metric.
4 Measuring calibration
SimpleQA measures calibration through stated confidence and repeated answers, finding that larger models are better calibrated but still tend to overstate confidence.
- SimpleQA measures calibration by comparing models’ stated confidence with their actual accuracy.A perfectly calibrated model would match accuracy to stated confidence, such as 75% accuracy when confidence is 75%.
- Models show positive confidence–accuracy relationships but perform below y = x, indicating systematic overstatement of stated confidence.The paper identifies substantial room to improve calibration based on stated confidence.
- Accuracy rises with response frequency when models answer the same question 100 times, providing a second calibration measure.The repeated-answer analysis uses temperature 1, string matching, and the most-frequent answer for each question.
- o1-preview is more calibrated than o1-mini, while gpt4o is more calibrated than gpt4o-mini under both calibration approaches.The repeated-response analysis finds o1-preview’s response frequency roughly matches response accuracy.
5 Related work and discussion
The paper positions SimpleQA as a concise factuality benchmark that updates older question-answering evaluations and supports calibration research, while acknowledging a narrow scope.
- SimpleQA is presented as a simple benchmark for measuring language-model factuality alongside prior world-knowledge benchmarks.The paper identifies TriviaQA and Natural Questions as similar older benchmarks that are now too easy for today’s models.
- The paper’s calibration experiments build on prior work studying whether neural networks and language models are calibrated.Its finding that answer frequency correlates with accuracy is described as consistent with earlier research.
- SimpleQA measures factuality only for short, fact-seeking queries with a single, verifiable answer.Whether short-answer factuality correlates with lengthy responses containing many facts remains an open research question.
A Template for ChatGPT grader
The grader template classifies predicted answers as correct, incorrect, or not attempted using semantic content, contradiction, and completeness criteria, with special handling for numbers and implied information.
- The grader assigns each prediction one of three labels: CORRECT, INCORRECT, or NOT_ATTEMPTED.The template presents examples before grading a new question and requests only the corresponding label.
- A correct answer must contain the question-relevant target information without contradiction, while capitalization, punctuation, grammar, and order do not matter.Hedging and guessing remain permissible when the target is fully included and no incorrect information appears.
- An answer is incorrect when it contradicts the gold target, including through hedged factual statements.The template explicitly treats statements such as “it is possible that” as incorrect when they contradict the target.
- An answer is not attempted when it omits important target information without contradicting the gold target.Expressions of uncertainty that fail to provide the requested information are examples of NOT_ATTEMPTED.
- For numeric targets, answers must be correct to the gold answer’s last significant figure, while approximate responses can be NOT_ATTEMPTED.The examples classify 120k, 124k, and 115k as correct against a 120k target, but around 100k as not attempted.
B Guessing strategy and F-score
The paper shows that F-score rewards guessing whenever the probability of correctness exceeds one half, creating a loophole for models deciding whether to answer.
- F-score incentivizes a model to guess whenever it is at least 50% confident that its answer will be correct.The paper frames this as a problem with using F-score to balance correct, incorrect, and unanswered questions.
- The F-score uses c for correct answers, i for incorrect answers, and n for non-answered questions.These quantities appear in the paper’s expression for F-score.
- When correctness probability exceeds 1/2, guessing has higher expected F-score than leaving the question unanswered, regardless of c, i, and n.The paper supports this with an inequality comparing expected F-score after guessing with the score from not answering.
C Calibration Prompt
The calibration prompt asks for an answer and a confidence score, with the response formatted as JSON containing both fields.
- The prompt presents a question and requests the model’s best guess together with a confidence score from 0% to 100%.
- The required response format includes an "answer" field containing the answer text.
- The required response format includes a numeric "confidence_score" field.