Source-linked AI summary
Do Large Language Model Benchmarks Test Reliability?
Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander Madry
TL;DR
The paper asks whether existing benchmarks can establish LLM reliability when their remaining errors may reflect label noise or ambiguity. It constructs platinum versions of fifteen benchmarks by correcting or removing problematic examples, then evaluates frontier models. The cleaned evaluations still reveal widespread failures on simple tasks and recurring failure patterns.
Problem
Existing benchmarks emphasize capability and may treat high but imperfect performance as saturation, leaving limited evidence about whether frontier models are reliably correct on simple, clearly defined tasks.
Method
The authors create platinum benchmarks by manually reviewing model-disagreement cases, correcting mislabeled solutions, and removing ambiguous or poorly written questions.
Results
Almost every model makes mistakes on almost every dataset after cleaning, while up to 30% of examples in several reading-comprehension datasets contain identified issues.
Takeaways & Limitations
Reliability evaluation requires carefully curated benchmarks because original benchmark errors can obscure genuine model failures and systematic weaknesses.
Takeaways & Limitations
Because the revision process re-labels examples only when some evaluated model fails, poorly written questions may remain when all models agree with the stated ground truth.
Abstract
from arXiv · showhide
When deploying large language models (LLMs), it is important to ensure that these models are not only capable, but also reliable. Many benchmarks have been created to track LLMs' growing capabilities, however there has been no similar focus on measuring their reliability. To understand the potential ramifications of this gap, we investigate how well current benchmarks quantify model reliability. We find that pervasive label errors can compromise these evaluations, obscuring lingering model failures and hiding unreliable behavior. Motivated by this gap in the evaluation of reliability, we then propose the concept of so-called platinum benchmarks, i.e., benchmarks carefully curated to minimize label errors and ambiguity. As a first attempt at constructing such benchmarks, we revise examples from fifteen existing popular benchmarks. We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems. Analyzing these failures further reveals previously unidentified patterns of problems on which frontier models consistently struggle. We provide code at https://github.com/MadryLab/platinum-benchmarks
1 Introduction
Existing benchmarks track capability gains but are poorly suited to determine whether frontier LLMs are reliable. The paper argues that label noise can obscure genuine failures and introduces platinum benchmarks to measure reliability more rigorously.
- Motivation: LLMs show impressive capabilities but still fail on tasks simpler than their frontier abilities, creating reliability concerns in high-stakes applications.Errors in healthcare, finance, insurance, and legal services can have serious consequences.
- Research question: The central question is on what kinds of tasks frontier models are actually reliable.
- Benchmark gap: Older benchmarks are often considered saturated when models reach roughly 90–95% performance, with remaining errors attributed to label noise or ambiguity.This leaves uncertainty about whether the remaining errors reflect model unreliability.
- Platinum benchmarks: Platinum benchmarks minimize label errors and ambiguity so that 100% performance is attainable and reliability remains measurable beyond capability saturation.They target the reliability frontier: the hardest tasks models can perform consistently without error.
- Contributions: Revising fifteen saturated benchmarks shows that many contain errors, while genuine failures remain on simple tasks and reveal consistent failure patterns.The paper reports chronological questions where multiple models answer with the first event over 85% of the time.
2 Cleaning Up Noisy Benchmarks
The paper constructs platinum benchmarks by identifying mislabeled solutions and poorly written questions, then correcting or removing them. Across fifteen benchmarks, substantial noise is found, including up to 30% problematic examples in reading comprehension datasets.
- Experimental setup: The study revises fifteen benchmarks spanning mathematics, logic, table understanding, reading comprehension, commonsense reasoning, and vision.It evaluates a variety of proprietary and open-weights frontier models.
- Reliability criterion: Benchmark reliability is defined as near-perfect correctness on unambiguous, clearly defined questions, so every incorrect answer should indicate a model failure.
- Cleaning protocol: Mislabeled solutions are corrected, while questions with contradictions, ambiguity, or construction flaws are removed.
- Error identification: The scalable detection strategy sends each question to several frontier LLMs and manually inspects examples where at least one model errs.Inspected examples are either removed as poorly written or relabeled when the solution is incorrect.
- Noise findings: Up to 30% of examples in SQuAD2.0, HotpotQA, and DROP contain identified issues.Reading-comprehension problems often involve open-ended answers or ambiguous unanswerable questions.
3 Evaluating Reliability with Platinum Benchmarks
Platinum benchmarks measure reliability across difficulty levels by recording manually verified model errors. They show that frontier models still make widespread simple mistakes, while many errors on original benchmarks disappear after cleaning.
- Reliability frontier: Reliability frontiers are estimated by finding the most difficult platinum benchmark a model can pass with 100% accuracy.
- Reliability frontier: Six revised mathematics benchmarks range from single operations to high-school mathematics, allowing the difficulty at which reliability declines to be localized.
- Findings: Almost every model makes simple mistakes on almost every dataset, except particularly simple mathematics datasets such as SingleOP, SingleEq, and MultiArith.Every reported platinum-benchmark error was manually verified.
- Findings: For most original benchmarks, the majority of model errors are attributable to mislabeling or poorly written questions rather than genuine model failures.
Appendix B.1.2 for details.
The paper identifies recurring failure patterns that persist after benchmark cleaning, including strong first-event bias and incorrect rounding behavior. These patterns show that frontier models can fail systematically on simple tasks despite advanced capabilities.
- First event bias: Chronological questions expose a first-event bias in which multiple models answer with the first event instead of the second over 85% of the time.The models can explicitly state reasoning that identifies the first event while answering a question about the second.
- First event bias: Gemini 1.5 Flash, Gemini 1.5 Pro, and Mistral Small exhibit this first-event failure pattern.
- Rounding up primes: Claude 3.5 Sonnet often rounds up a quotient even when division yields a whole number, producing an answer one greater than the solution.
- Rounding up primes: When the correct answer is prime, Claude 3.5 Sonnet makes this rounding error 20% of the time.
- Relation to prior work: The identified failures extend prior observations that LLMs can break down on simple reasoning tasks despite solving harder versions.
5 Discussion and Conclusion
The paper frames platinum benchmarks as an initial approach to measuring LLM reliability, while identifying coverage, sample-size, residual-labeling, difficulty, and prompt-brittleness limitations. Its findings also expose a gap between frontier models’ capabilities and reliability, motivating reliability as a deployment metric.
- Capabilities and levels of difficulty covered: The fifteen benchmarks omit capabilities such as coding and tool use, focusing broad difficulty coverage primarily on mathematics.
- Number of examples per benchmark: Some revised benchmarks contain as few as 100 examples, limiting certainty when distinguishing zero-percent from one-percent error rates.
- Only re-labeling errors: Although every reported model error is genuine after revising failed examples, unrevised questions may still contain ambiguity or poor wording.
- Benchmark difficulty: Current frontier models still fail sufficiently simple questions, but future platinum benchmarks for expert capabilities may require expensive expert annotation.
- Discussion: Frontier LLMs can solve graduate-level problems yet still fail basic logic and elementary problems, revealing a capability–reliability gap not seemingly addressed by further scaling.
- Prompt brittleness: Prompt choices may affect reliability even when instructions are clear, so the paper calls for studying prompting strategies and less prompt-brittle models.
- Framing reliability as a deployment metric: Reliability is framed as a deployment metric analogous to uptime in site reliability engineering, with platinum benchmarks intended as an initial step toward quantifying it for LLMs.
- Conclusions: The paper proposes carefully curated platinum benchmarks and revises fifteen existing datasets to test reliability against high standards for real-world applications.
A The Life Cycle of Benchmarks
The appendix compares benchmark accuracies at release and currently, illustrating how academic LLM benchmarks can move through a development, progress, and eventual-retirement life cycle.
- The table is used to show accuracies across benchmarks at two points in their apparent life cycle.
- Table A.1 reports popular academic LLM benchmarks’ accuracies at their release time and currently.
B.1 Benchmark processing
The benchmark-processing pipeline applies benchmark-specific cleaning and labeling choices, including manual single-label annotation for selected VQA queries and restrictions intended to reduce ambiguity.
- The processing section includes additional cleaning steps tailored to particular benchmarks.
- Re-labeling: The revised VQA subset manually assigns one ground-truth label to each included image-question query rather than relying only on existing annotations.
- Selection of queries: VQA v2.0 image pairs are randomly selected, with pairs rejected when either image is judged ambiguous.
- Selection of queries: The VQA subset is restricted to yes/no questions because open-ended queries have greater ambiguity and may have multiple correct answers.
B.1.2 Reading Comprehension Benchmarks
The revision treats reading-comprehension answers as potentially having multiple valid forms and updates labels accordingly, while excluding irreducibly ambiguous questions. Evaluation uses answer-equivalence matching and standardized prompting.
- Reading-comprehension questions can have multiple valid answers, such as both “Salma Hayek Pinault” and “Salma Hayek.”
- The revised labels list valid equivalent responses, while questions with too many plausible answers are marked as bad.
- Only examples whose original label is absent from the revised valid-label list are counted as mislabeled.
- For original benchmarks, GPT-4o identifies answers equivalent to the benchmark solution before errors are counted.
- Evaluation generally uses chain-of-thought prompts, while VQA V2.0 omits them because its questions rarely require multiple reasoning steps.
C.1 First event bias
The analysis identifies systematic failures on simple chronological and arithmetic tasks. Models often answer the first event instead of the second, while Claude 3.5 Sonnet rounds up whole-number quotients more often when solutions are close to prime.
- First event bias: Models answer the first event even when asked which event happened second, and their reasoning explicitly identifies that first event.
- First event bias: The event-order experiment repeats each question with both event orderings to reduce ordering bias.
- First event bias: Three models fail on over 85% of 100 order-flipped event questions, despite random guessing achieving 50%.The models are Gemini 1.5 Flash, Gemini 1.5 Pro, and Mistral Small.
- Rounding up primes: Claude 3.5 Sonnet (June) incorrectly rounds up whole-number quotients in elementary division problems.
- Rounding up primes: Failures become more frequent as the quotient has fewer non-trivial divisor pairs, meaning it is closer to being prime.
D Examples of Benchmark Label Errors
The revision process found label errors across fifteen benchmarks spanning mathematics, reading comprehension, and visual question answering. The examples include mislabeled, poorly written, and ambiguous questions.
- The study identifies label errors across fifteen benchmarks in six capability categories.
- Mathematics examples include mislabeled or poorly written questions from GSM8K and SVAMP.
- Reading-comprehension examples include label errors in SQuAD2.0, DROP, and HotPotQA.
- VQA V2.0 contains examples of ambiguous questions.
E Examples of Model Failures
Revised benchmark examples expose frontier-model failures on simple tasks, including coreference, logical deductions, arithmetic, and object counting. Model responses show that these failures persist across many leading systems, even when the questions are elementary.
- Object ordering: All listed models select the same option on a three-object ordering question, illustrating that some simple benchmark examples are solved consistently.The example states that the minivan is second-newest and offers three possible newest vehicles.
- Coreference resolution: The revised examples also test pronoun resolution where models consistently select Billy as the referent of “he.”The phrase says John cannot see the stage with Billy in front of him because he is short; the listed models choose option B.
- Coreference resolution: Most evaluated state-of-the-art models fail a basic coreference-resolution task, including o3-mini, Gemini 2.0 Flash, and DeepSeek-V3.The task asks whether “it” refers to the newspaper or the backpack; the cited responses select the newspaper.
- Logical deductions: Many frontier models answer a simple logical-deduction question incorrectly despite a context that identifies the correct response as unanswerable.The context concerns utility-line verification, while the revised answer states that the requested conclusion is not supported.
- Object counting: The examples include basic addition errors in BIG-bench Object Counting responses.The figure presents response snippets from multiple models to highlight these errors.