Source-linked AI summary
Benchmarks Saturate When The Model Gets Smarter Than The Judge
Marthe Ballon, Andres Algaba, Brecht Verbeken, Vincent Ginis
TL;DR
Current LLM benchmarks are undermined by dataset and judge errors, especially as tasks become harder and models approach saturation. Omni-MATH-2 audits the dataset and compares two judges, finding substantial benchmark noise and that Omni-Judge is wrong in 96.4% of disagreements.
Problem
Open-ended, difficult LLM benchmarks face dataset and judge errors that complicate reliable evaluation of model abilities.
Method
The paper manually revises Omni-MATH, creates cleaned and tagged subsets, and evaluates five models with Omni-Judge and GPT-5 mini.
Results
Judge disagreement increases with difficulty, while Omni-Judge is wrong in 96.4% of disagreements and both judges miss incompatibilities in tagged problems.
Takeaways & Limitations
Accurate benchmark measurement requires auditing problem quality and using judges competent enough to distinguish model abilities as difficulty rises.
Takeaways & Limitations
The study does not revise all reference answers and examines only two judges under specific prompts and settings.
Abstract
from arXiv · showhide
Benchmarks are important tools to track progress in the development of Large Language Models (LLMs), yet inaccuracies in datasets and evaluation methods consistently undermine their effectiveness. Here, we present Omni-MATH-2, a manually revised version of the Omni-MATH dataset comprising a clean, exact-answer subset ($n{=}4181$) and a tagged, non-standard subset ($n{=}247$). Each problem was audited to ensure LaTeX compilability, solvability and verifiability, which involved adding missing figures or information, labeling problems requiring a proof, estimation or image, and removing clutter. This process significantly reduces dataset-induced noise, thereby providing a more precise assessment of model performance. The annotated dataset also allows us to evaluate judge-induced noise by comparing GPT-5 mini with the original Omni-Judge, revealing substantial discrepancies between judges on both the clean and tagged problem subsets. Expert annotations reveal that Omni-Judge is wrong in $96.4\%$ of the judge disagreements, indicating its inability to differentiate between models' abilities, even well before saturation of the benchmark occurs. As problems become more challenging, we find that increasingly competent judges become essential in order to prevent judge errors from masking genuine differences between models. Finally, neither judge identifies the present failure modes for the subset of tagged problems, demonstrating that dataset quality and judge reliability are both critical to develop accurate benchmarks of model performance.
1 Introduction
As mathematical benchmarks approach saturation, dataset and judge errors increasingly shape measured performance rather than true model differences. Omni-MATH-2 addresses these problems through manual revision, clean and tagged evaluation subsets, and analysis of dataset–model–judge interactions.
- Motivation: Open-ended, difficult benchmarks probe deeper reasoning but introduce answer-extraction and equivalence-judgment errors.These additional evaluation complexities create new sources of error beyond fixed-answer formats.
- Motivation: Dataset-induced errors, including ambiguous statements and unsolvable items, remain widespread in established benchmarks.Audits of MMLU and HellaSwag found item, ground-truth, or instance problems.
- Implications: Judge reliability remains task-dependent despite proposed ensembles and committees, so evaluation outcomes can reflect the pipeline rather than model capability.The paper frames benchmarks as dataset–model–judge triplets and motivates dataset audits, judge calibration, and robust uncertainty reporting.
- Motivation: As model accuracy approaches saturation, mathematical benchmark scores become increasingly bottlenecked by dataset- and judge-induced errors.GSM8K and MATH are described as saturated, motivating more challenging datasets such as FrontierMath and Humanity’s Last Exam.
- Contribution: Omni-MATH-2 preserves Omni-MATH’s original size of n=4,428 while editing 647 problems (14.6%) and tagging 247 as non-standard (5.6%).The revision targets LaTeX compilability, interpretability, and suitability for automated judging, and releases evaluation-ready subsets including Omni-MATH-2-Filtered (n=4,181).
2 Results
Cleaning exposed substantial dataset-induced errors, while judge comparisons showed that evaluation outcomes and model rankings can vary sharply by judge, domain, and difficulty. GPT-5 mini generally outperformed Omni-Judge, especially on flawed tagged problems, though reference-answer errors remained an additional limitation.
- Dataset revisions: 14.6% of problem statements were edited to fix LaTeX, add missing information, or tag problematic questions.Dataset errors included degenerate questions, missing images or options, and proof or estimation questions judged against exact answers.
- Tagged failure modes: 61 problems contained images, and missing attachments could make problems unsolvable while causing judges to mark correct responses incorrect.One example involved a model correctly identifying insufficient information, but the judge labeling that response incorrect.
- Tagged failure modes: 18.44 out of 20 points was warranted for one estimate under the problem’s scoring rule, yet the judge marked the model’s answer incorrect.Omni-MATH contained 54 estimation problems whose grading compared estimates with exact final answers.
- Inter-judge comparisons: On Omni-MATH-2-Filtered (n=4,181), the two judges produced different model evaluations: Gemini 3 Pro ranked first with GPT-5 mini, while Claude Sonnet 4.5 and DeepSeek v3.2 changed substantially.Differences were negligible for GPT-5 and Kimi K2 Thinking, even after accounting for 95% confidence intervals in the other comparisons.
- Inter-judge comparisons: The largest inter-judge differences occurred for Calculus and Algebra, and for all models they were greatest on Tier 4 problems.Tier 4 consists of the hardest problems, so judge conclusions became more important as question difficulty increased.
- Judge reliability: 64.8% of 176 disagreements on tagged questions were attributed to Omni-Judge, versus 6.8% to GPT-5 mini.The remaining disagreements were too ambiguous to judge or lacked a reference answer; sampled disagreements also found 14% dataset errors, including 11 wrong or incomplete references and 3 ill-posed statements.
3 Discussion
The discussion frames benchmark saturation as an interaction among dataset quality, model capability, and judge competence, with dataset defects and judge errors obscuring genuine performance differences. It recommends stronger, redundant, and more flexible evaluation designs while noting unresolved reference-answer limitations.
- Discussion: Benchmark saturation becomes an interaction of dataset, model, and judge once evaluation becomes the bottleneck.Saturation is therefore not primarily a property of model capability alone.
- Discussion: 14.6% of Omni-MATH problems were edited, including cases with missing images, missing options, and mismatches between requested outputs and exact-answer verification.These defects propagate through the evaluation pipeline rather than remaining merely cosmetic.
- Discussion: 96.4% of Omni-Judge disagreements on the cleaned exact-answer subset were judged wrong, preventing reliable differentiation between model abilities.Judge disagreement increased with problem difficulty across all evaluated models on the n=4,181 subset.
- Discussion: The authors recommend evaluator margins or redundancy, including partial credit, uncertainty, abstention, and multi-judge frameworks for contested items.These designs address increasingly important judge-quality problems in challenging, rapidly saturating benchmarks.
- Limitations: The study did not explicitly revise Omni-MATH solutions and reference answers because ground truth was unavailable for some proofs and Tier 4 items.Incomplete reference answers were reported when they became visible through annotating judge disagreements.
A Methods · A.1 The original Omni-MATH dataset
Omni-MATH is a 4,428-problem Olympiad-level mathematics benchmark assembled from contest pages, AoPS Wiki, and the AoPS forum, with each entry pairing an exact answer and written solution with metadata.
- A.1 The original Omni-MATH dataset: 4,428 Olympiad-level math problems comprise the original Omni-MATH benchmark.The problems were crawled from contest pages, AoPS Wiki, and the AoPS forum.
- A.1 The original Omni-MATH dataset: The dataset draws problems from contest pages, AoPS Wiki, and the AoPS forum.Forum-extracted problems were reformatted with GPT-4o.
- A.1 The original Omni-MATH dataset: GPT-4o reformatted the problems extracted from the AoPS forum.
- A.1 The original Omni-MATH dataset: Each dataset entry contains a problem, an exact answer, and a written-out solution.
- A.1 The original Omni-MATH dataset: Entries include metadata fields for domain, difficulty, and source.The paper refers to Figures A1 to A3 for these fields.
- A.1 The original Omni-MATH dataset: Figure 3 uses only the primary domains and joins the Calculus and Pre Calculus domains.
A.2 Omni-MATH-2
Omni-MATH-2 is a manually revised Omni-MATH dataset designed to reduce evaluation noise by ensuring problems are compilable, solvable, and verifiable, while tagging tasks that require proofs or estimation.
- Dataset revision: Omni-MATH-2 revises the original dataset after identifying errors including missing images, multiple-choice options, and LaTeX clutter.The cleaning process exposed both obvious and subtle problem errors propagating through evaluation.
- Cleaning process: The cleaning process targeted three requirements: LaTeX compilability, solvability from the stated information, and verifiability with final-answer-based judges.Problems were compiled using XeLaTeX in Texmaker, with code also compatible with pdfLaTeX.
- Solvability and verifiability tags: Tags identify problems requiring proofs or estimation because final-answer verifiers compare responses with exact reference answers.Such tasks are mismatched with a verifier that expects an exact final answer, despite the dataset containing written solutions.
A.3 Models
The evaluation uses five state-of-the-art language models to solve Omni-MATH-2-Filtered and Omni-MATH-2-Tagged. All models receive the same instruction to solve each problem and enclose the final answer in a boxed environment.
- Models: Claude Sonnet 4.5 uses 64,000 output tokens and a 25,000-token thinking budget through the Claude Batch API.The model identifier is claude-sonnet-4-5-20250929.
- Models: DeepSeek V3.2 uses 64,000 shared output tokens with an implicit thinking budget through the DeepSeek API.The model identifier is deepseek-reasoner.
- Models: Gemini 3 Pro uses 64,000 output tokens and a high thinking level through the Gemini Batch API.The model identifier is gemini-3-pro-preview, with no thinking budget specified.
- Models: GPT-5 uses medium reasoning effort with no token limit through the OpenAI Batch API.The model identifier is gpt-5-2025-08-07.
- Models: Kimi K2 Thinking uses 128,000 output tokens and a 256,000-token context window through the Moonshot API.The model identifier is kimi-k2-thinking, with the output limit specified via max_tokens.
- Prompt: Each model receives the same prompt: solve the problem and enclose the final answer in a boxed environment.The instruction uses a boxed LaTeX environment for the final answer.
A.4 Judges
The evaluation compares the open-source Omni-Judge with GPT-5 mini for correcting five language models’ responses on filtered Omni-MATH-2. Excluding dataset errors, Omni-Judge is wrong in 96.4% of subsampled disagreements, mainly because it fails to recognize equivalent final and reference answers.
- Evaluation setup: Five state-of-the-art language models’ responses are evaluated using Omni-Judge and OpenAI’s GPT-5 mini.Both judges are applied to responses on the filtered Omni-MATH-2 dataset.
- Omni-Judge: Omni-Judge is an efficient, low-cost open-source model trained to assess answer correctness from the problem and reference answer.The evaluation uses the authors’ few-shot prompt and requests through the kbsdjames.omni-judge API.
- GPT-5 mini: GPT-5 mini is a faster, more cost-efficient GPT-5 version designed for well-defined tasks and precise prompts.It receives an adapted evaluation prompt and returns a JSON schema containing the extracted final answer, reasoning, and correctness.
- Judge disagreements: 96.4% of subsampled judge disagreements are incorrectly decided by Omni-Judge after excluding dataset errors.The primary cause is Omni-Judge’s inability to assess equivalence between a model’s final answer and the reference answer.