Source-linked AI summary
LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, Micah Goldblum
TL;DR
Test-set contamination and biases in human or LLM-based evaluation make conventional LLM benchmarks unreliable. LiveBench responds with frequently updated, objectively scored questions spanning diverse challenging tasks, and its top models remain below 70% accuracy while rankings stay highly stable across updates.
Problem
Public benchmark questions can enter model training data and inflate performance, while human or LLM judging introduces mistakes and biases into evaluation.
Method
LiveBench uses frequently updated questions from recent information sources, objective ground-truth scoring without LLM judges, and six diverse task categories.
Results
o1-preview-2024-09-12 performs best overall, 6% better than all other models, while no current model exceeds 70% accuracy.
Takeaways & Limitations
LiveBench’s monthly updates make the benchmark harder over time while preserving highly consistent model rankings across successive updates.
Takeaways & Limitations
Ground-truth scoring cannot cover use cases such as writing a travel guide where defining a ground truth is difficult, and task biases remain across LLM families.
Abstract
from arXiv · showhide
Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.
1 INTRODUCTION
LiveBench addresses contamination and evaluation-bias problems by combining frequently updated, objectively scored questions with diverse challenging tasks. Its questions draw on recent information sources and harder versions of prior benchmark tasks.
- Traditional benchmarks can become unreliable when their publicly available questions enter LLM training data, artificially inflating performance through test set contamination.
- LLM and human prompting or judging can introduce mistakes and biases, including preferences for verbose or self-generated answers and sensitivity to formatting or tone.
- LiveBench minimizes contamination and evaluation pitfalls through recent questions, automatic scoring against objective ground truth, and six diverse categories.
- No current model exceeds 70% accuracy, while questions are added or updated monthly alongside new tasks and harder task versions.
- LiveBench includes 18 tasks across math, coding, reasoning, language, instruction following, and data analysis, combining recent-information questions with harder versions of existing benchmarks.
- The authors evaluate proprietary and open-source models ranging from 0.5B to 8x22B parameters and release the questions, code, and model answers.
2 LIVEBENCH DESCRIPTION
LiveBench organizes six categories around recent information sources and more challenging benchmark variants. Its tasks test mathematical, coding, reasoning, data-analysis, instruction-following, and language-comprehension abilities with objectively checkable outputs.
- LiveBench has six categories with two to four tasks each, and every task uses recent information sources or a harder, more diverse existing benchmark variant.
- Math: Its math tasks modify recent competition problems, mask equations from recent olympiad solutions, and generate harder AMPS-style questions from challenging primitives.
- Coding: Its coding tasks assess writing correct solutions from competition statements and completing partially correct code using recent LiveCodeBench problems and GitHub solutions.
- Reasoning: Its reasoning tasks include harder Boolean word problems with deductive components and red herrings, handwritten spatial deductions, and procedurally generated Zebra puzzles.
- Data Analysis: Its data-analysis tasks cover column-type annotation, table reformatting, and table-join mapping using recent Kaggle or Socrata datasets.
- Instruction Following: Instruction-following tasks ask models to paraphrase, simplify, generate stories, or summarize Guardian articles while satisfying multiple verifiable instructions.
- Language Comprehension: Language-comprehension tasks test word-puzzle completion, typo correction without unrelated stylistic changes, and reordering scrambled plots of unknown movies.
3 EXPERIMENTS
Experiments evaluate 40 models across LiveBench, compare performance patterns with other benchmarks, and analyze category correlations, model-specific strengths, judging difficulty, and monthly-update stability.
- Experimental Setup: 40 LLMs are evaluated across LiveBench using single-turn, temperature-0 inference with model-specific templates and automated parsing checks.The evaluated set combines proprietary and open-source models, including small and large systems.
- Discussion of Results: o1-preview-2024-09-12 performs best overall, 6% better than all other models, while leading data analysis, language, and math.Claude-3-5-sonnet leads coding, and o1-mini is strongest in reasoning.
- Discussion of Results: Llama-3.1-405b-instruct and qwen2.5-72b-instruct virtually tie as the best open-source models and outperform gpt-4-turbo.Phi-3.5-moe-instruct, with 6.6B active parameters, outperforms gpt-3.5 and matches mixtral-8x22b.
- Correlation Analyses: Math, coding, and reasoning correlate with one another, whereas instruction following correlates relatively weakly with other categories.Math comp has the highest correlation with average LiveBench performance among tasks.
- Correlation Analyses: Residuals from best-fit lines identify models disproportionately strong in particular categories, including o1 and phi in reasoning and Llama, gemini, and command-r in instruction following.The analysis also reports each model’s relative best and worst tasks from highest and lowest residuals.
- Comparison to Other LLM Benchmarks: LiveBench scores correlate 0.91 with ChatBot Arena and 0.88 with Arena-Hard, while some models perform noticeably better on one benchmark than another.Both GPT-4 models are substantially better on Arena-Hard than LiveBench.
- Comparison to Other LLM Benchmarks: Experiments provide evidence that models struggling with hard math or reasoning questions also struggle to judge answer correctness, although the results are not definitive.This preliminary finding concerns zebra puzzles and AMC/AIME questions.
- Monthly Updates: Rank correlations exceed 0.997 across consecutive monthly updates, while mean and median scores fall about 1.2% from the original to the latest questions.The results indicate stable rankings alongside increasing benchmark difficulty over updates.
4 RELATED WORK
Related work contrasts static benchmarks and LLM- or human-judged evaluations with LiveBench’s regularly refreshed, objectively scored, and broad task design.
- Static Benchmarks: Static benchmark suites such as Big-Bench Hard and MMLU-Pro are useful for tracking performance but remain prone to test set contamination.Their datasets are published and static, making exposure during training possible.
- LLM-as-a-Judge: LLM-as-a-judge benchmarks efficiently evaluate open-ended outputs but introduce self-preference, judge-dependent variation, and grading errors.The related-work discussion specifically contrasts GPT-4 and Claude judge behavior.
- Human Evaluation: Human judging can be labor-intensive for complex tasks, while annotation choices and inter-rater variability can produce high or unpredictable error rates.The cited examples include complex math, coding, and long-context reasoning.
- Other Benchmarks: LiveCodeBench regularly releases ground-truth-judged questions but is limited to coding, whereas LiveBench spans six categories with objective correctness-based scoring.Other comparisons include Omni-MATH, SEAL, and broader benchmark efforts with different task or scoring scopes.
5 CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
LiveBench is designed to reduce contamination and judging-related problems through frequently updated, objectively scored, diverse tasks. The authors identify remaining gaps in language coverage, ground-truth applicability, and fairness across model families.
- Conclusions: LiveBench uses frequently updated questions, objective ground-truth scoring, and challenging tasks spanning six categories.Its questions draw on recent math competitions, arXiv papers, and datasets, alongside harder versions of earlier benchmarks.
- Conclusions: Questions are added and updated monthly, with new and harder tasks intended to distinguish model capabilities as models improve.
- Limitations and Future Work: LiveBench would benefit from non-English language tasks and cannot use ground-truth scoring for some open-ended applications.The paper gives writing a travel guide to Hawaii as an example where defining ground truth is difficult.
- Limitations and Future Work: The benchmark may retain biases because some LLM families favor particular prompt types.The authors plan to update prompts as new prompt strategies are developed and continue updating the leaderboard as new models are released.
6 REPRODUCIBILITY STATEMENT
The project is broadly reproducible because its benchmark materials, evaluation code, model outputs, and leaderboard are publicly available. Newly added monthly questions are partly withheld to preserve private test data.
- Reproducibility: The authors open-source the leaderboard, questions, model-running code, model outputs for 40 models, and scoring code.They describe every part of the project as publicly available through LiveBench.
- Reproducibility: New monthly question sets are withheld as the benchmark becomes more popular so that some questions remain private.
7 ETHICS STATEMENT
The paper reports no inherently negative broader societal impacts and hopes LiveBench will support more rigorous evaluation and future research. It also presents the framework as a possible way to mitigate contamination effects.
- Ethics Statement: The authors report no inherently negative broader societal impacts from the work.
- Ethics Statement: The authors hope the benchmark will accelerate research and enable more comprehensive and rigorous evaluations of existing and future models.
- Ethics Statement: They also hope that frequently updated questions from new information sources will be adopted more broadly to mitigate contamination effects.
A ADDITIONAL DETAILS ABOUT LIVEBENCH EXPERIMENTS
The appendix documents additional experiment materials, result displays, evaluated models, verifiable instructions, and correlation analyses. It defines comparisons against overall LiveBench performance at both category and task levels.
- Additional Details About LiveBench Experiments: The appendix provides further details about the LiveBench benchmark and experiments.
- Results and Materials: The appendix includes additional LiveBench comparisons in Figure 6, a full bar plot in Figure 5, and the full results table in Table 2.
- Results and Materials: Table 2 reports each of 40 models’ results for every main category and its overall LiveBench score.
- Results and Materials: Table 3 lists 25 instructions from prior work and the 16 real-world, automatically verifiable instructions used in LiveBench.
- Results and Materials: Table 4 lists evaluated models across LiveBench versions and their citations.
- Correlation Analyses: Tables 5 and 6 report Pearson correlations and standard errors between category- or task-level scores and overall LiveBench performance across 40 models.Table 7 further identifies each model’s relatively strongest and weakest task using residuals from a best-fit line.
A.2 DETAILS FROM ABLATION STUDIES
The ablation study examines whether LLMs can reliably judge difficult math and reasoning answers. Its results indicate substantial judging errors, while alternative prompts may affect the outcome.
- Ablation question: LLM judges are tested on whether they can evaluate challenging math and reasoning answers correctly.The study asks whether difficulty solving a problem also implies difficulty determining whether an answer is correct.
- Results: LLM-as-a-judge scoring produces error rates far above a reasonable level across the evaluated tasks.The reported tasks include challenging math and Zebra puzzle reasoning questions.
- Comparison: The comparison reports model performance under both ground-truth and LLM-judging evaluation.Table 9 presents results for math and reasoning tasks using both scoring approaches.
- Caveat: A more detailed, task-specific judging prompt could change the observed results.The authors identify this as a possible alternative experimental setup.
A.3.1 MATH CATEGORY
LiveBench’s math category combines recent competition problems, proof-oriented olympiad tasks, harder AMPS questions, and synthetic problems. It uses answer formats and ordering tasks that support automated evaluation while testing varied mathematical abilities.
- Math category: LiveBench’s math category includes recent competition questions, olympiad problems, and a harder version of AMPS.The competition sources include AMC12, SMC, and AIME; olympiad questions draw from USAMO and IMO.
- Examples: Math examples include formatted competition answers and an indefinite-integral problem from AMPS Hard.AIME-style answers require exactly three digits, while AMPS Hard requests a boxed final answer.
- Proof-based questions: Olympiad tasks mask proof equations, scramble them, and ask models to restore their correct order.The task evaluates predicted ordering against ground-truth ordering, with hardness variants masking 10%, 50%, or 80% of equations.
- Synthetic questions: Synthetic math questions are generated from random problem primitives such as differentiation, integration, completing the square, and polynomial factoring.The generation process creates problems with varying solution techniques and difficulty.
A.3.3 REASONING CATEGORY
LiveBench’s reasoning category combines harder logical-language tasks, Zebra puzzles, and spatial reasoning questions. The benchmark increases difficulty through red herrings, additional deductions, randomized constraints, and geometric questions.
- Reasoning tasks: The reasoning category includes a harder Big-Bench Hard task, Zebra puzzles, and spatial reasoning.These tasks test logical deduction, constraint following, and deductions about geometric shapes.
- Web of Lies v2: Web of Lies v2 adds red herrings, asks for three people’s truth values, and includes an additional deductive component.Red herrings are designed not to affect the logic while potentially misleading models.
- Zebra puzzles: Zebra puzzles require models to deduce requested attributes from constraints over people and categories.The generation repository randomizes people, attributes, values, and constraint levels, while a solver ensures a unique solution.
- Zebra puzzles: LiveBench generates Zebra puzzles with three or four people, three or four attributes, and constraint levels uniformly sampled from 10 to 20.The authors report that larger puzzles were exceedingly difficult for top-performing LLMs in preliminary testing.
- Spatial reasoning: Spatial reasoning uses 50 handwritten questions about intersections and orientations of common two- and three-dimensional shapes.Examples ask about shapes formed by tangent points and pieces created by cuts through a polygon.
A.3.4 DATA ANALYSIS
LiveBench’s data-analysis tasks use recent Kaggle and Socrata datasets for column annotation, table joins, and table reformatting. The broader benchmark also evaluates instruction following and language comprehension with verifiable transformations and structured scoring.
- Data analysis: The data-analysis category contains column type annotation, join-column prediction, and table reformatting tasks using recent Kaggle or Socrata datasets.These tasks require models to infer column types, identify joinable columns, or preserve table contents while changing schemas.
- Data analysis: Table reformatting converts tables between plaintext-readable schemas while keeping their contents unchanged.Pandas is used for conversions among formats including JSON, JSONL, Markdown, CSV, and TSV.
- Instruction following: Instruction-following tasks use recent Guardian articles for paraphrasing, summarizing, simplifying, and story generation under composed verifiable instructions.The benchmark samples between two and five instructions for these tasks.
- Instruction following: Instruction-following performance is scored by both prompt-level accuracy and instruction-level accuracy.Prompt-level scoring requires every instruction to be followed, while instruction-level scoring measures the fraction handled correctly.
- Language comprehension: Language comprehension tasks cover word-group puzzles, typo correction, and scrambled movie-plot ordering.Synopsis ordering uses fuzzy matching for transcription and a normalized Levenshtein-distance score for sentence order.
- Language comprehension: Synopsis unscrambling can remain useful for comparing models even if the available sentences do not determine a unique causal ordering.Models may use plot plausibility and subtle cues, and the task can function as a relative metric despite a score ceiling below 100%.
A.7 CONTAMINATION IN LIVEBENCH
LiveBench distinguishes test set contamination from task contamination and adopts safeguards aimed at limiting similarity-based contamination. The benchmark nonetheless retains a limited contamination risk because some questions are older or only lightly modified.
- Task contamination concerns similarity between training and test distributions, including whether models generalize across superficially altered questions.
- LiveBench limits task contamination by withholding question-generation code, modifying it for updates, and delaying public release of new questions for one month.
- A limited fraction of LiveBench is likely contaminated because some coding questions date from November 2023 and AMC questions received only low modification.
- The section points readers to the project repository for full documentation, including the datasheet and benchmark materials.
B.5 BENCHMARK STATISTICS
LiveBench reports task-level question and token statistics, alongside approximate API-model evaluation costs. Its current problem set contains 1,000 questions with average input and output lengths reported for each question.
- 1,000 questions comprise the current LiveBench problem set.The reported mean per-question lengths are 1,612 input tokens and 395 output tokens.
- Table 15 reports each task’s question count and gpt-4-turbo-2024-04-09 output-token mean and standard deviation.
- Table 16 gives approximate costs for running GPT and Claude models on LiveBench as of October 1, 2024.The calculations use the gpt-4-turbo tokenizer, making prices for other models approximate.