Source-linked AI summary
CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, Junyang Lin
TL;DR
Existing competition-level coding benchmarks lack private-test access, special-judge support, and aligned execution settings needed to evaluate increasingly capable reasoning models. CodeElo addresses these gaps with a CodeForces-based benchmark, direct platform judging, and human-comparable Elo ratings; o1-mini and QwQ-32B-Preview stand out, while most models struggle on easy problems.
Problem
Existing coding benchmarks do not adequately standardize competition-level evaluation because they lack private test cases, special judges, and aligned execution environments.
Method
CodeElo compiles CodeForces contest problems with detailed classifications, submits model solutions directly to the platform, and calculates lower-variance Elo ratings aligned with platform results.
Results
o1-mini and QwQ-32B-Preview stand out among 30 open-source and 3 proprietary LLMs, while most models struggle even with the easiest problems.
Takeaways & Limitations
CodeElo provides standardized human-comparable measurements and analysis across contest divisions, difficulty levels, algorithms, and programming languages.
Takeaways & Limitations
The benchmark allows only eight submissions per problem, which may make tested Elo ratings slightly lower than models’ actual ratings.
Abstract
from arXiv · showhide
With the increasing code reasoning capabilities of existing large language models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3, there is a growing need to develop more challenging and comprehensive benchmarks that effectively test their sophisticated competition-level coding abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to the unavailability of private test cases, lack of support for special judges, and misaligned execution environments. To bridge this gap, we introduce CodeElo, a standardized competition-level code generation benchmark that effectively addresses all these challenges for the first time. CodeElo benchmark is mainly based on the official CodeForces platform and tries to align with the platform as much as possible. We compile the recent six months of contest problems on CodeForces with detailed information such as contest divisions, problem difficulty ratings, and problem algorithm tags. We introduce a unique judging method in which problems are submitted directly to the platform and develop a reliable Elo rating calculation system that aligns with the platform and is comparable with human participants but has lower variance. By testing on our CodeElo, we provide the Elo ratings of 30 existing popular open-source and 3 proprietary LLMs for the first time. The results show that o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of 1578 and 1261, respectively, while other models struggle even with the easiest problems, placing in the lowest 25 percent among all human participants. Detailed analysis experiments are also conducted to provide insights into performance across algorithms and comparisons between using C++ and Python, which can suggest directions for future studies.
1 Introduction
CodeElo addresses gaps in competition-level coding benchmarks by standardizing CodeForces-based evaluation with platform judging and human-comparable Elo ratings. Its experiments show strong performance from o1-mini and QwQ-32B-Preview, while most models struggle on easy problems and vary by algorithm and language.
- Motivation: Existing benchmarks face false positives from weak or self-generated tests, lack special-judge support, and may use misaligned execution settings.CodeContests can still produce wrong answers or time-limit errors despite passing its tests, according to the cited report.
- Benchmark and evaluation: CodeElo standardizes competition-level coding evaluation using CodeForces problems categorized by contest division, difficulty rating, and algorithm tags.The benchmark provides a structured problem set for comprehensive evaluation and analysis.
- Benchmark and evaluation: Direct platform submission achieves zero false positives, supports special judges, and aligns the execution environment with CodeForces and human participants.The method uses an automatic bot to submit model solutions and obtain official platform judgments.
- Experimental findings: 1578 Elo places o1-mini near the 90th human percentile, while 1261 places QwQ-32B-Preview around the 60th percentile.Most other models struggle to pass even the easiest problems and fall within the lowest 10th percentile of human Elo ratings.
- Experimental findings: Models perform well on math and implementation problems but struggle with dynamic programming and tree problems.The analysis also finds that C++ generally outperforms Python for most models on competition code problems.
- Benchmark and evaluation: Human-comparable Elo ratings are provided for existing open-source and proprietary LLMs with lower variance while remaining aligned with the CodeForces platform.This rating system is designed to fairly compare model performance with human participants.
2 Related Work
Related competition-code benchmarks draw from public coding websites and generally evaluate solutions offline. CodeElo differs through direct platform judging, special-judge support, aligned execution, and standardized human-comparable Elo ratings.
- Competition-code benchmarks: APPS, CodeContests, xCodeEval, TACO, LiveCodeBench, and USACO are representative prior competition-code benchmarks.Their sources include Codewars, AtCoder, Kattis, CodeForces, LeetCode, and the USA Computing Olympiad.
- Competition-code benchmarks: Existing benchmarks generally source problems from open-access competition websites and conduct offline evaluations.LiveCodeBench and USACO update their benchmarks through newly released or re-scraped versions.
- Limitations of prior work: Offline benchmarks often rely on generated tests because online judges hide test cases, producing less robust validation and false-positive judgments.The paper notes that original competition tests are often adversarial and labor-intensive to create.
- Limitations of prior work: Prior benchmarks do not support special judges, which are needed for problems without unique correct outputs.About 30% of competition problems reportedly require special judging codes.
- Limitations of prior work: Individual-machine execution can make runtime comparisons sensitive to differing hardware performance, and prior benchmarks lack standardized human-comparable Elo ratings.CodeElo is presented as addressing these evaluation differences through platform-aligned judging and ratings.
- General code benchmarks: General code benchmarks such as HumanEval, MBPP, and BigCodeBench typically test small functions, whereas competition benchmarks emphasize complete algorithmic solutions.The paper distinguishes competition coding by its focus on problem analysis and algorithm design.
3 CODEELO Benchmark
CODEELO is a CodeForces-based benchmark that combines detailed problem classifications, direct platform judging, and a human-comparable Elo rating system for competition-level code generation.
- Problem Collection and Classification: CODEELO sources problems from CodeForces and organizes them by contest division, problem difficulty rating, and algorithm tags.The benchmark preserves problems’ original HTML structure by default while supporting flexible prompt restructuring.
- Problem Collection and Classification: Problem ratings represent difficulty: a competitor with rating x has a 50% first-attempt probability of passing a problem rated x.These ratings are derived from actual human contest performance and differ from participant ratings.
- Solution Submission and Judgment: CODEELO submits model-generated solutions directly to the official CodeForces platform, aligning execution environments and supporting special judges.A solution counts as accepted only when it passes all test cases, matching the platform’s criteria.
- Elo Rating Calculation System: The benchmark uses an Elo calculation similar to CodeForces to produce ratings comparable across humans, models, and contests.Treating contests independently preserves the official method’s expected rating while yielding significantly lower variance; binary search determines the rating value.
- Elo Rating Calculation System: Unlike pass@n alone, the Elo system accounts for multiple attempts, sampling diversity, and penalties for failed attempts before success.This provides a broader evaluation of competition performance than pass@1.
4 Evaluation on Existing LLMs
The evaluation tests LLMs on recent CodeForces contests under standardized C++ settings and compares performance across model sizes, contest divisions, problem difficulties, and sampling budgets.
- Experiment Setup: 54 contests containing 387 problems from May 4 to November 4, 2024 were selected after excluding Div. 1 contests.The exclusion followed observations that most models struggled with even the simplest Div. 1 problems.
- Experiment Setup: Each model receives up to eight attempts per problem, with no time penalties but penalties retained for failed attempts before a successful submission.These settings align with the official platform where applicable.
- Experiment Setup: 30 open-source and 4 proprietary models were evaluated using a common chain-of-thought prompt, with C++ selected because it generally elicited the best performance.The prompt requests an expert solution and complete C++ code.
- Main Results: o1-mini achieved the highest overall Elo rating at 1578, while QwQ-32B-Preview led open-source models at 1261.The ratings are presented with human-participant percentile comparisons in the main-results table.
- Main Results: Models generally perform best in divisions matching their skill level; most peak in Div. 4, whereas o1-mini performs best in Div. 3.Superior models consistently outperform inferior ones across contest divisions.
- Main Results: Easy problems are challenging for most models, medium problems distinguish several advanced models, and only o1-mini and QwQ-32B-Preview substantially pass hard problems.The difficulty bands are Easy [800, 1000), Medium [1000, 1300), and Hard [1300, 3500].
- Main Results: Most models improve pass rates as sampling increases from pass@1 to pass@8, although pass@n and Elo ratings do not always align.The metrics use different calculation methods.
5 Analysis Experiments
The analysis examines model performance across algorithm categories, programming languages, and rating variance. Models generally perform best on simpler categories and achieve higher ratings with C++ than Python, while broader contest coverage reduces rating variance.
- Performance across algorithms: Models achieve their highest pass rates on math, implementation, and sorting problems but often fail to solve dp, dfs and similar, and trees problems.
- Comparison between C++ and Python: Over 95% of unconstrained model outputs use Python, whereas 201 of 250 sampled human submissions use C++.
- Performance across algorithms: Table 4 reports pass@1 across 16 major algorithm categories, each represented by at least 30 tested problems.
- Comparison between C++ and Python: All tested models achieve higher Elo ratings with C++ than Python, consistent with C++ being more efficient under competition runtime constraints.
- Rating Variance: 300–500 standard deviations across individual contests decrease to around 50 when ratings are averaged over 54 contests.
6 Discussion
The discussion highlights CodeElo’s detailed problem metadata, platform-based evaluation, human-comparable ratings, and findings about model scale and language choice. It also identifies submission limits and reliance on CodeForces as key limitations.
- Contributions: CodeElo provides updateable CodeForces problems with contest divisions, difficulty ratings, and algorithm tags for comprehensive evaluation.
- Contributions: Submitting solutions directly to CodeForces addresses hidden tests, special judges, runtime constraints, and environment misalignment.
- Contributions: CodeElo supplies human-comparable Elo ratings and detailed analyses across contest divisions, problem difficulties, and algorithm tags.
- Findings: Models often perform better in C++ than Python, indicating that Python-only benchmarks may not measure their best competition-code performance.
- Limitations: Limiting models to eight submissions per problem may slightly lower measured Elo ratings, but the limit balances performance alignment against platform contamination.
- Limitations: The evaluation relies on CodeForces because hidden tests and special judges are difficult to access independently.
7 Conclusion
CodeElo combines CodeForces problems and direct platform judging with an Elo system for evaluating competition-level code generation. The benchmark finds strong results for o1-mini and QwQ-32B-Preview, alongside differences across algorithms and languages.
- CodeElo uses CodeForces submissions to provide zero false positives, special judge support, and an aligned execution environment.
- The benchmark evaluates 30 open-source and 3 proprietary LLMs, finding that o1-mini and QwQ-32B-Preview stand out significantly.
- Most models struggle even with the easiest problems and fall in the lowest 20 percent among human participants.
- Models show different performance across algorithm tags and achieve their best performance in C++ rather than Python.
8 Ethical Statement
The ethical statement limits CodeElo to controlled academic use because its judging process interacts with CodeForces. The authors defer releasing the full evaluation scaffold pending risk assessment and platform permission.
- CodeElo is intended for virtual participation only and must not be used for in-contest testing under CodeForces rules on AI use.
- The benchmark should be used sparingly to avoid affecting other users’ access to CodeForces.
- The complete submission and evaluation scaffold is withheld until risk assessment and permission from CodeForces are obtained.
A Model Cards
The paper lists the tested models and reports distinct inference settings for proprietary and open-source models.
- Table 5 lists and cites all tested models.
- Proprietary models use API calls with default parameters.
- Open-source models use temperature=0.7, top_p=0.8, top_k=20, and repetition_penalty=1.1.The maximum output length is 4,096 tokens for most models and 32,768 for QwQ-32B-Preview.
C Analysis of Our Elo Rating Calculation System
The benchmark’s Elo system is designed to align with CodeForces ratings while reducing variance through repeated contest evaluation, enabling comparison with human participants.
- The proposed Elo calculation is proven equivalent to the original CodeForces calculation under the same divisions.All LLMs attend the same contests, supporting fair comparison within this analysis.
- For each model, expected ratings are calculated after contests and then averaged under an IID assumption.The average rating retains expected value r while its variance decreases with the number of contests.
- As the number of contests approaches infinity, averaged ratings converge to expected value r.
- The platform update moves each rating halfway toward the contest’s expected rating.
- Increasing the number of contests preserves CodeForces’ expected results while significantly reducing rating variance.
- Standardized Elo ratings are comparable with human participants using publicly available CodeForces ratings.Human rating percentiles are reported in Table 6 from data collected in November 2024.
- CODEELO problems include structured fields such as title, limits, description, input/output formats, examples, and optional notes.
F Special Judge
Special judges are necessary for competition problems with multiple valid outputs, and CODEELO incorporates them to evaluate such problems correctly.
- A special judge verifies output validity when multiple outputs are correct for the same input.Reference-output comparison alone cannot evaluate these problems reliably.
- 30 out of 100 randomly selected competition problems required special judges.
- Previous competition-level benchmarks could not handle special-judge problems and therefore did not accurately assess the full capabilities of models.
- Figure 5 illustrates a problem where several strings derived from the input are acceptable outputs.The example demonstrates why a predetermined correct solution is insufficient for evaluation.
- Interactive problems are also represented by an example hosted on CodeForces.