Source-linked AI summary
Competitive Programming with Large Reasoning Models
OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, Wenda Zhou
TL;DR
The paper asks whether domain-specific, human-engineered inference strategies outperform learned reasoning for difficult coding tasks. It compares reinforcement-learned reasoning models with specialized o1-ioi across competitive programming and software-engineering benchmarks. The results show that scaling reinforcement learning and test-time compute consistently improves performance toward elite human levels, while later general-purpose reasoning can develop effective strategies autonomously.
Problem
The paper examines how domain-specific hand-engineered inference strategies compare with learned approaches that models generate and execute themselves.
Method
The study evaluates o1, specialized o1-ioi, and o3 across CodeForces, IOI, and coding benchmarks including Astra, using reinforcement learning and test-time compute.
Results
Across coding and reasoning tasks, increasing reinforcement-learning and test-time compute consistently boosts performance, with o3 reaching 2724 (99.8th percentile) on CodeForces and o1-ioi scoring 395.64 points at IOI under the 50-submission limit.
Takeaways & Limitations
Scaling general-purpose reinforcement learning offers a robust path toward state-of-the-art reasoning performance without relying on hand-crafted domain-specific inference heuristics.
Takeaways & Limitations
The reported IOI o3 results came from a later model version than the CodeForces results and used additional fresher training data.
Abstract
from arXiv · showhide
We show that reinforcement learning applied to large language models (LLMs) significantly boosts performance on complex coding and reasoning tasks. Additionally, we compare two general-purpose reasoning models - OpenAI o1 and an early checkpoint of o3 - with a domain-specific system, o1-ioi, which uses hand-engineered inference strategies designed for competing in the 2024 International Olympiad in Informatics (IOI). We competed live at IOI 2024 with o1-ioi and, using hand-crafted test-time strategies, placed in the 49th percentile. Under relaxed competition constraints, o1-ioi achieved a gold medal. However, when evaluating later models such as o3, we find that o3 achieves gold without hand-crafted domain-specific strategies or relaxed constraints. Our findings show that although specialized pipelines such as o1-ioi yield solid improvements, the scaled-up, general-purpose o3 model surpasses those results without relying on hand-crafted inference heuristics. Notably, o3 achieves a gold medal at the 2024 IOI and obtains a Codeforces rating on par with elite human competitors. Overall, these results indicate that scaling general-purpose reinforcement learning, rather than relying on domain-specific techniques, offers a robust path toward state-of-the-art AI in reasoning domains, such as competitive programming.
1 Introduction
Competitive programming offers an objectively gradable test of advanced reasoning and coding. The paper examines whether human-designed inference strategies or learned reasoning approaches better solve these challenging tasks.
- Competitive programming demands advanced computational thinking and problem-solving skills while providing objective grading for AI evaluation.
- Earlier program-synthesis models improved log-linearly with model size, while fine-tuning significantly boosted accuracy.Related systems included general code-generation models and AlphaCode’s large-scale generation with inference heuristics.
- The paper compares general-purpose o1, specialized o1-ioi, and early o3 checkpoints to study hand-engineered versus learned inference strategies.o1-ioi targets IOI 2024 and uses test-time strategies similar to AlphaCode.
- o3 develops complex test-time reasoning strategies through end-to-end reinforcement learning rather than relying on coding-specific strategies defined by humans.
- The report surveys reasoning in coding, progress in OpenAI’s reasoning models, and evaluation results across competitive programming and coding benchmarks.
2 OpenAI o1
The paper introduces o1 as a reinforcement-learned reasoning model that uses extended chain-of-thought and code execution to solve complex tasks. On CodeForces, o1 substantially outperforms gpt-4o and o1-preview under simulated contest conditions.
- OpenAI o1: o1 uses extended internal chain-of-thought reasoning to break down tasks, correct errors, and explore alternate solution paths.
- OpenAI o1: o1 executes generated code in a secure environment to check compilation, test cases, and correctness, then iteratively refines solutions.
- CodeForces Benchmark: CodeForces hosts internationally competitive live contests frequented by some of the world’s best competitive programmers.
- CodeForces Benchmark: The evaluation simulated CodeForces contests using complete test suites and appropriate time and memory constraints.
- CodeForces Benchmark: The study evaluated post-cutoff Division 1 contests and performed an embedding-based contamination check on test problems.
- CodeForces Benchmark: 1673 (89th percentile) was o1’s CodeForces rating, compared with 1258 (62nd percentile) for o1-preview and 808 (11th percentile) for gpt-4o.
3 OpenAI o1-ioi
The o1-ioi system combines additional coding-focused reinforcement learning with hand-engineered test-time selection for IOI and CodeForces. These specialized methods improved performance, but live IOI results remained constrained by the submission limit.
- Scaling Compute: Increasing reinforcement-learning compute and test-time inference compute consistently improved o1’s performance.
- System Design: o1-ioi extends o1 with coding-focused reinforcement learning and specialized inference strategies engineered for competitive programming.
- System Design: The coding-focused training emphasized challenging programming problems, C++ generation, runtime checks, and IOI submission formatting.
- System Design: o1-ioi improved solutions by writing and executing C++ programs during inference and iteratively refining them.
- Test-Time Strategy: The system divided IOI problems into subtasks, sampled 10,000 solutions per subtask, and used clustering and reranking to select submissions.
- Test-Time Strategy: Reranking used learned solution quality, generated-test errors, public-test failures, and cluster-level penalties tuned on prior IOI problems.
- Test-Time Strategy: The system submitted up to 50 solutions in round-robin order across subtasks, beginning with the hardest.
- CodeForces Benchmark: 2214 (98th percentile) was o1-ioi’s CodeForces rating with the complete test-time strategy, up from 1807 (93rd percentile) after additional coding RL alone.
4 OpenAI o3
The section evaluates whether reinforcement learning alone can produce competitive-programming reasoning strategies that match or exceed domain-specific pipelines. o3 surpasses o1-ioi on CodeForces and IOI 2024 while using simpler, non-hand-engineered strategies.
- CodeForces Benchmark: RL-trained o3 was evaluated on CodeForces using prompts containing problem descriptions, constraints, and available sample test cases.
- CodeForces Benchmark: 2724 (99.8th percentile) was o3’s CodeForces rating, up from o1-ioi’s 2214 (98th percentile).The increase reflects improved performance on a wider range of complex algorithmic problems.
- Reasoning Strategy: o3 independently generated sophisticated verification strategies, including brute-force implementations that cross-check optimized solutions.The model writes and executes code, then refines its approach based on verification.
- IOI 2024 Benchmark: Under official IOI rules, o3 was evaluated on the same six 2024 problems with a maximum of 50 submissions per problem.The comparison followed the official submission limit; later o3 results used fresher training data than the CodeForces evaluation.
- IOI 2024 Benchmark: 395.64 points placed o3 above the approximately 360-point gold threshold under the 50-submission limit.o1-ioi scored 213 points with 50 submissions and 362.14 points with 10K submissions.
- IOI 2024 Benchmark: o3 outperformed o1-ioi without IOI-specific hand-crafted strategies, replacing its clustering and selection pipelines with techniques emerging during training.The reported replacement included generating brute-force solutions to verify outputs.
5 Software Engineering Evaluations
The paper tests whether reasoning improvements extend from algorithmic programming to practical software-development tasks. Evaluations on HackerRank Astra and SWE-bench report gains for reasoning models over non-reasoning baselines.
- Evaluation Scope: The evaluation covers HackerRank Astra2 and SWE-bench Verified to assess reasoning on real-world coding tasks.Astra contains 65 project-oriented, multi-file challenges spanning React.js, Django, and Node.js.
- HackerRank Astra: HackerRank Astra omits public test cases, preventing reliance on hand-crafted test-time tactics.Its long-context, multi-file setting is intended to mirror actual development environments.
- HackerRank Astra: 63.92% pass@1 and 75.80% average score were achieved by o1 after further reinforcement-learning fine-tuning.These results were reported alongside o1-preview’s 9.98% pass@1 improvement and 6.03-point average-score gain over GPT-4o.
- SWE-bench Verified: SWE-bench Verified is a human-validated 500-task subset designed to correct grading, specification, and unit-test problems in SWE-bench.
- SWE-bench Verified: 22.8% improvement over o1 was reported for o3 on SWE-bench, following an 8.1% o1-preview improvement over GPT-4o and an 8.6% further improvement for o1.The evaluation used five candidate-patch attempts and averaged results over three trials.
6 Conclusion
The o-series models show that chain-of-thought reasoning improves coding performance across competitive programming and software engineering tasks. Increasing reinforcement learning and test-time compute boosts performance toward the strongest human competitors.
- Chain-of-thought reasoning improves performance across CodeForces, IOI, SWE-bench, and Astra coding tasks.
- Increasing reinforcement learning training compute and test-time compute consistently raises model performance.
- The resulting performance nearly matches the best humans in the world across the evaluated coding domains.
A Authorship, credit attribution, and acknowledgments
The report credits contributors across data preparation, model training, infrastructure, test-time strategy, evaluation, leadership, and o3 development. It also describes simulated contests as the basis for comparing models with human competitive programmers.
- Data preparation, IOI model training, sampling infrastructure, test-time strategy, and evaluation are credited to distinct contributor groups.
- Leadership and o3 model development receive separate attribution.
- Model-to-human comparisons are performed by simulating competitive programming contests and calculating scores and ratings.
B.1 Data
The evaluation uses post-cutoff Division 1 CodeForces contests, full grading environments, and a rating procedure based on simulated rankings. Several design choices adjust model–human comparability and validate the estimated ratings.
- The test set consists primarily of post-cutoff Division 1 contests, with embedding search used to check that problems were unseen during training.
- The evaluation runs complete test suites in an environment closely matching official CodeForces grading.
- Ten independent submissions are allowed per problem, marking a problem solved if any submission passes the full test set.
- Model scores use matched human scores to reduce the advantage of parallel thinking and simultaneous problem attempts.
- Ratings are estimated by maximizing the likelihood of observed rankings and human ratings, then averaging across contests so larger contests are not overweighted.
- The rating procedure recovers known human ratings and agrees with ratings predicted from average solve rates.
B.5 Percentile performance
The percentile analysis compares o3’s estimated rating and solve rate with active CodeForces competitors, while the accompanying material includes a detailed per-problem performance table and implementation excerpts. The results place o3 among the top 200 active participants, although the strongest humans remain substantially better.
- Percentile performance: o3 would rank among the top 200 active CodeForces participants by both estimated rating and solve rate.The comparison includes participants who entered at least 8 of 12 unseen test contests.
- Percentile performance: The strongest human competitors remain much stronger than o3, with solve rates above 85%.
- Percentile performance: The analysis reports a detailed per-problem breakdown of o3 performance from simulated contest participation.
- Percentile performance: The section also presents o1-ioi solutions from the 2024 International Olympiad in Informatics, including a single-submission perfect score on Nile.
C.2 Message
The message solution first identifies a known safe column, then uses it to recover sabotage information and message length before reading message bits across safe columns. The submission earned 79.64 out of 100, with full marks on subtask 1 and partial credit on subtask 2.
- 79.64 out of 100 points were scored, with full marks on subtask 1 and partial credit on subtask 2.
- The sender transmits four packets with distinct binary sequences to identify a known safe column among the 31 columns.Each sequence encodes one bit position of the safe-subset index.
- Correctness requires finding the known safe column; otherwise the method cannot proceed as intended.The receiver also expects to recover all message bits by the end of decoding.
- The next 31 rows store the sabotage subset in the known safe column, allowing the receiver to reconstruct the safe columns.Columns with false sabotage bits are added to safe_cols.
- The following 11 rows encode message length in the known safe column, while other safe columns carry message bits.Remaining message bits are then sent using all 16 safe columns.
- The receiver groups column sequences from the first four rows and selects a column whose sequence represents a value below 16.It also checks that the candidate column belongs to the safe subset.
C.6 Sphinx
The system scored 71.5 points on Sphinx across two submissions.
- 71.5 points were scored on Sphinx across two separate submissions.
C.6.1 Submission 1
Submission 1 achieved 50 out of 100, while its implementation constructs graph connectivity and recursively identifies merges before assigning component colors.
- 50 out of 100 was achieved, with 50% partial credit on all subtasks.
- The submission builds an undirected adjacency graph and supports connectivity queries with a UnionFind structure.Edges are stored in both adjacency lists, while path compression and rank-based union maintain components.
- Finally, the recursive merge search divides roots into left and right groups and assigns root and color identifiers.The procedure estimates merges in the left group before recursing on both halves.
C.6.2 Submission 2
Submission 2 achieved 43 points on subtasks 1, 2, and 3, using graph representations, group-testing queries, color recovery, and a randomized independent-set heuristic.
- The implementation represents the graph with adjacency lists and a Boolean adjacency matrix, while maintaining final colors and unassigned vertices.
- Group-testing queries count vertices of a specified color within a subset, and binary search isolates individual vertices with that color.The recovered vertices are assigned colors and removed from the remaining subset.
- A randomized heuristic repeats shuffled independent-set construction ten times and retains the largest successful attempt.The heuristic greedily accepts vertices when they remain compatible with the current attempt.
- If no color is found for the remaining subset, the assignment loop breaks and those vertices remain unresolved.