PaperlayerView exact revision on arXiv

Source-linked AI summary

GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning

Ornith Team, Xiaoya Li, Guoyin Wang, Songqiao Su, Chris Shum, Jiwei Li

arXiv:2604.02721v3cs.AI

TL;DR

Competitive programming still separates leading AI systems from the best human competitors, especially under live contest conditions. GrandCode combines coordinated agentic modules and reinforcement learning with Agentic GRPO for delayed rewards and off-policy drift. It ranked first in all three recent live Codeforces rounds, while offline and test-time RL produced further gains, particularly on hard problems.

  • Problem

    Leading AI systems still fall short of top human competitors in competitive programming, with recent results not established under live contest conditions.

  • Method

    GrandCode combines a coordinated loop of hypothesis generation, solving, summarization, test-case generation, verification, post-training, and online test-time RL with Agentic GRPO.

  • Results

    GrandCode ranked first in all three recent live Codeforces rounds, outperforming all human contestants, including top legendary grandmasters.

  • Takeaways & Limitations

    The results suggest that agentic reinforcement learning combined with verification and online adaptation can push coding systems beyond top human performance in real-time environments.

  • Takeaways & Limitations

    Multiple-submission penalties are not reflected in the reported contest scores, although the authors believe their effect is small.

Abstract

from arXiv · show

Competitive programming remains one of the last few human strongholds in coding against AI. The best AI system to date still underperforms the best humans competitive programming: the most recent best result, Google's Gemini~3 Deep Think, attained 8th place even not being evaluated under live competition conditions. In this work, we introduce GrandCode, a multi-agent RL system designed for competitive programming. The capability of GrandCode is attributed to two key factors: (1) It orchestrates a variety of agentic modules (hypothesis proposal, solver, test generator, summarization, etc) and jointly improves them through post-training and online test-time RL; (2) We introduce Agentic GRPO specifically designed for multi-stage agent rollouts with delayed rewards and the severe off-policy drift that is prevalent in agentic RL. GrandCode is the first AI system that consistently beats all human participants in live contests of competitive programming: in the most recent three Codeforces live competitions, i.e., Round~1087 (Mar 21, 2026), Round~1088 (Mar 28, 2026), and Round~1089 (Mar 29, 2026), GrandCode placed first in all of them, beating all human participants, including legendary grandmasters. GrandCode shows that AI systems have reached a point where they surpass the strongest human programmers on the most competitive coding tasks.

1 Introduction

GrandCode addresses the gap between strong AI coding systems and top human competitive programmers with a multi-agent reinforcement learning system. It combines coordinated agent modules, Agentic GRPO, and post-training or online adaptation, and reports first place in three recent live Codeforces contests.

  • Current AI systems remain behind top human competitors, with Gemini 3 Deep Think reaching eighth place only on historical problems.
  • GrandCode coordinates hypothesis generation, solving, summarization, and adversarial test generation in an agentic reasoning, verification, and feedback loop.
  • Agentic GRPO combines immediate reward updates with delayed correction to improve credit assignment in long, multi-stage rollouts under severe off-policy conditions.
  • GrandCode placed first in Rounds 1087, 1088, and 1089, outperforming every human participant, including legendary grandmasters, under live contest conditions.
  • The system builds on Qwen 3.5 and incorporates models, modules, and implementation ideas from several prior AI systems.

2 Codeforces Competition Results

GrandCode was evaluated in live Codeforces contests using separate and joint scoring views. It achieved the best score and finished first across all three reported rounds, with multiple-submission penalties not reflected in the reported scores.

  • Participating Results: The three contests used contestant IDs averyjones1, yokeko, and Vortex1 in Rounds 1087, 1088, and 1089, respectively.
  • Scoring: S(separate) sums task scores at completion, whereas S(joint) scores the full submission set in one account and is lower because of waiting time.
  • Scoring Caveat: Reported scores do not reflect multiple-submission penalties, although the authors consider this effect small because tasks used at most four attempts and early acceptance is rewarded more strongly.
  • Participating Results: GrandCode achieved the best score and was first to finish all tasks in each of three Codeforces contests.
  • Participating Results: The corresponding S(separate) scores were 9269, 16511, and 11596 for Rounds 1087, 1088, and 1089, respectively.

3 System Overview

GrandCode uses a difficulty-aware, multi-agent system that combines a main solver with hypothesis generation, summarization, and adversarial test-case generation. Its workflow spans post-training and live solving with direct generation or test-time reinforcement learning.

  • Architecture: The system combines three learned policies with a test-case generation module.
  • Agent Modules: The main solver generates reasoning traces and code, while the hypothesis model proposes structural conjectures for verification and prompt injection.
  • Agent Modules: The summarization model compresses long reasoning traces, and test-case generation produces adversarial, attack, and stress tests for candidate programs.
  • Post-Training: Post-training includes continued training on competitive-programming data, supervised fine-tuning on reasoning-solution triples, and multi-component reinforcement learning.
  • Test-Time Solving: Live solving uses difficulty-based routing, direct generation for easy instances, and online test-time reinforcement learning with verification feedback for harder instances.

4 Agentic GRPO with Immediate Reward and Delayed Correction

Agentic GRPO targets slow, multi-stage competitive-programming rollouts whose rewards arrive after repeated code evaluation and whose asynchronous training creates severe off-policy drift. It updates from intermediate rewards immediately and applies final-reward corrections after the rollout completes.

  • Motivation: Multi-stage code rollouts can be extremely slow because generated sequences require repeated compilation and execution, with one evaluation often exceeding one minute.
  • Motivation: Asynchronous training creates severe off-policy drift because sequences may be generated under multiple policy versions during in-flight updates.
  • Agentic GRPO: Agentic GRPO handles a rollout sequence s1, r1, s2, r2, ..., sN, rN by updating each stage with its immediate reward and later correcting it using the final reward.
  • Immediate Reward and Delayed Correction: Immediate updates avoid waiting for the full rollout, while delayed corrections use rN−rt for earlier stages after the final reward becomes available.
  • Asynchronous Training: The method is designed to work with asynchronous training, combining timely credit assignment with overlapping sampling and training for higher throughput.

5 Test Case Generation

GrandCode generates adversarial and stress tests to expose logical bugs, boundary failures, and incorrect complexity assumptions before submission. Difference-driven generation, solution attack, and online feedback improve test effectiveness on real Codeforces problems.

  • Motivation: Adversarial tests target logical bugs, boundary failures, and incorrect complexity assumptions before solutions are submitted.Large inputs are difficult to verify when brute-force solvers time out, so test generation must address both adversariality and scalability.
  • Adversarial Test Case Generation: Difference-driven generation and solution attack are the two strategies used to produce adversarial test cases.Difference-driven tests seek inputs that distinguish candidate solutions, while solution attack compares candidates against a gold solution to identify exploitable bugs.
  • Adversarial Test Case Generation: Generated tests are produced by multiple language models, filtered or refined by a validator, and executed across sampled candidate solutions.The pipeline retains tests that expose differences, treating them as informative indicators of edge cases.
  • Evaluation: 50 real Codeforces problems were evaluated using the Codeforces judge as the final criterion.The evaluation checks whether solutions passing generated tests also pass hidden official tests.
  • Evaluation: 42 to 48 tests passed after difference-driven generation and solution attack, rising to all 50 after online submission feedback and additional test generation.For the remaining two failures, submission feedback enabled further online test generation.
  • Online Feedback: Time-limit feedback can support larger-input evaluation by using a faster candidate solver when it outperforms the brute-force baseline.This allows testing inputs too large for brute-force execution.

6 Prelude: Hypothesis Generation

GrandCode proposes intermediate structural hypotheses and verifies them on small instances before solution synthesis. A lightweight hypothesis model is trained with supervised fine-tuning and reinforcement learning, while validated hypotheses guide the main solver.

  • 6.1 Overview: Hypothesis generation is the first workflow stage, producing intermediate claims or compact characterizations that are checked on small instances.Examples include identifying dynamic-programming structure or confirming graph properties using brute-force computation.
  • 6.1 Overview: A proposed characterization is tested on random small inputs against exact brute-force answers, and mismatches trigger iterative hypothesis revision.Only hypotheses consistent with the verification instances are promoted to solution synthesis.
  • 6.1 Overview: Validated hypotheses are injected into the main solver’s prompt to provide verified structural insights for subsequent solution synthesis.The stage can also use symbolic tools when discovering a formula or invariant is the main difficulty.
  • 6.2 Training Hypothesis Generation Model: The hypothesis model is a lightweight Qwen-3.5-27B component rather than a large model allocated to every instance.It is trained with supervised fine-tuning on verified hypotheses and then optimized with GRPO using verification-based rewards.
  • 6.2 Training Hypothesis Generation Model: The current independent training setup is imperfect because a correct hypothesis does not necessarily imply that it helps generate the correct solution.The full system later jointly trains the hypothesis model with the overall solver during RL.
  • 6.3 Evaluation: Pass@1 and pass@5 improve consistently from the base Qwen-3.5-27B model to +SFT and then +SFT+RL on 200 evaluation problems.Pass@1 measures the first generated hypothesis succeeding, while pass@5 measures at least one success among five hypotheses.

7 Post Training with Continue Training and SFT

GrandCode’s post-training pipeline combines broad continued training, supervised fine-tuning, solution matching, and progressive summarization to improve competitive-programming performance while controlling long reasoning traces.

  • Evaluation benchmark: 70%–75% overall accept rate and 35%–40% of Level 5 problems solved characterize current frontier models on the 100-question benchmark.The benchmark equally distributes questions across five difficulty categories and reports accept rate, Level 5 solutions, and difficulty-weighted score.
  • Continued training and SFT: Continued training expands competitive-programming data, while SFT trains the main model to generate a thinking process before producing the solution.The continued-training corpus combines existing datasets, expanded problems, and generated thinking-process/solution tuples; SFT uses question, thinking, and solution triples.
  • SFT data construction: Solution matching retains reasoning traces whose generated implementations are equivalent, comparable, or more efficient than reference solutions.The filtering procedure becomes insufficient on harder tasks because generated solutions frequently fail to match the gold solution.
  • Training the summarization model: Progressive summarization compresses chunked reasoning into states that preserve information for later reasoning and answer generation.The summarizer is first trained locally with GRPO, then trained end to end using final answer likelihood as the terminal reward, and finally integrated into joint RL.
  • Evaluation: 64% to 71% accept rate and 52.2 to 61.0 weighted score follow continued training, while SFT reaches 73% accept rate and 62.5 weighted score.The summary-augmented configuration produces a minor performance degradation in this ablation.
  • Multimodal problem inputs: Direct multimodal processing outperforms converting contest images into text descriptions because visual structure can be difficult to describe faithfully.The system uses Qwen’s native multimodal input protocol when problems contain images or diagrams.

8 Multi-Component RL Orchestration

GrandCode orchestrates separate hypothesis, solver, summarization, and test-generation components, using verification and downstream solution quality to shape their interaction.

  • Multi-component design: Separate auxiliary models are pretrained before joint RL, reducing compute needs and making module-specific effects easier to isolate.The main solver uses a large MoE model, while hypothesis and summarization models are smaller and separately initialized.
  • Reward evaluation: Generated code receives reward through executability, correctness, and subsequent evaluation stages.Non-compilable or non-executable snippets receive score 0; correctness uses gold outputs during training and brute-force outputs at test time.
  • Hypothesis generation: Hypotheses continue only after passing verification tests, then condition the main solver’s reasoning and candidate-solution generation.The system also compares solution quality with and without hypotheses to encourage downstream usefulness, not merely local validity.
  • Hypothesis generation: Multi-round self-correction lets the hypothesis generator refine initially incorrect conjectures toward locally correct and globally useful hypotheses.The iterative process is explicitly designed to recover from initial mistakes.
  • Summarization orchestration: Summarization is triggered for long reasoning sequences and becomes more frequent as training shifts toward harder problems.Summary-conditioned rollouts use the same final reward as the main solver.
  • Systems implementation: Asynchronous serving places the large MoE solver on a dedicated GPU mesh and the smaller auxiliary policies on separate pools.This avoids fragmenting the main mesh and keeps rollout and training pipelines saturated.

9 RL Infrastructure

The infrastructure targets asynchronous off-policy drift and long-context computation through staleness-aware updates, pipelined context parallelism, dynamic parallelism, and fixed expert routing.

  • Asynchronous training: Severe off-policy drift arises because asynchronous rollouts span in-flight weight updates and multiple policy versions.Agentic GRPO is combined with pipeline-RL to improve efficiency and address this setting.
  • Asynchronous training: Staleness weighting downweights older tokens and drops them after a hard maximum lag threshold.Intermediate rewards can be used immediately, whereas delayed correction terms may be discarded when they arrive too late.
  • Pipelined context parallelism: Pipelined context parallelism preserves DeltaNet’s sequential recurrent-state semantics while overlapping multiple micro-batches across context-parallel ranks.A representative configuration with 4 CP ranks and 4 micro-batches achieves about 90% overall utilization.
  • Dynamic parallelism: Dynamic context parallelism adjusts CP size by batch difficulty because reasoning length varies substantially across difficulty levels.Fixed CP sizes can over-parallelize easy batches or under-parallelize hard batches.
  • Expert routing: Freezing the router during RL prevents routing drift and changing expert-load patterns from adding training instability.Only expert feed-forward parameters are updated.

10 Test-Time/Live-contest RL

Test-time RL adapts optimization to each contest problem by emphasizing the best rollout, conditioning on search history, and selectively applying lightweight updates when direct generation is insufficient.

  • Test-time objective: Post-training RL targets general ability across problems, whereas test-time RL optimizes separate parameters for the specific problem instance.The test-time objective emphasizes obtaining the single best solution rather than improving average rollout reward.
  • Best-of-N smoothing: Rank-based weighting smoothly shifts optimization from average reward toward best-of-N reward as λ increases.When λ = 0, weights are uniform; increasing λ concentrates weight on higher-ranked rollouts.
  • Search-history conditioning: Test-time RL updates LoRA parameters and conditions on historical candidate solutions, their scores, and an online summary of the search history.The summary records explored strategies and what has or has not worked.
  • Selective application: Direct generation is attempted first on easier problems, while test-time RL is reserved for cases where direct generation is insufficient.This policy reflects the expense of test-time RL and the value of early accepted submissions.
  • Results: 85% overall accept rate, 15/20 solved Level 5 problems, and 73.5 weighted score follow test-time RL, up from 81%, 13/20, and 72.3 after full RL training.The reported progression starts from 72% accept rate, 7/20 Level 5 problems, and 61.7 weighted score after post-training.

11 Conclusion

GrandCode combines a coordinated multi-agent loop with Agentic GRPO to address delayed rewards and off-policy drift in long agentic rollouts. It ranked first in all three recent live Codeforces contests and outperformed all human contestants.

  • GrandCode combines reasoning, hypothesis generation, summarization, test-case generation, and verification in a coordinated agentic loop, alongside Agentic GRPO for delayed rewards and off-policy drift.The system jointly uses multiple agentic stages and a reinforcement-learning method designed for long, multi-stage rollouts.
  • GrandCode ranked first in all three recent Codeforces live rounds and outperformed every human contestant, including top legendary grandmasters.The reported contests were Round 1087, Round 1088, and Round 1089.
  • Offline benchmarks, continued training, supervised fine-tuning, full RL, and test-time RL each contributed meaningful gains, especially on hard problems.

C Analysis on agentic-GRPO

The analysis contrasts standard GRPO's final-reward update with Agentic GRPO's immediate and delayed updates for multi-stage rollouts. Independent phase normalization gives immediate quality and future-stage contribution signals separate scales for more balanced credit assignment.

  • Standard GRPO updates every token in a full trajectory using only the final reward and a single normalized advantage.
  • Agentic GRPO updates each stage first with its immediate reward, then applies a delayed correction based on the final reward after the rollout completes.This design enables updates before the full sequence finishes while correcting them with later outcomes.
  • Independent phase normalization prevents either narrow-range immediate rewards or high-variance corrections from dominating the other signal.Per-phase normalization gives each stage equal voice in the gradient despite differing reward scales.
  • S(separate) sums task scores when each solution is completed, whereas S(joint) scores the full single-account submission set and is strictly lower because of waiting time.
Loading 2604.02721v3…