PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeClash: Benchmarking Goal-Oriented Software Engineering

John Yang, Kilian Lieret, Joyce Yang, Carlos E. Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, Diyi Yang

arXiv:2511.00839v2cs.SEcs.AI

TL;DR

Existing coding benchmarks focus on concrete, well-specified tasks, leaving open how models perform when software development requires pursuing high-level goals without explicit guidance. CodeClash evaluates models in multi-round tournaments where their codebases compete in arenas and logs inform subsequent edits. Models show diverse development styles but shared strategic limitations, and even the best model fails to win a round against an expert human-written bot.

  • Problem

    Coding benchmarks emphasize focused tasks with explicit instructions, leaving evaluation of models pursuing high-level software goals without explicit guidance an open challenge.

    Source: §1 · p. 1

  • Method

    CodeClash has language models iteratively edit codebases that compete head-to-head in multi-round arenas, with competition logs available to inform later edits.

    Source: §1 · pp. 1–2

  • Results

    Models exhibit diverse development styles but share limitations in interpreting feedback, validating changes, and maintaining organized codebases; Claude Sonnet 4.5 fails to win a single round against an expert human-written bot.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CodeClash provides a testbed for studying autonomous software development systems through tournaments in which codebases compete directly.

    Source: §7 · p. 11

    CodeClash’s arenas are relatively smaller and more self-contained than most real-world software systems.

    Source: §7 · p. 11

Abstract

from arXiv · show

Current benchmarks for coding evaluate language models (LMs) on concrete, well-specified tasks such as fixing specific bugs or writing targeted tests. However, human programmers do not spend all day incessantly addressing isolated tasks. Instead, real-world software development is grounded in the pursuit of high-level goals, like improving user retention or reducing costs. Evaluating whether LMs can also iteratively develop code to better accomplish open-ended objectives without any explicit guidance remains an open challenge. To address this, we introduce CodeClash, a benchmark where LMs compete in multi-round tournaments to build the best codebase for achieving a competitive objective. Each round proceeds in two phases: agents edit their code, then their codebases compete head-to-head in a code arena that determines winners based on objectives like score maximization, resource acquisition, or survival. Whether it's writing notes, scrutinizing documentation, analyzing competition logs, or creating test suites, models must decide for themselves how to improve their codebases both absolutely and against their opponents. We run 1680 tournaments (25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal that while models exhibit diverse development styles, they share fundamental limitations in strategic reasoning. Models also struggle with long-term codebase maintenance, as repositories become progressively messy and redundant. These limitations are stark: top models lose every round against expert human programmers. We open-source CodeClash to advance the study of autonomous, goal-oriented code development.

1 Introduction

CodeClash addresses the gap between coding benchmarks built around explicit, focused tasks and real-world development driven by high-level goals. It evaluates whether language models can iteratively improve codebases in competitive, multi-round arenas, where results reveal shared strategic weaknesses and a gap from expert humans.

Motivation

  • Real-world software development pursues high-level goals and requires decomposing them, prioritizing actions, and iterating based on feedback, unlike fine-grained tasks with explicit instructions.The introduction identifies evaluating models under these conditions as an unaddressed benchmarking challenge.Source: §1 · p. 1

Benchmark

  • In CodeClash, agents edit codebases before they compete head-to-head in arenas that determine winners by objectives such as score maximization, resource acquisition, or survival.Competition logs are copied back into codebases between rounds.Source: §1 · p. 2, Figure 1 · p. 2
  • Models must choose how to improve their codebases, including whether to discover arena information, analyze past-round logs, and create notes, tests, or analyses.Agents receive only a brief setting description, while mechanics, example bots, and recommended strategies are available in starter code.Source: §1 · p. 2

Results

  • Claude Sonnet 4.5, the best model, fails to win a single round against an expert human-written bot, while models share limitations in interpreting feedback, validating changes, and maintaining organized codebases.Models also develop increasingly dissimilar solutions round over round, even against the same opponent in the same arena.Source: §1 · p. 2

Evaluation

  • CodeClash evaluates 8 frontier LMs across 6 arenas and 1,680 tournaments, finding increasingly dissimilar solutions round over round—even against the same opponent in the same arena—and diverse styles alongside shared limitations.Source: §1 · p. 2

Release

  • The authors release CodeClash as an open-source toolkit with code, arena logs, and a leaderboard to support study of self-evolving language-model software-engineering agents.Source: §1 · p. 2

2 CodeClash

CodeClash formalizes software engineering as multi-round competition between codebases, with arena outcomes and logs feeding an agent’s next edits. Its arenas combine open-ended objectives, varied feedback, adversarial opponents, self-crafted memory, and self-directed improvement.

2.1 Formulation

  • Each round has an edit phase, where players modify their codebases, followed by a competition phase in which the arena executes them and determines a winner or tie.Players receive competition results and logs as their sole new information; they cannot see opponents’ codebases, and prior actions are not otherwise persistent.Source: §2.1 · p. 3

2.2 Technical Details

  • Players use mini-SWE-agent to interact with codebases through bash actions, while a lightweight arena interface needs only commands to run competitions and determine winners.The flexible interface allows existing competitive games and tasks to be incorporated into CodeClash.Source: §2.2 · p. 3

2.3 Features

  • CodeClash’s six arenas use open-ended objectives, requiring models to turn outcomes such as profit or survival into actionable subtasks and intermediate metrics rather than optimize for unit tests.Source: §2.3 · p. 3
  • Arena diversity spans codebase structure, interfaces, and feedback, while head-to-head play requires adapting to opponents whose behavior and countermeasures are not directly visible.Early wins do not ensure continued dominance.Source: §2.3 · p. 3
  • Because models have no persistent cross-round memory, they must record useful insights in their codebases and independently decide how to improve, for example by analyzing logs or testing candidates.The representation of retained knowledge is left to each model.Source: §2.3 · pp. 3–4

3 Experiments

The experiments evaluate eight language models using multi-round head-to-head tournaments across six arenas. Performance is summarized through tournament win rates and Elo scores fitted from those win rates, with bootstrap checks of ranking stability.

Models and agent

  • The evaluation includes eight language models from Anthropic, OpenAI, Google, Qwen, and x.ai, using mini-SWE-agent with a maximum of 30 edit-phase turns per round.Source: §3 · p. 4

Tournament configuration

  • The main leaderboard uses 10 tournaments per model pair per arena, with 15 rounds per tournament, for 25,200 total rounds.Tournament runtime averages 75 minutes per tournament.Source: §3 · p. 4

Performance metrics

  • A model’s win rate is its fraction of tournaments won, and Elo scores are fit to win rates using maximum likelihood rather than sequential updates.Parametric and non-parametric bootstrapping yield more than 98% pairwise order agreement.Source: §3 · p. 4

4 Results

The main leaderboard shows strong overall performers but arena-specific variation; additional evaluations expose persistent gaps against human experts and challenges in adapting competitively.

4 Results

  • Claude Sonnet 4.5 leads overall, followed by o3 and GPT-5; after a 100-Elo gap, Claude Sonnet 4 and GPT-5 mini follow, while Claude ranks fourth in Poker.Figure 2 reports Claude Sonnet 4.5’s highest average win rate at 69.9% across 240 tournaments.Source: §4 · p. 4, Table 1 · p. 5, Figure 2 · p. 5
  • Across rounds, some models excel early then plateau, while others improve steadily; Claude Sonnet 4.5 gains momentum and GPT-5 loses it.Source: §4 · p. 4, Figure 3 · p. 5

4.1 Ablations

  • CC:Ladder tests progression against ranked human solutions, but no model completes either ladder: GPT-5 mini reaches rank 57/58 in RobotRumble and 260/264 in Core War.Source: §4.1 · p. 5, Table 2 · p. 6
  • Most runs terminate by failing to win the final round rather than losing a majority, suggesting models are competitive but cannot consolidate gains within the round window.Source: §4.1 · p. 6
  • With opponent code visible, GPT-5 won 74.6% of rounds (+7.8%), yet accessed opponents’ code in only 12.8% of rounds; frequent inspection did not guarantee advantage.Source: §4.1 · p. 6
  • In six-player Core War, lead changes occurred in 48.4% of rounds versus 18.2% in two-player play, and winners captured 28.6% versus 78.0% of total points.Source: §4.1 · p. 6

5 Analysis

Across competitive dynamics, models show varied development styles but struggle to recover from losses and maintain orderly codebases. In strategic reasoning, they often fail to ground edits in logs, make unsupported failure diagnoses, and validate changes.

5.1 Competitive Dynamics

  • Editing styles vary substantially—from o3’s average of two files and 51 changed lines per round to GPT-5’s five to six files and over 400 lines for Qwen3 Coder or Claude Sonnet models—with no observed correlation to win rates.Claude Sonnet 4.5 typically uses 23 of 30 editing turns, while GPT-5 and o3 typically use 15.Source: §5.1 · pp. 6–7
  • After one loss, Claude Sonnet 4.5’s next-round win probability is below one third and o3’s is 26%; after five losses, rates fall below 15% and 10%, respectively.The overall round win rates are 71% for Claude Sonnet 4.5 and 65% for o3.Source: §5.1 · p. 7
  • Solutions generally become more dissimilar each round, while round-one similarity ranges from 0.63 for o3 to 0.41 for GPT-5.The analysis compares BattleSnake main.py solutions across 10 samples for each model, opponent, and round.Source: §5.1 · p. 7, Figure 5 · p. 7
  • Agent-created file counts grow almost linearly with rounds; Claude Sonnet 4.5 averages more than 30 files per tournament, GPT-5 21, and o3 fewer than 5.Claude’s files accumulate at the repository root, while GPT-5’s include output and temporary files in separate directories.Source: §5.1 · p. 8, Figure 6 · p. 8
  • Filename redundancy reaches 59% for Qwen3 Coder and 35% for Claude Sonnet models, and Claude Sonnet 4.5 and GPT-5 average 18 and 15 throwaway files per tournament.Most agent-created files are not referenced, reused, or modified in later rounds; o3 remains near the bottom in throwaway-file counts.Source: §5.1 · p. 8, Figure 7 · p. 8

5.2 Strategic Reasoning Limitations

  • Edits by models other than Claude Sonnet are ungrounded in more than 65% of rounds, and o3’s are ungrounded in almost 80%.Most models often stop at the first lines of logs or calculate only the last round’s win rate despite access to detailed records.Source: §5.2 · p. 9
  • Claude Sonnet 4.5 makes uncorroborated claims about why a game was lost in over 17% of rounds on average, rising to 46% in BattleSnake.Such claims often infer causes from opening log lines that omit the deciding moment.Source: §5.2 · p. 9, Figure 8 · p. 9
  • Only Claude Sonnet 4.5 validates changes in a majority of rounds (56%), while Gemini 2.5 Pro and o3 validate in about one round out of five.Across models, more than 85% of generated actions execute successfully, and the authors attribute performance differences to strategic reasoning and code quality rather than bash capabilities.Source: §5.2 · pp. 9–10

6 Related Works

CodeClash differs from software engineering benchmarks that specify objectives and often recommended approaches, and from optimization benchmarks that lack direct competition. It also connects interactive coding with game evaluation and offers a continuing relative-performance signal for self-improving agents.

Software engineering benchmarks

  • Unlike issue-resolution benchmarks such as SWE-bench, CodeClash leaves the notion of improved code and refinement strategy to the model, testing strategic thinking, opponent adaptation, and long-term planning.Prior benchmarks generally specify an objective and often a recommended approach.Source: §6 · p. 10

Performance optimization

  • Unlike code-optimization benchmarks, CodeClash has directly competing codebases, opponent adaptation, and objectives broader than runtime performance.Its arenas have flexible win conditions, while optimization tasks typically evaluate narrower objectives independently.Source: §6 · p. 10

Game playing

  • CodeClash studies interactive coding and gaming together, with codebases competing toward goals intended to represent competitive software development beyond games.Prior game-playing work has AI systems play games directly.Source: §6 · p. 10

Self improving agents

  • Unlike fixed unit-test feedback, CodeClash’s evolving opponents provide a relative learning signal that does not saturate once tests pass.The authors propose it as a potential training ground for self-improving SWE agents.Source: §6 · p. 10

7 Discussion

This section describes CodeClash’s competitive arenas and their core mechanics, and discusses evaluation choices, infrastructure design, and benchmark directions. It highlights persistent model weaknesses in strategic reasoning and codebase maintenance while considering how to make the benchmark more realistic and useful for training.

7 Discussion

  • CC:Ladder rankings broadly match the main leaderboard, scaffold changes shift scores by at most two ranks, and dominant failures are strategic rather than interface-related.The authors also propose more realistic multi-objective arenas, multimodal feedback, and training from editing traces or through self-play and reinforcement learning; models struggle to interpret logs and maintain organized codebases.Source: §7 · pp. 10–11

A Infrastructure

  • CodeClash uses mini-SWE-agent to provide execution feedback and codebase interaction through bash actions, aiming to support self-directed editing without favoring particular tools.With Claude 4 Opus, the scaffold scores 67.6% on SWE-bench Verified, which the authors take as evidence that bash-only interaction can work with few syntax failures.Source: A Infrastructure · pp. 17–18
  • The infrastructure validates submissions and excludes invalid codebases or awards wins and ties according to which players remain valid; it also shuffles player order to remove positional advantage.Each round runs 1,000 simulations and declares the winner by the most frequent outcome; randomizing order restored MIT Battlecode 2025’s measured win rate to 50%.Source: A Infrastructure · pp. 18–19

B Arenas

  • CodeClash’s implemented arenas span varied objectives, programming languages, and challenges, including opponent analysis, adaptation, and codebase management.The arena cards describe objectives, mechanics, log formats, and effective strategies.Source: Table 3 · p. 21

B.1 MIT Battlecode 2025 (Battlecode, 2025)

  • MIT Battlecode 2025 is a real-time strategy arena where teams coordinate specialized robots and resources to paint 70% of the map to win.Its mechanics require balancing resource use, unit roles, offense, and defense.Source: §B.1 · pp. 21–22

B.2 Battlesnake (Chung et al., 2020)

  • Battlesnake requires each player to control a grid-based snake that survives by avoiding collisions and outlasting opponents; the last surviving snake wins, or the longest wins at the turn limit.Effective strategies balance safety, space control, and movement, often using area estimation, pathfinding, look-ahead, and risk-aware heuristics.Source: §B.2 · p. 23
  • Battlesnake is played on an 11×11 grid, where snakes seek food and avoid collisions to survive as the last snake remaining.Food restores health and grows a snake; head-to-head collisions eliminate the shorter snake, or both snakes if they are the same length.Source: §B.2 · p. 24

B.3 Core War (Jones & Dewdney, 1984)

  • In Core War, Redcode programs take turns executing in shared memory, aiming to eliminate opponents and remain the last living warrior.Invalid instructions or overwritten processes can kill a warrior; games may end in a draw at the cycle limit.Source: §B.3 · pp. 25–26

B.4 Halite I (Truell & Spector, 2016)

  • Halite bots compete on a rectangular grid to claim territory and build strength through production, while battling opponents for control.If the turn limit is reached or multiple bots are eliminated simultaneously, players are ranked by territory, with total Strength as a rare tiebreaker.Source: §B.4 · pp. 27–28

B.5 Poker (Husky Hold’em Bench) (Kumar et al., 2025)

  • The poker arena uses no-limit Texas Hold’em, and the overall winner is the player with the largest chip count when the match ends.Players receive two private cards and five community cards are revealed across four betting stages; a hand can also be won by making opponents fold.Source: §B.5 · pp. 29–30

B.6 RoboCode (Hartness, 2004)

  • RoboCode tanks compete to outlast and outscore opponents using movement, detection, and weapons, with scores awarded for survival, damage, and placement.Firing uses energy, and bullets take time to travel, so tanks must account for energy costs and opponents’ anticipated positions.Source: §B.6 · p. 31
  • RoboCode awards cumulative multi-round wins based on survival, bullet and ram damage, and a last-survivor bonus.Bots can also improve their chances through predictive targeting and movement that avoids predictable patterns.Source: §B.6 · pp. 31–32

B.7 RobotRumble (Outkine & Oxer, 2020)

  • RobotRumble rewards effective unit positioning and coordinated focus fire to defeat enemies while avoiding friendly fire.Because attacks deal 1 damage, multiple robots can coordinate to eliminate a 5-HP enemy in one turn.Source: §B.7 · pp. 32–33
  • RobotRumble's winner is the last surviving team, with remaining health or damage used to decide matches that reach the limit.Matches run for 100 turns, and robots have 5 health.Source: §B.7 · pp. 32–33

C Evaluation

The evaluation uses mini-SWE-agent to let models investigate and edit codebases over multiple turns, then compares their submissions in configurable tournaments. Prompts provide limited arena guidance and encourage agents to use codebase resources and preserve knowledge across rounds.

C.1 mini-SWE-agent Configuration

  • Each editing round allows at most 30 interactive turns and $1 in model costs; cost limits were almost never reached, while some models frequently exhausted their turn limits.The cost cap was intended to discourage flooding the interaction buffer and encourage selective use of context and code.Source: §C.1 · p. 34
  • Agents receive a brief arena description and access to documentation, logs, and edit histories, but must discover useful information themselves and store knowledge in the codebase because memory does not persist across rounds.The arena description is intentionally sparse to test whether agents can identify and fill gaps using available resources.Source: §C.1 · pp. 35–36

C.2 Tournament Configuration

  • Tournament settings are configurable; the reported main-results setup used 9 models, 6 arenas, 10 tournaments, 2 players, and 15 rounds, totaling 32,400 rounds.The configuration can also set arena mechanics and simulation counts, and player configurations can select different mini-SWE-agent prompts.Source: §C.2 · pp. 37–38

C.3 Evaluation Metrics

The evaluation defines round and tournament outcomes, estimates model strength with Elo, and quantifies uncertainty in those rankings. Bootstrap analyses report high pairwise rank agreement, while sequential play makes round scores dependent.

C.3.1 Definitions

  • A tournament consists of 15 rounds; a model wins a round by being the only valid submission or by scoring above all opponents, and wins the tournament by taking more rounds.Tied round totals go to the model with the last win; tournaments with only drawn rounds are extremely rare.Source: §C.3.1 · pp. 38–39
  • Round scores are not independent: most arenas show heavily bimodal score distributions rather than the uniform distribution expected under independence.The sequential structure of tournaments is associated with this dependence.Source: §C.3.1 · p. 39, Figure 24 · p. 39, Figure 25 · p. 40
  • Win rate is the fraction of tournaments won, while Elo estimates absolute model strength from Bradley–Terry win probabilities.The Elo presentation uses a starting rating of 1200 and slope of 400; these choices affect readability, not model predictions.Source: §C.3.1 · pp. 39–40

C.3.2 Statistical uncertainties

  • The strength covariance matrix is derived from the log-likelihood Hessian, which is singular under shift invariance and therefore inverted within the constrained subspace where strengths sum to zero.The resulting strength variances are scaled to Elo uncertainties reported in Table 4.Source: §C.3.2 · p. 40, Table 4 · p. 42

C.3.3 Statistical validation and rank stability

  • Bootstrapping gives 98% pairwise order agreement for the Elo ranking, with bootstrap-derived uncertainties agreeing well with Hessian-based estimates.The analysis uses both non-parametric resampling of tournaments and parametric replicas from the fitted Bradley–Terry model; Table 5 reports rank-stability metrics.Source: §C.3.3 · p. 42, Table 5 · p. 42, Figure 27 · p. 43

D Extended Results

  • The extended-results section introduces further analyses and visualizations of model behavior and performance beyond those in Section 4.The supplied section passage characterizes the scope of these additional analyses but gives no specific result.Source: D Extended Results · p. 44

D.1 Interaction Trends

Models vary in how much they edit, how many turns they use, and how long their thought traces are. Most recover quickly from errant actions, which the analysis does not identify as a significant source of performance difficulty.

Interaction patterns

  • Editing styles range from o3 and Gemini 2.5 Pro’s two to three files and fewer than 100 lines per round to larger, long-tailed changes by Claude Sonnet 4.5 and GPT-5.The long tail often reflects test suites, competing bot versions, or markdown notes; competition-logic edits generally decline as rounds progress.Source: Figure 29 · p. 44, Figure 30 · p. 44, Figure 32 · p. 44
  • The Anthropic models and Qwen3-Coder typically use 22–27 of 30 turns, while Gemini 2.5 Pro and GPT-5 mini rarely exceed 15; turn use is fairly steady across rounds.The $1 cost cap never automatically terminated a trajectory, so these trends reflect models’ own stopping behavior.Source: §D.1 · p. 45, Figure 33 · p. 45, Figure 34 · p. 45
  • Most models produce 35–55-word thought responses, while Gemini 2.5 Pro averages about 95 words and o3 just under 19; o3’s apparent brevity is likely underestimated because intermediate tokens are hidden.The thought-length calculation excludes the action produced in the same response.Source: §D.1 · pp. 45–46, Figure 36 · p. 46, Figure 35 · p. 46
  • At least 90% of actions are well formed and execute successfully, and the next action succeeds more than 80% of the time after an errant action.By the third subsequent step, nearly zero occurrences show continued difficulty producing a well-formed action; the authors find errant actions are not a significant performance factor.Source: §D.1 · pp. 46–47, Figure 37 · p. 46, Figure 38 · p. 46

D.2 Additional Ablations

The additional analyses compare multi-player competition with head-to-head play and examine how models inspect visible opponents’ codebases. They also identify TrueSkill ratings as an additional analysis of six-player Core War tournaments.

D.2 Additional Ablations

  • Figure 42 reports TrueSkill ratings for models in 20 six-player Core War tournaments, representing each player’s skill as a Gaussian with mean and uncertainty.The ratings update after rounds based on match outcomes and final placement.Source: Figure 42 · p. 48
  • Six-player Core War has more frequent lead changes and a much lower winner share of total points than head-to-head play; multi-way competition also brings analytical complexity and was not comprehensively tested across other arenas.BattleSnake, Halite, Poker, and RoboCode support two-player tournaments, but the authors leave broader multi-player analysis for future work.Source: §D.2 · pp. 47–48
  • In three-model Halite tournaments, Claude Sonnet 4.5 checks opponents’ codebases nearly every round, while GPT-5 and Gemini 2.5 Pro check less often mid-tournament before increasing later.The main results assume opponents cannot view one another’s codebases; this ablation examines the effect of making them visible.Source: §D.2 · p. 48, Figure 41 · p. 48

D.3 Analyzing trajectories using LMs as a judge

  • The following section describes trajectory observations obtained using a language model as judge.The supplied passage states the analysis setup but no findings.Source: §D.3 · p. 48

D.3.1 Additional results

This section presents additional results on how agent behavior varies across arenas and tournament progression, alongside the rubric and procedures used to assess edits, validation, and hallucinations. It also defines how agent claims and cited evidence are labeled and clarifies how to interpret reported scores and Figure 8(b)’s hallucination bars.

D.3.1 Additional results

  • Up to 45% of BattleSnake rounds elicited hallucinations from Claude Sonnet 4.5, while RoboCode showed particularly low edit-validation rates across models.Groundedness, hallucinated loss causality, and validation results are shown across arenas in Figures 43 and 44.Source: §D.3.1 · pp. 48–49, Figure 43 · p. 49, Figure 44 · p. 50

D.3.2 Groundedness of edits and validation of edits

  • The groundedness and validation study uses structured trajectory review to classify final main-player-file edits and assess their motivation and testing.The schema records edit category, motivation by logs, insights, or old static messages, whether edits were reverted based on insights, and unittest or simulation testing. The rubric counts final edits, treats logs as motivating only when they reveal a failure mode linked to the edit, and distinguishes evidence-based insights from generic static advice.Source: §D.3.2 · pp. 49, 53, 54, 55

D.3.1 Additional results

  • As tournaments progress, models shift from feature-heavy edits toward smaller tweaks or fixes, with more rounds making no meaningful change to the main player file.Figure 45 categorizes rounds by the kind of change made to the main player file.Source: §D.3.1 · p. 49, Figure 45 · p. 51
  • Models’ turn activities differ, and read operations increase over time; the average number of actions and effort on testing, analysis, and test matches also vary across models.Figure 46 compares average actions in early (round ≤7) and late tournament periods.Source: §D.3.1 · p. 49, Figure 46 · p. 52

D.3.2 Groundedness of edits and validation of edits

  • A simulated game validates an edit only when the updated core implementation is played and the outcome is verified; against an opponent, the new version must win or achieve a good win rate.Failed runs, clearly worse performance, or unverified winners count as false; unit tests are not simulations, and no final main-file edits is a special case counted as true.Source: §D.3.2 · pp. 55–56
  • Unit-test validation requires tests of the final main-file edits’ new behavior, while a framework improvement must leave the next-round player with significantly more useful tools.A game win-rate run or unrelated tests do not qualify; saved additional tests, broader analysis, or reusable simulation-and-parsing scripts can qualify, whereas documentation or round-specific scripts do not.Source: §D.3.2 · p. 56

D.3.3 Hallucinations

  • For the hallucination study, GPT-5 with high reasoning judges agent trajectories for concrete factual claims that are unsupported by available information, not inferable by reasoning, obtainable in principle, and relevant to winning.The rubric excludes hypotheses, vague overconfidence, and recoverable mistakes; agents’ trajectories consist of thought, action, and execution output, and the judging model receives actions and outputs with thoughts stripped to avoid sycophancy.Source: §D.3.3 · pp. 57–58
  • Use loss_reason when an agent attributes its last-round loss to a specific cause, and possible_improvement for suggested code improvements not necessarily tied to that round.Source: §D.3.3 · p. 59
  • Scores are typically calculated as (wins + 0.5 * ties) / total_games * 100, though agents may also report win rates.Scores may be None, 0%, or 100% even when no games were played if a player submitted an invalid submission and was disqualified, so numbers should be judged in context.Source: §D.3.3 · p. 59
  • For each claim, annotators choose only its single best-fitting source category, or none if the agent cites no specific source.Categories include game logs, documentation, source code, test output, analysis output, and sparingly used misc.Source: §D.3.3 · p. 59

D.3.4 Action space analysis

The action-space analysis defines a structured taxonomy for labeling LM agent actions, with categories covering file access, edits, execution, and submission. It specifies distinctions within those action types and rules for assigning each action a category.

D.3.4 Action space analysis

  • Actions are categorized as search, navigation, reading, writing, execution, submission, or other operations.The analysis of Figure 46 groups navigation, search, and reading together as “read.”Source: §D.3.4 · pp. 60–62
  • Read operations are split by target—source, logs, docs, or other—and by whether the file was created during the trajectory.A script created during the trajectory is “new”; one created earlier is “old.”Source: §D.3.4 · p. 61
  • Write targets distinguish documentation, main code, backups, opponents, analysis scripts, tests, and other files.Unit tests are distinguished from analysis scripts by their predefined pass-or-fail outcome.Source: §D.3.4 · p. 61
  • Write actions are further labeled as creating a file, modifying an old file, or modifying a file created during the trajectory.A copied file counts as a creation; an overwritten pre-existing file remains an old-file modification.Source: §D.3.4 · p. 61
  • Execution categories cover game runs and setup, analysis, unit tests, or other execution targets.Each execution is also classified as in-memory, new, or old, according to when its script was created.Source: §D.3.4 · p. 62
  • When an action combines operation types, execution takes precedence over writing, which takes precedence over reading; every action must receive exactly one category.Responses also record the base action and, when applicable, success, notes, and target paths.Source: §D.3.4 · pp. 62–63

D.4 Additional Analyses

Models’ codebases diverge even when they face the same opponent, and their repositories tend to become more cluttered and redundant over time. These analyses show both variation in model development and recurring codebase-maintenance problems.

Codebase diversity

  • GPT-5’s code similarity fell from 0.409 in round 1 to 0.163 by round 15, while model solutions became increasingly dissimilar across repeated tournaments.Round 1 solutions were already divergent, and the opponent appeared to have little impact on how similarly a model started.Source: §D.4 · p. 63, Figure 47 · p. 64, Figure 48 · p. 64

Codebase organization

  • Five of eight models fell in the undesirable combination of low file reuse and high root-level clutter, while Claude Sonnet 4.5 had the highest root-level ratio.The analysis defines desirable practice as high file reuse with low root-level clutter.Source: §D.4 · p. 63, Figure 49 · p. 64
  • Models often create single-use analysis and testing files that are rarely reused, and the paper proposes CodeClash as a testbed for studying interventions that might encourage better practices.The authors did not explore prompting to mitigate this behavior.Source: §D.4 · p. 64

Filename redundancy

  • Redundantly named files increased at different rates across all models, and Claude Sonnet 4.5 created 13 “analyze_” files that mostly performed the same task.The inspected codebase also had similar redundancy among “check_” and “ROUND_” files.Source: §D.4 · p. 64, Figure 50 · p. 64

D.5 Agreement with Human Judgement

The authors assess the reliability of GPT-5’s trajectory annotations and describe how they construct and use human-solution ladders for comparison. The ladder protocol preserves a model’s evolving codebase as it attempts progressively stronger opponents.

D.5 Agreement with Human Judgement

  • Three authors independently annotated 100 randomly sampled trajectories, stratified by model and arena, on groundedness, hallucination, and validation.For annotation tractability, validation was labeled as a single binary outcome rather than split by technique.Source: §D.5 · p. 65
  • Agreement between human annotators and GPT-5 was substantial to almost perfect across all annotation dimensions, supporting the reliability of GPT-5 labels used in Section 5.2.Hallucination was harder to annotate than groundedness or validation, though agreement remained substantial; GPT-5 slightly more often flagged incidents humans did not, making reported hallucination rates conservative estimates.Source: §D.5 · p. 65

D.6.1 Constructing human solution ladders

  • The authors construct human-solution ladders by ranking publicly available RobotRumble and Core War solutions from all-pairs matchups using Elo ratings fitted to pairwise wins.The collected sets contain 58 RobotRumble solutions and 264 Core War solutions.Source: §D.6.1 · p. 67

D.6.2 Evaluation protocol

  • A model starts with the weakest human solution, plays seven rounds per opponent, and advances only if it wins more than ⌊7/3⌋ rounds and wins the final round; its codebase carries over between opponents.Each ladder is run five times, with the best score reported; the score is the rank of the highest opponent defeated.Source: §D.6.2 · p. 67

D.6.3 Results

CC:Ladder rankings broadly preserve the main leaderboard’s tier structure, although their relative ordering is not identical.

D.6.3 Results

  • CC:Ladder rankings broadly match the main leaderboard’s tier structure, but the relative ordering differs because the ladder measures progression against human opponents rather than head-to-head model competition.The full ranked human-solution lists and reproduction scripts are available in the project repository.Source: §D.6.3 · p. 67

D.6.4 Scaffold ablation: mini-SWE-agent vs. SWE-agent

Replacing mini-SWE-agent with SWE-agent produces similar CC:Ladder results across six model–arena combinations. The authors also report limited use of SWE-agent’s additional navigation tools.

D.6.4 Scaffold ablation: mini-SWE-agent vs. SWE-agent

  • Across six model–arena combinations, SWE-agent scores were within two positions of mini-SWE-agent, matching in four cases and performing marginally better in two.Scores are ranks of the highest human opponent defeated, with the best run reported.Source: Table 7 · p. 68, §D.6.4 · p. 68
  • SWE-agent’s editor sometimes conflicted with models’ preferred workflows, while models rarely used its tree-view and AST-search tools.The authors suggest that the small arena codebases may have been navigable through bash alone.Source: §D.6.4 · pp. 67–68
Loading 2511.00839v2…