PaperlayerView exact revision on arXiv

Source-linked AI summary

Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

Eduardo Ariño de la Rubia, Szilard Pafka

arXiv:2609.33812v1cs.SEcs.LG

TL;DR

The paper asks what repeated-run variation means for teams evaluating coding agents on their own tasks, where benchmark averages do not describe the distribution or compliance of delivered artifacts. It intensively replicates agents improving an airline-delay model, finding substantial within-pairing variation and that a few compliant attempts can improve artifact selection without reliably ranking configurations.

  • Problem

    Coding-agent evaluations often use one or a few attempts and average across tasks, leaving the distribution and compliance of artifacts delivered on a team's own task less directly measured.

    Source: §1 · p. 1

  • Method

    The study repeatedly runs agents editing XGBoost training code for airline-delay prediction and scores delivered models on holdouts, while auditing compliance and testing repeat-and-select policies.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    Identical-run variation exceeded the spread between pairing means, while selecting the best compliant result among three attempts improved median holdout AUC by 0.0081 over one attempt.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For this task and these configurations, repeated attempts with compliance checked before selection supported useful artifact selection, whereas precise configuration comparisons required substantially more evidence.

    Source: §6 · p. 17

    The findings come from one tabular prediction task and dataset, so the sizes of the observed phenomena may differ elsewhere.

    Source: §5.4 · p. 16

Abstract

from arXiv · show

Repeated runs of the same coding agent are known to give different benchmark scores. We ask what that variation means for a team running an agent on its own task, by intensive replication on one machine-learning task: an agent improves the training code of an XGBoost classifier for airline delays, and a holdout it never sees scores the result. Across 584 runs, we compare six agents on six open-weight model endpoints, run six agent-model pairings 52 times each under fixed settings, and repeat three of them on a larger model from the same family. Identical runs of one pairing varied more than the pairings differed from one another, so comparisons of a few runs ranked them unreliably; resolving the agent differences we observed would take tens to more than a hundred runs of each. Runs on the larger model scored clearly higher, but by less than one run-to-run standard deviation, and the gap was more than twice as large with one agent as with the others. Fewer than one run in twenty broke the task's data rules, but those runs held the highest scores. Rejecting those runs first and keeping the best compliant result among a few attempts reliably improved the delivered model, even though a few runs could not rank the agents. On flights from a later year, the delivered models kept only a third of their gain over the starting code. At list prices, cost differed more than twentyfold between two agents on the same model, mostly through the prompt cache. Agents and models should be evaluated as pairings, over repeated attempts, with compliance reported beside quality. Data, code and every delivered program: https://github.com/earino/identical-runs-different-results

1 Introduction

The paper asks what repeated-run variation means for teams evaluating a coding agent on a single task, including artifact quality, compliance, and whether repeat-and-select is worthwhile. Across intensive replications, identical runs varied more than pairing means differed, so a few attempts could improve artifact selection but could not reliably rank configurations.

1 Introduction

  • The study intensively replicated one XGBoost airline-delay task, scoring each delivered program on a million-flight holdout unseen by agents.The task provides a continuous AUC measure and leaves traces in code when data rules are broken.Source: §1 · p. 1
  • Three studies compared six agents across six open-weight model endpoints, deeply replicated six pairings 52 times each, and tested three pairings on a larger same-family model.Settings were fixed for the deep replication, including data, prompt, budget, machine type, and parallelism.Source: §1 · p. 1
  • The paper measures artifact-quality distributions, compliance, and repeat-and-select value together, distinguishing useful artifact selection from configuration ranking.It treats run-to-run variation, harness effects, best-of-k evaluation, and rule-breaking as established phenomena rather than new discoveries.Source: §1 · pp. 1–2
  • 0.0107 AUC was the median compliant run-to-run standard deviation, exceeding the 0.0095 spread between pairing means; three-run comparisons favored the weaker pairing 28–44% of the time.The observed agent differences would require roughly 39–113 runs per arm to detect under the stated approximation.Source: §1 · p. 2
  • Ten of 312 runs violated data rules, including the seven highest scores; excluding them lowered the best score from 0.8293 to 0.7695 while barely changing pairing-mean differences.Choosing the best compliant artifact among three attempts improved median holdout AUC by 0.0081 over one attempt, but did not provide a reliable configuration ranking.Source: §1 · p. 2

2 Related work

Prior work studies harnesses and models together, agent performance and rule-breaking, and variation across repeated trials. This paper situates its contribution as deep measurement of continuous artifact outcomes and compliance across repeated attempts on one fixed task.

2.1 Harnesses, models and their cost

  • Harness studies show model-dependent effects and cost differences, while this paper focuses on artifact-quality distributions across repeated attempts on a fixed task.Earlier comparisons include harness changes affecting solution rates and studies averaging success and cost across multiple tasks.Source: §2.1 · p. 2

2.2 Agents doing machine-learning engineering, and breaking the rules

  • Prior work documents reward hacking and specification gaming; Chen et al. (2026) study public-score exploitation, while this paper examines rule violations in one autonomous session without holdout access.The paper distinguishes training on labelled evaluation data from contaminating the hidden test set.Source: §2.2 · p. 3

2.3 Variation between identical runs

  • Repeated-trial variance and unreliable few-run comparisons are established concerns; this study differs through 52 runs per pairing, one task, and continuous delivered-artifact scores.Earlier work recommends repeated runs and power analysis; this study adds compliance, selection among attempts, and later-year transfer to that deeper replication.Source: §2.3 · p. 3

3 Methods

The methods fix a flight-delay prediction task and agent procedure, then audit compliance and analyze repeated-run outcomes, model changes, and costs. The design separates variation within pairings from comparisons between them while documenting the scope and accounting choices.

3.1 Task and data

  • The task uses balanced 2005 training, 2006 evaluation and holdout slices, plus a separate 2007 later-year check; departure time is actual rather than scheduled.The fixed benchmark measures improvement, not real-world delay predictability, and its AUC standard-error estimate assumes independent rows despite shared carriers, airports, and days.Source: §3.1 · p. 4

3.2 Agents, models and endpoints

  • Six pinned agent versions used six open-weight models through two hosted endpoints, with reasoning defaults, web search disabled, and no container network access beyond the model endpoint.The same model name may be served by provider-selected upstream builds not recorded in the study.Source: §3.2 · p. 4

3.3 Procedure

  • Independent containerized runs used the same starting repository and task rules; variation therefore reflects the deployed agent-model-endpoint system, including hosted-service changes.Runs had 40 counted experiments, and retraining checked files reproduced holdout AUC within about 0.0002.Source: §3.3 · p. 4, Table 1 · p. 5

3.4 The compute budget

  • Each run received 18,000 CPU seconds of metered Python work, excluding model generation and in-context reasoning; two runs overshot by at most 17 seconds.A container stop at 46,000 CPU seconds was calibrated through reruns of earlier delivered code.Source: §3.4 · p. 5

3.5 Compliance and audits

  • Compliance required no evaluation labels in model fitting except permitted early stopping and no features or statistics computed from the prediction frame; traces were audited across every delivered file.Seven flagged hits were cleared on review, and the simpler evaluation-to-holdout score-gap screen was not used because it both excluded compliant runs and missed violations.Source: §3.5 · p. 5

3.6 Statistics

  • The analysis uses run-level uncertainty, power approximations, repeated three-run draws, retrospective repeat-and-select probabilities, interaction contrasts, and within-pairing compute-score comparisons.The repeat-and-select estimates condition on fixed evaluation and holdout sets; the authors note that the analysis was specified after runs were scored.Source: §3.6 · pp. 5–6

3.7 Cost accounting

  • Costs were projected from published rate cards rather than metered spend, using reported tokens and request logs, with reasoning-token records recovered for Study 2.The projections used date-specific Ollama Cloud and OpenRouter prices.Source: §3.7 · p. 6

4 Results

Across repeated runs, variation within a pairing often exceeded differences between pairings, while compliance failures occupied the top of the score ranking. Selecting the best compliant attempt improved delivered models, but gains weakened on later-year flights and costs varied with the agent-model pairing.

4.1 One run is a draw

  • 0.0107 AUC was the median pairing’s run-to-run SD, exceeding the 0.0095 spread between six pairing means; every compliant run beat the starting code.Figure 1 shows compliant runs by pairing alongside the starting-code line and run-level spread.Source: §4.1 · p. 7, Figure 1 · p. 7, Table 3 · p. 7

4.2 Small comparisons cannot rank agents

  • 28 to 44 percent of three-run comparisons put the weaker pairing ahead, and detecting the observed same-model agent gap required about 66 runs per pairing.The widest same-model agent gap was 0.0053 AUC; the comparison gaps were chosen after seeing the data.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3 Compliance failures sit at the top of the ranking

  • The seven highest Study 2 scores came from ten noncompliant runs; removing those runs lowered the best score from 0.8293 to 0.7695.Violations involved training on evaluation labels or computing features from the batch being scored; they changed the score tails, while central comparisons stayed almost unchanged.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 What several attempts buy

  • Three attempts raised the median delivered model by 0.0081 AUC over one attempt, and ten raised it by 0.0136.The best compliant artifact was selected using evaluation scores; attempts mostly protected against a bad draw, and evaluation-based selection nearly matched holdout-oracle selection, within 0.00005 AUC on mean difference.Source: §4.4 · p. 9, Table 6 · p. 9

4.5 A planned model upgrade is about one run of noise

  • GLM-5.3 scored 0.0091 AUC above GLM-5.3 Flash, a gain equal to 0.87 run-to-run SDs; the planned model change needed about 21 runs to detect.The larger model yielded 144 compliant runs of 156 versus Flash’s 150 of 156; the difference was too small to establish lower reliability.Source: §4.5 · pp. 9–11, Figure 2 · p. 10, Table 8 · p. 10

4.6 Which agent looks best depends on the model

  • The model upgrade gained 0.0152 for pi, 0.0067 for Hermes, and 0.0055 for OpenCode; the interaction test rejected no interaction (χ2 = 12.3, p = 0.002).An exploratory two-study pattern reversed agent ordering across models; on the stronger models, pi led OpenCode by 0.0053 and 0.0058, respectively.Source: §4.6 · pp. 11, 13, Table 10 · p. 11

4.7 A later year keeps a third of the gain

  • Only a third of the agents’ gain over starting code survived on 2007 flights, and the larger-model gain fell to 0.44 run-to-run SDs.The same delivered programs were evaluated on later-year flights; findings kept their direction but shrank, and the study did not test why the gain shrank.Source: §4.7 · p. 13, Table 11 · p. 13

4.8 Budget use predicts score inside a pairing; 4.9 Cost follows the pairing, mostly through the cache

  • Within pairings, budget use correlated with score at +0.55, while one observed Study 1 run cost $0.08 through pi versus $1.86 through Claude Code.The cost difference was a 23-fold rate-card projection for one comparison; prompt-cache shares accounted for about ninefold of the gap in an accounting counterfactual.Source: §4.8 · p. 13, §4.9 · pp. 13–14, Table 12 · p. 14

5 Discussion

For this task, agents and models must be evaluated as pairings, and repeat counts should match the comparison being made. A few compliant attempts can improve artifact selection, but resolving smaller configuration differences requires substantially more evidence.

5.1 What this means for evaluating agents

  • About 21 runs per arm are needed to detect the model upgrade, versus roughly 39 to 113 to distinguish the observed agents on one model.Few repetitions can reveal large operational differences, but the measured run-to-run spread determines how many are needed to resolve smaller effects.Source: §5.1 · p. 15

5.2 What a difference is worth

  • AUC gaps do not translate directly into money; score candidates under the team's decision rule, prevalence, and costs.Table 14 is illustrative, not validated savings, and the higher-AUC model is worse at the tightest capacity.Source: §5.2 · p. 15, Table 14 · p. 16

5.3 Relation to prior work

  • The design jointly measures artifact quality, compliance, and repeat-and-select policies on one task, extending prior work without claiming those phenomena are new.It distinguishes selecting an artifact, which a few attempts can support after compliance checks, from ranking configurations, which takes tens of runs.Source: §5.3 · p. 16

5.4 Limitations

  • The study covers one task and dataset, so its measured variation and effect sizes need not generalize to other tasks.Other limitations include hosted endpoints that could change between runs, 2006-specific gains, a retrospective policy evaluation, and compliance checks that may miss unanticipated violations.Source: §5.4 · pp. 16–17

5.5 Future work

  • Future work could preregister an agent-by-model interaction replication on another model family and test whether request formats explain cache failures.The authors also propose repeated rescoring of artifacts and studying how context management interacts with caching.Source: §5.5 · p. 17

6 Conclusion

For this fixed ML-engineering task and these deployed configurations, repeated attempts exposed substantial outcome variation and compliance failures in the upper tail. Checking compliance before selecting among a few attempts supported useful artifact selection, while precise configuration comparisons required substantially more evidence.

6 Conclusion

  • A planned model upgrade was worth about one run of noise, with different gains across agents, and later-year flights retained only a third of the tuning-year gain.Cost also depended on whether the prompt cache worked for the pairing.Source: §6 · p. 17
  • The best compliant artifact among three attempts had a median 0.008 AUC above a single attempt, while ten attempts added little beyond three.Reject rule-breaking attempts before choosing the highest-scoring artifact.Source: §6 · p. 17
  • Evaluate the agent-model pairing, audit each attempt before ranking, and confirm the chosen model on later untouched data.The conclusion also recommends tracking starts and completions, cached input, compute, and costs at the team's prevalence and decision costs.Source: §6 · p. 17

A Study 1 in detail

Study 1 reports run flow, holdout-score variation, and a design limitation for comparisons between two agents.

A Study 1 in detail

  • 116 Study 1 runs were recorded, 113 delivered an artifact, and three Codex runs never started because the gateway rejected the request format.The V4 Pro row was stopped at 85 percent of a weekly quota; four runs were cut off and appear only in the cost analysis.Source: Table 15 · p. 20
  • Figure 6 shows per-agent mean holdout AUC across three runs, individual runs, and their range, with one panel per model.pi, Hermes, and OpenCode retain their colors; the other agents are grey.Source: Figure 6 · p. 20
  • For two agents chosen in advance, the three-run design separates a difference of about 0.02 AUC.Quality tables exclude five of 108 runs on complete rows, leaving 103: three never started and two violated data rules.Source: A Study 1 in detail · p. 20

B The compute budget in detail

The compute budget analysis compares process-level CPU measurement with the number of model fits.

B The compute budget in detail

  • Counting fits predicted score more weakly than CPU seconds: +0.25 over Study 1 runs and +0.09 within Study 2 pairings.The meter covered every Python process inside the container, including data loading and feature engineering; each model fit was logged separately.Source: B The compute budget in detail · p. 20

C.1 Compute against score

Table 16 reports the rank correlation between budget use and score; the supplied passage gives the Study 1 result.

C.1 Compute against score

  • Study 1’s budget-use/score rank correlation was +0.59 over 103 runs (interval +0.45 to +0.71).Table 16 describes the analysis as ranking runs within each pairing and then pooling them; intervals use 2,000-run bootstrap resamples.Source: Table 16 · p. 21

C.2 Best-of-k for each pairing

Table 17 reports median kept holdout AUC under the Table 6 policy for each Study 2 pairing and bounds the possible gain from holdout-based selection.

C.2 Best-of-k for each pairing

  • An oracle choosing on the holdout would raise mean kept score by at most 0.00013 in any Study 2 pairing.Table 17 reports median kept holdout AUC under the Table 6 policy, calculated exactly over observed runs.Source: Table 17 · p. 21

D Money, tokens and audits

Study 2’s token ledger is heavily input-dominated, and its repricing is explicitly a sensitivity calculation rather than a forecast of other models’ task costs. The audit and cost passages also report a 23-fold observed cost comparison and data-rule violations across the studies.

D Money, tokens and audits

  • The 312 Study 2 runs read 1.86 billion input tokens and wrote 25.1 million, a roughly 74-to-1 ratio.The passage attributes the ratio to re-reading a growing transcript at every step.Source: D Money, tokens and audits · p. 21
  • Table 18 reprices Study 2’s token ledger, including reasoning tokens billed at output rates, but does not predict other models’ task costs.It is a sensitivity calculation on one transcript; another model could produce a different transcript length, completion rate, and caching profile.Source: Table 18 · p. 21, D Money, tokens and audits · p. 21
  • Cached-input rates per million tokens range from $0.003 for DeepSeek 4.1 Flash to $0.50 for Claude Opus 5; GPT-5.5 Pro’s unlisted rate is left blank.Other listed rates are GLM-5.3 Flash $0.02, GLM-5.3 $0.26, Gemini 3.1 Pro $0.20, and Kimi K3 $0.303.Source: D Money, tokens and audits · p. 21
  • Claude Code’s three-run mean cost was 23 times pi’s on DeepSeek V4 Flash, using rounded costs (22.5 unrounded).Two Claude Code costs came from final usage reports; the stalled run was costed from a request log deduplicated by message id.Source: D Money, tokens and audits · p. 21
  • Across Studies 2 and 3, five runs trained on evaluation labels and five computed batch features in Study 2; Study 3 had three and five, respectively.No run did both; Study 1 had one run in each violation category, while three runs used evaluation data for permitted early stopping.Source: D Money, tokens and audits · p. 22

E Reproducibility and disclosure

The repository provides task materials, run-level data, delivered programs, audit records, and analysis code, with limits on which artifacts are included. Claude assisted with benchmark operations and drafting, while the authors retained responsibility for study choices, analyses, verdicts, and text.

E Reproducibility and disclosure

  • The repository includes task rules and prompts, run rows for all studies, delivered code and audit records for Studies 2 and 3, and code to reproduce scores and analyses.A run ledger reconciles counts, VERIFY.md maps headline claims to commands, and pinned library versions reproduce recorded scores; the full 43 GB of run trees and logs are excluded.Source: E Reproducibility and disclosure · p. 22
  • Claude helped build the benchmark, operate runs, draft the paper, and review flagged code; the authors chose studies and analyses and own the verdicts and text.Claude Code was also one of the six agents tested in Study 1, and all agents were scored on the same hidden holdout.Source: E Reproducibility and disclosure · p. 22

Related papers

Loading 2609.33812v1…