PaperlayerView exact revision on arXiv

Source-linked AI summary

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Kirill Brilliantov, Alejandro Hernández-Cano, Emmanuel Abbé

arXiv:2609.40303v1cs.AI

TL;DR

This paper asks whether elaborate MLE harnesses remain useful when coding agents can work directly in an execution environment. Through controlled ablations and matched benchmark comparisons, it finds that a minimal coding-agent session matches or beats tested harnesses at frontier backbones.

  • Problem

    MLE harnesses have grown increasingly elaborate, while coding agents with direct execution-environment access have received little attention in comparisons.

    Source: §1 · p. 1, §6 · p. 9

  • Method

    The authors ablate harness interventions at matched budgets and backbones, compare Malena with open-source MLE harnesses, and analyze agent traces.

    Source: §1 · p. 2, §5 · p. 7, §5.2 · p. 8

  • Results

    Across tested frontier backbones, no tested harness significantly outperforms Malena; with GLM 5.2, Malena earns medals on 62.5% of competitions versus 47.1% for the best external harness.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    A single long-running coding-agent session with minimal machinery is a strong baseline, and the standard components tested did not improve on it.

    Source: §6 · p. 9

    Most results rely on MLE-bench, some confidence intervals remain too wide for strong effect-size claims, and most tested interventions operate within a single worker.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents - where LLMs have direct access to the execution environment through read, write, and bash primitives - has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

1 Introduction

As coding agents gained the ability to execute, inspect, and debug within one session, it became less clear which harness machinery still helps. This introduction motivates controlled comparisons and reports that a minimal coding-agent session matches or outperforms tested harnesses at frontier backbones.

1 Introduction

  • Modern coding agents can read and write files, execute code, and debug in one session, capabilities that earlier single-completion harnesses had to supply.The paper also questions whether long-horizon work requires dedicated memory and context management when million-token windows are affordable.Source: §1 · p. 1
  • Existing MLE-harness comparisons are difficult to interpret because they often vary in backbone and hardware, use few seeds, and can cost thousands of GPU hours to reproduce.Coding-agent comparisons in MLE have only recently appeared and remain sparsely documented.Source: §1 · pp. 1–2
  • The authors hold backbone, hardware, and time budget fixed while comparing a ladder of interventions from chat-based iteration to multi-agent coding-agent orchestration.Their evaluation uses a harder and broader task set than Lite.Source: §1 · p. 2
  • With frontier backbones, Malena matches or outperforms tested harnesses on MLE-bench and NatureBench, while the coding-agent environment is the largest measured intervention.Beyond the coding-agent environment, no tested intervention significantly improves performance.Source: §1 · pp. 2–3
  • Trace analysis suggests the agent performs adaptive search without harness-imposed structure, balancing rare techniques against performance-oriented refinement and reusing prior artifacts.The introduction also describes this as a trace analysis of coding-agent search.Source: §1 · pp. 2–3

2 Related work

Prior MLE harnesses generally use an outer program to prompt an LLM for code and manage execution, scoring, and exploration. Their interventions target limitations in unaided LLMs, while another research line trains the backbone itself.

2 Related work

  • MLE harnesses commonly prompt an LLM for code and handle execution, scoring, and exploration, with interventions aimed at what LLMs cannot reliably do unaided.These systems trace back to tree search over single-file programs and follow a shared outer-harness template.Source: §2 · p. 3
  • Harness interventions commonly manage accumulated state or prescribe search topology through tree search, bandit rules, hypothesis search, and multi-agent orchestration.The paper notes that harness-directed search can limit adaptability.Source: §2 · p. 3
  • A parallel line trains MLE backbones with supervised fine-tuning and reinforcement learning, but collecting long-run traces and training on them is costly.This study focuses instead on interventions in hand-crafted harnesses.Source: §2 · p. 3

3 Experimental settings

The experiments use an OpenCode-based harness and MLE-bench as the primary benchmark. Performance is reported under defined submission-selection and evaluation conventions, with resources held fixed.

3 Experimental settings

  • The harness uses OpenCode v1.15.6, and MLE-bench is the primary benchmark.Task selection is described in Appendix A.1.Source: §3 · p. 3
  • Unless otherwise specified, results use self-selection, choosing the submission with the best validation score; oracle selection instead uses the best hidden-test score.Percentile measures the share of human leaderboard entries scoring below the agent.Source: §3 · p. 3
  • The experiments report 95% confidence intervals on fixed-task macro-averages, bootstrapped over available seeds, with resources fixed at 1×A100 80GB, 12 CPU cores, and 144GB RAM.Unless stated otherwise, the backbone is GLM 5.2.Source: §3 · p. 3

4 What interventions actually matter?

The section uses controlled, same-codebase ablations to test MLE harness interventions. It identifies coding-agent access as the highest-leverage intervention and examines whether added search, autonomy, and orchestration improve performance.

4 What interventions actually matter?

  • The coding agent environment is the highest-leverage intervention, and additional harness machinery yields no statistically significant gain as the agent receives more autonomy.The authors argue that common interventions may be redundant in coding-agent settings.Source: §4 · p. 4

4.1 Coding agent environment

  • Frontier models improve dramatically with direct execution access, and weaker models also benefit, as shown by Chat and Oneshot performance across backbones.The authors also report larger improvement for DeepSeek V4 models than for their Preview versions.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 Search primitives

  • No search strategy significantly outperforms another in percentile or medal rate; UCB1 leads Best-of-N by 2.71 percentile points (95% CI −1.44 to +6.60).Best-of-N’s self-selected performance falls 6.832pp short of its oracle; the authors identify reducing selection gaps as a design priority, while noting some noise is unavoidable.Source: §4.2 · p. 5

4.3 Autonomy

  • Malena achieves the highest medal rate and significantly exceeds manual search methods by 0.4%–5.6% in medal rate, depending on the method.Its percentile advantage over UCB1 and Greedy is statistically indistinguishable; it has the lowest validation gap at 1.876pp.Source: §4.3 · pp. 5–6, Figure 3 · p. 6

4.4 Multi-agent orchestration

  • Base Malena performs no significantly worse than any tested orchestration; parallelism raises oracle-selected percentile ceiling, but its larger selection gap leaves self-selected quality similar.The authors leave effective coordination of strong agents as an open question.Source: §4.4 · p. 6, Table 1 · p. 7

5 Generalization to production harnesses

Across tested harness–backbone pairs, Malena generally matches or beats the production harnesses, with one smaller-backbone exception. Its traces show a single pipeline refined through code-patch search, artifact reuse, and a late shift toward performance-oriented techniques.

5 Generalization to production harnesses

  • The comparison tests Malena against MLEvolve, AiScientist, Arbor, and ScienceFlow under the same compute resources, time budget, and backbone models.The baselines span single-completion and multi-session harness designs; Malena is a minimal single-session harness, not a chat-only baseline.Source: §5 · p. 7

5.1 Results

  • Across 17 MLEbench and 6 NatureBench harness–backbone pairs, Malena matches or beats every harness; only Gemma 4 31B favors MLEvolve by mean estimate, with overlapping medal-rate confidence intervals.The comparison covers 30 MLEbench tasks and 40 NatureBench tasks.Source: Table 2 · p. 7, §5.1 · p. 7

5.2 Trace Analysis

  • Malena and MLEvolve are the top two methods in technique rarity, even as Malena becomes more performance-oriented late in the run.Figure 5 computes technique rarity per task across the combined runs of all method–backbone groups.Source: Figure 5 · p. 8, §5.2 · p. 9
  • Malena builds one pipeline across a task’s natural stages, then searches code patches while reusing checkpoints and utility code across iterations.The passage says this reuse improves efficiency and narrows the generalization gap.Source: §5.2 · p. 8
  • LLM-driven methods begin with balanced pipeline-stage techniques and shift toward ensembling and other performance-oriented methods; the late shift is more pronounced for Malena, while MLEvolve stabilizes after about 10% of the run.Figure 6 plots recency-decayed category shares across checkpoint fractions.Source: §5.2 · pp. 8–9, Figure 6 · p. 9

6 Conclusion

The conclusion finds that a minimal-harness coding agent is a strong baseline: after giving the model an execution environment, added harness machinery yields little improvement. The authors qualify this conclusion with limits involving benchmark reliability, statistical power, coordination scope, and tuning asymmetry.

6 Conclusion

  • The paper tests harness layers by ablation at a fixed model, budget, and task set.It asks which accumulated machinery remains useful when the model can work autonomously in a coding environment.Source: §6 · p. 9
  • A single long-running agent session with minimal machinery is a strong baseline, and none of the standard components added improved on it.The conclusion attributes nearly all measured gain to the runtime and LLM backbone, with little to added scaffolding.Source: §6 · p. 9
  • With GLM 5.2, Malena earns medals on 62.5% of competitions versus 47.1% for the best external harness; across backbones, it is favored or statistically indistinguishable.The comparison holds model, hardware, and budget fixed.Source: §6 · p. 9
  • The authors conclude that elaborating harnesses around models already strong at autonomous work has poor practical returns; leverage lies in the model and runtime.Source: §6 · pp. 9–10
  • The authors flag MLE-bench task-preparation and contamination risks, limited power for effect-size claims, mostly single-worker interventions, and possible tuning asymmetry.They say the weaker no-worse-than claims are well supported and report mitigation steps for benchmark issues and external-harness tuning.Source: §6 · p. 10

Reproducibility statement

  • The reproducibility statement points to appendices for agent tools, search strategies, model configurations, task lists, fixes, and statistical methods.Source: Reproducibility statement · p. 10

A.1 MLE-bench tasks used

The appendix specifies the MLE-bench task subsets and fixes, the NatureBench evaluation cohort, backbone setup, and statistical comparisons. It documents the task and evaluation choices underlying the paper’s experiments.

A.1 MLE-bench tasks used

  • The MLE-bench suite contains 75 tasks, with experiments focused on a 30-task subset split into 5 Lite, 21 Medium, and 4 High tasks; the default29 split excludes two tasks.Results in Section 4 use the original benchmark without intervention on the default29 split.Source: §A.1 · p. 16, Table 3 · p. 17

A.2 Fixes applied to MLE-bench tasks

  • Four MLE-bench competitions with breaking issues or cheating risks were fixed, and results for comparisons against existing baselines use those fixed versions.The issues included missing training inputs and leaked ground-truth information.Source: §A.1 · p. 16, Table 3 · p. 17

A.3 NatureBench tasks used

  • NatureBench evaluates whether agents surpass published state of the art on held-out protocols, using 40 licensed tasks and an 8-hour budget for GLM 5.2 and Kimi K3.The reported beat-SOTA rate is macro-averaged; a task counts as surpassed when normalized gain g > 0.1.Source: §A.3 · p. 18, Table 4 · p. 19

A.4 LLM backbones used

  • Unless otherwise stated, experiments use GLM 5.2; backbones are served locally with vLLM, and the appendix lists the models and configurations used.GLM 5.2 and Kimi K3 identifiers and September 2026 effective token prices are among the listed details.Source: §A.4 · pp. 18–19

A.5 Statistical protocol

  • Performance is reported as a macro-task average, with bootstrap confidence intervals and paired-by-task differences used to assess statistical significance.A difference is significant when its 95% confidence interval excludes zero; comparisons use the same task set.Source: §A.5 · pp. 19–20

B Implementation details

This section details the implementation of agent iterations, infrastructure, and search strategies, and examines time limits, method comparisons, and interventions related to the hidden validation gap. It also considers how allocated time affects oracle performance for a bare Malena iteration and, at 24 hours, how backbone size and model choice shape comparison with Best-of-N.

B.1 Iterations

  • The appendix details three iterations: Chat produces and runs a script without tools, Oneshot allows one production run, and Malena can iterate freely throughout the time budget.Oneshot and Malena receive data paths, while Chat receives an inline data overview; Oneshot’s improve variant uses a parent node’s documented design as its starting point.Source: §B.1 · pp. 20–21

B.2 Infrastructure

  • The coding-agent infrastructure uses OpenCode’s read, write, edit, and bash tools, with a custom bash implementation for resource reporting and watchdog termination; optional tools and interventions vary by configuration.The appendix describes submission, system, jobs, and skill tools, plus interventions including delegation, time and resource nudges, strict validation guidelines, and a minimal prompt.Source: §B.2 · pp. 21–22

B.3 Search strategies

  • UCB1 scores every tree node globally using rank-based values and best-descendant score backups, while Best-of-N composes independent Oneshot runs into trajectories matched to consumed time.The exploration constant is fixed at c = 0.75; the rank transform supports transfer across differently scaled metrics, and trajectory results are averaged across bootstrapped sets.Source: §B.3 · p. 23

C.1 Oneshot time limit effect

  • Per-iteration performance improved monotonically with longer budgets, but aggregated Best-of-N performance quickly saturated at three hours, making six hours a cost-performance compromise.Even at a 12-hour per-iteration limit, average effective time use remained much lower, enabling aggregation at a six-hour budget.Source: Figure 7 · p. 24, §C.1 · p. 24

C.2 Pairwise Oneshot search strategy and Malena agent comparison

  • Tables 5 and 6 provide detailed paired-by-task comparisons of Oneshot search strategies and Malena, including the selection gap, on the 24-hour default29 setting.Table 5 reports percentile differences with 95% confidence intervals; Table 6 reports selection gaps with 95% confidence intervals.Source: Table 5 · p. 24, §C.2 · p. 24, Table 6 · p. 25

C.3 Hidden validation gap

  • Best-of-N Oneshot had the highest oracle percentile ceiling; strict validation guidelines slightly improved self-selection, while a baseline iteration produced the smallest selection gap but a significantly worse ceiling.The authors hypothesize that shared validation splits make scores more comparable, but say this mechanism is untested and the experiments lack power for strong conclusions.Source: Table 7 · p. 25, §C.3 · p. 25

C.4 Malena budget scaling

  • The budget-scaling study uses bare Malena with submission, system, and jobs tools disabled, so it reports an oracle ceiling because there is no self-selection mechanism.The authors describe this as a realistic range for temporal behavior because this Malena iteration had the lowest validation gap among tested iterations.Source: §C.4 · p. 26
  • Malena’s oracle percentile rises steadily with more time, while performance at identical timeframes varies by total budget, suggesting budget-dependent exploration–exploitation adaptation.The comparison uses a DeepSeek V4 Pro (Preview) backbone on the default29 split.Source: §C.4 · p. 26, Figure 8 · p. 26

C.5 Malena with different backbones

  • At 24h, GLM 5.2 and Kimi K3 benefit clearly from Malena, while Gemma 4 31B favors Best-of-N in percentile but retains a better Malena medal rate.The authors suggest larger models may better leverage single-session autonomy, or Gemma’s 256k context window may bottleneck as compaction loses details.Source: §C.5 · p. 26, Figure 9 · p. 27, Table 8 · p. 27

C.6 Malena infrastructure ablation

Malena’s infrastructure ablations find no significant performance gains from added interventions, while shrinking context is accompanied by lower performance despite repeated compaction. The constrained-context results suggest sensitivity near a hard context ceiling, though confidence intervals overlap.

C.6 Malena infrastructure ablation

  • None of the tested infrastructure interventions significantly improved performance; removing the system tool or adding time nudges hurt, while removing jobs was marginally better.The authors report no significant improvements at current statistical power; the system-tool result suggests the agent struggles to track time without it.Source: §C.6 · p. 27, Table 9 · p. 28

C.7 Malena under a constrained context budget

  • 9.5 compactions per 24h run on average; medal-reaching runs progressed through 3.25 compactions on average before their first medal.All 40 shrunk-context runs compacted; among the 20/40 runs earning a medal, the count before first medal ranged from 0 to 19.Source: §C.7 · p. 28
  • Performance drops under the shrunk budget at both selection strategies, although the 95% CIs overlap.The comparison used GLM 5.2 at its native 1M context limit versus a 64k-triggered compaction setting on the full fixed10 subset; the authors read the drop as sensitivity near a hard context ceiling.Source: §C.7 · p. 28

C.8 Oneshot infrastructure ablation

This appendix compares infrastructure interventions and external MLE harnesses, then examines tuning choices, Kimi K3 trajectories, and performance across task-difficulty clusters.

C.8 Oneshot infrastructure ablation

  • Most Oneshot infrastructure interventions modestly improve point estimates, with only H significant for GLM 5.2 and I for DeepSeek V4 Pro; the larger gain appears when all interventions run together.The reported averages are per iteration, with comparisons on the default29 split.Source: §C.8 · pp. 28–29, Table 13 · p. 30, Table 14 · p. 30

D.1 Detailed external harness description

  • The compared harnesses range from MLEvolve’s UCT candidate tree and retrieval memory to AiScientist’s sequential, agent-directed workflow, Arbor’s coordinator-managed idea tree, and ScienceFlow’s parallel workers.AiScientist’s agent, rather than harness code, selects its final candidate.Source: §D.1 · pp. 29–31

D.2 Detailed MLE-bench results

  • Malena beats every presented baseline on GLM 5.2, has strictly better percentile scores on Kimi K3, and beats or ties baselines statistically on DeepSeek V4 Flash.On Gemma 4 31B, the weakest tested backbone, MLEvolve leads on percentile while medal rates are equivalent at current statistical power.Source: §D.2 · p. 31, Table 15 · p. 32, Table 16 · p. 33
  • No-submission runs receive the worst percentile, so unlike failure modes can tie and a few large gaps can disproportionately affect estimates; uneven task seed counts are also treated equally.The latter applies to per-task-mean-based methods.Source: §D.2 · p. 31

D.3 Tuning effort invested in baseline harnesses

  • On DeepSeek V4 Flash, medium reasoning cut median call length about 5× and increased mean completed MLEvolve search nodes per run from 18.5 to 62.8.Any-medal rate rose from 0% to 25.0%, while buggy-node rates were similar at 61% versus 62%.Source: §D.3 · pp. 31–32, Table 17 · p. 33
  • Arbor’s 12-hour executor timeout and disabled early stopping left medal rate unchanged; ScienceFlow’s worker-count comparison was statistically indistinguishable, and its resource-policy change showed no reliable improvement.The 1-worker ScienceFlow point estimate for any-medal rate was 43.9% versus 40.0% for two workers.Source: §D.3 · pp. 33–34, Table 20 · p. 34, Table 18 · p. 34

D.4 Trajectory and token usage with Kimi K3

  • At Kimi K3, quality rises quickly in the first hours and then plateaus, while MLEvolve’s uncached, output-heavy tree search is the most expensive system shown.Kimi K3’s output pricing is $13.99/M, versus GLM 5.2’s $3.123/M.Source: §D.4 · p. 34, Figure 10 · p. 35

D.5 Quality by task difficulty

  • Malena’s clearest cluster lead is on cluster 3; its lead is smaller on clusters 2 and 4, while all methods approach ceiling performance on cluster 1 by 24 hours.On cluster 2, Malena and MLEvolve sustain an edge over Arbor and AiScientist; on cluster 4, Malena’s edge is consistent but smaller.Source: §D.5 · p. 35

D.6 NatureBench: evaluation protocol

NatureBench permits repeated live evaluation and reports each run’s best score, while the trace-analysis appendix details how techniques are labeled and compared over time and across tasks.

D.6 NatureBench: evaluation protocol

  • NatureBench grants unrestricted /evaluate access and extends runs to 8 hours; its best-score metric rewards the number of scoring opportunities, which may mechanically disadvantage MLEvolve’s lower call rate.Median calls per run were 16 for Malena, 18 for AiScientist, and 8 for MLEvolve.Source: §D.6 · pp. 35–36, Table 21 · p. 36

E Trace Analysis

  • Trace analysis labels checkpoints from 10 original MLE-bench tasks, a set distinct from the paper’s fixed10 challenging split.GLM 5.2 is the default labeling and judge model.Source: E Trace Analysis · p. 36

E.1 Checkpoint mining

  • Checkpoints snapshot repositories at method-specific moments, and sequential labeling carries forward entries from the nearest earlier checkpoint.Technique entries separate a task-agnostic mechanism from task-specific instantiation and code evidence; offline, per-task clustering and review canonicalize entries before a second labeling pass.Source: §E.1 · pp. 36–37

E.2 Main-text plot methodology

  • Category-share decay weights techniques by when they appear in a run, while rarity measures how many pooled runs on the same task do not exhibit a technique.The decay plot uses half-life 0.25 in the main text; rarity is calculated within each task across methods and backbones.Source: §E.2 · pp. 37–38

E.3 Robustness of the category-share-decay curves

  • The appendix checks whether Malena’s late-run shift toward ensembling and model selection persists at half-lives 0.1, 0.15, and 0.2, both pooled and on aptos2019 and nfl individually.The figures retain checkpoint fraction on the x-axis and decayed technique-category share on the y-axis.Source: §E.3 · p. 38, Figure 12 · p. 38

E.4 Per-task technique-category composition across methods

The per-task comparison separates task-driven technique choices from differences among methods, finding a shared task signal alongside method-specific departures.

E.4 Per-task technique-category composition across methods

  • The profiles count whether a method tried a category-tagged technique at all, not how often it used that technique during a run.A technique cluster with multiple category tags contributes to each tagged category rather than being split fractionally.Source: §E.4 · p. 38
  • Task identity strongly shapes technique-category profiles: feature engineering dominates champs, h&m, and nfl, while ensembling dominates petfinder and several other tasks for Malena and AiScientist.On aptos2019, cassava, and jigsaw, ensembling dominates specifically for Malena and AiScientist.Source: §E.4 · pp. 38–39
  • Malena and AiScientist have similar profiles on most tasks, whereas MLEvolve often shows more data preprocessing and less ensembling.This pattern is consistent with MLEvolve’s pooled under-representation of ensembling growth in Figure 6.Source: §E.4 · p. 39
  • Jigsaw has the largest cross-method spread: Oneshots has less ensembling share, offset by a larger loss/optimization block.Source: §E.4 · p. 39

E.5 Illustrative session excerpts

Session excerpts illustrate artifact-based refinement, a stable multi-stage solution, and independent rediscovery of a rare technique with differing amounts of surrounding engineering.

E.5 Illustrative session excerpts

  • After reviewing summaries and histories for eight models, the subagent chose artifact-based ensemble reweighting rather than changing training code.Its session record reports 14 bash calls and six read-file calls, with no edit-file calls.Source: §E.5 · p. 41
  • On whale, Malena settled within 17 checkpoints on an embedder-plus-nearest-neighbor pipeline and used the remaining 63 checkpoints for targeted patches rather than new stages.The fixed pipeline includes a learned novelty detector for unseen whales.Source: §E.5 · p. 41
  • Malena’s GLM 5.2 and Kimi K3 runs independently rediscovered soft-label self-training on aptos2019, a technique absent from the other listed methods’ runs on that task.The cluster’s reported rarity is 0.9.Source: §E.5 · p. 42
  • The two aptos2019 runs shared the pseudo-labeling mechanism but differed in their added engineering: GLM 5.2 used iterative rounds and combinatorial search, while Kimi K3 used per-fold warm-starting.The GLM 5.2 run reported OOF 0.7212 versus 0.7199 for the earlier pseudo-label version.Source: §E.5 · pp. 39, 42

F Contamination check

Containment-based code matching and chunk-embedding checks find no anomalous similarity between Malena-generated code and real solutions; a behavioral comparison also supports iterative problem-solving rather than recall.

F Contamination check

  • The checks compare Malena against a curated corpus of 1,350 distinct real solutions spanning all 30 fixed30 tasks, using both token-based and embedding-based signals.The embedding comparison represents each solution as a bag of chunk embeddings and uses three similarity metrics.Source: F Contamination check · pp. 42–43
  • Zero of 924,750 Malena-vs-real pairs were flagged by containment, with scores below the real-vs-real baseline at every reported percentile.Containment uses the smaller file’s token overlap and longest matching run, addressing default Dolos similarity’s tendency to miss partial reuse from much larger files.Source: F Contamination check · p. 43, Table 22 · p. 43
  • At p95 and p99, Malena-vs-real embedding similarity is below the real-vs-real baseline for all three metrics, with no anomalously heavy right tail.At the median, the similarities are comparable; mean-pooling cosine remains comparable through p90.Source: F Contamination check · p. 43, Table 22 · p. 43
  • At 6h budgets, Best-of-N Oneshot iterations significantly outperform a single Chat iteration, contrary to the prediction that memorized solutions need no environment feedback.The authors interpret this pattern as consistent with iterative problem-solving rather than recalling known answers.Source: F Contamination check · pp. 43–44

G.1 Limitations

The study’s conclusions are bounded by benchmark exposure, statistical power, coordination scope, and possible tuning asymmetry. The authors present post-training substitution and a bitter-lesson pattern as hypotheses for future work, not established claims.

G.1 Limitations

  • Most results rely on MLE-bench, a public Kaggle benchmark with preparation-script issues and plausible pretraining contamination, despite task fixes and two similarity checks.NatureBench adds recent research questions without a public solution corpus, but covers fewer models and methods.Source: §G.1 · p. 44
  • At 24h, only 3–4 seeds per task were affordable, leaving some confidence intervals too wide for strong effect-size claims.The authors consider their weaker claim that Malena is no worse than a baseline well supported, but urge caution about exactly how much better it is.Source: §G.1 · p. 44
  • The claim that most harness interventions add little is based almost entirely on a single worker, leaving broader multi-worker coordination research for future work.The authors note that coordination may require different useful interventions and a larger compute budget.Source: §G.1 · p. 44
  • Possible tuning asymmetry remains a concern, although external harnesses received backbone-specific verification and targeted configuration checks.The ScienceFlow and Arbor ablations changed outcomes by no more than run-to-run noise at the available sample sizes.Source: §G.1 · p. 44

G.2 Broader Outlook

  • The authors hypothesize that coding-agent post-training substitutes for scaffolding; DeepSeek V4 comparisons offer suggestive, not conclusive, evidence for this account.Preview releases showed little Chat–Oneshot gap, while final releases with more coding-agent post-training showed a clear, though narrower, gap than GLM 5.2.Source: §G.2 · pp. 44–45
  • For capable tool-using models, the authors argue that hand-designed search and coordination can have poor or negative returns as a plain coding agent closes the gap.They frame this as a test-time-inference version of the bitter lesson, while noting weaker backbones still benefit from workflow priors.Source: §G.2 · p. 45

Related papers

Loading 2609.40303v1…