Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits

1 INTRODUCTION

• Across six models, response tokens per answer decline as score gains diminish, while specialized harnesses’ relative benefits diminish or reverse as the base model improves.The paper names these patterns long-horizon deceleration and harness staleness, respectively.Source: §1 · pp. 1–2
• Persistent task artifacts let successive agents build on prior work, so the agent that discovers a useful candidate need not be the one best suited to refine it.The paper asks whether structured alternation can consistently outperform the best single agent under a matched budget.Source: §1 · p. 2
• HMA models alternating sessions as a CR-SMDP and uses two native agents, a shared workspace, context resets, and caps of at most k experiments per session.The analysis establishes sufficient conditions and experiment-budget bounds for exceeding single-agent reference limits, and characterizes when fixed-k interruption preserves the advantage.Source: §1 · pp. 2–3, Figure 1 · p. 2
• 86.4% of mean response tokens per answer are retained from early to late execution by HMA, versus 60.7% for matched single-agent baselines.The paper interprets this as more persistent exploration across sessions.Source: §1 · p. 3
• 78.2% any-medal rate on MLE-bench, 8.0 points above the strongest published open-source workflow; HMA also improves over constituent baselines across six heterogeneous configurations.The average improvement across those configurations is 6.4 points.Source: §1 · p. 3

Harness staleness

• Specialized harness benefits relative to the official harness diminish as the backbone improves, and the stronger backbone favors the native harness despite longer specialized-workflow budgets.On the 16-task MLE-bench subset, some specialized harnesses match or exceed the native baseline with the weaker backbone, while the stronger backbone consistently favors it.Source: Table 1 · p. 3, §2 · p. 4

Motivation

• The section frames these limits as motivation for HMA, which coordinates alternating agents from different providers to extend exploration beyond an individual model–harness system.The harness comparison disables task-specific cold-start knowledge for fairness.Source: §2 · pp. 3–4

Long-horizon deceleration

• Across six single-session configurations under /goal, mean output tokens per response in a trailing 30-minute window—a proxy for reasoning intensity—decrease by 29.8%–45.0% from hours 0–2 to hours 3–5, revealing diminishing returns from continued single-agent exploration.Source: §2 · p. 3

3.1 HMA PRELIMINARIES

• The CR-SMDP state tracks the shared workspace, retained result, and active agent’s context; option boundaries preserve the workspace and result while resetting context.The analysis defines expected quality and target-success probability after a given experiment budget.Source: §3.1 · p. 4

3.2 ALTERNATION BEYOND SINGLE-AGENT LIMITS

• Under best-result retention and cooperative access and use, the theorem gives sufficient conditions for alternation to exceed the single-agent reference ceiling.The reference family comprises single-agent workflow variants, including variants with ordinary or stronger harnesses.Source: §3.2 · pp. 4–5
• The corollary gives a sufficient experiment-budget bound for reaching target-success probability p0; setting p0 = (U* + η)/τ guarantees mean quality U* + η.The budget form also applies to a single agent with a uniform conditional success probability of at least a > 0.Source: §3.2 · p. 5

3.3 FIXED-K INTERRUPTION AND EMPIRICAL CHOICE

• Fixed-k interruption exceeds the natural-termination reference when its expected net gain plus the NTA margin is positive; geometrically decaying interruption losses ensure some finite cap can suffice.The theorem compares capped alternation with natural-termination alternation for the optimal initial agent.Source: §3.3 · p. 5
• 97.83% of observed gains are retained at k = 5 with 73.77% mean within-option experiment retention across eleven configurations, motivating candidate caps k ≤5.The configurations comprise six goal and five NTA setups evaluated on 75 MLE-bench tasks.Source: Figure 3 · p. 6, §3.3 · p. 6

3.4 HMA WORKFLOW OVERVIEW

• The workflow alternates agents until the session cap or exploration deadline, resets context at handoff, then reserves final review for the other agent to select from the candidate archive.The latest candidate is the fallback, and review proceeds without test-set scores.Source: §3.4 · p. 6

4.1 EXPERIMENT SETUP

• MLE-bench evaluation covers 75 Kaggle competitions, with one NVIDIA A10 GPU and six hours per run; agents alternate at k = 5 with context reset at handoff and final review in the last 15 minutes.Test-set scores are unavailable to agents and used only for offline evaluation.Source: §4.1 · pp. 6–7

4.2 MAIN COMPARISON

• 78.2% overall medals for Opus-first HMA exceeds the strongest single-agent baseline (72.4%) and highest published workflow baseline (70.2%), with a six-hour budget versus 12–24 hours for published workflows.The workflow uses no task-specific knowledge bases or additional prompt engineering.Source: Table 2 · p. 7, §4.2 · pp. 7–8
• All six HMA configurations improve overall medal rate over their stronger constituent; Opus–GPT in either starting order reaches 78.2% or 73.3%, versus 72.4%.Across five model pairs, constituent medal rates range from 54.2% to 72.4%, and HMA also improves Gold, Med+, and mean percentile over constituent averages.Source: §4.2 · p. 8

4.3 EXPERIMENT ANALYSIS AND ABLATION STUDY

• HMA retains 86.4% of mean response length from early to late execution, versus 60.7% for matched goal references, while all six configurations continue improving through six hours.Within two hours, HMA configurations reach 42.7%–60.0% medals.Source: Figure 4 · p. 8, §4.3 · p. 8
• Final review changes 8.9% of selections and raises any-medal outcomes from 311 to 319, with no losses relative to the pre-review candidate.Among 330 runs with a medal-winning exploration candidate, final selection retains a medal in 319 (96.7%), compared with 311 (94.2%) before review.Source: §4.3 · p. 8

4.4 CASE STUDY

• On INGV, alternation improves the historical best MAE in five of six post-handoff options, reaching 34,234—46.5% below the best of six constituent goal runs.Response length increases on four of five returns to a previously used model.Source: Figure 5 · p. 9, §4.4 · p. 9

4.5 EXPLORATORY APPLICATION BEYOND MLE-BENCH

• In a selected 48-hour AOPT run, Opus–GPT reaches 878 cycles after 34 hours 33 minutes and retains that score in its final candidate.The score is 14 cycles (1.62%) above the community leaderboard leader’s 864 cycles, but this is not a verified leaderboard placement; the case is a single-run preliminary application beyond MLE-bench.Source: Figure 6 · p. 9, §4.5 · pp. 9–10

• Recent MLE systems modify how one model acts within its harness through search, reflection, memory, branch reuse, recoverable workspaces, or task-conditioned interfaces; HMA instead coordinates complete native model–harness pairs.Source: §5 · p. 10
• Prior long-horizon workflows use reusable experience, reflection, search, communication, role assignment, checkpointing, or review; HMA studies fixed-cap alternation of complete native agents on one shared workspace.HMA resets context at every handoff.Source: §5 · p. 10
• Semi-Markov processes model variable-duration actions and options add an internal policy and termination rule; HMA treats each session as an option in fixed-cap alternation with context reset.Source: §5 · p. 10

• HMA alternates native agents, resetting context while preserving shared task state; its CR-SMDP analysis gives conditions and budget bounds for exceeding single-agent reference limits and preserving the advantage under fixed-cap interruption.Source: §6 · p. 10
• Across six MLE-bench configurations, HMA outperforms the stronger constituent in medal rate under equal six-hour budgets, reaching 78.2% at best.Source: §6 · p. 10
• A supplementary AOPT case illustrates kernel-optimization application over 48 hours, but comparative gains beyond MLE-bench remain future work.Source: §6 · p. 10

AI USE STATEMENT

• Generative AI assisted with language polishing, grammar checking, and portions of code; the authors say they developed the research and verified generated code.Two authors tested the LLM-generated code for correctness.Source: AI USE STATEMENT · p. 10

A THEORETICAL DETAILS AND PROOFS

• The formal supplement follows the Method section, covering finite-budget CR-SMDP definitions, the alternation guarantee and budget bound, and fixed-cap interruption conditions.Source: A THEORETICAL DETAILS AND PROOFS · p. 15

A.1 MODEL DETAILS AND EXACT PERFORMANCE

• The model-details appendix includes finite-budget definitions, option recursion, and exact expressions for target-attainment probability and expected final quality.Source: §A.1 · p. 15

A.2 ALTERNATION GUARANTEE AND EXPERIMENT BUDGET

• Under Assumptions 3.1 and 3.2, alternating agents reach expected quality at least τ, exceeding the single-agent reference limit U* when τ > U*.The proof uses a cooperative-success probability lower bound a = αβ and quality retention.Source: §A.2 · p. 16
• Under the theorem’s conditions with F0 = 1 and fixed p0 ∈ (0, 1), when 0 < a ≪ 1, a sufficient experiment budget for pµk(N; τ) ≥ p0 has leading form (2k/a) log(1/(1 − p0)) + O(k).Source: §A.2 · p. 16
• Initial-agent choice can change startup progress and finite-budget quality, even when later conditional success rates—and thus subsequent budget scales—are similar.The later budget scales are 2k/a.Source: §A.2 · p. 16

A.3 INTERRUPTION GUARANTEE AND SUFFICIENT CAPS

• Natural-termination alternation reaches quality at least τ above U* under the access assumptions, with almost-surely finite option durations.If both natural termination and a finite cap have access across the relevant quality range, their limiting expected qualities equal the common upper bound U†.Source: §A.3 · p. 17
• With geometrically decaying interruption loss, an explicit cap preserves at least half the NTA margin; under uniform constants and margin bounds, the sufficient cap is O(log N).The finite-candidate condition also guarantees some cap beats U* when its worst-case accumulated loss is below the NTA margin.Source: §A.3 · p. 18

B PREFIX RETENTION AND EMPIRICAL CAP SELECTION

• Prefix-retention analysis measures observed gains preserved by short session prefixes using task-level session and quality records.The appendix presents prefix retention as evidence for choosing candidate caps.Source: B PREFIX RETENTION AND EMPIRICAL CAP… · p. 19, §B.1 · p. 19

B.1 RETENTION STATISTICS AND CONFIDENCE BOUNDS

• The simultaneous lower-confidence construction assumes independence across tasks for each prespecified policy, cap, and statistic, while allowing dependence across configurations and caps.Under that assumption, the lower bounds hold jointly with probability at least 1 − δ.Source: §B.1 · p. 19

B.2 CONFIGURATIONS AND OBSERVED PREFIX RESULTS

• The analysis covers 75 MLE-bench tasks and 1,065 recorded NTA options, with prefix-curve means aggregated across configurations.The curves average experiment fractions horizontally and task-level gain retention vertically.Source: §B.2 · p. 19
• At k = 1, 3, and 5, mean gain retention is 63.15%, 93.35%, and 97.83%, using 24.76%, 54.88%, and 73.77% of experiments, respectively.At k = 5, individual configurations retain 96.11%–99.20% of gains, including 98.74% for Opus.Source: §B.2 · p. 20
• At k = 5, 87.84%–98.67% of tasks retain at least 90% of gains; simultaneous lower bounds are 65.08%–83.30%.The bounds use δ = 0.05 across eleven configurations, fifty caps, and two statistics.Source: §B.2 · p. 20

C.1 HUMANIZE AND EXECUTION ENVIRONMENT

• HMA uses Humanize to coordinate shared-workspace execution, fixed-k interruption, and final review; k = 5 and 900 of the 21,600 seconds are reserved for review.Exploration ends at 20,700 seconds.Source: §C.1 · p. 21

C.2 AGENT AND BASELINE CONFIGURATIONS

• The runs use native model-harness pairs at maximum reasoning effort, while published baseline results retain their reported inference settings.The comparison includes ScienceFlow, MLEvolve, and ML-Master 2.0.Source: §C.2 · p. 21

C.3 EVALUATION PROTOCOL AND REPEATS

• All metrics use the returned candidate, and each configuration is evaluated in three independent repeats over the fixed 75-task suite.Gold measures tasks earning gold medals; Med+ measures tasks outperforming the human leaderboard median.Source: §C.3 · pp. 21–22

D ADDITIONAL EVIDENCE FOR THE MOTIVATION

• The response-length analysis uses six single-agent configurations, each evaluated on 75 tasks with three repeats and max reasoning effort.The /goal protocol has no model alternation or fixed-cap interruption and runs until natural termination or six hours of active runtime.Source: §D.1 · p. 23

D.1 DECLINING RESPONSE LENGTH IN SINGLE-AGENT RUNS

• Across all six single-agent configurations, mean output tokens per response decline by 29.8%–45.0% from hours 0–2 to hours 3–5.The interval statistics use raw event totals divided by response counts before temporal smoothing.Source: §D.1 · p. 23

D.2 HARNESS STALENESS ACROSS MODEL VERSIONS

• The harness comparison tests three research harnesses against the official DeepSeek Harness on a 16-task subset, retaining original workflows while disabling task-specific cold-start knowledge where present.It uses DeepSeek-V4-Flash and DeepSeek-V4.1-Flash; the reported comparisons are distinct from the full 75-task results.Source: §D.2 · p. 23

E ADDITIONAL MAIN-COMPARISON ANALYSES

• Matched group-average curves weight each HMA configuration equally and give each model equal weight in the corresponding goal reference; HMA retains 86.4% of early response length versus 60.7% for goal.Mean output tokens per response are 610.9 and 527.6 for HMA, and 650.3 and 394.9 for goal, in the early and late intervals.Source: E ADDITIONAL MAIN-COMPARISON ANALYSES · p. 25
• At two hours, the six HMA configurations attain 42.7%–60.0% any-medal rates and finish at 62.7%–78.2%; GPT–Kimi HMA gains medals on 5, 4, and 6 tasks and loses them on 4, 3, and 2.The corresponding net gains are 1, 1, and 4 tasks, within the evaluated six-hour task union.Source: E ADDITIONAL MAIN-COMPARISON ANALYSES · p. 25

F.1 STARTING-ORDER EFFECTS

• With k = 5, HMA narrows the observed starting-order medal-rate gap from 21.3 to 4.0 percentage points, while the six-hour horizon limits second-agent participation under NTA.NTA results are 77.3% with GPT first and 56.0% with Kimi first; HMA results are 70.7% and 66.7%, respectively.Source: §F.1 · p. 26

F.2 EFFECTS OF THE FIXED INTERRUPTION CAP

• For GPT-first HMA, k = 5 yields the highest medal rate among tested finite caps: 70.7%, versus 65.3% at k = 1 and 66.7% at k = 3.The k = 5 rate is compared with the GPT goal mean of 68.0%; the authors note cap choice and starting agent are jointly relevant under the limited horizon.Source: Table 11 · p. 27, §F.2 · pp. 27–28

F.3 NATURAL TERMINATION BEFORE THE CAP

• At k = 5, 45.5% of all pre-deadline option closures are natural exits; the rate is 12.8% for initial options and 60.3% for later options.Of 655 natural exits, 248 produce no accepted experiment.Source: §F.3 · p. 28, Table 12 · p. 28

F.4 FINAL REVIEW AND HISTORICAL ANY-MEDAL RETENTION

• Historical any-medal retention is the fraction of medal-positive runs whose selected candidate earns a medal; repeated receipts for a reviewed candidate add no new historical outcome.Historical status is computed from offline evaluator records before final review.Source: §F.4 · p. 28
• 319/330 historical medal-winning runs retain a medal after final review (96.7%), up from 311/330 (94.2%), recovering eight medals without losing any held immediately before review.The comparison holds exploration trajectories fixed and measures final selection rather than additional search time.Source: §F.4 · pp. 28–29
• Final review changes 40 of 450 selections (8.9%); 311/319 final medals (97.5%) were already held, and missed historical medals fall from 19 to 11.Across all runs, the returned candidate matches historical any-medal status in 439/450 (97.6%); 120 runs never produce a medal.Source: §F.4 · p. 29

G CASE STUDIES OF HETEROGENEOUS ALTERNATION

• The case studies cover four model pairs across five model families, with GLM–Kimi added using the same k = 5 workflow.Runs were selected for inspectable artifacts and logs; scores were evaluated offline, and final review was excluded from token statistics.Source: G CASE STUDIES OF HETEROGENEOUS… · p. 30
• These case studies illustrate observed mechanisms rather than estimating their average causal effect.Response renewal is compared within the same model because tokenizers and policies differ.Source: G CASE STUDIES OF HETEROGENEOUS… · p. 30

G.1 INGV VOLCANIC ERUPTION PREDICTION

• After Opus first reaches 41,214 MAE, alternation lowers the score to 34,234, a further 16.9% improvement and 46.5% below the best of six constituent goal runs (63,991).Five of six post-handoff options improve the historical best; shared features and cached predictions support successive refinements.Source: §G.1 · p. 30
• Mean response length increases on four of five INGV returns by factors of 1.6–3.6; the final Opus return has a ratio of 0.94.The 27 intervals contain 826 responses, excluding 31 trailing responses; final review is excluded.Source: §G.1 · p. 30

G.2 GOOGLE QUEST Q&A LABELING: GPT–DEEPSEEK

• The returned Google QUEST candidate scores 0.50747 Spearman correlation, versus 0.47252 for the best of six constituent goal runs.Sessions reuse saved predictions for target-specific blending, calibration, and larger-encoder refinement; the final two options produce no accepted experiment.Source: §G.2 · p. 30, Figure 8 · p. 31

G.3 PETFINDER PAWPULARITY: GLM–KIMI

• On Pawpularity, the returned candidate scores 16.85639 test RMSE and earns gold, while neither constituent earns a medal in its three goal runs.Artifact reuse improves the search trajectory, but the returned candidate is worse than the best observed RMSE of 16.77953.Source: §G.3 · p. 31, Figure 9 · p. 31

G.4 GOOGLE SMARTPHONE DECIMETER: DEEPSEEK–KIMI

• Kimi’s smartphone-positioning candidate reaches 2.77310 mean haversine error and bronze, a 42.1% reduction from the best constituent endpoint (4.78536).The late correction replaces straight-line extrapolation through an outage with a direction-matched route from public training trajectories; returning response length does not rebound.Source: §G.4 · p. 32, Figure 10 · p. 32