MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution

Motivation

• Harness engineering is manual and expert-driven, explores a combinatorial design space, and must be repeated because harnesses are model-specific.Harness design can improve performance with less compute than model training, but a harness tuned for one model may be suboptimal for another.Source: §1 · p. 2
• Prior automated search is limited by narrow mutations, failure-blind memory, fixed or only partly adaptive strategies, and accuracy-focused objectives that neglect latency and tokens.For strategy-level discovery, global structural mutations matter; each harness evaluation can take hours, so failed candidates also need to inform search.Source: §1 · p. 3

Key Insight

• MILO combines isolated islands with distinct seeds, lineage memory that retains rejected candidates, and an orchestrator that revises mutators, memory, and curriculum.Candidates are admitted only when they advance a Pareto frontier in accuracy, tokens, and latency.Source: §1 · p. 3

Contributions

• MILO formalizes harness engineering as strategy-level discovery and uses non-competing lineages to preserve diversity, prune failed patterns, and encourage structural exploration.Its orchestrator evolves the search strategy, and the paper reports state-of-the-art results on software-engineering benchmarks and open mathematical problems.Source: §1 · p. 4
• MILO outperforms eight state-of-the-art harnesses and six search methods on Terminal-Bench 2.1, PaperBench, and DeepSWE with Opus 4.8 and gpt-oss-120b.With gpt-oss-120b, it reaches 15.6% on DeepSWE, 20× its initial harness, while prior search stays below 1%.Source: §1 · p. 4

Instance-level discovery

• Instance-level search ranges from greedy refinement (Stages 0–I) to island populations (Stage II), but retains survivors rather than failed candidates and generally cannot react to stalls.Stages III–IV add online adaptation, with self-adaptive search rewriting strategy when progress stalls.Source: §2 · p. 4

Strategy-level discovery

• Strategy-level methods often tune prompts, skills, or memory, while whole-harness approaches remain at lower adaptiveness stages.These methods range from single-harness refinement to archive-based selection with hard-coded adaptation.Source: §2 · pp. 4–5

Comparison framework

• Table 1 classifies methods by discovery level and adaptiveness; stages 0–II mostly fix strategy, while online-adaptive stages III–IV are mostly instance-level.The table distinguishes fixed-rule adaptation from self-adaptation by meta-evolution.Source: Table 1 · p. 5

Summary

• Methods at both discovery levels suffer diversity decay, while strategy-level methods remain at low adaptiveness stages; MILO self-adapts more broadly than prior evolutionary search.The figure presents a general evolutionary-search skeleton, and the table classifies existing methods by adaptiveness.Source: Figure 2 · p. 4, §2 · p. 5

Agents and harnesses

• An agent pairs a model with a harness, the software layer that governs interaction with the environment by executing the model’s proposed actions.With the model fixed, the paper identifies the agent with its harness; the harness may be a program, configuration file, or tool/skill repository.Source: §3 · p. 5

Agent evaluation

• Evaluation uses benchmark tasks with initial states and task-specific verifiers, recording execution traces, accuracy, reports, and nonnegative costs such as tokens or latency.Because execution is stochastic, the paper runs multiple rollouts per task and aggregates accuracy, cost, and evidence over the benchmark.Source: §3 · p. 6
• Accuracy should be high and cost low when comparing harnesses.The evaluation aggregates both dimensions over the benchmark.Source: §3 · p. 6

Problem statement

• Given a fixed model and target task distribution, automated harness discovery seeks a replacement harness that advances the agent’s accuracy–cost frontier.The search uses a benchmark split into search and validation sets, validity and generality constraints, and a finite budget; its output is assessed on validation tasks.Source: §3 · p. 6

4.1.1 Hierarchical Lineage Memory

• Each island keeps an append-only tree of harnesses and edit histories, including rejected candidates, so mutators can learn from successful and failed changes.Nodes record evaluation and admission verdicts; edges record the source-code edit and resulting gains.Source: §4.1.1 · pp. 6–7

4.1.2 Evidence-Driven Mutator Agents

• Per-island coding agents use a parent harness, its failure evidence, and the island snapshot to make targeted fixes or structural redesigns.MILO's mutators can work over multiple turns with coding tools, unlike one-shot mutation calls described for prior methods.Source: §4.1.2 · pp. 7–8

4.1.3 Meta-Evolutionary Orchestrator Agent

• When an island stalls, the orchestrator diagnoses its bottleneck and can reassign its mutator, graft another island's harness, speciate a niche, or revise the task curriculum.Curriculum revision favors high-regret tasks, where some admitted harnesses succeed and others fail.Source: §4.1.3 · p. 8

4.2 Evolution Loop

• Each round selects a parent, mutates and evaluates a child, admits it based on fitness, records it in memory, and monitors admitted harnesses on held-out tasks.Fitness tracks accuracy and cost, while parent selection favors candidates with fewer dominators and distinct behavior.Source: §4.1.3 · p. 9, §4.2 · p. 9

• The mutator edits the parent harness source into a child that satisfies the constraints, using parent failure evidence and the island snapshot.Source: §2 · p. 9
• The child runs k rollouts on every task in the search benchmark, recording per-task accuracy, cost, and traces.Source: §2 · p. 9
• The recorded outcomes are aggregated into the harness's multi-objective fitness vector.Source: §2 · p. 9

4. Child acceptance. Child h enters L(r+1)

• A child is admitted only when its verdict indicates it enlarges the admitted set, corresponding to positive hypervolume contribution.Admitted harnesses are rescored on held-out tasks; an island's stall counter resets when a held-out gain occurs.Source: §4 · p. 10

4.3 Generalization to Other Domains

• MILO is presented as model-agnostic and domain-agnostic, with evaluation across four benchmarks and potential applicability to other solver families.Source: §4.3 · p. 10
• For instance-level discovery, MILO evolves the agent that writes optimizer programs, which are then run from supplied starting solutions and scored by normalized gain.Figure 3 contrasts prior evolutionary search, which evolves π, with MILO, which evolves Ah to generate π.Source: §4.3 · p. 10, Figure 3 · p. 10

5.1 Benchmarks and Tasks

• The benchmarks cover command-line tasks, paper replication, and multi-file software engineering, with Frontier-Bench used to test transfer without search.EinsteinArena is used for instance-level discovery on open mathematical problems.Source: §5.1 · p. 10

5.2 Evaluation Metrics

• Across k=5 or 3 attempts, pass@k counts any full solve, PR@k gives per-attempt partial credit, and RR@k counts full solves, with benchmark-specific scoring.For Terminal-Bench, Frontier-Bench, and DeepSWE, scores are fractions of tests passed; PaperBench uses its official rubric-weighted score.Source: §5.2 · pp. 10–11

5.3 State-of-the-Art Baselines and Experimental Setup

• Comparisons include eight SoTA harnesses and six search methods, run from shared expert-designed seeds under a 72-hour limit; Self-Harness, DarwinX, and AIDE2 were unavailable.The three expert-designed DeepAgents harnesses are the shared seeds, and each search reports its best candidate.Source: §5.3 · p. 11

5.4 Experimental Results

• Seven of eight SoTA harnesses underperform the minimal harness on PaperBench RR@3 with Opus 4.8, while six exceed it on DeepSWE by up to 41.3%.On TB2.1, all eight improve RR@5 over the minimal harness with Opus 4.8, but four underperform it with gpt-oss-120b.Source: §5.4 · p. 11
• +12.0%, +28.3%, and +10.3% RR@k gains over Best-of-3 on TB2.1, PB, and DeepSWE; the best prior-search gains were +4.5%, +18.3%, and 0%.With gpt-oss-120b, MILO improves PR@k by +11.8%, +8.6%, and +14.8% on the same three benchmarks.Source: §5.4 · p. 11, Table 2 · p. 12
• +3.8% RR@3 over Best-of-3 and +8.6% over Mini-SWE-Agent on harder Frontier-Bench, using the TB2.1-evolved harness without further search.The harness was evaluated unchanged on Frontier-Bench.Source: §5.4 · p. 12, Table 4 · p. 14
• After all islands stalled in rounds 6–14, Graft and Reassign interventions resumed progress; the weakest seed (39.2%) yielded the 59.9% best harness.On TB2.1 with Opus 4.8, MILO gained +12.0% RR@5 over Best-of-3 while using 0.74× its tokens.Source: §5.4 · pp. 11–12, Table 3 · p. 13
• MILO's instance-level variant set official-verified records on three EinsteinArena problems, surpassing best-known bounds, including those of AlphaEvolve, TTT-Discover, and EvoX.Table 5 evaluates MILO against state-of-the-art instance-level evolution on open mathematical problems; lower values are better.Source: §5.4 · p. 13, Table 5 · p. 14

6 Conclusion

• MILO’s island-partitioned lineage memory preserves diversity and rejected mutations, while an orchestrator adapts memory, mutators, and curriculum when progress stalls.Mutator agents use this memory for structural rewrites, evolving the search strategy alongside the harnesses.Source: §6 · p. 14
• MILO-discovered harnesses improve results across Terminal-Bench 2.1, PaperBench, and DeepSWE, transfer to Frontier-Bench without further search, and tighten three EinsteinArena bounds.The conclusion attributes gains beyond prompt or skill refinement alone to structural harness redesign and self-adaptive search.Source: §6 · p. 14

Appendix

• The appendices cover reproducibility, benchmark details, search baselines, the minimal-harness baseline, ablations, seed and evolved harnesses, prompts, and open-problem records.They also include the best harness evolved by each search method and by MILO.Source: Appendix · p. 21

• Experiments use a CPU-only x86-64 host with 192 vCPUs, 1.5 TiB of memory, and a 1 TB SSD-backed volume; model inference is remote.The host runs UNIX, Docker, and Python.Source: §A.1 · p. 21
• Models are accessed through Amazon Bedrock, and the harnesses use DeepAgents while accepting any LangChain chat model.The harness source is a single Python file; MILO requires an editable source and runnable build_agent entry point, not a particular framework.Source: §A.1 · pp. 21–22
• Tasks run in separate Docker containers; evolved harnesses stay host-side, and evaluation runs 16 tasks concurrently with no other tasks on the machine.During harness evolution, up to eight tasks run concurrently per island; DeepSWE containers are fully sealed.Source: §A.1 · pp. 21–22
• MILO uses three islands, a 72-hour search limit, and six coding-agent mutators; its orchestrator can reassign mutators and apply up to two moves per intervention.Mutations and orchestration calls are capped at 45 minutes, and the same three expert-designed harnesses initialize every compared search method.Source: §A.1 · p. 22

• At each benchmark’s strict time limit, an attempt is graded in whatever state it has reached, so partial progress is scored.Source: §A.2 · p. 22
• Terminal-Bench 2.1 has 89 command-line workflow tasks, 85 rated medium or hard, with per-task limits from 15 minutes to three hours and hidden-test scoring.Source: §A.2 · p. 22
• Frontier-Bench is the harder successor to Terminal-Bench 2.1, with 74 disjoint tasks and 30-minute-to-eight-hour limits; this evaluation uses its 70 non-GPU tasks.Source: §A.2 · p. 22
• PaperBench-CodeDev asks agents to reimplement 20 ICML 2024 Spotlight and Oral papers in 12 hours, scoring replication with an author-co-developed rubric of 8,316 binary leaves.Source: §A.2 · p. 22
• DeepSWE comprises 113 tasks across 91 repositories and five languages, with 90 minutes per task and hidden tests; its reference solutions modify 5.5× as many lines as SWE-bench Pro’s.Terminal-Bench and Frontier-Bench allow full internet access, PaperBench blocks each target paper’s code repository, and DeepSWE seals its container entirely.Source: §A.2 · p. 23

A.3 Setup of the State-of-the-Art Evolutionary Search Baselines

• Every search method gets the same three expert-designed seeds, official scorer, and 72-hour limit; single-LLM proposers use Opus 4.8, with stated exceptions for multi-model methods.Source: §A.3 · p. 23
• OpenEvolve uses three MAP-Elites islands with ring migration and full-rewrite mutations, while ShinkaEvolve uses three migrating islands and a four-model UCB bandit.OpenEvolve retains its two-model ensemble; ShinkaEvolve retains novelty rejection.Source: §A.3 · p. 23
• GEPA is run with both prompt-only and whole-harness optimization; EvoX uses a single-population loop with strategy meta-evolution.GEPA’s gpt-oss-120b reflection minibatch is increased from three to 12 tasks because the default minibatch is usually all-fail and prevents its acceptance gate from firing.Source: §A.3 · p. 23
• A-Evolve changes skill and memory files while freezing code, whereas Meta-Harness retains its released loop and keeps the best harness after each proposal.Source: §A.3 · p. 23

A.4 Minimal-Harness Baseline

• The minimal baseline is a bare model loop with read_file, write_file, and bash, without planning tools, sub-agents, skills, memory, middleware, or context summarization.The tools support reading and writing task files and executing commands; the loop repeats until the model stops calling tools.Source: §A.4 · p. 24

A.5 Search-Mechanism Ablation

• RR@5 rises 76.4 → 79.1 → 79.3 → 80.5 → 86.1 across the five configurations, with the largest gain from adding the orchestrator.The ladder progresses from an LLM mutator over flat memory through an agentic mutator, lineage memory, multiple islands, and the orchestrator.Source: §A.5 · p. 25
• At a 4K per-call output cap, Best-of-3 achieves only 34.4% RR@5, while MILO still evolves an 81.1% harness.Source: §A.5 · p. 25

B.1 Base DeepAgents Harness

• Stock DeepAgents uses defaults; its agent loop lives in LangGraph outside the harness, while editable surfaces include prompts, tools, sub-agents, skills, and long-term memory.Source: §B.1 · p. 25

B.2 Expert-Designed Harnesses (B1–B12)

• At 4K, expert harnesses B1–B12 fell to 20.2–34.4 RR@5 from 70.5–75.0 at 128K, while MILO reached 81.1 RR@5 and preserved pass@5 at 93.2.MILO’s 81.1 RR@5 was within 5.0% of its 86.1 at 128K; its pass@5 of 93.2 matched exactly across caps.Source: §B.2 · p. 26, Table 6 · p. 26

B.3 The Three Initial Harnesses for Evolutionary Search (B10–B12)

• B10 (sub-agent roles), B11 (plan-and-solve gate), and B12 (reproduce-first rubric gate) seed every evolutionary search; single-seed methods use benchmark- and backbone-specific Best-of-3.B10 delegates to explorer, coder, and verifier sub-agents with isolated contexts, and its completion gate prevents stopping before verification is dispatched.Source: §B.3 · pp. 26–27
• B12 scored 74.1 RR@5 and was the strongest Opus 4.8 Terminal-Bench 2.1 seed and the Best-of-3 seed for DeepSWE.Its reproduce-first protocol uses a rubric grader to check the transcript against completion criteria whenever the agent tries to finish.Source: §B.3 · p. 30
• The initial harness rubric requires reproducing the failure before editing, making a minimal targeted change, and rerunning the exact check to verify success.A wrapper supplies the default rubric when the caller omits one, while allowing an explicit rubric override.Source: §B.3 · p. 30

C Prompts Used by MILO

• MILO’s appendix reproduces the prompts used in its implementation, including the mutator-agent prompt.Source: C Prompts Used by MILO · p. 32, §C.1 · p. 32

C.1 Mutator-Agent Prompt

• The mutator prompt directs each round to use parent failures and population history to diagnose one cause, then choose a targeted refinement or structural redesign.It prioritizes pass rate over cost and calls for stronger architectural changes when the parent is stuck.Source: §C.1 · p. 32

C.2 Orchestrator-Agent Prompt

• At an island stall, the orchestrator diagnoses the search and selects at most two interventions; for a graft, it merges the donor’s distinct mechanism into the destination harness.The graft prompt supplies both parent harnesses, failure evidence, and the diagnosis; the resulting child faces the standard admission test.Source: §C.2 · pp. 33–34

D.1 Best Harness Evolved by Each SoTA Evolutionary Search

• +7.5 RR@5: MILO leads the compared searches by rebuilding the harness assembly, while every baseline rewords or lightly extends the seed.The four identified failure modes are premature completion, deadline overrun, lax verification, and missing know-how; MILO replaces the seed’s plan gate with an independent verifier gate, a scaffold floor, and a deadline governor.Source: §D.1 · p. 34, Table 8 · p. 35
• A rubric middleware invokes an independent grader whenever the agent would finish, returning specifics when rubric criteria remain unmet.The rubric requires visible evidence of reproduction, a targeted change, rerunning the same check, and checking requested outputs against the requirements.Source: §D.1 · p. 35
• MILO’s prompt requires agents to reproduce or pin down the task, state a hypothesis, make a minimal change, and rerun the exact check.It also says to treat errors as evidence and not declare success unless the rerun check passes.Source: §D.1 · p. 36
• OpenEvolve returns a compressed B12 with a four-step protocol, a thinking-free grader capped at two iterations, and a lightly refactored rubric-agent wrapper.The rubric-gate assembly is kept.Source: §D.1 · p. 39
• RR@5 was 77.7 for ShinkaEvolve and 76.1 for OpenEvolve, each reported as a diff versus Best-of-3 (B12).ShinkaEvolve expands the prompt protocol; OpenEvolve compresses the harness while retaining the rubric gate.Source: §D.1 · pp. 39, 41
• ShinkaEvolve grows B12’s four-step protocol into a six-step playbook and expands the rubric to five criteria, while leaving code unchanged.Its rubric instructs the grader not to bounce the agent for process or style alone.Source: §D.1 · p. 41
• The underlying scientist harness makes reproduce–hypothesize–minimal-change–re-verify the loop’s spine and uses a rubric-checking grader to demand transcript evidence before completion.The prompt and rubric reinforce one another: the grader checks the transcript whenever the agent would finish and returns specifics when criteria are unmet.Source: §D.1 · p. 41
• EvoX rewrites the prompt while leaving code untouched, reshaping the rubric, limiting feedback to one bounce, and adding a whole-task coverage rule.Its rubric uses four end-state criteria and identifies the single most important missing action for the bounce.Source: §D.1 · p. 43
• Meta-Harness makes reproduce, hypothesize, minimal change, and re-verify the solver protocol, with an independent rubric grader checking transcript evidence before completion.The grader bounces unmet criteria with specifics; the rubric is injected per invocation through a wrapper.Source: §D.1 · p. 43
• Its rubric prioritizes verified end state and explicit sub-requirements, passing when these are met rather than bouncing solely for process or style.A failure gets only one more agent cycle, so the grader should name the single most important concrete action still needed.Source: §D.1 · p. 44
• RR@5 was 78.2 for Meta-Harness and 76.8 for EvoX, each reported as a diff vs. Best-of-3 (B12, Listing 5).The Meta-Harness score is reported in Listing 13; the EvoX score is reported in Listing 12.Source: §D.1 · pp. 43, 45

• Meta-Harness adds per-call elapsed-time readouts and escalating convergence guidance to B12, while inheriting its protocol, rubric gate, and grader unchanged.The guidance encourages banking a verified passing state before refining the solution.Source: §D.1 · p. 45
• All 25 failures were timeouts; 20 ended before a passing state, while five were genuine wrong answers, making deadline loss the target failure mode.The solver lacked elapsed-time feedback and could continue exploring or verifying until its deadline, before banking a passing state.Source: §D.1 · p. 45
• Pacing escalations apply only until the primary requirement is verified and use elapsed-time thresholds that scale with task budgets when those budgets are available.The clock starts at the first model call and uses a monotonic timer.Source: §D.1 · p. 45

D.2 Best Harness Evolved by MILO

• RR@5 = 86.1: MILO’s best Terminal-Bench 2.1 harness with Opus 4.8 uses general search and verification machinery, with no task-specific logic.The listing presents the harness as a diff against its island’s seed.Source: §D.2 · pp. 47–48
• The harness rewrites the prompt into floor, clock, and verifier phases, combining deadline triage with an independent-verifier gate that relaxes near the deadline.It also adds anti-stub checks; the verifier gate requires fresh adversarial sign-off before stopping, except near the deadline.Source: §D.2 · pp. 47–48
• TIMEOUT-ZEROS, not semantic rubber-stamping, dominated residual losses: at least 14 of 21 failed tasks hit exact agent_seconds caps with 0 tests passing.The harness targets these failures because correctly shaped deliverables can earn partial credit even before their hard logic is correct.Source: §D.2 · p. 48
• A one-shot scaffold floor creates a minimal valid version of every deliverable before deep work, so a timeout can earn partial credit instead of leaving an empty graded state.The directive fires on the first model call; the deadline governor then keeps the artifact saved and helps land the run near its deadline.Source: §D.2 · p. 48
• The harness floors a minimal valid version of every required deliverable early and keeps it valid while incrementally improving it, protecting against timeout with no scoreable artifact.It calls for confirming each deliverable’s path, format, and structure with a concrete command.Source: §D.2 · p. 51
• It requires an exhaustive plan based on the task’s exact end state, followed by stepwise execution with concrete checks and plan revisions when results contradict expectations.The todo list is updated as work proceeds, and each intermediate result is checked before moving on.Source: §D.2 · pp. 50–51
• Before finishing, the lead must dispatch a fresh-context verifier to independently re-derive requirements and try to falsify the solution, reporting discrepancies with concrete evidence.The verifier checks actual outputs against task semantics and gives a SATISFIED verdict only after independently checking every requirement; otherwise it reports NOT SATISFIED or unresolved checks.Source: §D.2 · p. 51
• Verification is mandatory while time allows, but the harness warns against starting a long re-verification pass near the deadline and requires a final check that the deliverable is in place.The prompt also instructs agents to heed wall-clock notes, stop exploring as time runs low, and prioritize a correct verified result over risking a timeout.Source: §D.2 · p. 51

E.1 Records against the Best Known Bounds

• Each task pairs a starting construction with an optimizer-writing agent under a 60 s budget; trusted subprocesses and the arena verifier validate gains across a 30-task bank.The arena verifier reproduced leaderboard scores to the last digit when checked on two dates.Source: §E.1 · p. 52
• Three EinsteinArena problems receive MILO records below prior bests, with every margin exceeding the arena’s minimum for a new #1.All three are minimization problems, so lower scores are better.Source: Table 9 · p. 52, §E.1 · p. 52
• Tab. 5 lists TTT-Discover’s Erdős score of 0.3808753 and first-autocorrelation score of 1.5028629, plus EvoX’s third-autocorrelation score of 1.4558.EvoX’s published discretized objective matches the arena’s.Source: §E.1 · p. 52

• AlphaEvolve’s final program is shown for the first autocorrelation inequality; comparisons for the other two problems are at the construction level because optimizer code is unavailable.For those problems, the cited passage says AlphaEvolve and TTT-Discover publish constructions, EvoX a one-sentence description, and arena leaders are anonymous.Source: §E.2 · p. 52
• MILO improves Erdős minimum overlap from 0.3808586 to 0.3808568, the first autocorrelation inequality from 1.50274365 to 1.50274360, and the third from 1.4508066 to 1.4488860.Source: §E.2 · pp. 53–55
• For Erdős minimum overlap, the 512-grid-point optimizer combines an exact-verifier ratchet with softmax-weighted FFT-gradient momentum, followed by FISTA and Newton stages.The FISTA ladder and equioscillation Newton stage are not shown in the excerpt.Source: §E.2 · p. 53
• For the first autocorrelation inequality, the 65,536-grid-point optimizer uses annealed logsum-exp Gauss–Newton steps, FFT derivatives, truncated conjugate gradients, and backtracking acceptance.Rejected steps advance the annealing schedule faster.Source: §E.2 · p. 54
• For the third autocorrelation inequality, the 25,600-grid-point optimizer combines a log-sum-exp objective and FFT gradient with a fixed-iteration melt ladder and warm-floor hops that re-melt and publish improvements.Source: §E.2 · p. 55

E.2 Best Optimizers Discovered by MILO

• The optimizer geometrically cools from a hot to a cold temperature across stages, after validating that the temperatures are positive and ordered.The stage-to-stage ratio is (t_cold / t_hot)^(1 / (stages - 1)).Source: §E.2 · p. 55
• The search checks deadlines and remaining budget, and publishes the ratchet’s best point when its score improves over the prior score.The run path receives a deadline, and the publishing branch is conditional on the best score improving.Source: §E.2 · p. 55
• After many unsuccessful hops with little score gain, the optimizer raises the melt temperature because this family’s dead region is at the mild end.The recovery branch checks the score gap and failure count, then increases heat when it is not already sufficiently hotter than the warm temperature.Source: §E.2 · p. 55
• To prevent deterministic hops from repeatedly landing at the same point, the optimizer adds random multiplicative jitter whose amplitude scales with melt strength.The comments report that repeated landings caused chains to stall after 50 consecutive dead hops.Source: §E.2 · p. 55