PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale

Xinyu Zhu, Yuzhu Cai, Zexi Liu, Cheng Wang, Fengyang Li, Wenkai Jin, Wanxu Liu, Zehao Bing, Bingyang Zheng, Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xianghe Pang, Yaxin Du, Tingjia Miao, Yuzhi Zhang, Ruoxue Liao, Zhaohan Ding, Linfeng Zhang, Yanfeng Wang, Weinan E, Siheng Chen

arXiv:2604.17406v4cs.AI

TL;DR

Existing agent frameworks are largely static and single-pass, unlike science’s iterative trial-and-error process. EvoMaster provides a foundational, self-evolving harness and scores best on nine of ten benchmarks, with the highest average among four agents.

  • Problem

    Existing agent frameworks predominantly use single-pass execution and lack mechanisms to learn from failures or accumulate insights across attempts, despite scientific discovery’s iterative, long-horizon nature.

    Source: §1.2 · p. 3

  • Method

    EvoMaster combines a foundational harness for diverse disciplines with continuous self-evolution, enabling agents to iteratively refine strategies and accumulate experience.

    Source: §1.3 · p. 3

  • Results

    EvoMaster scored best on nine of ten benchmarks and averaged 58.02%, compared with Codex at 40.29%, OpenHands at 39.84%, and OpenClaw at 32.39%.

    Source: §6.3 · p. 10, §7 · p. 17

  • Takeaways & Limitations

    The evaluation spans varied scientific workflows, while the SciMaster ecosystem’s incubation supports EvoMaster’s cross-disciplinary scalability.

    Source: §7 · p. 17

    EvoMaster is primarily optimized for in silico and computational workflows and lacks native support for tasks requiring direct manipulation of physical experimental apparatus.

    Source: Limitations · p. 17

Abstract

from arXiv · show

The convergence of large language models and agents is catalyzing a new era of scientific discovery: Agentic Science. While the scientific method is inherently iterative, existing agent frameworks are predominantly static, narrowly scoped, and lack the capacity to learn from trial and error. To bridge this gap, we present EvoMaster, a foundational evolving agent framework engineered specifically for Agentic Science at Scale. Driven by the core principle of continuous self-evolution, EvoMaster empowers agents to iteratively refine hypotheses, self-critique, and progressively accumulate knowledge across experimental cycles, faithfully mirroring human scientific inquiry. Crucially, as a domain-agnostic base harness, EvoMaster is exceptionally easy to scale up -- enabling developers to build and deploy highly capable, self-evolving scientific agents for arbitrary disciplines in approximately 100 lines of code. Built upon EvoMaster, we incubated the SciMaster ecosystem across domains such as machine learning, physics, biology, web research, and general science. Evaluations on ten benchmarks spanning scientific research/coding/experimentation, scientific reasoning and information search, and practical scientific problem solving compare EvoMaster against OpenHands, OpenClaw, and Codex. EvoMaster achieves the highest score on nine of the ten benchmarks and the strongest average score (58.0\%) among the four agents, validating its efficacy and generality as the premier foundational framework for the next generation of autonomous scientific discovery.

1. Introduction … 3. Design Principles

EvoMaster is presented as a foundational framework for scaling scientific agents across disciplines and enabling them to improve through iterative experimentation. Its architecture combines reusable, experiment-ready components with self-evolution and multi-agent collaboration; evaluations report top performance on nine of ten benchmarks.

1. Introduction

  • Agentic Science extends AI from answering human-posed questions to conducting research cycles, and scaling it across disciplines could reduce discovery’s dependence on human bandwidth.The envisioned cycle includes literature review, hypothesis generation, experimental design, code execution, result analysis, and paper writing.Source: §1.1 · p. 2
  • EvoMaster addresses domain silos that duplicate harness engineering and stateless agents that cannot learn from failures or accumulate insights across research iterations.These limitations make supporting new disciplines costly and hinder work on complex, open-ended scientific problems.Source: §1.2 · pp. 2–3

1. The EvoMaster foundational agent framework.

  • EvoMaster provides a shared foundational harness and continuous evolution, enabling developers to build domain agents in approximately 100 lines of code and invoke the framework as a skill.Its design translates these aims into modular composability, an experiment-ready harness, iterative self-evolution, and multi-agent collaborative evolution.Source: §1.3 · p. 3, §1 · p. 4

2. The SciMaster agent ecosystem.

  • The SciMaster ecosystem comprises 6+ agents spanning machine learning, general scientific research, web retrieval, physics reasoning, and embodied intelligence.The shared harness is intended to support rapid expansion into additional scientific domains.Source: §1.3 · p. 3, §2 · p. 4

3. Comprehensive experimental validation.

  • EvoMaster scores best on nine of ten benchmarks and averages 58.0% across the suite; it also progressively improves on long-horizon MLE-Bench.The evaluations compare EvoMaster with OpenHands, OpenClaw, and Codex across ten benchmarks; Figure 2 illustrates MLE-Bench performance over time.Source: §1.3 · p. 4, Figure 2 · p. 4, §3 · p. 5

2. Related Work

  • Unlike general or software-focused frameworks and end-to-end scientific agents, EvoMaster serves as a reusable framework layer for scientific workflows and cross-domain adaptation.It abstracts common tool and harness management while supporting agents that iterate through experiments.Source: §2.1 · p. 5, §2.2 · p. 5

3. Design Principles

  • Modular components across Playground, Experiment, and Agent layers, together with registries and MCP and Skill standards, support swapping models, environments, and tools.YAML manifests configure parameters, prompts, and tools; structured trajectory logs record turns, tool calls, and token statistics for auditability and reproducibility.Source: §3.1 · p. 6, §3.2 · p. 6
  • Reactive multi-turn loops let agents use tools, observe outcomes, and self-critique, while context summarization and compression sustain long horizons and dynamic agent teams support collaborative problem-solving.The context manager is designed to prevent degradation over hundreds of experimental turns; specialized agents can be coordinated through declaratively defined Agent Slots.Source: §3.3 · p. 6, §3.4 · p. 6

4. Architecture · 5. The SciMaster Agent Ecosystem

EvoMaster combines decoupled execution and capability layers with an engine and orchestration designed for continuous evolution, enabling reusable scientific agents. The SciMaster ecosystem applies this shared harness across scientific domains, reducing development effort and enabling shared upgrades and cross-domain tool use.

4. Architecture

  • EvoMaster’s architecture combines decoupled execution and capability layers with continuous-evolution mechanisms to support reusable scientific agents across disciplines.The framework is designed to align structural solutions with research bottlenecks while keeping deployment overhead minimal.Source: §4 · pp. 6–7

4.1. Execution Architecture: Scaling Through Decoupling

  • The execution architecture separates orchestration, experiment lifecycle management, and iterative agent reasoning into three layers.Adding a discipline requires defining a new Playground, while foundational improvements can benefit all scientific domains.Source: §4.1 · pp. 7–8

4.2. Agent Engine: The Evolution Core

  • The Agent Engine runs a multi-turn reason–tool-use–observation–self-critique loop and uses context management to support long experimental horizons.Dynamic LLM-based summarization and sliding windows help prevent context degradation without exceeding context limits.Source: §4.2 · p. 8

4.3. Capability Layer: Universal Tools and Skills

  • The capability layer provides universal interfaces for scientific tools, hierarchical skills, and interchangeable language models.MCP-compatible tools integrate through an Action-Execution-Observation pattern; skills follow emerging specifications, and the unified LLM interface supports 100+ models.Source: §4.3 · p. 8

4.4. Playground Orchestrator: Collaborative Evolution

  • The Playground orchestrator assigns specialized agent roles and supports sequential handoffs, parallel exploration, and iterative peer review.The AgentSlots mechanism gives roles independent LLM and tool configurations, while custom workflows are discovered at runtime; complex teams take approximately 100 lines of orchestration code.Source: §4.4 · p. 8

4.5. Experiment Harness: Rigor and Reproducibility

  • The experiment harness makes agent runs reproducible and auditable through YAML configuration manifests and structured trajectory logs.The logs record multi-turn dialogues, tool invocations, and token statistics in thread-safe JSON.Source: §4.5 · p. 9

5. The SciMaster Agent Ecosystem

  • The SciMaster ecosystem extends EvoMaster across scientific domains and reduces specialized-agent development from several days to mere hours.Shared engine and orchestration upgrades, reduced harness coding, and cross-domain tool interoperability let researchers focus on domain-specific logic; the ecosystem is still expanding.Source: §5 · p. 9

6. Experiments

Across ten benchmarks, EvoMaster leads on nine and has the highest average score; results cover scientific research and coding, reasoning and information search, and practical scientific problem solving.

6.1. Benchmarks

  • The ten benchmarks span research, coding, and experimentation; scientific reasoning and information search; and practical scientific problem solving.The suite includes PaperBench (CodeDev), PostTrainBench, MLE-Bench (Lite), Frontier Science, BrowseComp, BrowseComp-ZH, HLE, PRL-Bench, Combinatorial Construction, and BiomniBench-DA.Source: §6.1 · p. 10

6.2. Experimental Setup

  • All agents use GPT-5.4 with medium reasoning effort, benchmark-specific time limits, scoring scripts, and task splits.Where permitted, agents share external tools and retrieval channels; MLE-Bench has a strict 24-hour limit. EvoMaster uses benchmark-tailored SciMaster agents, including X-Master, ML-Master, Browse-Master, PhysMaster, and a minimal agent for several benchmarks.Source: §6.2 · p. 10

6.3. Results and Analysis

  • EvoMaster leads on nine of ten benchmarks and averages 58.02%, ahead of Codex (40.29%), OpenHands (39.84%), and OpenClaw (32.39%).Codex leads on PostTrainBench.Source: §6.3 · p. 10, Table 2 · p. 11

6.3.1. Scientific research, coding, and experimentation.

  • EvoMaster scores 74.40 on PaperBench (CodeDev) and 13.83 on PostTrainBench, while Codex leads PostTrainBench at 15.92.On CodeDev, EvoMaster’s 25.33 gain over Codex accompanies a loop of paper-specific planning, repository construction, static auditing, and judge-guided repair. On PostTrainBench, iterative data preparation, proxy evaluation, and checkpoint promotion help, though results remain sensitive to dataset choice, answer formatting, and final-model validation.Source: §6.3.1 · p. 11, Table 3 · p. 12, Table 4 · p. 13
  • EvoMaster reaches a 75.76% any-medal rate on MLE-Bench (Lite), above MLE-STAR and R&D-Agent (both 68.18%).It also records a 100.00% valid-submission rate, an 84.85% above-median rate, and a 31.82% silver-medal rate under the 24-hour budget; its loop uses executable drafts, validation feedback, debugging, and research-plan refinement.Source: §6.3.1 · pp. 11–12, Table 5 · p. 13

6.3.2. Scientific reasoning and information search.

  • EvoMaster leads all three baselines on Frontier Science (55.00%), BrowseComp (74.25%), BrowseComp-ZH (73.36%), and HLE (41.10%).Its browsing process preserves intermediate evidence, revises search directions, and synthesizes information across sources; the HLE SOLVE–CRITIQUE–REWRITE–SELECT pipeline supports iterative multi-agent refinement and correction of initial errors.Source: §6.3.2 · pp. 12–14

6.3.3. Practical scientific problem solving.

  • EvoMaster tops PRL-Bench (51.08%), Combinatorial Construction (56.04%), and BiomniBench-DA (65.36%).It leads four of six PRL-Bench subfields, reaches 67.95% in quantum information, and scores 80.00% in BiomniBench-DA immunology; Codex leads some physics and biomedical domains, and margins are smaller on practical tasks. Combinatorial Construction’s benchmark score is the Sum-Difference score normalized by the reference upper bound 2.Source: §6.3.3 · pp. 14–16

7. Conclusion · A. Benchmark Descriptions

EvoMaster combines modularity and continuous self-evolution to support scientific agents, and its evaluation reports the best score on nine of ten benchmarks and the highest average among four agents. The appendix describes benchmarks spanning research workflows, scientific reasoning and information search, and practical scientific problem solving.

7. Conclusion

  • EvoMaster combines foundational modularity and continuous self-evolution with iterative refinement, experiment-ready harnesses, and multi-agent workflows.These mechanisms support agents in hypothesizing, experimenting, self-critiquing, and evolving from trial and error.Source: §7 · p. 17
  • Across ten benchmarks using GPT-5.4 (medium), EvoMaster scores best on nine and has the highest average among EvoMaster, OpenHands, OpenClaw, and Codex.The suite spans research coding and experimentation, scientific reasoning and information search, and practical scientific problem solving.Source: §7 · p. 17
  • The authors envision EvoMaster as a substrate for autonomous scientific discovery, expanding to disciplines such as biochemistry, materials science, and complex physical systems.Their stated goal is to shift the bottleneck of scientific progress from human bandwidth to scalable, self-evolving artificial intelligence.Source: §7 · p. 17

Limitations

  • EvoMaster is primarily optimized for in silico and computational research, without native support for direct manipulation of physical experimental apparatuses.Future work could extend the Session abstraction to interface with laboratory automation protocols.Source: Limitations · p. 17

A. Benchmark Descriptions

  • PaperBench CodeDev [34], PostTrainBench [32], and MLE-Bench [9] assess research-code implementation, bounded-compute model post-training, and end-to-end machine-learning competition workflows.They respectively focus on Code Development rubric nodes, iterative experimental design, and the overall average any medal rate.Source: A. Benchmark Descriptions · p. 18
  • BrowseComp [41], BrowseComp-ZH [46], Frontier Science [38], and Humanity’s Last Exam [30] cover complex web retrieval, Chinese-web multi-hop search, frontier science reasoning, and broad closed-book knowledge.Frontier Science includes Olympiad and research tracks; Humanity’s Last Exam spans academic disciplines and includes multimodal questions.Source: A. Benchmark Descriptions · pp. 18–19
  • PRL-Bench [24], Combinatorial Construction [44], and BiomniBench-DA assess theoretical and computational physics, evaluator-driven mathematical construction, and biomedical data-analysis trajectories.The public BiomniBench-DA release used in the experiments contains 50 tasks, scored against expert-designed rubrics.Source: A. Benchmark Descriptions · p. 19

B. Implementation Details

EvoMaster is configured differently across benchmark settings, with task-specific workflows, tools, resource budgets, and scoring protocols. The setups range from repository construction and competition submissions to scientific reasoning and practical problem solving.

PaperBench (CodeDev)

  • PaperBench CodeDev evaluates 20 isolated implementation tasks with the rubric hidden, iterative repository-continuation rounds, a quality gate, and code-only scoring.Each task runs in a Docker container, and missing or unevaluable submissions count as failures.Source: PaperBench (CodeDev) · pp. 19–20
  • Local experiments allocate each task an A800 GPU, 16 CPU cores, and 10 hours; official evaluate.py scripts score submissions, with GPT-5.4 judging when GPT-5-mini is unavailable.The minimal agent uses GPT-5.4 with medium reasoning effort and supports file editing, shell execution, and Python execution.Source: PaperBench (CodeDev) · p. 20
  • MLE-Bench Lite uses 22 low-complexity tasks, the official offline environment, two RTX 4090 GPUs per task, and the official 24-hour runtime budget.The default ML-Master 2.0 workflow allows up to 20 research rounds; official grading checks submission schemas, while private labels and scores remain hidden.Source: PaperBench (CodeDev) · p. 20
  • BrowseComp uses a single GPT-5.4 agent with Serper search and Jina web retrieval, capped at 100 turns; GPT-5 judges answers as correct or incorrect.The same configuration and workflow are used for another task whose name is not specified in the supplied passage.Source: PaperBench (CodeDev) · p. 20
  • Frontier Science’s 60-task Research Track uses GPT-5.4 with web and scholarly retrieval, local computation, and file access; GPT-5 judges solutions against task-specific rubrics.Tasks span physics, chemistry, and biology, and missing or invalid solution files count as failed executions.Source: PaperBench (CodeDev) · pp. 20–21
  • Humanity’s Last Exam uses a four-stage solver-critic-rewriter-selector pipeline with five parallel agents per stage, and GPT-5 scores final answers against gold answers.The local-session workflow uses GPT-5.4 agents and enables web search, file inspection, and shell execution.Source: PaperBench (CodeDev) · p. 21
  • PRL-Bench uses an offline, MCTS-driven Phys-Master workflow with two parallel workers and up to five rounds; Combinatorial Construction searches four SimpleTES task directories.PRL-Bench answers are judged against original rubrics; construction runs use local deterministic evaluators and return the highest-scoring valid construction.Source: PaperBench (CodeDev) · p. 21
  • BiomniBench-DA evaluates 50 public tasks in independent workspaces, requiring trace.md and answer.txt; its process-level judge scores expert-rubric criteria and averages task scores.GPT-5.4 runs with medium reasoning effort and a 150-turn limit; failed evaluations receive score 0.Source: PaperBench (CodeDev) · p. 22

C. Combinatorial Construction Case Studies … 4. Long-run search after plateaus.

These case studies show EvoMaster improving executable combinatorial constructions through evaluator-checked search, retained high-scoring candidates, and continued exploration after plateaus. The supplied examples cover a sum-difference set and a discrete Hadamard determinant task.

C. Combinatorial Construction Case Studies

  • The appendix presents two evaluator-driven case studies: improving a finite integer set under exact recomputation and iteratively repairing discrete matrix constructions.Aggregate Combinatorial Construction results are reported in the main experimental discussion.Source: C. Combinatorial Construction Case… · p. 22

1. Respect the exact evaluator contract before optimizing.

  • The sum-difference evaluator recomputes |A + A|, |A − A|, and C(A) from the submitted set, preventing credit for an inflated reported exponent.The task requires a finite integer set with 2 ≤ |A| ≤ 512 and entries in [−106, 106].Source: C. Combinatorial Construction Case… · p. 22

2. Use scored candidates as construction memory.

  • EvoMaster’s first major improvement raises C(A) from 1.0597930945 to 1.0734301121 at generation 33, then reaches 1.0772571565 at generation 289.The search repeatedly samples high-scoring inspirations, mutates the set, and preserves strong constructions for further variants.Source: §2 · p. 23

3. Continue after apparent plateaus.

  • After many non-improving valid candidates, the best-so-far archive preserves progress as C(A) rises to 1.0807374883, 1.0869151792, and 1.0959684666.These improvements occur at generations 609, 625, and 806, respectively.Source: §3 · p. 23

4. Long-run search after plateaus.

  • The representative trajectory continues past generation 800 and ends at C(A) = 1.1057713861 after three late best-score improvements.The late scores are 1.0998382358 at generation 870, 1.1032786881 at generation 886, and 1.1057713861 at generation 918.Source: §4 · p. 23
  • 56.04% is the normalized benchmark score for the final GPT-5.4-medium EvoMaster result, which reaches C(A) = 1.1207 after rounding.The benchmark result uses the sum-difference set task.Source: C. Combinatorial Construction Case… · p. 22, §4 · p. 23

C. Combinatorial Construction Case Studies

  • The search starts with an executable contract and valid construction, then maintains an archive, mutates constructions, rejects non-improving candidates, and searches beyond long plateaus.The initial construction has C(A) = 1.0597930945.Source: §1 · pp. 22–23, §4 · p. 23

4. Long-run search after plateaus.

  • The Hadamard case asks for a 29 × 29 matrix with entries in {±1} that maximizes |det(H)|, with invalid or numerically unstable candidates receiving no credit.The evaluator checks entry constraints and computes a determinant-based ratio.Source: §4 · p. 23

1. Start from a valid but weak discrete matrix. … 5. Probe tools, then adapt.

The Hadamard construction evolves from a valid but weak matrix to a determinant ratio of 0.9210526316; a separate BiomniBench-DA case illustrates iterative single-cell analysis and receives a judge score of 93/100.

1. Start from a valid but weak discrete matrix; 2. Replace ad hoc generation with determinant-aware search.

  • 0.1432748538 was the initial valid determinant ratio; determinant-aware evaluation and local flip search raised it to 0.5504599401 by generation 25.Intermediate ratios were 0.3288543767 at generation 3 and 0.4623936618 at generation 8.Source: §1 · p. 23, §2 · p. 23

3. Use evaluator feedback to switch construction families; 4. Consolidate the best valid matrix.

  • After early local gains plateaued, stronger structured matrices lifted the ratio to 0.6482062272, 0.7594555052, and 0.8596491228 before a later candidate reached 0.9210526316.The archive preserved valid improvements even as many later candidates scored below the current best.Source: §3 · p. 23, §4 · pp. 23–24

4. Consolidate the best valid matrix.

  • 0.9211 was reached by EvoMaster, Codex, and OpenClaw on the auxiliary task, while OpenHands reached 0.8596.The representative Hadamard trajectory reached 0.9210526316, matching the best GPT-5.4-medium score reported for this task.Source: §4 · p. 24
  • The construction loop used evaluator feedback to replace weak families, preserve high-determinant valid matrices, and promote the best validated artifact instead of the latest generated code.The case highlights avoiding regressions from invalid or numerically unstable candidates.Source: §4 · p. 24

D. BiomniBench-DA Case Studies

  • In a celiac-disease scRNA-seq communication analysis, EvoMaster-Bio inferred ligand-receptor communication between specified T-cell populations, and the benchmark judge scored the answer 93/100.The 46-step, 51-tool-call case received full-credit assessments for QC, clustering, marker validation, ligand-receptor scoring, and mechanistic interpretation.Source: D. BiomniBench-DA Case Studies · p. 24

1. Inspect task and raw matrices; 2. Convert inspection into an explicit plan; 3. First executable experiment: load, QC, and all-cell cluster.

  • The agent inspected ten metadata-free matrices, planned a staged Scanpy workflow, and executed QC and clustering, reducing 72,177 cells to 67,284 and identifying 11 clusters.The workflow proceeded from raw-data inspection to processing all matrices, annotating major cell types, and planning T-cell subclustering.Source: §1 · p. 24, §2 · p. 24, §3 · p. 24

4. Self-correction: all-cell clusters were too coarse for the question; 5. Probe tools, then adapt.

  • When all-cell clusters lacked resolution for the target populations, the agent narrowed analysis to T cells and switched from unreliable online OmniPath access to available LIANA resources and a curated pathway set.T-cell subclustering produced clusters 0–16, with disease-skewed cytotoxic populations appearing.Source: §4 · pp. 24–25, §5 · p. 25

6. Refine the sender population. … 2. Identity verification before answering.

Across the cases, EvoMaster refined its analyses by revising cohorts, features, and experimental choices in response to observations. Its final sepsis clustering identified two meta-clusters, while its directional ligand-receptor analysis highlighted predicted pathways beyond MHC-I.

6. Refine the sender population.

  • The sender analysis required conventional CD8/TCRαβ identity, finding 465 KIR+ conventional CD8 cells, including 255 CeD and 210 HC cells.This excluded gamma-delta-like cytotoxic cells that also expressed KIR transcripts; t_leiden 14 was 50.6% KIR+ conventional CD8.Source: §6 · p. 25

7. Refine the receiver population.

  • Without tetramer labels, the receiver analysis rejected the FOXP3+ regulatory cluster and selected a CeD-enriched activated CD4 proxy.cd4_leiden 2 had 1,317 CeD versus 296 HC cells and expressed CD40LG, IL32, TIGIT, TNFRSF4, and HLA-DP+.Source: §7 · p. 25

8. Final experiment: directional ligand-receptor scoring.

  • Beyond MHC-I, the strongest predicted pathways were CCL5→CCR4, LGALS9→CD44, TNFSF14/LIGHT→TNFRSF14/HVEM, IFNG→IFNGR1/2, and FASLG→FAS.Scoring compared KIR+CD8→pathogenic CD4 against a KIR−CD8 background, including CCL5-CCR4 score 0.0114 ratio 3.17 and IFNG-IFNGR2 ratio 16.60.Source: §8 · p. 25

1. Full schema audit.

  • The sepsis task clustered continuous endotyping signatures, but the audited table had 3,948 rows and 3,218 patients, with repeated rows complicating patient-level analysis.The baseline_infected subset had 2,214 rows, 2,203 unique patients, and 11 duplicate baseline patients.Source: Case 2: Correlation-Based Sepsis… · p. 25, §1 · p. 25, §2 · p. 25

3. Initial clustering exposes the need for refinement.

  • All-row clustering mixed controls and follow-up samples, prompting baseline-infected patient filtering, duplicate-row averaging, and removal of redundant score transforms.The refined cohort had n=2,203 and k=2 silhouette 0.536203; all-row sensitivity gave k=2 silhouette 0.514525 with the same broad split.Source: §3 · p. 26, §4 · p. 26, §5 · p. 26

6. Final clustering experiment plus sensitivity check.

  • The sepsis analysis identified inflammatory/detrimental/SRS-like myeloid-dysregulation and adaptive/protective meta-clusters, with k=2 silhouette 0.536 on the refined cohort.Strong concordant pairs included mod4_score–lymphoid_protective_score (ρ=0.960), detrimental_score–myeloid_detrimental_score (ρ=0.946), and davenport_SRSq–cano_SRSq (ρ=0.928); the judge awarded 100/100.Source: §6 · p. 26

E. BrowseComp Case Studies

  • In the BrowseComp case study, EvoMaster identified Ken Walibora in nine steps but withheld its answer until the probation-officer years were verified.The initial broad search found the candidate but left the requested years unverified; the appendix presents qualitative trajectories rather than aggregate benchmark results.Source: E. BrowseComp Case Studies · p. 26, §1 · p. 27

2. Identity verification before answering.

  • Fetching the 2018 interview verified the distinctive “no sell by date” clue and strengthened the identification of Ken Walibora, while the employment years remained unverified.The agent fetched the interview page instead of relying on search snippets.Source: §2 · p. 27

3. Cross-check identity clues and expose conflicting dates. … 7. Final missing clue.

Across the cases, EvoMaster kept candidate hypotheses open, tracked missing evidence, and rejected incomplete matches rather than finalizing from a single clue. It finalized only after tool-based checks resolved the remaining constraints.

3. Cross-check identity clues and expose conflicting dates; 5. Conflict-focused search.

  • Conflicting secondary-source date ranges prompted a targeted CV search, whose snippets suggested that teaching and probation periods had been swapped.The conflicting probation ranges were 1988–1996 and 1985–1986; the CV snippets separately listed teaching in 1985–86.Source: §3 · p. 27, §5 · p. 27

4. Use evidence to separate identity facts from answer facts.

  • Ohio State and Riara University sources confirmed Walibora’s death and lecturer role, but the fetched pages did not establish the probation years.One fetched page gave 1988–1996, which remained insufficiently authoritative for the exact range.Source: §4 · p. 27

6. Final verification from the CV.

  • The CV resolved the date conflict: probation work was 1988–1996, while high-school teaching was 1985–1986.The CV lists the two roles separately.Source: §6 · p. 27

1. Noisy broad search.

  • A broad search across all four Lillian Karabaic clues returned noisy results, so EvoMaster kept the answer open and pivoted to more discriminative clues.Search results included spam-like pages and benchmark mirrors without reliably identifying a person.Source: §1 · p. 28

2. False lead from the finance-column clue; 3. Reject the incomplete candidate.

  • Martha Harris Myron matched the 2023 finance-column clue but lacked support for the other clues, so EvoMaster rejected her as incomplete.Follow-up searches found no reliable links to the radio resignation, bike ride, or forum presentation.Source: §2 · p. 28, §3 · p. 28

4. Second partial lead and rejection; 5. Candidate convergence; 6. Batch verification across independent sources.

  • JJ Yore fit a radio-management clue but not the bike-ride or Pay Dirt clues; searches then made Lillian Karabaic the leading candidate for source verification.Batch fetching verified three clues: the 2008 ride, June 2020 XRAY.FM finance-manager resignation, and March 2023 final Pay Dirt column.Source: §4 · p. 28, §5 · p. 28, §6 · p. 28

7. Final missing clue.

  • Lillian Karabaic was the verified answer: sources confirmed her 2008 Bowie Vs Prince ride, 2020 XRAY.FM resignation, 2023 final Pay Dirt column, and 2014 forum presentation.The speaking page confirmed the final missing clue, the 2014 Women’s Bicycling Forum presentation.Source: §6 · p. 28, §7 · pp. 28–29
  • The case trajectory made the answer inspectable: EvoMaster recorded hypotheses, evidence IDs, and information gaps, then finalized after tool-based verification.The report states all four constraints were verified before returning the full name.Source: §7 · p. 29

F. Frontier Science Case Studies … 4. Let the assay table, not memory, drive the structure–activity relationships.

The appendix presents two successful Frontier Science trajectories: iterative evidence repair for a Super-Kamiokande neutrino-burst problem and evidence-adaptive synthesis planning and SAR analysis for NS9283 analogues. It illustrates self-evolution through source checking, targeted extraction, and refinement from supplied assay data; aggregate benchmark evaluation is reported in the main experimental section.

1. Start broad, but use the quantitative variables explicitly requested by the problem; 2. Use the historical paper for context, then realize it is insufficient.

  • The agent searched for the prompt’s exact numerical hooks, then used the 2007 burst paper for context but found it insufficient for newer efficiency and timing details.The initial search targeted the trigger, efficiencies, cluster timing, and background terms rather than relying on general supernova physics.Source: §1 · p. 29, §2 · p. 29

3. Reject an incorrect retrieval after title inspection.

  • After a plausible arXiv identifier led to an unrelated tensor-network paper, the agent checked the title, discarded that source, and found the Super-Kamiokande IV paper by exact title.The recovered source was identified as the paper on Super-Kamiokande IV supernova bursts.Source: §3 · p. 29

4. Extract the quantitative answer from targeted PDF reads.

  • The targeted PDF read identified a single-event trigger of 47 PMT hits in 200 ns and the requested efficiencies of 54% above 5.5 MeV and 99% above 8.5 MeV.This goal-directed extraction followed a shift from broad retrieval to searching for the exact trigger and efficiency values.Source: §4 · pp. 29–30

5. Repair a missing detail through a narrower search.

  • A failed keyword search and noisy web results prompted a narrower PDF search for spallation and direct inspection of the cluster-search pages, recovering the >50-μs time cut.The relevant table also listed burst-search cluster windows.Source: §5 · p. 30

6. Convert evidence into a requirement-coverage checklist; F. Frontier Science Case Studies

  • Before writing the neutrino answer, the agent enumerated all seven requested slots and checked coverage of trigger, efficiencies, cluster scale, scaling, backgrounds, and event separation.The case-study overview contrasts this iterative evidence repair with chemistry’s evidence-strategy adaptation and data-driven hypothesis refinement; aggregate benchmark results remain in the main experimental section.Source: F. Frontier Science Case Studies · p. 29, §6 · p. 30

6. Convert evidence into a requirement-coverage checklist.

  • 54% efficiency above 5.5 MeV and 99% above 8.5 MeV; the final Super-Kamiokande answer also gave a ≥47-PMT-hit, 200-ns trigger and a >50-μs event-separation cut.It additionally described optically obscured core collapses, O(103) detected interactions over roughly 100 s for a Galactic-center burst, inverse-square distance scaling, and manually removed PMT-malfunction and data-acquisition backgrounds.Source: §6 · p. 30, §4 · p. 30, §5 · p. 30

1. Try to locate the primary analogue paper; 2. Adapt the evidence strategy after source-access failures.

  • After failing to retrieve the analogue article and supporting information, the agent separated accessible literature evidence, table-derived SAR trends, and explicitly labeled compound-specific hypotheses.A targeted procedure search supported a general strategy of converting nitriles to amidoximes and carboxylic acids to acyl chlorides, followed by reaction and cyclization; individual precursor mappings remained proposals.Source: §2 · pp. 30–31, §1 · p. 30

4. Let the assay table, not memory, drive the structure–activity relationships.

  • 3.4 for Comp-4 exceeded 3.212 for NS9283 in the parsed assay results, while the reported inactive tail included Comp-17, Comp-18, Comp-19, Comp-22, and Comp-24.The passage reports values of 1.01, 0.89, 0.90, 0.96, and 0.82 for those compounds, respectively; it frames the supplied table as the basis for structuring SAR trends.Source: §4 · p. 31

5. Refine mechanistic hypotheses by cross-checking the binding-site literature. … 2. Critique stage corrects over-strong mechanistic inference.

The HLE cases show EvoMaster iteratively grounding and revising claims, while the chemistry trajectory distinguishes assay evidence and literature-grounded hypotheses from proposed synthesis routes.

5. Refine mechanistic hypotheses by cross-checking the binding-site literature.

  • Binding-site literature places NS9283 at the stoichiometry-selective α4–α4 interface, so Comp-4 should be interpreted through substituent hypotheses rather than response magnitude alone.The proposed explanations include substituent vector, electron withdrawal, and potential polar contacts, not experimentally established mechanisms.Source: §5 · p. 31

6. Handle structural edge cases instead of forcing one template.

  • Comp-4’s mean response was 3.40 versus 3.212 for NS9283, but variability and the single-point assay did not establish a significant advantage or higher intrinsic potency.For analogues retaining the common scaffold, meta-NO2 was a potentially favorable substitution; electron withdrawal, orientation, and polar contacts remained hypotheses.Source: §6 · p. 31
  • The proposed common route converts nicotinonitrile to amidoxime, forms acyl chlorides from carboxylic acids, and cyclizes to 3,5-disubstituted 1,2,4-oxadiazoles.Compound-specific precursor mappings were labeled as proposals, with ester saponification and other late-stage transformations retained for structural exceptions.Source: §6 · p. 31

G. HLE Case Studies; 4. Select stage performs final local verification.

  • The HLE case studies illustrate evidence inspection and revision; the selector checked local paper lines 386–393 and 490–493 before confirming B, number of characters.Across the cases, the process tightened claims from initial reasoning to auditable evidence; aggregate HLE results are left to the main experimental discussion.Source: G. HLE Case Studies · p. 32, §4 · pp. 32–33

1. Early solving: forming an initial candidate; 1. Initial solvers disagree.

  • An early solver guessed number of characters from paper-level patterns, while initial phage solvers disagreed between H and F based on partial readings of the experiments.The number-of-characters guess was later checked against exact table evidence; the phage critique separated supported results from an over-strong mechanistic claim.Source: §1 · pp. 32–33, §2 · p. 33

2. Critique stage upgrades the candidate into table-level evidence.

  • n-chars was significantly negative for LSTM, GPT-2, BiLSTM, and BERT (***, p<0.001) but positive for Children (**, p<0.01).The critique retrieved Table 2 significance markers, turning a plausible candidate into a verifiable finding.Source: §2 · p. 32, §3 · p. 32

3. Rewriting stage consolidates the answer.

  • The rewriting stage combined student agreement with Table 2 and discussion evidence to confirm number of characters as the answer.It connected the negative effect for all four language models with the contrasting effect for children.Source: §3 · p. 32

2. Critique stage corrects over-strong mechanistic inference.

  • The critique rejected H: RP reduced phage success, XY helped the phage counteract RP, and the 500 Da molecule appeared only in the 60-minute RP+ wild-type infection sample.The molecule’s formation therefore required RP, wild-type phage operon XY, and infection progress to 60 minutes.Source: §2 · p. 33, §1 · p. 33

3. Rewriting stage contracts to the most evidence-supported interpretation. … 2. Construct an executable baseline.

Across the examples, EvoMaster refines claims and experimental plans by checking evidence, correcting overreach, and retaining supported conclusions. Its case studies show both conservative interpretation of bacterial experiments and evolving MLE-Bench solutions, including a medal-threshold time-aware classifier and an executable plant-recognition baseline.

3. Rewriting stage contracts to the most evidence-supported interpretation.

  • Rewriting retained only jointly supported findings: XY helps phage growth in RP+ bacteria, while the 500 Da molecule requires RP, intact XY, and infection progression.The observations do not establish specific substrate/product mechanisms or that the molecule is produced in uninfected cells.Source: §3 · p. 33, §4 · p. 33

4. Final selection rejects over-inference and converges on the conservative answer.

  • The final bacterial answer selects F: RP increases resistance to phageDE3, but is not required for the phage’s maximal observed virulence.The trajectory reaches this conservative conclusion through solver disagreement, critique, rewriting, and final selection.Source: §4 · p. 34

H. MLE-Bench Case Studies; 1. Establish a lexical baseline.

  • In the 5,671-request pizza-fulfillment task, EvoMaster began with TF–IDF text features and request-level metadata such as sentiment, politeness, subreddit participation, and user activity.The appendix presents this as one of two representative MLE-Bench trajectories and omits routine implementation steps.Source: H. MLE-Bench Case Studies · p. 34, §1 · p. 34

2. Expand the model space after validation plateaued; 3. Reject expensive but ineffective directions.

  • After lexical additions yielded marginal gains, EvoMaster explored embeddings, boosted trees, Bayesian optimization, and transformer fine-tuning, but expensive transformer and embedding directions lacked stable validation gains.Learned subreddit embeddings and feature selection also produced inconsistent improvements, so those implementations were abandoned.Source: §2 · p. 34, §3 · p. 34

4. Identify missing temporal structure; 5. Correct the validation protocol.

  • A time-aware formulation added historical fulfillment rates using only earlier requests, then replaced random validation with chronological splits and past-only aggregate features; the final ensemble reached the MLE-Bench medal threshold.The temporal pipeline achieved the first medal-level result during Research Plan 4 and produced a valid submission.csv.Source: §4 · p. 34, §5 · p. 35

H. MLE-Bench Case Studies; 1. Retrieve task-relevant prior knowledge.

  • For herbarium recognition, the task involved about 32,000 plant species, over one million images, and a long-tailed class distribution; EvoMaster retrieved prior plant-seedlings experience to guide class-imbalance handling and validation.The retrieved practices included weighted sampling, focal loss, stratified validation, conservative augmentation, and selecting the model by validation macro F1.Source: §5 · p. 35, §1 · p. 35

2. Construct an executable baseline.

  • EvoMaster turned the retrieved guidance into a stable executable plant-recognition baseline using pretrained ResNet-50, weighted sampling, focal loss, and mixed-precision training.The initial experiment used image size 224 and a fixed train-validation split.Source: §2 · p. 35

3. Generate multiple improvement hypotheses. … 5. Metric-level reinterpretation instead of blind thresholding.

Across the case studies, EvoMaster iteratively tests and retains supported changes, audits artifacts, and repairs conclusions when implementation details or metric definitions could mislead. The examples include image-classification experiments and PRL-Bench physics analyses.

3. Generate multiple improvement hypotheses

  • 0.031: Raising image resolution to 384×384 improved validation macro F1, while strong Mixup/CutOut, EfficientNet-B4, and hierarchical auxiliary losses did not outperform the baseline.The experiments tested multiple proposed improvements rather than assuming each would help.Source: §3 · p. 35

5. Retain effective changes and discard harmful ones

  • Region metadata yielded a further validation improvement of 0.006; the final solution retained ResNet-50 at 384×384 with class-aware sampling and focal loss, discarding harmful additions.The final submission removed ineffective augmentations and auxiliary objectives.Source: §5 · pp. 35–36

I. PRL-Bench Case Studies

  • The PRL-Bench case studies illustrate iterative artifact audits, mismatch detection, repairs, and evidence-based refinement across AMO physics and quantum information.The two trajectories exemplify different self-evolution modes: revising a problem formulation and refining retrieved experience through controlled experiments.Source: §I · p. 36, §5 · p. 36

1. Self-correction after an initial file-path failure

  • After a missing-file error, EvoMaster recovered by locating and inspecting node_2/lk_analysis.py rather than stopping or inventing missing code.It found the implementation’s stationary-root, amplitude, DDE-solver, and spectral-stability routines.Source: §1 · p. 36

2. Evidence audit of numerical artifacts before rewriting the answer

  • EvoMaster checked which benchmark figures already existed, shifting from generating a solution to verifying claims against available artifacts.The audit found the required homogeneous and heterogeneous time-series and spectrum plots, plus additional plots.Source: §2 · p. 36

3. Root-level repair of Task 1 by explicit residual validation

  • Residual checks validated three recovered stationary roots at approximately 1e-14, 1e-14, and 1e-13, making the numerical claim auditable.The checks targeted the roots and their amplitude/carrier consistency.Source: §3 · p. 36

4. Branch-selection repair for Task 3

  • EvoMaster solved the heterogeneous 2M system and explicitly selected the minimum-|Ω| branch, with the selected candidate Ω = -6.050176959016639.The benchmark required exposing the branch-selection rule rather than merely supplying a solution.Source: §4 · p. 37

5. Metric-level reinterpretation instead of blind thresholding

  • 75/100: The PRL case’s final answer reports an intermediate disorder window around σ≈0.2–0.4, strongest near 0.3, and clarifies that σPS is uncentered RMS rather than centered linewidth.The benchmark’s 0.05 GHz threshold applies to centered width, so the stored σPS value alone does not establish that the line is broad.Source: §5 · p. 37

1. Normalization repair instead of silently mixing mathematical objects. … 2. Build a non-GPQA science data hypothesis, then repair dataset access.

The cases show EvoMaster correcting mismatches and consolidating solutions around verifiable evidence. The appendix also traces GPQA post-training from task and data inspection to a valid packaged model.

1. Normalization repair instead of silently mixing mathematical objects.

  • EvoMaster repairs normalization by distinguishing unnormalized assemblage elements from the normalized states used in the antidistinguishability lemmas.The revise node identifies a semantic mismatch between Eq. (4)'s raw outputs and the states used in the lemmas.Source: §1 · p. 37

2. Construction-level self-correction for the 𝑑= 6 theorem.

  • The d=6 branch had overlaps above the 1/4 threshold, so later revise nodes replaced it with an Eq. (10) specialization using explicit pairings.Examples of the earlier overlaps include 0.7274, 0.7935, and 0.8916.Source: §2 · pp. 37–38

3. Turning qualitative conclusions into scored numerical witnesses.

  • Node 10 adds AME = 1.000000 at x = 1.2, AMSmax ≈0.9773463613, a λ = 1/2, x = π/3 benchmark, and explicit construction and search artifacts.Those artifacts include d=4 code, a general d=4p scaling formula, and a pairing-based d=3 search with min(max overlap) = 1.Source: §3 · p. 38

4. Final repair targets the hardest missing evidence, not the whole answer.

  • The final answer supplies explicit witnesses and constructions across d=4, d=6, and d=8, and earns a benchmark-judge score of 65.38/100.It reports max overlap 0.174127 < 1/4 for the chosen d=8 example and restricts the d=3 no-go conclusion to the pairing-based search.Source: §4 · p. 38
  • Across the two cases, EvoMaster inspects, detects mismatches, repairs weak links, and consolidates around verifiable evidence rather than producing a single-pass derivation.The cases illustrate residual validation, branch-selection disambiguation, physical reinterpretation of a spectral metric, normalization repair, construction replacement, and artifact surfacing.Source: §4 · p. 38

J. PostTrainBench Case Studies

  • The PostTrainBench appendix presents two GPQA post-training trajectories that inspect the evaluation interface, test a candidate with a small proxy evaluation, diagnose failure, and revise the recipe.Aggregate PostTrainBench results are omitted from the case studies and discussed in the main experimental section.Source: J. PostTrainBench Case Studies · pp. 38–39

Case 1: Format Repair and Checkpoint Selection for GPQA Post-Training (gemma3_gpqamain)

1. Inspect the target and evaluation contract.

  • The agent inspected the task recipe, metadata, and evaluator, establishing that GPQA uses shuffled multiple-choice QA and requires a parseable final answer in the expected ANSWER format.Source: §1 · p. 39

3. Train the first candidate. … 6. Package a valid final artifact.

Across two GPQA post-training cases, EvoMaster used evaluation evidence to expose output-format failures, revise data or candidate selection, and promote models that achieved 30.13% and 22.77% accuracy on the full official evaluations.

3. Train the first candidate.; 5. Revise the training format instead of only training longer.

  • EvoMaster built a compact science/medical SFT corpus, then removed the instruction-like target stub and aligned targets to the evaluator’s final-line convention.The first corpus contained 9,275 training and 190 validation examples; the regenerated corpus had about 9.5k non-GPQA examples.Source: §3 · p. 39, §5 · p. 39

4. Use a small evaluation to expose a hidden failure mode.

  • On a 20-sample GPQA slice, the first candidate scored 0.0 accuracy (stderr 0.0) and copied the prompt instruction instead of answering.The sample output repeated the instruction to provide the final answer as ANSWER: $LETTER.Source: §4 · p. 39

6. Compare candidates and promote the best-evidenced checkpoint.

  • A 20-sample local evaluation gave sft_v2_ckpt150 accuracy 0.25 (stderr 0.0993); EvoMaster promoted it over sft_v2_ckpt290, which lacked a usable score.The selection rule favored the scored checkpoint rather than simply taking the latest file.Source: §6 · p. 39

6. Compare candidates and promote the best-evidenced checkpoint.; 6. Package a valid final artifact.

  • 30.13% GPQA accuracy (standard error 2.17%) for sft_v2_ckpt150 and 22.77% GPQA accuracy (standard error 1.98%) for lora_sci_v1_merged on the full official evaluations.These were the promoted models in the first and second cases, respectively.Source: §6 · pp. 40–41

2. Turn dataset errors into schema information.; 1. Audit the task before training.

  • EvoMaster identified the official evaluator and format constraint, then repaired dataset split and answer-encoding mismatches rather than discarding sources, producing 205,542 training examples.The audited recipe recommended multiple-choice data outside GPQA, short proxy evaluations, and explicit candidate tracking.Source: §2 · p. 40, §1 · p. 40

4. Evaluate the base model and the candidate on the same proxy.; 3. Train a format-aware QLoRA candidate.

  • On the 40-example proxy, lora_sci_v1_merged reached accuracy 0.675 and parsed 0.95, versus base-model accuracy 0.0 and parsed 0.0.The candidate used a multi-source science and medical QLoRA training recipe after an initial unstable run was patched.Source: §4 · p. 40, §3 · p. 40

5. Record both improvement and residual artifacts.

  • The candidate still appended repetitive REF text, and the proxy overestimated final GPQA accuracy despite correctly detecting and fixing the base model’s parseability failure.EvoMaster recorded the residual artifacts while selecting the candidate because it repaired the dominant benchmark-facing failure.Source: §5 · p. 40, §6 · p. 41

6. Package a valid final artifact.

  • The promoted candidate was hard-linked into final_model and passed sanity checks for model architecture and tokenizer loading.The package used the lora_sci_v1_merged source checkpoint.Source: §6 · pp. 40–41

Related papers

Loading 2604.17406v4…