PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale

Xinyu Zhu, Yuzhu Cai, Zexi Liu, Cheng Wang, Fengyang Li, Wenkai Jin, Wanxu Liu, Zehao Bing, Bingyang Zheng, Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xianghe Pang, Yaxin Du, Tingjia Miao, Yuzhi Zhang, Ruoxue Liao, Zhaohan Ding, Linfeng Zhang, Yanfeng Wang, Weinan E, Siheng Chen

arXiv:2604.17406v4cs.AI

TL;DR

Scientific-agent frameworks are often static and narrowly scoped despite scientific discovery requiring iterative learning and cross-domain scaling. EvoMaster addresses this with a foundational harness built around modularity, experiment readiness, continuous self-evolution, and collaboration. Across ten benchmarks, it leads on nine and records the highest average score, while its remaining scope excludes native physical-environment integration.

  • Problem

    Existing agent frameworks are predominantly static, narrowly scoped, and unable to learn from trial and error, limiting support for scalable scientific discovery.

  • Method

    EvoMaster provides a modular foundational harness with experiment-ready tracking, iterative self-critique loops, context management, and multi-agent workflows.

  • Results

    EvoMaster achieves the best score on nine of ten benchmarks and the highest average score among EvoMaster, OpenHands, OpenClaw, and Codex.

  • Takeaways & Limitations

    The SciMaster ecosystem demonstrates that EvoMaster can support self-improving scientific agents across diverse domains with minimal engineering overhead.

  • Takeaways & Limitations

    EvoMaster is primarily optimized for in silico and computational workflows and lacks native support for direct manipulation of physical experimental apparatuses.

Abstract

from arXiv · show

The convergence of large language models and agents is catalyzing a new era of scientific discovery: Agentic Science. While the scientific method is inherently iterative, existing agent frameworks are predominantly static, narrowly scoped, and lack the capacity to learn from trial and error. To bridge this gap, we present EvoMaster, a foundational evolving agent framework engineered specifically for Agentic Science at Scale. Driven by the core principle of continuous self-evolution, EvoMaster empowers agents to iteratively refine hypotheses, self-critique, and progressively accumulate knowledge across experimental cycles, faithfully mirroring human scientific inquiry. Crucially, as a domain-agnostic base harness, EvoMaster is exceptionally easy to scale up -- enabling developers to build and deploy highly capable, self-evolving scientific agents for arbitrary disciplines in approximately 100 lines of code. Built upon EvoMaster, we incubated the SciMaster ecosystem across domains such as machine learning, physics, biology, web research, and general science. Evaluations on ten benchmarks spanning scientific research/coding/experimentation, scientific reasoning and information search, and practical scientific problem solving compare EvoMaster against OpenHands, OpenClaw, and Codex. EvoMaster achieves the highest score on nine of the ten benchmarks and the strongest average score (58.0\%) among the four agents, validating its efficacy and generality as the premier foundational framework for the next generation of autonomous scientific discovery.

1 School of Artificial Intelligence, Shanghai Jiao Tong University 2 SciLand 3 DP Technology

EvoMaster addresses fragmented, static scientific-agent development with a foundational, evolving harness designed for cross-domain scaling. Its SciMaster ecosystem and benchmark results support broad applicability across scientific workflows.

  • Challenges: Existing scientific-agent frameworks are fragmented and domain-specific, while single-pass execution lacks mechanisms for learning from failures or accumulating insights.These limitations make cross-domain scaling costly and conflict with the iterative nature of scientific discovery.
  • Framework: EvoMaster unifies agent development across disciplines through modular composability, experiment-ready execution, iterative self-evolution, and multi-agent collaboration.Its design principles directly target siloed development and the need for repeated hypothesis, experimentation, and refinement cycles.
  • Ecosystem: The SciMaster ecosystem spans autonomous machine learning, general scientific research, web information retrieval, physics reasoning, and embodied intelligence training.The ecosystem is described as expanding to additional scientific domains through the shared foundational harness.
  • Evaluation: EvoMaster achieves the best score on nine of ten benchmarks and the highest average score at 58.0% across research coding, information search, reasoning, and practical scientific problem solving.The comparison includes OpenHands, OpenClaw, and Codex, with all agents using GPT-5.4 as the backend model.
  • Framework: New domain agents can be built in approximately 100 lines of code using EvoMaster’s foundational harness.The framework is presented as a reusable base rather than a domain-specific agent system.

3. Design Principles

EvoMaster’s design principles combine reusable modular structure, reproducible experimentation, iterative self-evolution, and collaborative multi-agent orchestration. These principles are implemented through decoupled layers and shared capability interfaces that support cross-domain reuse.

  • Modular Composability: EvoMaster separates Playground, Experiment, and Agent layers to decouple orchestration, experiment execution, and iterative intelligence.This orthogonal structure reduces coupling between decision logic and domain-specific tools.
  • Modular Composability: Unified registries, MCP, and Skill standards let developers swap models, environments, and toolsets while building domain-specific agents in approximately 100 lines of code.The interfaces make capabilities reusable across scientific domains.
  • Experiment-Ready Harness: YAML-based manifests and structured trajectory recording provide parameter agility, auditability, and reproducibility for multi-turn agent experiments.The recording system logs conversational turns, tool invocations, and token statistics in structured JSON.
  • Iterative Self-Evolution: The Agent Engine runs a reason → invoke tools → observe → self-critique loop, with context summarization and sliding windows supporting hundreds of interaction turns.Context management allows agents to accumulate insights and refine strategies without exceeding context limits.
  • Capability Layer: The capability layer exposes universal interfaces for tools, skills, and language models, making domain-specific extensions immediately available across disciplines.MCP-compatible tools are standardized, while skill instructions can be loaded on demand for context efficiency.
  • Multi-Agent Collaborative Evolution: The Playground orchestrator uses declarative AgentSlots to assign specialized roles and support dynamic multi-agent collaborative evolution.Roles can include solver, critic, and rewriter agents with independent model and tool configurations.

6. Experiments

EvoMaster is evaluated across ten benchmarks covering scientific research, reasoning and information search, and practical scientific problem solving. It leads on nine benchmarks, with especially strong results on long-horizon engineering and retrieval-intensive reasoning.

  • Overall results: EvoMaster achieves the best score on nine of ten benchmarks, while Codex leads on PostTrainBench.Across all ten benchmarks, EvoMaster averages 58.02%, compared with 40.29% for Codex, 39.84% for OpenHands, and 32.39% for OpenClaw.
  • Scientific research, coding, and experimentation: 74.40% on PaperBench (CodeDev) and 75.76% on MLE-Bench (Lite) are EvoMaster’s strongest research coding and experimentation results.CodeDev combines paper-specific planning, repository construction, static auditing, and judge-guided repair; MLE-Bench uses iterative improvement through validation feedback, debugging, and research-plan refinement.
  • Scientific research, coding, and experimentation: 75.76% on MLE-Bench (Lite) includes a 100.00% valid-submission rate and an 84.85% above-median rate under the same 24-hour budget.EvoMaster also reaches a 31.82% silver-medal rate, exceeding OpenHands and Codex at 40.91% any-medal and OpenClaw at 18.18%.
  • Scientific reasoning and information search: EvoMaster leads all baselines on Frontier Science, BrowseComp, BrowseComp-ZH, and HLE, scoring 55.00%, 74.25%, 73.36%, and 41.10%, respectively.The largest gains occur on retrieval-heavy BrowseComp tasks requiring search, filtering, cross-checking, and synthesis across multiple sources.
  • Practical scientific problem solving: EvoMaster achieves 51.08% on PRL-Bench, 56.04% on Combinatorial Construction, and 65.36% on BiomniBench-DA, the top score on each benchmark.These margins are smaller because baselines are also strong on concrete coding, data-analysis, and evaluator-driven optimization workflows.
  • Overall results: The results show the clearest advantages on long-horizon ML engineering and retrieval-intensive reasoning, with smaller but consistent gains on practical scientific tasks.This pattern supports EvoMaster’s utility across research coding, information-intensive reasoning, and concrete scientific analyses and construction searches.

7. Conclusion

EvoMaster combines foundational modularity with continuous self-evolution to support autonomous scientific workflows across disciplines. Its evaluation spans ten benchmarks, while its current scope remains primarily computational and in silico.

  • EvoMaster integrates iterative refinement, experiment-ready execution, and multi-agent collaboration so agents can hypothesize, experiment, self-critique, and evolve from trial and error.
  • The evaluation suite covers scientific research, coding, experimentation, scientific reasoning, information search, and practical scientific problem solving across ten benchmarks.
  • EvoMaster achieves the best score on nine of ten benchmarks and the highest average score among EvoMaster, OpenHands, OpenClaw, and Codex.
  • The framework incubates domain-specific benchmark agents and is presented as a scalable foundation for deploying self-improving scientific agents with minimal engineering overhead.
  • EvoMaster is currently optimized for in silico and computational workflows and lacks native support for directly manipulating physical experimental apparatuses.

B. Implementation Details

Implementation uses benchmark-specific EvoMaster workflows under controlled model, tool, time, and evaluation settings. The configurations range from isolated coding environments to iterative search and process-level judging.

  • PaperBench runs in isolated Docker containers with read-only paper packages, writable submission repositories, and artifact logging, while the rubric remains hidden during generation.
  • BrowseComp and Frontier Science use GPT-5.4 single-agent workflows with web or scholarly retrieval, local computation, up to 100 turns, and 128k-token contexts.
  • PRL-Bench uses a Phys-Master MCTS workflow with five rounds, two parallel workers, and GPT-5.4 at medium reasoning effort.
  • Combinatorial Construction edits evolvable programs, runs deterministic local evaluators, and returns the highest-scoring valid construction found.
  • BiomniBench-DA uses up to 150 turns and process-level rubric scoring, while MLE-Bench Lite uses iterative research rounds and official local submission grading.

C. Combinatorial Construction Case Studies

The construction case studies show EvoMaster improving executable mathematical artifacts through evaluator feedback, best-so-far preservation, and continued search beyond early plateaus.

  • Sum-Difference Set Improvement: The sum-difference task evaluates submitted integer sets by recomputing |A+A|, |A−A|, and C(A), preventing credit from inflated self-reported objectives.
  • Sum-Difference Set Improvement: The representative sum-difference trajectory improves C(A) from 1.0597930945 to 1.0734301121 at generation 33 and continues through later generations.
  • Sum-Difference Set Improvement: The trajectory reaches C(A) = 1.1057713861, while the final benchmark result reaches C(A) = 1.1207 after rounding, corresponding to 56.04%.
  • Takeaway: Across both cases, the self-evolving loop preserves validated high-scoring constructions and promotes the best artifact rather than the latest generated code.
  • Discrete Determinant Repair: In Hadamard search, EvoMaster constructs 29 × 29 {±1} matrices and uses evaluator feedback to reject invalid or numerically unstable candidates.

1. Start from a valid but weak discrete matrix.

The discrete-matrix trajectory begins with a valid weak construction, then improves through evaluator-guided local and structured search while preserving valid high-scoring candidates.

  • The initial valid Hadamard construction has determinant ratio 0.1432748538, providing an executable baseline for subsequent search.
  • Candidate programs using stable determinant evaluation and local flips improve the ratio to 0.4623936618 by generation 8 and 0.5504599401 by generation 25.
  • Later structured-matrix exploration raises the determinant ratio to 0.6482062272, 0.7594555052, and 0.8596491228.
  • A later candidate reaches determinant ratio 0.9210526316, matching the best GPT-5.4-medium score reported for the auxiliary task.
  • EvoMaster, Codex, and OpenClaw reach 0.9211 after rounding, while OpenHands reaches 0.8596 on the auxiliary task.
  • The case illustrates evaluator-driven construction through preserving valid high-determinant matrices and avoiding regressions from invalid or unstable candidates.

6. Refine the sender population.

The trajectory refined heterogeneous immune populations into conventional KIR+ CD8 and CeD-enriched activated CD4 groups, then compared pathway-specific ligand–receptor scores against a KIR− CD8 background.

  • 6. Refine the sender population.: 5,808 conventional CD8 cells included 465 KIR+ cells, with 255 from CeD and 210 from healthy controls.
  • 6. Refine the sender population.: The agent selected a CeD-enriched activated CD4 proxy rather than the FOXP3+ regulatory cluster because tetramer labels were unavailable.The selected cluster contained 1,317 CeD versus 296 healthy-control cells and expressed CD40LG, IL32, TIGIT, TNFRSF4, and HLA-DP.
  • 6. Refine the sender population.: Pathway scoring compared KIR+ CD8→pathogenic CD4 interactions with a KIR− CD8 background, making the analysis pathway-specific rather than expression-specific.CeD group sizes were 255 KIR+ CD8, 2,668 KIR− CD8, and 1,317 pathogenic CD4 cells.
  • 6. Refine the sender population.: The strongest predicted pathways included CCL5→CCR4, LGALS9→CD44, TNFSF14/LIGHT→TNFRSF14/HVEM, IFNG→IFNGR1/2, and FASLG→FAS.

2. Identity verification before answering.

EvoMaster verifies identities before answering by escalating from broad, conflicting search results to direct source retrieval and targeted chronology checks. It finalizes only after the requested facts are independently supported.

  • 2. Identity verification before answering.: The agent verified Ken Walibora’s interview clue but withheld the answer because the probation-officer years remained unconfirmed.
  • 2. Identity verification before answering.: Conflicting secondary sources proposed 1988–1996 or 1985–1986, so the agent treated the exact employment range as unresolved.
  • 2. Identity verification before answering.: A curriculum vitae resolved the ambiguity: probation work was 1988–1996, while teaching belonged to 1985–1986.
  • 2. Identity verification before answering.: For Lillian Karabaic, the agent rejected candidates lacking the full clue set and verified the four constraints across multiple sources before answering.The verified clues covered the 2008 bicycle ride, 2020 XRAY.FM resignation, 2023 Pay Dirt column, and 2014 forum presentation.
  • 2. Identity verification before answering.: The cases make the final answer inspectable because EvoMaster records hypotheses, evidence IDs, information gaps, rejected explanations, and verification artifacts.

F. Frontier Science Case Studies

The Frontier Science case studies show EvoMaster adapting its evidence strategy across physics and chemistry tasks. It narrows retrieval to authoritative sources in physics and separates evidence-supported chemistry routes from hypotheses after access failures.

  • F. Frontier Science Case Studies: In the physics case, EvoMaster moved from broad web retrieval to the correct Super-Kamiokande IV paper and targeted extraction of trigger values.
  • F. Frontier Science Case Studies: 54% efficiency above 5.5 MeV and 99% above 8.5 MeV were reported for the single-event trigger using 47 PMT hits in 200 ns.
  • F. Frontier Science Case Studies: The physics workflow explicitly enumerated all seven requested answer slots, including trigger values, event scaling, backgrounds, and the event-separation cut.
  • F. Frontier Science Case Studies: The chemistry case separated a literature-supported general reaction from proposed compound-specific synthesis mappings after the target document could not be reliably retrieved.The unavailable document was associated with a timed-out download, a 0-byte PDF, and a 403 HTML response.

5. Refine mechanistic hypotheses by cross-checking the binding-site literature.

The chemistry trajectory used prior binding-site literature to frame a mechanistic interpretation of the nitro analogue, while clearly separating supported reaction evidence from proposed analogue-specific routes. The broader HLE cases likewise tightened conclusions through critique and evidence checks.

  • 5. Refine mechanistic hypotheses by cross-checking the binding-site literature.: Prior work located NS9283 activity at the stoichiometry-selective α4–α4 interface, providing the literature context for interpreting Comp-4.
  • 5. Refine mechanistic hypotheses by cross-checking the binding-site literature.: The final chemistry response marked compound-specific precursor mappings and late-stage transformations as proposed rather than literature-established routes.
  • 5. Refine mechanistic hypotheses by cross-checking the binding-site literature.: The chemistry case revised an initial common-route hypothesis when structurally exceptional analogues required separate late-stage transformations.
  • 5. Refine mechanistic hypotheses by cross-checking the binding-site literature.: Table 2 identified number of characters as significantly negative for four language models and significantly positive for children.The reported thresholds were p<0.001 for the models and p<0.01 for children.
  • 5. Refine mechanistic hypotheses by cross-checking the binding-site literature.: Critique rejected an unsupported mechanistic claim about operon XY while retaining the data-supported conclusion that RP reduces phage success.

3. Rewriting stage contracts to the most evidence-supported interpretation.

EvoMaster rewrites candidate interpretations to retain only claims jointly supported by heterogeneous experimental evidence. In the time-aware classification case, it also revises the task formulation and validation procedure after identifying temporal structure and leakage risk.

  • EvoMaster removes unsupported mechanistic claims and retains only conclusions jointly established by the two experiments.The evidence supports RP as a defense system, XY as a counter-defense operon in RP+ bacteria, and a molecule associated with their interaction.
  • In the pizza-request task, static lexical and user-history features provide only marginal gains, while transformer experiments produce unstable validation improvements.RoBERTa- and DeBERTa-based approaches increase computational cost without stable gains, and learned subreddit embeddings are inconsistent.
  • EvoMaster models fulfillment behavior with rolling historical success rates computed from requests preceding each current request.This revision addresses temporally varying outcomes that static text features do not capture.
  • Chronological validation and past-only aggregate features prevent future information from leaking into historical predictors.The leakage-safe temporal pipeline reaches the first medal-level result during Research Plan 4.
  • The final pizza-request solution combines textual, metadata, and leakage-safe temporal features and reaches the MLE-Bench medal threshold.Its decisive change is the chronological formulation and validation of time-varying fulfillment behavior.

2. Construct an executable baseline.

For fine-grained plant recognition, EvoMaster constructs a stable pretrained baseline, tests targeted improvements, and retains only changes supported by validation evidence. Higher resolution helps, while destructive augmentations and auxiliary objectives are rejected.

  • EvoMaster builds a stable ResNet-50 baseline with weighted sampling, focal loss, mixed-precision training, and a fixed validation split.The pipeline uses 224-pixel images and addresses the long-tailed class distribution through class-aware sampling and focal loss.
  • 384 × 384 resolution improves validation macro F1 by 0.031, confirming the importance of fine-grained visual details.Mixup, CutOut, EfficientNet-B4, and hierarchical auxiliary losses fail to outperform the baseline under the available budget.
  • The trajectory exemplifies self-evolution through retrieving related experience, running controlled experiments, and retaining effective modifications while rejecting harmful ones.This process follows the inspect, hypothesize, execute, critique, and refinement loop.
  • EvoMaster removes ineffective augmentations and auxiliary objectives while retaining the higher resolution and ResNet-50 backbone.Region metadata adds a further validation improvement of 0.006 as a complementary feature.
  • The final solution uses pretrained ResNet-50 at 384 × 384 with class-aware sampling and focal loss.Harmful augmentations and ineffective auxiliary objectives are removed before submission generation.

1. Normalization repair instead of silently mixing mathematical objects.

In the measurement-antidiscrimination case, EvoMaster repairs a semantic mismatch between unnormalized assemblage elements and normalized states. It replaces an under-verified d=6 construction and adds targeted artifacts needed for auditable theorem evaluation.

  • EvoMaster distinguishes unnormalized assemblage elements from the normalized states required by antidistinguishability lemmas.The repair corrects the semantic interface between Eq. (4) outputs and the theorem’s mathematical objects.
  • The earlier d=6 branch is rejected because several outcome-wise pairwise overlaps exceed the required 1/4 threshold.Examples include overlaps of 0.7274, 0.7935, and 0.8916.
  • EvoMaster replaces the broken d=6 branch with a corrected Eq. (10) specialization using explicit pairings.The replacement is artifact-backed rather than a polishing pass over the under-verified construction.
  • The repaired answer reports AME = 1.000000, AMSmax ≈0.9773463613, explicit d=4 code, arbitrary-d=4p scaling, and a pairing-based d=3 search.These additions make the construction and theorem checks more auditable.
  • The final repair adds all 18 d=6 reduced states, a six-row overlap table, a verified d=8 example with ω=0.20, and a scoped d=3 conclusion.The d=6 overlap maxima are all below 1/4, while the d=3 claim is restricted to the searched pairing-based family.

3. Train the first candidate.

EvoMaster trains an initial science and medical multiple-choice candidate, evaluates it on a proxy, and revises the training targets after observing behavioral failure. It then selects a scored checkpoint rather than blindly promoting the latest artifact.

  • The first Gemma candidate scores 0.0 accuracy on a 20-sample GPQA slice because it copies the prompt instruction instead of answering.Its output repeats the expected answer-format instruction rather than producing a letter.
  • EvoMaster removes the instruction-like answer stub and aligns supervised targets with the evaluator’s final-line convention.The revised data builder regenerates approximately 9.5k non-GPQA supervised examples.
  • The corrected Gemma checkpoint reaches 0.25 accuracy on the local 20-sample slice, while the later merged checkpoint lacks a reliable small-evaluation score.EvoMaster therefore chooses the scored checkpoint rather than the latest file.
  • 30.13% GPQA accuracy is achieved by the promoted sft_v2_ckpt150 model on the full official evaluation.The reported standard error is 2.17%.
  • The SmolLM3 case uses a parse-oriented proxy that separates answer correctness from extractable ANSWER formatting.The base model records 0.0 accuracy and 0.0 parsed rate, while the candidate reaches 0.675 accuracy and 0.95 parsed rate.
  • The candidate is promoted despite repetitive REF text because it fixes the dominant benchmark-facing failure of unparsable answers.The residual output artifact remains recorded as a limitation.
Loading 2604.17406v4…