Source-linked AI summary
EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery
Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo, Wenxiang Hu, Jan Piotrowski, Jakub Kaliski, Jacopo Urbani, Zaiqiao Meng, Lun Zhou, Xiaohui Yan
TL;DR
Existing AI scientist systems commonly rely on static pipelines and do not adapt their strategies from accumulated outcomes and failures. EvoScientist introduces a self-evolving multi-agent framework with persistent memories, and experiments report stronger idea-generation quality and code execution success than established baselines.
Problem
Most AI scientist systems use static, hand-designed pipelines and rarely adapt idea- or code-generation strategies from accumulated interaction histories.
Method
EvoScientist coordinates Researcher, Engineer, and Evolution Manager Agents with ideation and experimentation memories that store and retrieve reusable strategies.
Results
EvoScientist outperforms 7 open-source and commercial systems on novelty, feasibility, relevance, and clarity, while achieving higher code execution success rates.
Takeaways & Limitations
Persistent memory and self-evolution support continuous improvement in idea quality and code execution success for end-to-end scientific discovery.
Takeaways & Limitations
Evaluation focuses on computational research tasks; generalization to physical experimentation remains open and requires laboratory workflows and real-world feedback.
Abstract
from arXiv · showhide
The increasing adoption of Large Language Models (LLMs) has enabled AI scientists to perform complex end-to-end scientific discovery tasks requiring coordination of specialized roles, including idea generation and experimental execution. However, most state-of-the-art AI scientist systems rely on static, hand-designed pipelines and fail to adapt based on accumulated interaction histories. As a result, these systems overlook promising research directions, repeat failed experiments, and pursue infeasible ideas. To address this, we introduce EvoScientist, an evolving multi-agent AI scientist framework that continuously improves research strategies through persistent memory and self-evolution. EvoScientist comprises three specialized agents: a Researcher Agent (RA) for scientific idea generation, an Engineer Agent (EA) for experiment implementation and execution, and an Evolution Manager Agent (EMA) that distills insights from prior interactions into reusable knowledge. EvoScientist contains two persistent memory modules: (i) an ideation memory, which summarizes feasible research directions from top-ranked ideas while recording previously unsuccessful directions; and (ii) an experimentation memory, which captures effective data processing and model training strategies derived from code search trajectories and best-performing implementations. These modules enable the RA and EA to retrieve relevant prior strategies, improving idea quality and code execution success rates over time. Experiments show that EvoScientist outperforms 7 open-source and commercial state-of-the-art systems in scientific idea generation, achieving higher novelty, feasibility, relevance, and clarity via automatic and human evaluation. EvoScientist also substantially improves code execution success rates through multi-agent evolution, demonstrating persistent memory's effectiveness for end-to-end scientific discovery.
1 Introduction
EvoScientist addresses the limitations of static AI scientist pipelines by using multi-agent evolution and persistent memory to improve idea generation and experiment execution over time. Its evaluations report higher-quality ideas and code execution success than strong open-source and commercial baselines.
- Existing AI scientist systems use largely static pipelines that rarely learn from accumulated outcomes and failures, causing repeated failures, missed directions, and infeasible ideas.
- EvoScientist coordinates a Researcher Agent, Engineer Agent, and Evolution Manager Agent for idea generation, experiment execution, and reusable knowledge distillation.
- Its ideation and experimentation memories store feasible directions, unsuccessful directions, and effective execution strategies for retrieval on future tasks.
- EvoScientist outperforms 7 open-source and commercial baselines on idea-generation quality across novelty, feasibility, relevance, and clarity under automatic and human evaluation.
- Three self-evolution mechanisms—idea direction, idea validation, and experiment strategy evolution—learn from accumulated outcomes and failures to improve discovery over time.
2 Related Work
Related work has progressed from LLM assistance on individual tasks to end-to-end and multi-agent scientific discovery systems. However, existing approaches generally lack persistent evolution across ideation and experiment execution under one unified objective.
- AI Agents for Scientific Discovery: Recent systems automate multiple research stages, including ideation, literature review, experimental design, coding, analysis, and manuscript preparation.
- AI Agents for Scientific Discovery: Representative systems use end-to-end pipelines, tree search, specialized multi-agent roles, iterative collaboration, debate, or human expert feedback.
- AI Agents for Scientific Discovery: Existing AI scientist systems typically keep agent roles and decision policies fixed while rarely distilling interaction outcomes and failures into reusable experience.
- Self-Evolving Agents: Self-evolving agents use mechanisms such as memory and adaptive tool use, but are predominantly evaluated on single-stage or narrowly scoped tasks.
- Self-Evolving Agents: EvoScientist addresses the gap by learning from accumulated interaction histories across the full pipeline, covering ideation and experiment-execution strategies under a unified objective.
3 Method
EvoScientist formulates end-to-end scientific discovery as a two-stage, verifiable pipeline and equips three agents with persistent memories for evolving ideas and experiment execution. The evolution manager distills outcomes and failures into reusable ideation and experimentation strategies retrieved by the researcher and engineer agents.
- Problem Formulation: The pipeline transforms a user goal into a research proposal and executable experiments that produce verifiable outputs and an execution report.Idea generation creates an idea and full proposal; experiment execution searches for and runs code to produce logs, metrics, and diagnoses.
- EvoScientist Framework: Three agents divide responsibilities: the RA generates proposals, the EA executes experiments and code, and the EMA distills interaction histories into persistent memories.The RA retrieves ideation knowledge, while the EA retrieves execution strategies before generating and running code.
- Researcher Agent: The RA uses ideation memory to retrieve relevant directions, then performs literature-grounded propose–review–refine tree search and Elo-based ranking of candidate ideas.Candidate ideas store review feedback, and ranking uses novelty, feasibility, relevance, and clarity before retaining top ideas.
- Engineer Agent: The EA retrieves experimentation strategies and searches through four experiment stages, iteratively revising executable code after diagnosing failures from structured execution results.The stages are initial implementation, hyperparameter tuning, proposed method, and ablation; execution histories record run status, logs, and metrics.
- Evolution Manager Agent: The EMA evolves three capabilities by summarizing feasible directions, recording failed proposals, and distilling data-processing and model-training strategies into persistent memories.Experiment strategies are derived from best-performing code and full code-search trajectories, while idea validation uses execution reports and baseline comparisons.
4 Experimental Setup
The experiments evaluate EvoScientist across idea generation, code implementation, and end-to-end discovery because no public dataset covers the complete pipeline. Evaluation combines automated and expert judgments and compares EvoScientist with seven open-source and commercial systems under consistent protocols.
- Research Questions: The evaluation addresses idea quality, code execution reliability, end-to-end paper production, and the contribution of multi-agent evolution to idea improvement.Idea quality is assessed through novelty, feasibility, relevance, and clarity, while code reliability is measured by execution success rates.
- Evaluation Set: The multi-level evaluation set covers idea generation, code implementation, and end-to-end scientific discovery because no public dataset spans the complete pipeline.The idea-generation portion contains 30 research queries solicited from experienced AI researchers across contemporary artificial-intelligence topics.
- Baselines: EvoScientist is compared with four open-source systems—Virtual Scientist, AI-Researcher, InternAgent, and AI Scientist-v2—and three commercial systems—Hypogenic, Novix, and K-Dense.The comparison includes seven representative baseline systems across the evaluation tasks.
- Evaluation Protocol: Idea quality is evaluated with automated LLM-based pairwise comparisons and expert human judgments, using swapped positions to reduce positional bias in LLM evaluations.The LLM judge scores ideas on a 1–10 scale across the reported quality dimensions.
- Implementation: The implementation uses specialized language models and retrieval settings for literature review, idea generation, code generation, manuscript authoring, and memory indexing.Experiments maintain consistent task settings and evaluation protocols across EvoScientist and baseline systems.
5 Experimental Results
EvoScientist improves scientific idea quality, code execution, and end-to-end discovery outcomes through memory-driven multi-agent evolution. Ablations indicate idea evolution particularly benefits originality and feasibility.
- 5.1 Idea Generation Performance: EvoScientist achieves positive average gaps against all compared open-source and commercial systems across novelty, feasibility, relevance, and clarity.Average gaps range from +29.17 to +93.34 against open-sourced baselines and from +46.00 to +80.83 against commercial baselines.
- 5.1 Idea Generation Performance: Expert human judges consistently prefer EvoScientist over strong baselines for novelty and feasibility.
- 5.2 Code Generation Performance: 34.39 to 44.56: mean execution success rate increases across four experiment stages after experiment strategy evolution.The evolution manager distills outcomes and failures into experimentation memory for retrieval by the engineer agent.
- 5.3 End-to-end Scientific Discovery Performance: All six EvoScientist-generated papers were accepted to ICAIS 2025, with one Best Paper Award and another AI Reviewer’s Appraisal Award.The AI Scientist Track received 82 submissions and accepted 26 papers.
- 5.4 Ablation Studies: Removing idea direction evolution reduces novelty and feasibility, while the main benefits of idea evolution arise in originality and feasibility rather than surface-level clarity.
6 Conclusions
EvoScientist addresses static AI scientist pipelines by combining three specialized agents with persistent memory and self-evolution. It improves idea-generation quality and code execution success through reusable knowledge from prior interactions.
- EvoScientist coordinates Researcher, Engineer, and Evolution Manager Agents to generate ideas, execute experiments, and distill interaction histories into reusable knowledge.
- Two persistent memories store feasible and unsuccessful ideation directions plus effective experimentation strategies, which the agents retrieve to improve future performance.
7 Limitations and Ethical Considerations
EvoScientist is intended to support human-led scientific discovery, but its evaluation scope and deployment requirements limit how its results should be interpreted.
- Limitations: EvoScientist’s evaluation covers computational tasks testable through simulation and code execution, leaving generalization to physical experimentation open.Extension to materials science and drug discovery requires laboratory workflows and real-world feedback.
- Ethical Considerations: EvoScientist should support rather than replace expert judgment, with human verification, safeguards against misuse, and monitoring for reproduced biases.
8 GenAI Disclosure
GenAI was used only for language refinement and manuscript polishing; the authors retain responsibility for the scientific content and conclusions.
- LLMs did not contribute to conceptualization, experimental design, data analysis, or interpretation, and no generated text affects the work’s originality or intellectual contribution.
A Details of Datasets
The evaluation uses 30 AI research queries and compares EvoScientist with open-source and commercial scientific-discovery systems using automatic and human assessments.
- Evaluation set: 30 research queries span contemporary AI topics including translation, speech recognition, healthcare agents, retrieval-augmented generation, safety, and alignment.
- Baselines: EvoScientist is compared with open-source baselines including Virtual Scientist, AI-Researcher, InternAgent, and AI Scientist-v2.
- Baselines: Commercial baselines include Hypogenic, Novix, and K-Dense, covering idea generation and broader research-lifecycle automation.
- Evaluation protocol: Idea-generation evaluation uses pairwise LLM judgments across novelty, feasibility, relevance, and clarity, with answer order swapped to reduce positional bias.
- Evaluation protocol: Human evaluation involved three PhD-level AI annotators who could verify literature-related claims online when needed.
C.3 Agreement between LLM Evaluation and Human Evaluation for Idea Generation
LLM-based idea-generation judgments show strong agreement with expert human assessments across the four evaluation dimensions.
- 90.0% overall agreement was achieved between LLM and human judgments across 120 evaluated idea pairs.
- 87.3% average agreement was obtained across all four dimensions and 600 dimension-level judgments.
- Clarity had the highest dimension-level agreement at 90.8%, followed by novelty at 88.3%.
- Relevance and feasibility showed agreement rates of 84.2% and 83.3%, respectively.
D Details of Implementation
The implementation details describe prompts for evolving ideas and experiment strategies, alongside representative accepted papers illustrating EvoScientist’s research outputs.
- Evolution prompts: Figures 10 and 11 provide prompts for idea-direction and idea-validation evolution, respectively.
- Evolution prompts: The idea-evaluation prompt asks an analysis engine to compare two technical ideas against a user research goal.
- Evolution prompts: Figure 12 provides the prompt details for experiment-strategy evolution.
- Case studies: Table 4 summarizes six accepted EvoScientist-generated ICAIS 2025 papers, while Figures 13 and 14 analyze two representative cases.
- Case studies: In the Best Paper case, the Researcher Agent retrieved direction-level insights from ideation memory to refine a clinically meaningful formulation balancing personalization and uncertainty calibration.
- Case studies: Reviewers praised the contribution’s validity and balance of methodological novelty with engineering practicality, while requesting clearer formalization, metric specification, and reproducibility details.
E.2 AI Reviewer’s Appraisal Award Case: Hierarchical Change Signature Analysis
The appraisal-award case links experiment-memory reuse with robust empirical implementation, while reviewer feedback identifies protocol consistency and reproducibility as continuing requirements.
- AI Reviewer’s Appraisal Award Case: Experiment memory summarized early execution failures and configuration issues, helping the Engineer Agent converge toward a robust, deployment-oriented implementation.
- Reviewer diagnostics: Reviewer feedback identified ambiguities in stability gating, metric reporting, and baseline fairness as protocol-level issues.
- Reviewer diagnostics: The case reflects strengths in empirically strong, practically relevant systems alongside limits in protocol alignment and documentation.
- Reviewer diagnostics: Across the two cases, sustained reviewer confidence depends on internal consistency, protocol-faithful baselines, and reproducibility-complete reporting.