PaperlayerView exact revision on arXiv

Source-linked AI summary

An AI system to help scientists write expert-level empirical software

Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Marc Coram, Hao Cui, Jake Garrison, Renee Johnston Anton Kast, Cory Y. McLean, Peter Norgaard, Zahra Shamsi, David Smalling, James Thompson, Subhashini Venugopalan, Brian P. Williams, Chujun He, Sarah Martinson, Martyna Plomecka, Lai Wei, Yuchen Zhou, Qian-Ze Zhu, Matthew Abraham, Erica Brand, Anna Bulanova, Jeffrey A. Cardille, Chris Co, Scott Ellsworth, Grace Joseph, Malcolm Kane, Ryan Krueger, Johan Kartiwa, Dan Liebling, Jan-Matthis Lueckmann, Paul Raccuglia, Xuefei, Wang, Katherine Chou, James Manyika, Yossi Matias, John C. Platt, Lizzie Dorfman, Shibl Mourad, Michael P. Brenner

arXiv:2509.06503v3cs.AIq-bio.QM

TL;DR

Scientific software is slow and difficult to create, limiting systematic exploration of computational solutions. ERA combines a Large Language Model with Tree Search to generate and improve scorable scientific software, outperforming state-of-the-art methods across diverse scientific tasks.

  • Problem

    Scientific software is slow and difficult to create, limiting systematic exploration of alternative computational approaches.

  • Method

    ERA drives Tree Search with a Large Language Model to generate, score, and iteratively improve scientific software while integrating external research ideas.

  • Results

    ERA produced expert-level software across diverse scientific tasks, including 40 methods beating the best known scRNA-seq integration method and 14 outperforming the CDC ensemble.

  • Takeaways & Limitations

    ERA can accelerate solution exploration for scientific tasks whose outputs can be scored automatically, reducing searches from weeks or months to hours or days.

  • Takeaways & Limitations

    The evaluated problems primarily emphasize advanced empirical software engineering rather than genuine scientific discovery involving theories, causal mechanisms, or mathematical frameworks.

Abstract

from arXiv · show

The cycle of scientific discovery is frequently bottlenecked by the slow, manual creation of software to support computational experiments\cite{hannay2009how}. To address this, we present Empirical Research Assistance (ERA), an AI system that creates expert-level scientific software whose goal is to maximize a quality metric. The system uses a Large Language Model (LLM) and Tree Search (TS)\cite{silver2016mastering} to systematically improve the quality metric and intelligently navigate the large space of possible solutions. ERA achieves expert-level results when it explores and integrates complex research ideas from external sources. The effectiveness of tree search is demonstrated across a diverse range of tasks. In bioinformatics, ERA discovered 40 novel methods for single-cell data analysis that outperformed the top human-developed methods on a public leaderboard. In epidemiology, ERA generated 14 models that outperformed the CDC ensemble and all other individual models for forecasting COVID-19 hospitalizations. ERA also produced expert-level software for geospatial analysis, neural activity prediction in zebrafish, and numerical solution of integrals, and a novel rule-based construction for time series forecasting. By devising and implementing novel solutions to diverse tasks, ERA represents a significant step towards accelerating scientific progress.

Introduction

Empirical software is central to scientific discovery but remains slow and difficult to create. ERA addresses this bottleneck by using an LLM and Tree Search to systematically generate and improve software, achieving expert-level performance across scientific tasks.

  • Scientific importance: Empirical software is central to science, supporting models of complex systems and Nobel Prize-winning advances in chemistry.Examples include Density Functional Theory, molecular dynamics simulation, and protein structure prediction.
  • The bottleneck: Creating domain-specific empirical software is slow and difficult because it requires tedious work and lacks systematic searches over alternatives.Design choices are often driven by intuition or expediency rather than exhaustive experimentation.
  • ERA approach: ERA uses an LLM to rewrite software and Tree Search to select candidate solutions for further exploration.Research guidance can be injected by users or automatically retrieved from the literature through a search engine.
  • Main result: ERA produces software that outperforms the state-of-the-art across scorable tasks spanning broad scientific disciplines.The system’s expert-level performance comes from exhaustively searching for high-quality solutions at unprecedented scale.

Results

ERA achieved expert-level performance across diverse scientific software tasks, outperforming established methods and benchmarks in single-cell analysis, COVID-19 forecasting, time-series forecasting, and other domains. Tree search generated diverse candidates, enabled effective recombination, and discovered high-performing hybrid strategies.

  • Kaggle competitions: ERA substantially outperformed a single LLM call, best-of-1000 LLM calls, and AIDE across 16 Kaggle Playground competitions.Its advantage came from maintaining a diverse candidate tree and backtracking when code mutations plateaued.
  • Single-cell analysis: 14% overall improvement over ComBat was achieved by BBKNN (TS), which matched or exceeded published BBKNN on every dataset and 11/13 metrics.The method combined ComBat-corrected PCA embeddings with BBKNN, rather than simply implementing BBKNN.
  • Method recombination: 44% of recombination solutions outperformed both base methods, while 22 of the remaining 31 outperformed one base method.These comparisons used average scores across metrics successfully computed for all three methods.
  • COVID-19 forecasting: 26 average WIS was achieved by ERA, outperforming the official CovidHub Ensemble’s average WIS of 29.ERA also generated 26 hybrid models, of which 11 exceeded both parent models.
  • COVID-19 forecasting: 14 distinct forecasting strategies emerged from systematic exploration using tree search, recombination, and externally generated research ideas.The strongest hybrids frequently combined the climatology-based CMU-climate_baseline with the autoregressive UMass-ar6_pooled model.
  • Additional scientific tasks: ERA outperformed the entire May 18, 2025 leaderboard, including foundation models, deep-learning models, and standard time-series methods.Across geospatial segmentation, vertebrate neural activity forecasting, and numerical analysis, ERA achieved expert-level performance.

Discussion

The discussion presents ERA as an LLM-driven Tree Search system that iteratively creates and improves empirical scientific software by integrating external research ideas. It distinguishes empirical optimization from genuine scientific discovery, highlights broad performance and speed, and notes safety risks from lowering the expertise barrier.

  • Contributions: ERA combines an LLM with Tree Search to rewrite software whose outputs maximize a quality score, turning scientific software creation into a scorable task.Its rewriting approach supports flexible integration of domain knowledge and external research ideas.
  • Contributions: The authors report that ERA beats human performance across a wide range of relevant, well-studied scientific problems and often improves on methods from cutting-edge papers.This breadth distinguishes ERA from systems specialized for a single scientific domain.
  • Limitations: ERA targets empirical predictive-model optimization rather than genuine scientific discovery, which requires reasoning about theories, causal mechanisms, and mathematical frameworks.The evaluated problems primarily emphasize advanced empirical software engineering because they support rigorous automated scoring.
  • Limitations: Autonomous production of expert-level empirical software creates safety risks by lowering the technical expertise required to execute sophisticated computational tasks.The same democratization that may accelerate beneficial discovery can also lower barriers to deploying advanced models in domains affecting human well-being.
  • Contributions: ERA can search over externally supplied research ideas from published literature, research agents, or solutions it discovers itself.The system creates code that follows specific ideas, enabling systematic exploration of those ideas.
  • Implications: ERA reduces exploration of research ideas from weeks or months to hours or days, accelerating trial-and-error workflows in machine-scorable scientific fields.The authors argue that this acceleration could have profound consequences for scientific advancement.

Methods

ERA prompts an LLM to generate and execute scientific code, then uses scores and search feedback to iteratively improve solutions. Its PUCT-based tree search balances exploration and exploitation while recombining expert ideas through code-generating prompts.

  • System workflow: ERA prompts an LLM with a task description, evaluation metric, and data, then executes and scores the generated Python code in a sandbox.Search uses scores, output logs, and other information to hill climb toward better solutions.
  • Tree search: The system uses PUCT tree search with a UCB-inspired acquisition score to balance exploitation of high-scoring solutions against exploration of the search space.The selected node receives one novel child solution generated by the language model; the code is executed, scored, and added to the tree.
  • Tree search: PUCT converts task-specific scores into rank scores, uses a flat prior over executed candidates, and selects the node with the globally maximum PUCT score.The exploration constant c_puct was tuned on the Kaggle benchmark, where c_puct = 1 worked well.
  • Solution generation: Mutations occur at the code level, while ideas are expressed in prompts that generate code, which is then scored and iterated upon.Scores saturate after 300-1000 nodes, and the system searches over ideas by combining different tree searches with different prompts.
  • Solution generation: ERA recombines expert ideas directly in conceptual space through LLM prompts rather than solely in distilled model space.This recombination approach is conceptually related to evolutionary methods such as RHEA but differs from using the LLM only as a mutation and crossover operator.

Code Availability

ERA’s reference implementation and best candidate solutions for all six scientific problems are publicly available, with an interface for examining representative tree-search runs.

  • Code Availability: The ERA implementation, six best candidate solutions, and representative tree-search data are publicly available online.The accompanying interface supports inspection of solution progression and breakthrough plots as tree search proceeds.

Author Contributions Statement

The authors contributed across system development, empirical applications, evaluation, interfaces, design, program management, and leadership. Contributions were distributed among named collaborators for each activity.

  • Research and system development: Named authors contributed to the Code Mutation System, single-cell RNA-seq batch integration, COVID forecasting, geospatial analysis, ZAPBench, GIFT-Eval, and integral computation.The listed contributors are identified for each research activity.
  • Support and leadership: Additional contributions covered user interfaces, graphical design, program management, and leadership.These roles are assigned to named authors in the contribution statement.

Competing Interest Declaration

The authors affiliated with Google are Google employees and hold Alphabet stock.

  • The authors affiliated with Google are employees of Google Inc. and hold Alphabet stock.

Extended Data Figure Legends · A. Supplementary Notes

The extended data legends document ERA’s replicate performance, search trajectories, ablations, recombination experiments, forecasting evaluations, and expert inspection across several scientific tasks. They emphasize how tree search and hybridization were evaluated for scRNA-seq integration, COVID-19 forecasting, and time-series prediction.

  • Extended Data Figure Legends: Extended Data Fig. 1 reports replicate-level overall scores on holdout OpenProblems datasets, with bars showing each method’s highest-validation-performance replicate.Three full-process replicates were performed for tree-search implementations.
  • Extended Data Figure Legends: Extended Data Fig. 2 visualizes BBKNN tree-search progress through maximum score versus node count and the corresponding solution-tree structure.Green dots mark abrupt score increases caused by code improvements.
  • Extended Data Figure Legends: Extended Data Fig. 3 gives an ablation analysis of the top-performing BBKNN method and describes its scaling, log1p, standardization, ComBat+PCA, and batch-balanced k-nearest-neighbor steps.The listed transforms include scaling to 10^4 total counts and standardization to mean 0 and unit variance.
  • Extended Data Figure Legends: Extended Data Fig. 4 compares two base methods with recombination across all 55 pairwise combinations of 11 base methods on the OpenProblems test dataset.Recombination tree searches were seeded with top variants from two base-method runs and evaluated over an intersection of successfully calculated metrics.
  • Extended Data Figure Legends: Extended Data Fig. 5 presents WIS by reference date for the best retrospective COVID-19 hospitalization forecast replicate from each validation window.Models selected using six weeks of validation dates are evaluated on the next two weeks of unseen reference dates.
  • Extended Data Figure Legends: Extended Data Fig. 6 shows the COVID-19 prediction search’s maximum score as a function of nodes alongside its solution-tree structure.Green dots identify abrupt score increases associated with code improvements.
  • Extended Data Figure Legends: Extended Data Fig. 7 compares average WIS for COVID-19 forecasting recombination models with their constituent CovidHub baseline models.The recombination bars represent hybrid models synthesizing novel forecasting strategies.
  • Extended Data Figure Legends: Extended Data Fig. 8 depicts GIFT-Eval tree-search progress through maximum score versus node count and the associated solution tree, while Table 1 records expert inspection of scRNA-seq implementation adherence.The breakthrough plots mark abrupt score increases caused by code improvements.

A.1. Geospatial Analysis: Segmentation of Remote Sensing Images

ERA applied tree search to semantic segmentation of high-resolution remote-sensing images using the DLRSD dataset and a reproducible 80/20 split. Its three best solutions achieved mIoU greater than 0.80, surpassing reported academic results through established architectures, powerful encoders, and test-time augmentation.

  • Task: ERA addressed semantic segmentation, assigning land-cover class labels to every pixel in high-resolution remote-sensing images despite substantial visual heterogeneity.Image differences can arise from time of day, season, and weather.
  • Dataset: The DLRSD dataset densely labels UC Merced imagery at the pixel level with 17 class labels for remote-sensing applications.DLRSD supports tasks including multi-label classification, image retrieval, and semantic segmentation.
  • Evaluation: Models were trained with a reproducible 80/20 imagery split and evaluated on 420 held-out test images using mean intersection over union.The 420 test images were randomly selected.
  • Results: mIoU greater than 0.80 was achieved by all three top tree-search solutions, significantly outperforming reported results on the DLRSD benchmark.Solutions 1 and 3 used UNet++ or U-Net with ImageNet-pretrained encoders, while Solution 2 used SegFormer; all employed extensive test-time augmentation.

A.2. Neuroscience: Whole-Brain Neural Activity Prediction

ERA was applied to cellular-resolution whole-brain neural activity forecasting in larval zebrafish, producing tree-search-derived models that combine temporal and cross-neuron information. These solutions were substantially faster to train than the benchmark’s best video model while supporting multi-step and one-step prediction settings.

  • Benchmark: ZAPBench records cellular-resolution activity across an entire larval zebrafish brain during two hours of visual stimulation using 3D light-sheet microscopy.The benchmark captures responses under distinct visual stimulus conditions designed to elicit different behaviors.
  • Forecasting setup: ERA forecast all neurons up to 32 time steps ahead from their preceding 4 activity steps using ZAPBench’s stimulus-specific train, validation, and test splits.The splits allocate 70% for training, 10% for validation, and 20% for testing per stimulus condition.
  • ERA models: Tree search produced a multi-step model combining temporal convolutions, a learned global brain state, neuron-specific embeddings, weight-shared residual blocks, and a dense output layer.The model generates multi-step predictions in one shot.
  • ERA models: A separate tree-searched one-step model used dynamic attention for global-context aggregation and FiLM-like conditioning of feature representations.Both tree-searched architectures generate a learned global context vector to inform per-feature predictions.
  • Efficiency: Less than two hours on a single T4 GPU versus 36 hours on 16 A100 GPUs was required to train ERA’s solutions and the best-performing video model, respectively.The ERA solutions also use cross-neuron information to generate predictions.

A.3. Numerical Analysis: Library for Numerical Evaluation of Difficult Integrals

ERA evolved a general-purpose numerical integration method for difficult oscillatory integrals where scipy.integrate.quad() failed, using tree search to improve a benchmark-scored implementation. The resulting method correctly evaluated 17 of 19 held-out integrals within 3% fractional error while retaining quad() behavior for less pathological cases.

  • QUADPACK81, underlying scipy.integrate.quad(), can fail to converge, miss important integrand features, or lose precision through cancellations.
  • 38 oscillatory integrals with infinite upper limits formed the benchmark because quad() returned incorrect answers despite known analytic solutions.
  • ERA trained on 19 integrals and evaluated on 19 held-out integrals, starting from quad() and conducting a 1000-node tree search scored by logarithmic absolute fractional error.
  • The evolved method partitions the infinite domain into contiguous finite subintervals, converts the integral into an infinite series, and extrapolates its limit by repeatedly averaging adjacent terms.
  • 17 of 19 held-out integrals were evaluated correctly within a fractional error of less than 3%.
  • The evolved code applies quad() first and invokes specialized fallbacks only for large error estimates, NaN or Inf results, or exceptions, enabling drop-in replacement use.

A.4. Towards Genuine Discovery · A.5. Computational Cost and Resource Utilization

ERA’s tree-search deployments demonstrate potential for genuine scientific discovery, including novel symbolic derivations yielding an exact closed-form solution. The approach is more resource-intensive than single-shot or zero-shot prompting because it iteratively generates and empirically evaluates candidate solutions.

  • A.4. Towards Genuine Discovery: ERA’s tree-search algorithm demonstrated potential for genuine scientific discovery across several recent deployments.
  • A.4. Towards Genuine Discovery: Six novel analytical derivations produced a closed-form exact solution for the previously unsolved cosmic-string radiation-spectrum integral.The system searched symbolic mathematics without empirical data or predictive modeling.
  • A.4. Towards Genuine Discovery: Mathematical discovery required an objective that scored candidate solutions for discovering a closed-form analytical equation for cosmic-string radiation spectra.Designing such metrics for open-ended scientific domains requires domain expertise and scientific creativity.
  • A.4. Towards Genuine Discovery: The cosmic-string example illustrates how automated search can be extended from standardized predictive-modeling scores to novel, open-ended scientific objectives.
  • A.5. Computational Cost and Resource Utilization: Search-based LLM optimization is inherently more resource-intensive than single-shot or zero-shot prompting.Its cost arises from iterative generation and empirical evaluation.
  • A.5. Computational Cost and Resource Utilization: Representative computational costs per search node are reported for search runs across the evaluated benchmarks in Supplementary Table S1.

Supplementary Figures · B. Supplementary Tables

The supplementary figures document ERA’s experimental design, generated-method behavior, and performance across single-cell integration, COVID-19 forecasting, and additional scientific tasks. Supplementary tables provide computational-cost information, benchmark and prompting materials, and example generated code.

  • Supplementary Figures: ERA’s single-cell integration search used CELLxGENE data, 20,000-cell training and validation splits, 11 base methods, and three independent replicas per method.The top-performing node from each tree was selected using training scores before evaluation on holdout data.
  • Supplementary Figures: BBKNN (TS) produced cell-type-specific UMAP clusters while showing good batch mixing across the Immune Cell Atlas dataset.The projections were colored separately by cell type and data batch.
  • Supplementary Figures: ERA-generated methods formed major non-deep-learning and deep-learning clusters, while pairwise text-embedding similarities among single-cell solutions exceeded 0.85.The single-cell embeddings were hierarchically clustered, and Gemini classification confirmed the two-cluster interpretation.
  • Supplementary Figures: Three tree-search replicas showed consistent high-performing probabilistic forecasts for COVID-19 hospitalizations across rolling validation dates.Lower WIS indicates superior forecasting accuracy and calibration.
  • Supplementary Figures: The best COVID-19 recombination model, CEPH-Rtrend_covid x CMU-climate_baseline, was identified as second-best during validation and performed best on the held-out test set.Supplementary figures compare it with the CovidHub-ensemble and other forecasting models across jurisdictions and validation splits.
  • Supplementary Figures: The numerical integration routine solved 17 of 19 held-out integrals within 3 percent, whereas scipy.integrate.quad() failed in all these cases.The benchmark contained 38 oscillatory semi-infinite-domain integrals, none correctly solved by scipy.integrate.quad().
  • B. Supplementary Tables: Supplementary Table S1 reports per-node language-model tokens, sandbox duration, hardware requirements, and linear scaling of computational cost with explored nodes.The integral-task request token count is artificially inflated by image tokenization, and the study used NVIDIA Tesla T4 GPUs.
  • B. Supplementary Tables: Supplementary Tables S2–S8 contain the Kaggle Playground competitions, prompts, expert advice, boosted-decision-tree prompts, and example ERA-generated BBKNN (TS) code.The materials cover Season 3 competitions and include an example solution for single-cell batch integration.

Example code generated by ERA.

ERA-generated code implements a batch-integration workflow that preprocesses single-cell data, applies ComBat correction, and constructs a batch-aware nearest-neighbor graph. The code also includes safeguards for small datasets and configurable PCA and neighbor parameters.

  • Preprocessing and batch correction: The workflow normalizes, log-transforms, and scales single-cell data before applying ComBat batch correction to the gene-expression matrix.Scaling clips values at max_value =10, while normalization uses target_sum =1e4.
  • Embedding: The ComBat-corrected PCA embedding is assigned as X_emb for evaluation with metrics including ASW, LISI, and PCR.The code limits PCA components to the available observations and variables.
  • Edge-case handling: For datasets too small for meaningful PCA or graph construction, the code returns a trivial embedding and empty graph structures instead of failing.The fallback records a degenerate neighbor method with n_neighbors = 0 and n_pcs = 0.
Loading 2509.06503v3…