PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

Shanghua Gao, Ada Fang, Marinka Zitnik

arXiv:2605.28655v1cs.AI

TL;DR

Existing AI agents often struggle to sustain parallel, adaptive exploration during long-running scientific experimentation. AutoScientists uses decentralized, self-organizing agent teams that share evidence and critique proposals, improving over prior agents across biomedical machine learning, language-model training optimization, and protein fitness prediction under matched experimental budgets.

  • Problem

    Existing AI agents typically follow single trajectories or fixed decompositions, limiting parallel exploration as scientific directions change over time.

  • Method

    AutoScientists is a decentralized agent team that shares experimental state, critiques proposals before execution, and dynamically reorganizes around productive hypotheses.

  • Results

    AutoScientists improves over prior AI agents across biomedical machine learning, language-model training optimization, and protein fitness prediction, achieving a 74.40% mean leaderboard percentile across 24 BioML-Bench tasks.

  • Takeaways & Limitations

    AutoScientists supports long-running experimentation as a collective search process that records failures, reduces repeated dead ends, and continues finding productive modifications after individual agents plateau.

  • Takeaways & Limitations

    AutoScientists uses more LLM tokens than Autoresearch and is designed to improve experimental search under fixed experimental-compute budgets rather than maximize LLM-call efficiency.

Abstract

from arXiv · show

Scientific research proceeds through iterative cycles of hypothesis generation, experiment design, execution, and revision. AI agents can automate parts of this process, but existing approaches typically follow a single research trajectory or coordinate through a central planner with fixed objectives. As a result, they struggle to sustain parallel exploration, adapt as experimental evidence changes, or preserve knowledge of failed directions over long-running experiments. We introduce AutoScientists, a decentralized team of AI agents for long-running computational scientific experimentation. Agents interpret a shared experimental state, self-organize into teams around promising hypotheses, critique proposals before using experimental compute, and share successes and failures to reduce redundant exploration. Under matched experimental budgets, AutoScientists improves over prior AI agents across biomedical machine learning, language-model training optimization, and protein fitness prediction. On BioML-Bench, spanning biomedical imaging, protein engineering, single-cell omics, and drug discovery, AutoScientists achieves a mean leaderboard percentile of 74.4% across 24 tasks, improving over the strongest AI agent by +8.33%. On GPT training optimization, AutoScientists reaches a target validation bits-per-byte 1.9x faster than Autoresearch and continues discovering improvements from a starting champion where the single-agent approach finds none (7 vs. 0 accepted improvements). On ProteinGym fitness prediction, AutoScientists discovers a method for ACE2-Spike binding that improves over the current state-of-the-art model by +12.5% in Spearman correlation. Applied without modification across all 217 ProteinGym assays, the same method improves over the prior state of the art by +6.5% (Spearman correlation).

1 Introduction

AUTOSCIENTISTS is a decentralized, self-organizing agent team for long-running scientific experimentation, addressing the limits of single trajectories, fixed decompositions, and centralized coordination. Across benchmarks, it improves over existing AI agents, including on BioML-Bench and drug-discovery tasks.

  • Motivation: Long-running experimentation requires maintaining competing hypotheses, updating them as evidence changes, and using failures to redirect the search.Single-agent systems follow one search trajectory, while existing multi-agent systems coordinate through a central structure.
  • Present Work: AUTOSCIENTISTS lets agents interpret shared experimental state and dynamically form teams without a central orchestrator or user-specified decomposition.The shared state records proposals, experiments, results, failures, and the current champion.
  • Present Work: Peer critique filters experiment proposals before execution, allowing teams and experimental directions to emerge through interaction.Agents post proposals to a shared forum where peers critique them before experiments run.
  • Benchmark Results: 74.40% average leaderboard percentile across 24 biomedical ML tasks exceeded Autoresearch’s 66.07% under the same task interface, model backend, and hardware budget.AUTOSCIENTISTS achieved the highest average leaderboard percentile among the evaluated agents on BioML-Bench.
  • Benchmark Results: 64.52% leaderboard percentile in drug discovery improved over 46.16% for the comparison condition.The passage identifies drug discovery as the area with the largest performance improvements.

2 Related Work

Prior work applies AI agents to scientific workflows, including biomedical analysis and longer-horizon discovery. Multi-agent research also emphasizes that collaboration structure and agent composition can affect performance, and that interaction is not automatically beneficial.

  • AI Agents for Scientific Research: AI agents increasingly automate scientific workflows such as literature review, hypothesis generation, tool use, code execution, experimental design, benchmarking, and manuscript drafting.
  • AI Agents for Scientific Research: Biomedical agents combine multi-step reasoning with biomedical tools, literature grounding, omics analysis, code execution, and evidence reconciliation.
  • Coordination of Multi-Agent Systems: Multi-agent performance depends strongly on collaboration structure and agent composition, while interaction is not automatically beneficial.
  • Coordination of Multi-Agent Systems: Multi-agent systems have underperformed their best individual member on some tasks, motivating analyses of collaboration, competition, and comparisons with single-agent baselines.

3 AUTOSCIENTISTS: Long-Running Self-Organizing Agent Teams

AUTOSCIENTISTS formalizes scientific experimentation as a long-running iterative search in which persistent agents self-organize into teams, run parallel experiments, and update shared state from accumulated evidence. The system adapts team structure as directions succeed or stall while preserving successful and failed experimental knowledge.

  • Long-Running Search: AUTOSCIENTISTS uses n persistent agents that iteratively propose, train, and evaluate programs while maintaining internal state and adapting from accumulated experience.The search begins from a task description, optional initial program p0, dataset D, and evaluation metric ℓ.
  • Discussion and Self-Organization: Agents alternate discussion phases, where they debate directions and form teams without predefined partitions, with execution phases that run experiments in parallel and write results to shared state S.The roster can be adopted by agents on subsequent heartbeat cycles.
  • Discussion and Self-Organization: When evidence indicates a direction is exhausted, teams can create, merge, split, or rebalance around more productive hypotheses, while dead-end registries prevent repeating unproductive experiments.Reorganization may follow consistent failure to improve the current champion, such as no improvement in the last 10 experiments.
  • Long-Running Parallel Experiments: Teams use analyst and experiment roles to prioritize underexplored directions, execute queued changes, record outcomes, and confirm improvements within the empirical noise band on a second seed.All results, including failures, are visible across teams, enabling coordinated search under stochastic evaluation.
  • Output: The system outputs a final champion with reproducibility artifacts and preserves the search history, including failed directions, performance changes, rejection reasons, and mechanisms underlying successful modifications.These records document how hypotheses evolved and which directions were explored during the run.

4 Experiments

Across biomedical benchmarks, GPT training optimization, and protein fitness prediction, AUTOSCIENTISTS achieves strong results under matched experimental settings and sustains improvement beyond existing methods. Ablations indicate that analyst support, cross-agent feedback, and self-organization are important for different tasks.

  • BioML-Bench: AUTOSCIENTISTS evaluates 24 BioML-Bench tasks spanning biomedical imaging, drug discovery, protein engineering, and single-cell omics.The benchmark provides task descriptions, training data, test inputs, and example submission formats, while hidden labels remain external to the agent workspace.
  • BioML-Bench: 74.40 (6.20)% mean leaderboard percentile exceeds Autoresearch’s 66.07 (7.38)% by +8.33 leaderboard-percentile points, with all 24 tasks completed.AUTOSCIENTISTS achieves the highest aggregate leaderboard percentile among evaluated systems.
  • GPT training optimization: 1.9× fewer experiments are required for AUTOSCIENTISTS to reach val_bpb ≈0.978 from the Autoresearch baseline, using 34 experiments versus 65.In the champion regime, AUTOSCIENTISTS accepts seven improvements over 93 experiments and reaches val_bpb = 0.9730, while Autoresearch accepts zero over 100 experiments.
  • Protein fitness prediction: 12.5% relative improvement raises mean Spearman’s ρ from 0.747 for Kermut to 0.840 on the ACE2–Spike development assay.The discovered predictor is a three-GP ensemble combining Kermut’s structure-kernel with expanded zero-shot features, greedy diversity-based feature selection, and quantile-warped targets.
  • Ablations: The most damaging ablation varies by task: removing the analyst reduces TDC-hERG AUROC from 0.867 to 0.738, while removing self-organization changes GPT val_bpb from 0.9777 to 0.9833.Removing cross-agent feedback most damages Human Plasma-Protein Binding, reducing Pearson correlation from 0.8729 to 0.7144.

5 Limitations and Future Work

AutoScientists is not designed to reduce LLM-call costs relative to single-agent baselines. Instead, it uses additional token expenditure for parallel reasoning, discussion, and team reorganization to improve experimental search under a fixed experimental-compute budget.

  • Efficiency trade-off: AutoScientists is not more LLM-call efficient than single-agent baselines.Its objective is experimental search quality rather than minimizing language-model calls.
  • Efficiency trade-off: AutoScientists uses more LLM tokens than Autoresearch, though within the same order of magnitude.The additional usage reflects multiple agents conducting parallel reasoning, discussion, and team reorganization.
  • Future direction: AutoScientists is designed to improve experimental search under a fixed experimental-compute budget.Teams of agents explore and collaborate over the design space rather than following a single-agent trajectory.

6 Conclusion

AUTOSCIENTISTS is a self-organizing agent team for long-horizon scientific experimentation that improves over state-of-the-art AI agents across three experimental domains under matched budgets. Its collective search process evaluates proposals, shares evidence and failures, and reorganizes teams to sustain productive exploration.

  • 6 Conclusion: AUTOSCIENTISTS improves over state-of-the-art AI agents across BioML-Bench, GPT training optimization, and ProteinGym under matched experimental budgets.The system is designed for long-horizon scientific experimentation rather than isolated experiments.
  • 6 Conclusion: Agents evaluate proposals before execution, record successful and failed directions, share evidence through a common state, and reorganize teams when progress stalls.These mechanisms make long-running experimentation a collective search process.
  • 6 Conclusion: This design helps agents use experimental trials more effectively, avoid repeated dead ends, and continue identifying productive modifications after individual agents plateau.It addresses the difficulty existing AI agents face in sustaining experimental search as evidence accumulates and productive directions change.

Reproducibility Statement

The authors provide AutoScientists’ source code and launch scripts, alongside the official repositories used for the benchmarked systems and datasets.

  • AutoScientists’ repository includes launch scripts for running all outlined experiments and identifies the official repositories used for BioML-Bench, ProteinGym, Autoresearch, Biomni, and Kermut.Biomni is used with the base LLM Claude Sonnet 4.6.

Impact Statement

AutoScientists may accelerate machine learning and AI for Science by improving exploration, experimental compute efficiency, and documentation of successful and failed hypotheses. These benefits are especially relevant to costly computational biomedical and scientific ML experiments, while over-trusting automatically discovered models remains a risk.

  • Potential benefits: AutoScientists may help researchers explore modeling choices, use experimental compute more efficiently, and document successful and failed hypotheses.The proposed documentation includes shared logs, dead-end registries, model cards, and research reports.
  • Applicability: These benefits are most relevant in computational biomedical and scientific ML settings where experimental iteration is costly.
  • Risks: Over-trusting automatically discovered models is identified as a main risk of AutoScientists.

A Implementation Details and Algorithmic Protocols … B Extended Ablation Results

AutoScientists combines deterministic heartbeat execution with self-organized team formation, shared-state coordination, queue-based experimentation, noise-aware validation, and analyst-driven proposal generation. Extended ablations compare component removals, Autoresearch, and alternative team sizes.

  • A Implementation Details and Algorithmic Protocols: AutoScientists runs through launch, self-organized team formation, and repeated heartbeat-driven experimentation and analysis.Agents begin without team assignments, discuss research directions, vote on proposals, and write the team roster before normal cycles begin.
  • A.1 Setup and Launch: Each invocation reads shared state and enters discussion, no-team exit, pending-result posting, or role-specific normal-cycle branches.Long-horizon coordination emerges by repeatedly invoking single-session heartbeats that inspect persistent agent state and shared state S.
  • A.2 Heartbeat Protocol: Analysts trigger re-discussion under stagnation, audit untested parameters, rank directions by empirical effect, and queue two proposals under ambition and diversity rules.After a champion change, at least one proposal must target the property responsible for the improvement.
  • A.3 Self-Organized Team Formation: Teams are formed and reformed through agent discussion, ranked hypotheses, voting, and roster writing without an external coordinator.A majority [DISCUSS-DONE] vote ends discussion, and the alphabetically-last participating analyst consolidates the roster, which may create, merge, split, retire, or rebalance teams.
  • A.4 File Discovery: The list–decide–read protocol lets agents discover new shared-state artifact types without changing the access mechanism.Agents first retrieve lightweight metadata, select relevant items, and then fetch their contents.
  • A.5 Queue and Claim Protocol: Optimistic locking makes team-queue updates atomic and prevents duplicate experiment claims.Stale writes are rejected and retried against the latest queue version; records include experiment identifiers, priorities, diffs, proposing agents, and proposal links.
  • A.6 Noise-Aware Champion Validation: M = 2 defines the noise band: candidates with ∆ > Mσ are promoted, 0 < ∆ ≤ Mσ requires a second-seed confirmation, and ∆ ≤ 0 is rejected.The gate prevents champion pollution caused by promoting stochastic improvements that are only noise.
  • B Extended Ablation Results: Extended ablations compare three component removals, Autoresearch, and working-agent counts n ∈{2, 4, 14} against the default crew n = 9.The comparisons cover GPT training optimization and BioML-Bench, with the single-component removals reported in the main-paper ablation table.

B.1 Comparison Against Autoresearch at Matched and Extended Compute · B.2 Team-Size Sensitivity: Parallelism Gain and Oversubscription · C Run-to-Run Stability

Across matched and extended compute, team-size sweeps, and independent cold starts, AUTOSCIENTISTS improves GPT training optimization, exhibits task-dependent crew-size sensitivity with oversubscription at n=14, and shows stable run-to-run behavior. Larger crews reduce parallel-execution time, while three cold-start runs converge to a similar best-so-far region.

  • B.1 Comparison Against Autoresearch at Matched and Extended Compute: 0.9777 vs. 0.9790: after 50 experiments from the original Autoresearch setup, AUTOSCIENTISTS reaches lower val_bpb than Autoresearch.Both systems start from val_bpb = 0.998; AUTOSCIENTISTS reaches 0.9777, while Autoresearch reaches 0.9790.
  • B.1 Comparison Against Autoresearch at Matched and Extended Compute: 7 KEEPs: from the AUTOSCIENTISTS champion at val_bpb = 0.9777, 100 additional experiments reach 0.9730.This is the extended-compute setting initialized from AUTOSCIENTISTS’s matched-compute champion.
  • B.2 Team-Size Sensitivity: Parallelism Gain and Oversubscription: 0.780–0.867 AUROC: TDC-hERG shows substantial variation across crew sizes, while ProteinGym SPIKE-SARS2 spans 0.506–0.874 Spearman ρ.The corresponding leaderboard-percentile ranges are 14.3–85.7 and 72.7–100.0, respectively.
  • B.2 Team-Size Sensitivity: Parallelism Gain and Oversubscription: 0.9777–0.9821 val_bpb: GPT training optimization also varies by crew size, and n=14 degrades relative to its best score on every task.The crew size achieving the top score differs across tasks, indicating task-dependent sensitivity.
  • B.2 Team-Size Sensitivity: Parallelism Gain and Oversubscription: 0.9777, 0.9778, and 0.9821: the n=2, n=4, and n=9 crews achieve similar final GPT-training val_bpb, whereas n=14 is worse.Figure S1 reports 11 KEEPs / 51 exps for n=2, 7 KEEPs / 38 exps for n=4, 11 KEEPs / 71 exps for n=9, and 15 KEEPs / 50 exps for n=14.
  • C Run-to-Run Stability: 0.9784 mean final val_bpb: three independent cold-start runs have sample standard deviation 0.0010 and range 0.0018.Final values are 0.9777, 0.9795, and 0.9780 after 75, 64, and 62 experiments, with comparable acceptance rates of 10.7%, 10.9%, and 9.7%.
  • B.2 Team-Size Sensitivity: Parallelism Gain and Oversubscription: ≈4 heartbeats: under fully parallel execution, n=14 finishes fastest, followed by n=9 at ≈8, n=4 at ≈10, and n=2 at ≈26.One heartbeat is a synchronized cycle in which all n working agents fire one LLM call simultaneously.

D Per-Experiment Trajectories for Ablation Runs … E.1 Model Card

The ablation trajectories show that self-organization, communication, analyst support, and shared coordination each affect long-running GPT nanochat optimization, while the model card documents the resulting 87 M-parameter research artifact and its constrained evaluation setting.

  • D Per-Experiment Trajectories for Ablation Runs: Ablation trajectories compare running best-so-far validation val_bpb from the pristine nanochat baseline across attempted experiments and accepted KEEPs.Transparent markers denote attempts, opaque markers denote accepted improvements, and Autoresearch plus full AUTOSCIENTISTS provide reference trajectories.
  • D.1 No Self-Organization (abl-no-self-org): 5 KEEPs and final val_bpb of 0.9833 were achieved without self-organization over 47 unique experiments, versus 11 KEEPs and 0.9777 for full AUTOSCIENTISTS in 71 experiments.The ablated run tracked the full system for roughly the first 10 experiments before stalling.
  • D.1 No Self-Organization (abl-no-self-org): The no-self-organization run stopped at 47 experiments after a mid-run budget revision, leaving whether additional experiments would close the gap unresolved.The original HANDOVER target was 100 experiments.
  • D.2 No Cross-Agent Communication (abl-no-cross-agent): 9 KEEPs and final val_bpb of 0.9814 were obtained when cross-agent communication was removed, leaving coordination through shared artifacts בלבד.Only proposal and result posts remained enabled; other forum activity and stagnation-triggered re-discussion were disabled.
  • D.3 No Analyst Role (abl-no-analyst): Removing the analyst role left five experiment agents generating proposals, prioritizing axes, inducing post-KEEP insights, and coordinating through the shared workshop and suggestion threads.The analyst role was at-or-below replacement value in this template surface and 50-experiment budget, but n = 1 runs per condition do not support the gap statistically.
  • D.4 Independent Agents (abl-independent): Best-of-population val_bpb of 0.9833 was reached over 50 unique experiments by six independent agents lacking cross-agent coordination, shared artifacts, or visibility into one another’s state.Each agent ran an independent autoresearch loop with a private champion seeded from the pristine upstream.
  • E AUTOSCIENTISTS Output on GPT nanochat: AUTOSCIENTISTS descended from val_bpb = 0.998 to a final champion at val_bpb = 0.97769 over 75 experiments on GPT nanochat training optimization.The run also produced a model card and research insights document following the system’s released templates.
  • E.1 Model Card: The model card describes an 87 M-parameter decoder-only transformer trained for 300 s, processing 339 M tokens, and achieving val_bpb = 0.97769.The model is a short-budget research artifact, substantially underfit relative to publication-scale language models; val_bpb is not a language-task performance score.

E.2 Research Insights … F.4 Performance Across Wall-Clock Time

AutoScientists’ GPT nanochat run produced seven accepted improvements through decentralized experimentation, while BioML-Bench evaluation used matched compute budgets, task-specific metrics, repeated-run checks, and wall-clock tracking. Across the merged sections, the paper details the research trajectory, benchmark setup, datasets, evaluation procedures, and performance analyses.

  • E.2 Research Insights: 75 experiments yielded 8 accepted KEEP decisions, including 7 improvements, over approximately 6.4 hours using 9 agents.The team comprised 6 experiment agents and 3 analyst agents.
  • E.2 Research Insights: The accepted GPT nanochat improvements spanned throughput, capacity, and optimizer quality, with halving batch size from 219 to 218 approximately doubling optimizer steps and accounting for −0.0090.The run also found that step granularity mattered more than total-token volume under fixed wall-clock budgets.
  • E.2 Research Insights: Dead ends included 67 rejected experiments, while schedule-shape rewrites regressed sharply and capacity gains depended on accompanying learning-rate-scale corrections.The findings came from a single task, backend, and 300 s budget, so transferability and refinement-order effects remain untested.
  • F Implementation Details of BioML-Bench: BioML-Bench comparisons used 4 hours on 1 H100 GPU with 16 CPUs and 48 GB memory, except biomedical imaging, which received 16 hours.The protocol adapted tasks into iterative validation-set development loops requiring agents to write training scripts, report validation performance, and submit test predictions.
  • F.2.1 Biomedical Imaging: Biomedical-imaging tasks covered pathology, pulmonary-fibrosis, brain-tumor, and gastrointestinal-tract prediction using ROC-AUC, modified Laplace log-likelihood, and a Dice–HD3D composite metric.These tasks used patient- or training-level validation splits as specified for each dataset.
  • F.2 Task Datasets and Evaluation Metrics: Drug-discovery and single-cell-omics tasks spanned classification, regression, perturbation, communication, label projection, and spatial-gene analysis with task-specific metrics and validation schemes.Drug-discovery development used five Murcko-scaffold groups and 5-fold scaffold cross-validation; single-cell tasks used held-out sites, donors, batches, or within-cell-type splits.
  • F.2.4 Protein Engineering: ProteinGym evaluated out-of-fold predictions across prescribed cross-validation splits using mean Spearman correlation between predicted and measured fitness.BioML-Bench protein-engineering comparisons included matched-compute methods and task scores, leaderboard percentiles, median status, and medal status.
  • F.3 Performance Across Independent Runs: Across 3 independent tdcommons-herg runs, AUTOSCIENTISTS achieved AUROC values of 0.867, 0.830, and 0.862, while wall-clock plots tracked mean Spearman ρ against Autoresearch on six protein-engineering tasks.Protein-engineering tracking was used because identical cross-validation averaging linked validation progress to final leaderboard performance.

F.5 Behavior of AUTOSCIENTISTS on BioML-Bench Tasks · G AUTOSCIENTISTS-Kermut for ProteinGym ACE2-Spike binding DMS · G.1 Setup

Across BioML-Bench, AUTOSCIENTISTS uses task-specific combinations of pretrained representations, boosting, linear models, and handcrafted methods, while its ProteinGym champion extends Kermut into a frozen three-GP ensemble. The ProteinGym setup combines sequence and structure features with region-aware noise, fixed optimization, and per-fold cross-validation.

  • F.5 Behavior of AUTOSCIENTISTS on BioML-Bench Tasks: Foundation-model fine-tuning accounts for 3/4 biomedical-imaging tasks, whereas the fourth combines frozen CNN features, Ridge regression, and PCA-nearest-neighbour retrieval.The imaging pipelines therefore mix end-to-end fine-tuning with fixed embeddings and classical models.
  • F.5 Behavior of AUTOSCIENTISTS on BioML-Bench Tasks: 6/9 drug-discovery tasks use gradient-boosted trees on hand-crafted RDKit fingerprint stacks, while the remaining pipelines include ChemBERTa, residual MLP, and Chemprop models.Chemistry-specific D-MPNNs appear in three tasks.
  • F.5 Behavior of AUTOSCIENTISTS on BioML-Bench Tasks: Five of six ProteinGym BioML-Bench pipelines use frozen ESM-2 embeddings, while only PSAE_PICP2 relies exclusively on fine-tuned models.Two tasks include ESM-2 LoRA fine-tuning, and Ridge or stacked Ridge is the dominant downstream regressor in 3/6 tasks.
  • F.5 Behavior of AUTOSCIENTISTS on BioML-Bench Tasks: 10/24 tasks use Linear/Ridge models, while gradient-boosted trees are the most common primary strategy in 8/24 tasks, including 6/9 drug-discovery tasks.Linear/Ridge models usually serve as meta-learners or downstream probes rather than standalone predictors.
  • G AUTOSCIENTISTS-Kermut for ProteinGym ACE2-Spike binding DMS: AUTOSCIENTISTS-Kermut extends Kermut into a three-GP ensemble whose components share a structure kernel but differ in sequence-side features and kernel families.The ensemble configuration was selected during SARS2-Spike development and then frozen.
  • G.1 Setup: Each variant is represented using ESM-2 embeddings, ProteinMPNN mutation probabilities, ESM-2 zero-shot fitness, and fifteen additional zero-shot predictors.Preprocessing is performed per fold, with z-scored targets and clipped predictions for test-time evaluation.
  • G.1 Setup: Each GP mixes structure and sequence kernels through a learnable weight, while the ensemble also includes an ESM-2 embedding ARD kernel and an 18-dimensional linear-feature kernel.The compact feature vector retains selected log-probability, zero-shot, and contact-map features.
  • G.1 Setup: The model uses region-aware fixed noise and trains every GP for 1000 AdamW steps with a 10^-1-to-10^-3 cosine-style learning-rate decay across 5-fold ProteinGym cross-validation.Per-fold preprocessing, ensemble members, and target normalization are refit independently for each held-out fold.

G.2 Ablations of AUTOSCIENTISTS-Kermut

Ablations show that the ensemble, quantile-normalized targets, and auxiliary zero-shot predictors each meaningfully improve AUTOSCIENTISTS-Kermut’s ranking performance. The ensemble has the largest effect, while auxiliary predictors provide consistent gains across splits.

  • Overall ablation findings: All three ablated components contribute meaningfully to AUTOSCIENTISTS-Kermut’s ranking performance on the SARS2-Spike binding task.The comparison uses the same five-fold cross-validation setup for every ablation.
  • Target normalization: Quantile-normalizing training targets is critical for learning a well-calibrated ranking function, although removing it improves MSE by compressing prediction scale.The ablation replaces rank-quantile mapping to a standard normal with per-fold z-scoring.
  • Auxiliary zero-shot predictors: The 15 auxiliary zero-shot predictors consistently improve both Spearman ρ and MSE across all splits.Their gains suggest diverse evolutionary signal beyond the ESM-2 embedding rather than overfitting to a particular data regime.
  • Ensemble ablation: Removing the ensemble causes the steepest average Spearman ρ drop, from 0.8407 to 0.8231.The three Gaussian processes provide complementary inductive biases, especially on the harder contiguous split requiring sequence extrapolation.
Loading 2605.28655v1…