Source-linked AI summary
Science sandboxes measure the scientific capability of AI agents
Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai, Kenneth B. Hsu, Yasha Ektefaie, Shantanu Singh, Sangeeta N. Bhatia, Steven K. Reilly, Ryan Tewhey, Eric S. Lander, Pardis C. Sabeti
TL;DR
AI agents need to do more than optimize scientific scores: they must infer rules and use them to design informative experiments. This paper introduces science sandboxes, closed-loop evaluations spanning physical, predictive, and invented oracles, and applies them to biological settings. Agents often achieved strong quantitative performance using familiar biological priors, but their rule inference deteriorated on systems outside those priors.
Problem
Scientific capability requires evidence that an agent can learn rules explaining why solutions work, not merely raise a benchmark score.
Method
Science sandboxes evaluate agents through repeated cycles in which they choose specimens, receive feedback from a sealed oracle, revise hypotheses, and select subsequent experiments.
Results
Agents achieved high quantitative performance using familiar biological priors, but their ability to infer underlying rules collapsed on synthetic or non-canonical systems outside those priors.
Takeaways & Limitations
Science sandboxes make the frontier of scientific capability measurable by evaluating both score optimization and rule discovery across different forms of experimental feedback.
Takeaways & Limitations
Damp oracles inherit the data biases and assumptions of their predictive models, while dry oracles trade real-world physical complexity for exact ground truth.
Abstract
from arXiv · showhide
Scientific progress depends not only on finding solutions, but on learning the rules that explain why they work and using that understanding to design better experiments. We introduce science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision. Science sandboxes invite an agent to query the natural world in different ways, ranging from "wet" physical experiments, to "damp" predictive models trained on empirical data, to "dry" invented rules. By establishing a common experimental loop and a protocol for evaluating agents within it, science sandboxes allow assessment of both quantitative performance on specific metrics and qualitative scientific reasoning, across a spectrum of empirical verifiability. Here, we instantiate this framework in two biological settings, models of regulatory genomics and protein fitness prediction, and examine the capabilities of frontier agents. Across these settings, we could see when agents successfully optimized a quantitative metric without understanding the rules underlying the system. In particular, their scientific reasoning deteriorated when they encountered systems whose rules fell outside familiar biological priors. By highlighting such failure modes, science sandboxes make the frontier of scientific capability measurable and provide a controlled setting in which to study and ultimately expand it.
Introduction
Science requires discovering rules that explain why solutions work, not merely finding high-scoring solutions. Science sandboxes evaluate whether AI agents can infer hidden rules through successive experiments, feedback, and hypothesis refinement.
- Scientific progress depends on learning rules that explain why observed solutions work and guide the design of subsequent experiments.
- Science sandboxes preserve an investigator’s cycle of experimentation, interpretation, hypothesis revision, and experiment selection while hiding the system’s rules.
- MPRAbox tests informative regulatory-DNA library selection, whereas CodonBox tests de novo discovery of rules linking sequence to function in unfamiliar genetic systems.
Results
In MPRAbox, agents improved library-design performance and, over repeated experiments, developed and revised hypotheses about informative sequence content. Dry-oracle tests showed that framing could alter exploration and quantitative progress without reliably producing hidden-rule discovery.
- Single-round performance: Claude achieved median r = 0.774 without prior knowledge, exceeding the best human-selected strategy’s mean r = 0.763; Gemini and GPT reached r = 0.680 and r = 0.655.Neither Gemini nor GPT exceeded the best human reference.
- Single-round performance: Agents shifted toward genomic sequence after seeing human strategies, with GPT improving from r = 0.655 to r = 0.760 and Gemini from r = 0.680 to r = 0.751.Claude changed less because it had already favored genomic regulatory DNA; the human reference mean was r = 0.763.
- Multi-round hypothesis revision: The four 30-round runs exceeded the strongest human-selected strategy, but quantitative gains over single-round tests were modest; multi-round play primarily exposed evolving scientific reasoning.The runs showed how agents built, tested, and revised theories while using iterative feedback and a finite experimental horizon.
- Multi-round hypothesis revision: Agents increasingly used controlled experiments that changed one library component at a time, after recognizing that simultaneous changes could produce wrong conclusions.One agent explicitly identified changing cCRE and chicken sequences together as the source of an incorrect inference.
- Multi-round hypothesis revision: Across four 30-round runs, agents developed different hypotheses about informative libraries, including native genomic context, rare-class enrichment with broad coverage, cross-species augmentation, and GC-content stratification.The fourth agent found that stratifying sequences by GC content further improved performance.
- Dry-oracle rule discovery: Across 14 invented rules, task framing changed agents’ investigations and sometimes quantitative progress, but biological priors could provide useful proxies without yielding rule understanding.For the hidden English-word cipher, none of the three framings led the agent to consider character-pair encoding or decoded language.
Discussion
Science sandboxes distinguish optimizing benchmark scores from discovering the rules that explain system behavior. Their results show that agents can perform well within familiar biological priors while failing to infer unfamiliar rules, and that notebooks expose this gap.
- Science sandboxes test whether agents discover governing rules rather than merely optimize benchmark scores.
- Agents achieved high quantitative performance in familiar biological settings but their rule inference collapsed on unfamiliar positional, mathematical, or combinatorial systems.
- Agents often exhausted experimental budgets through brute-force search, revealing weaknesses in experimental strategy even when quantitative performance was strong.
- Wet, damp, and dry oracles trade physical realism, scalability, and exact ground truth in evaluating scientific capability.
- A minimal agent-and-oracle architecture supports sandboxes spanning physical laboratories, predictive surrogates, and synthetic rule systems.
Definition of a science sandbox
A science sandbox is a closed-loop environment where an agent selects specimens, receives sealed-oracle feedback, and revises its hypotheses across experiments.
- A science sandbox specifies specimens, assays, and a sealed oracle that returns assay reports.
- Each round links the agent’s specimen selection, oracle feedback, hypothesis revision, and next experiment.
- The framework evaluates both final performance and the sequence of hypotheses, experiments, interpretations, and revisions.
- Wet oracles perform physical experiments, damp oracles use empirical-data models, and dry oracles apply designer-invented rules.
The MPRAbox task and sandbox
MPRAbox treats regulatory-sequence library design as sequential experimentation: agents construct libraries, receive anonymized scores, and use repeated feedback to guide subsequent designs.
- MPRAbox asks agents to design libraries of 50,000 DNA sequences, each 200 bp long, for training sequence-to-activity models.
- Agents submit libraries through generate.py and prepare.py, which validates them and invokes the sealed in silico MPRA oracle.
- The oracle returns Pearson correlations for 14 evaluation sets and their mean overall performance score, without identities or sequence-level predictions.
- One-shot experiments compared agents with and without descriptions and scores for 14 human-selected strategies across 30 total runs.
- The long-horizon regime allowed Claude Opus 4.7 to submit 30 successive libraries, with four independent runs split across prior-knowledge conditions.
- Agents maintained append-only notebooks and reusable skill files, rereading them at the start of each long-horizon round.
In silico MPRA oracle
The MPRAbox oracle uses Malinois to simulate assay results, trains a new sequence-to-activity model on each submitted library, and evaluates it on hidden test sets.
- Malinois predicts MPRA activity for each 200 bp sequence across K562, HepG2, and SK-N-SH cells.
- The oracle uses predicted sequence activities to train a sequence-to-activity model from scratch for each candidate library.
- The resulting model is evaluated on 14 hidden test sets to score library informativeness.
Downstream model training and scoring
Each submitted library was used to train an independently initialized downstream sequence-to-activity model under a fixed architecture, objective, optimization procedure, and evaluation pipeline. Models predicted MPRA log2 fold-change for three cell types from 600 bp reporter-context inputs.
- Each submitted library trained a fresh model from random initialization without sharing weights with the Malinois oracle.
- Models trained only on 50,000 submitted sequences and Malinois-generated labels, using a 10% validation split and early stopping for up to 200 epochs.Downstream models used single-strand inputs and single forward-pass evaluation; reverse-complement averaging was restricted to Malinois label generation.
- The model embedded each 200 bp candidate sequence in Malinois’s fixed reporter-vector context to form a one-hot encoded 600 bp input.
- A convolutional encoder and shared fully connected layer fed separate K562, HepG2, and SK-N-SH output branches.Each branch returned predicted MPRA log2 fold-change for one cell type.
- Training used the inherited Malinois objective, combining L1 activity loss with a KL-divergence term, and optimized with Adam and AMSGrad.The learning rate was 3.27 × 10^-3, with weight decay 3.44 × 10^-4.
Evaluation suite
The evaluation suite tested trained models on 14 held-out sets spanning nine source distributions. It combined empirical MPRA labels for direct measurement-based evaluation with Malinois-generated labels for broader sequence distributions.
- Models were evaluated on 14 held-out evaluation sets derived from nine source distributions.Five sets used empirical MPRA measurements and nine used Malinois-generated labels.
- Empirical-label sets enabled direct evaluation against measured regulatory activity, while oracle-labeled sets extended testing to distributions absent from experimental data.
- The suite included held-out MPRA reporter sequences, UK Biobank and GTEx variants, DHS regions, Sei regulatory classes, random genomic windows, and synthetic random sequences.
- Chromosome-level holdout excluded chromosomes 7, 13, 19, 21, and X from Malinois training for genome-derived evaluation sets.Variant sets used 200 bp windows centered on candidate causal variants.
Human-curated baseline strategies
The study constructed a reproducible reference panel of 14 human-curated library-design strategies. Each strategy specified a sampling rule and was replicated across five independent seeds, with comparisons centered on 50,000-sequence libraries.
- Fourteen human-curated strategies defined sampling rules rather than fixed libraries, with five independent libraries generated per strategy.Replicates differed only in sequences sampled under the same design procedure.
- The strategies sampled accessible chromatin, Sei regulatory classes, synthetic random DNA, or previously assayed MPRA sequences.
- DHS strategies used NMF topic annotations to sample by topic loading or stratify elements across regulatory programs.The annotations summarized accessibility patterns across biosamples into 16 regulatory programs.
- Sei regions were sampled either by class frequency or equal allocation, while synthetic libraries drew nucleotides uniformly from {A, C, G, T}.
- Strategies were evaluated at library sizes from 10,000 to 300,000 sequences, with 50,000 as the primary comparison point.
Agent configuration and regimes
The study used three provider-native coding-agent harnesses in a sandboxed Linux environment. Agents could autonomously use standard computational and internet tools, without a third-party scaffold.
- The harnesses instantiated Claude Opus 4.7 in Claude Code, GPT-5.5 in Codex CLI, and Gemini 3.5 Flash in Gemini CLI.
- All agents operated in sandboxed Linux environments with shell, Python, filesystem, and internet access configured for autonomous tool use.
- The study otherwise used each provider’s default native-harness parameters and introduced no third-party agent scaffold.
Dry oracles with invented rules in MPRAbox
MPRAbox uses invented dry oracles to test whether agents can design informative sequence libraries when the underlying rules are hidden. Performance is evaluated through predictive-model correlation under multiple task framings.
- Dry-oracle construction: Fourteen invented rules served as dry oracles, replacing Malinois while preserving the library-design task and three-output activity format.The oracle-generated sequence labels were hidden from the agent.
- Scoring procedure: Each submitted library was represented by normalized 6-mer frequencies and used to train a three-output Ridge regression model with α = 1.0.The model was trained on submitted sequences and their oracle-generated labels.
- Scoring procedure: Performance was measured by Pearson correlation between predictions and oracle-generated labels, averaged across outputs and evaluation sets.The evaluation averaged results across the 14 MPRAbox evaluation sets.
- Experimental framings: Claude Opus 4.7 evaluated all 14 dry oracles under MPRA-framed, unframed, and symbolic conditions.The hidden rule was identical across framings, and each oracle-framing combination ran independently for 30 rounds.
CodonBox implementation
CodonBox evaluates rule discovery in invented genetic worlds by hiding the sequence-to-fitness mapping while keeping protein folding physics fixed. The worlds vary genetic-system properties, allowing agents to confront unfamiliar rule combinations.
- Sandbox design: CodonBox presents an agent with an invented biological world and returns a single fitness number for each submitted candidate sequence.A sealed oracle applies the world-specific hidden rule and folds the resulting product with a fixed physical model.
- Folding model: The physical substrate is Dill’s HP model, representing proteins as strings of hydrophobic (H) and polar (P) residues.Every translated product is a 16-residue H/P string folded on a two-dimensional square lattice.
- Folding model: Fitness equals the number of favorable non-consecutive H–H contacts in the lowest-energy fold, with a ceiling of 9 contacts.The contact value was enumerated for all 65,536 possible 16-residue H/P strings, fixing the folding physics across worlds.
- World construction: Eight worlds vary codon length, alphabet size, silent positions, and whether informative positions act additively or interact.A control world mirrors familiar genetics with a four-character alphabet and three-character codons.
- Reproducibility: The sandbox code is publicly available in the science-sandbox repository.The repository contains the code used to produce the presented sandboxes.
Supplementary Figures and Tables
The supplementary material details evaluation views for MPRAbox, agent instructions, and the construction of the 14-set evaluation used to score libraries. It also compares performance across library sizes and describes one-shot and iterative experimental settings.
- Evaluation and performance: Figure S1 varies training library size from 10,000 to 300,000 sequences for each of 14 human-curated strategies, showing means over five seeds with ±1 SD bands.The figure organizes one strategy per panel and displays performance as library size changes.
- Evaluation and performance: Figure S2 compares one-shot library performance by model across held-out empirical MPRA and oracle-labeled regulatory, variant, genomic, and synthetic sequence sets.The held-out evaluation includes multiple sequence-set types rather than a single assay category.
- Agent comparisons: Figure S4 places one-shot model results and best long-horizon designs against human-best and random-floor reference lines in blind and informed conditions.The comparison distinguishes individual one-shot points from best-of-run long-horizon designs.
- Agent instructions: The MPRAbox instructions require exactly 50,000 sequences of length 200, prohibit modifying the black-box evaluator, and permit only one evaluation call in the one-shot setting.Agents must design before evaluation and document their reasoning, analyses, decisions, and proposed next experiment.
- Evaluation and performance: Performance is reported across 14 anonymous evaluation sets, each returning mean_r, k562_r, hepg2_r, and sknsh_r, with no single primary metric.The evaluation uses hidden test sets and requires optimization across all reported metrics.