Science sandboxes measure the scientific capability of AI agents

• Science sandboxes assess scientific capability by requiring agents to conduct successive experiments, interpret hidden-rule feedback, revise hypotheses, and choose more informative experiments.This distinguishes growing understanding from optimization alone, which traditional benchmarks cannot separate.Source: Introduction · p. 2
• The framework spans physical experiments, computational models, and invented systems, preserving agents’ evolving hypotheses and experimental choices to reveal whether improved performance reflects understanding.The feedback source is hidden, so agents must infer system rules from evidence alone.Source: Introduction · p. 2
• The paper introduces MPRAbox and CodonBox to evaluate frontier agentic AI systems on informative regulatory-DNA library selection and de novo discovery of unfamiliar genetic translation rules.Together, the sandboxes test AI as a scientific investigator rather than merely as a solver of scientific tasks.9,10Source: Introduction · p. 2

Science sandboxes measure how agents learn from experiments

• Science sandboxes evaluate agents through closed-loop experiment selection, oracle feedback, hypothesis revision, and lab-notebook reasoning across wet, damp, or dry experimental settings.Their central aim is to distinguish score optimization through local search from forming, testing, and revising hypotheses that explain the system.Source: Science sandboxes measure how agents… · pp. 3, 5

MPRAbox is a science sandbox for regulatory sequence design

• MPRAbox uses a damp oracle to score 50,000-sequence libraries by retraining a sequence-to-activity model and averaging Pearson correlations across 14 hidden evaluation sets.The task asks agents to design informative libraries for generalizable model training while receiving only aggregate performance feedback.Source: MPRAbox is a science sandbox for… · p. 6

Performance of frontier agents in a single-round test with no prior knowledge

• Without prior knowledge, Claude achieved the highest median performance at r = 0.774, while Gemini and GPT reached r = 0.680 and r = 0.655 and neither exceeded the best human reference.Claude consistently selected broadly sampled human genomic regulatory DNA, whereas GPT used synthetic motif perturbations that lacked native genomic context and performed worse.Source: Performance of frontier agents in a… · pp. 9, 11

Performance in single-round testing with prior knowledge

• With prior knowledge, Claude, GPT, and Gemini reached median performances of r = 0.781, r = 0.760, and r = 0.751, respectively, versus r = 0.763 for the best human-selected strategy.Prior knowledge improved every agent, especially GPT and Gemini, which shifted from mostly synthetic designs toward genomic sequences.Source: Performance in single-round testing… · p. 12

Agents refine hypotheses about MPRA library design in multi-round testing

• All four 30-round Claude runs exceeded the strongest human-selected strategy, although quantitative gains over single-round testing were modest.The multi-round evaluation primarily exposed how iterative feedback changed scientific reasoning rather than measuring hill-climbing alone.Source: Agents refine hypotheses about MPRA… · p. 14
• Across rounds, Claude revised its library-design theory after surprising feedback, including recognizing that random DNA could score highly and testing composition-driven signal against regulatory grammar.The agent designed subsequent experiments to distinguish the contributions of synthetic sequences and regulatory sequence content.Source: Agents refine hypotheses about MPRA… · pp. 14–15
• Agents increasingly shifted from broad comparisons to controlled one-variable tests, recognizing that changing cCREs and chicken sequences together had produced a misleading conclusion.The agent explicitly identified isolating one variable at a time as an operational lesson for interpreting whether an intervention works.Source: Agents refine hypotheses about MPRA… · p. 15
• Across four MPRA runs, agents developed different library-design hypotheses: native motif context, rare-class enrichment with broad coverage, chicken-sequence augmentation, and GC-content stratification.The four runs produced different high-performing libraries rather than converging on one design.Source: Agents refine hypotheses about MPRA… · p. 15

Dry oracles with invented rules test the limits of agents’ logical inference

• Across 14 dry rules, task framing changed exploration and sometimes quantitative progress, but it did not reliably produce rule discovery.Biological framing prompted biology-inspired hypotheses, whereas unframed and symbolic presentations led toward nucleotide or symbol frequencies and short patterns; these could provide useful proxies without revealing the hidden rule.Source: Dry oracles with invented rules test… · pp. 17, 19, Figure 4 · p. 18

CodonBox: a science sandbox for invented biological rule discovery

• Agents correctly inferred codon length for k = 2, 3, and 4, but complete codon-by-codon mapping became ineffective as the table grew.With k = 4, the agent learned many codons without recovering the complete table; with j = 6 and k = 3, it failed to discover codon structure despite predicting some score changes.Source: CodonBox: a science sandbox for… · pp. 21–22
• Controlled perturbations enabled agents to recover general CodonBox rules, including silent positions and interactions between codon positions, whereas collecting individual examples alone did not.Changing outer positions revealed a silent middle-position rule, and a counterexample prompted the agent to test combinations and recover an interaction rule.Source: CodonBox: a science sandbox for… · p. 22
• Agents reached maximum CodonBox performance within 10 rounds in every run, yet the hardest 1,296-codon world still defeated recovery of the silent position and general mapping rule.The agent inferred four-nucleotide parsing but increasingly cataloged and reused successful codons, separating high scores from rule discovery.Source: CodonBox: a science sandbox for… · pp. 21–22

Motivation

• Science sandboxes test whether agents discover governing rules rather than merely constructing modern epicycles that optimize benchmark scores.The framework operationalizes the distinction between fitting outcomes and inferring the causes underlying system behavior.Source: Discussion · p. 23

Empirical findings

• Agents optimized target metrics in regulatory genomics but failed to infer unfamiliar positional, mathematical, or combinatorial rules, exposing a gap between performance and scientific understanding.In MPRAbox, agents matched or exceeded human-designed strategies using familiar biological priors, whereas synthetic MPRAbox and non-canonical CodonBox systems caused rule inference to collapse despite continued score optimization.Source: Discussion · p. 23

Qualitative evaluation

• Lab notebooks revealed that agents often exhausted budgets through brute-force search, whereas stronger trajectories combined structured exploration with targeted tests.Notebook inspection can be automated with independent AI judges scoring hypotheses against oracle ground-truth rules, removing the human bottleneck for high-throughput evaluation.Source: Discussion · p. 23

Framework limitations

• Oracle choice creates a realism–scalability–certainty tradeoff: wet experiments are costly and noisy, damp models inherit data biases, and dry rules provide exact ground truth but less physical complexity.The framework spans evaluations from known ground truth to settings enabling genuinely novel scientific discovery, with MPRAbox robustness tested on hidden empirical and diverse oracle-labeled sets.Source: Discussion · pp. 23–24

Future directions

• Science sandboxes require only an agent that selects specimens and an oracle that reports assay results, enabling deployment across physical labs, predictive surrogates, and synthetic rule systems.BroadBox is intended to release new sandboxes and encourage cross-disciplinary contributions for studying and expanding scientific capability at scale.Source: Discussion · p. 24

Definition of a science sandbox

• The sandbox evaluates how an agent learns through repeated cycles of experiment, feedback, and hypothesis revision.Its duration may be one round or many rounds, enabling evaluation of both final performance and the agent’s evolving scientific process.Source: Definition of a science sandbox · p. 25
• Each sandbox specifies a class of specimens, one or more assays, and a sealed oracle that performs assays and returns reports.Source: Definition of a science sandbox · p. 25
• On each round, the agent selects one or more specimens to test, receives the oracle’s assay report, and may use it to choose the next experiment.Source: Definition of a science sandbox · p. 25
• The framework permits assessment of both final performance and the sequence of hypotheses, experiments, interpretations, and revisions.Source: Definition of a science sandbox · p. 25
• Wet oracles perform physical experiments, damp oracles use computational models trained on empirical data, and dry oracles apply invented rules specified by the designer.Source: Definition of a science sandbox · p. 25
• The oracle supplies a verification mechanism for empirical science while allowing the same experimental loop to use different feedback sources.Unlike software or mathematics, empirical sciences such as biology and chemistry generally lack scalable, automated verifiers.Source: Definition of a science sandbox · p. 25
• Across oracle types, the sandbox keeps the underlying rules hidden from the agent while exposing assay results as feedback.Source: Definition of a science sandbox · p. 25

Task definition

• MPRAbox casts library design as sequential decision-making: each round produces 50,000 valid 200 bp DNA sequences to improve models across regulatory distributions and cell types.The target includes contexts that are not directly measured.Source: The MPRAbox task and sandbox · p. 25

Sandbox interface

• Agents submit libraries through generate.py and receive aggregate feedback from sealed prepare.py, which returns Pearson correlations on 14 evaluation sets and their mean overall score.The report withholds evaluation-set identities, sequences, labels, genomic coordinates, and sequence-level predictions.Source: The MPRAbox task and sandbox · pp. 25–26

Experimental regimes

• The one-shot regime used three models, two information conditions, and five replicates each, for 30 runs differing only in access to human-selected strategy descriptions and scores.Agents with prior knowledge received strategy descriptions and performance scores, but not the underlying library sequences.Source: The MPRAbox task and sandbox · p. 26
• The long-horizon regime allowed 30 successive submissions with feedback between rounds and used Claude Opus 4.7 in four independent runs, split between conditions with and without prior strategy knowledge.Long-horizon experiments were limited to one model because of computational cost and throughput.Source: The MPRAbox task and sandbox · p. 26

Agent state

• Long-horizon agents maintained an append-only notebook and reusable skill files, rereading both at the start of each round to retain hypotheses, observations, plans, and procedures.The notebook and skill files were agent-written state.Source: The MPRAbox task and sandbox · p. 26

Oracle model

• Malinois, a deep convolutional neural network trained on 776,474 assayed 200 bp sequences, served as the in silico MPRA measurement oracle.14The training data covered K562, HepG2, and SK-N-SH cells, with reported Pearson r ≈0.88 across the three cell types.Source: In silico MPRA oracle · p. 26
• For each submitted 200 bp sequence, Malinois predicted activity in K562, HepG2, and SK-N-SH cells.14The study used the published checkpoint downloaded directly from the original publication.Source: In silico MPRA oracle · p. 26

Prediction procedure

• Each candidate was embedded in the required fixed 600 bp reporter-vector context, and predictions were averaged across forward and reverse-complement orientations.14Source: In silico MPRA oracle · p. 26

• Each library trained a fresh model from random initialization with the Malinois architecture, but without sharing weights with the oracle.Training used only 50,000 submitted sequences and their Malinois-generated activity labels.Source: Downstream model training and scoring · p. 27
• The model encoded 600 bp inputs and used convolutional layers, a shared fully connected representation, and cell-type-specific heads for K562, HepG2, and SK-N-SH.Each output head predicted MPRA log2 fold-change for one cell type.Source: Downstream model training and scoring · p. 27
• A fixed evaluation pipeline trained models with a combined L1 and KL-divergence objective, Adam optimization, cosine-annealing warm restarts, batch size 512, and early stopping.Downstream models used single-strand inputs and a single forward-pass evaluation, whereas reverse-complement averaging was applied when generating Malinois labels.Source: Downstream model training and scoring · p. 27

Evaluation suite

• 14 held-out evaluation sets from nine source distributions used five empirical MPRA-label sets and nine Malinois-label sets.Empirical labels measured regulatory activity directly, while oracle-labeled sets covered sequence distributions absent from the experimental data.Source: Evaluation suite · p. 27
• The suite spanned reporter, variant, DNase I hypersensitive-site, regulatory-class, random-genomic-window, and fully synthetic random sequences.It included held-out Gosai et al. MPRA reporter sequences, UK Biobank and GTEx variant sequences, Meuleman et al. DHS data, and Sei-assigned regulatory classes.Source: Evaluation suite · pp. 27–28
• Chromosome-level holdout excluded chromosomes 7, 13, 19, 21, and X from Malinois training, while evaluation-set identities were hidden from the agent.Variant sets used 200 bp windows centered on candidate causal variants, retaining either one allele or both reference and alternate alleles depending on the set.Source: Evaluation suite · p. 28
• Performance was computed as Pearson correlation between predicted and labeled MPRA log2 fold-change across included sequences.Source: Evaluation suite · p. 28

• The reference panel comprised 14 human-curated strategies, each implemented as a sampling rule with five independent libraries differing only by sampling seed.The strategies were designed as a reproducible comparison panel rather than fixed libraries.Source: Human-curated baseline strategies · p. 28
• The strategies sampled four sequence sources: DHS-index accessible chromatin, Sei regulatory classes, uniformly random synthetic DNA, and previously assayed Malinois training MPRA sequences.Prior MPRA sequences were evaluated using either Malinois-generated labels or original empirical measurements.Source: Human-curated baseline strategies · p. 28
• DHS strategies used uniform, NMF-topic-weighted, or equal-per-topic sampling across 16 regulatory programs defined by accessibility patterns across biosamples.Topic-weighted sampling followed topic loadings, whereas topic-stratified sampling assigned elements to their maximum-loading topic before equal allocation.Source: Human-curated baseline strategies · p. 28
• Sei strategies sampled regions by observed class frequency or equal class allocation, while the full panel also included two- and three-source mixtures of DHS, Sei, and synthetic sequences.Synthetic libraries sampled each nucleotide independently and uniformly from {A, C, G, T}.Source: Human-curated baseline strategies · p. 28
• Libraries were evaluated at sizes from 10,000 to 300,000 sequences, with 50,000 as the primary comparison point matching agent submissions.The evaluated sizes were 10,000, 25,000, 50,000, 100,000, 150,000, 200,000, and 300,000 sequences.Source: Human-curated baseline strategies · p. 28

Agent configuration

• Three agents—Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash—ran in their respective native coding environments with autonomous access to shell, Python, filesystem, and internet tools.All operated in sandboxed Linux environments; permissions enabled autonomous tool use, while other settings remained at provider defaults.Source: Agent configuration and regimes · p. 29

Oracle construction

• Fourteen dry oracles replaced Malinois with hidden synthetic activity rules spanning sequence composition, patterns, positional rules, compressibility, number theory, cellular automata, and a nucleotide-pair cipher.Each oracle assigned three synthetic activity values per 200-bp sequence while preserving the same library-design task and three-cell-type output format.Source: Dry oracles with invented rules in… · p. 29

Scoring

• Agent performance was scored as the mean Pearson correlation between Ridge-model predictions and hidden oracle labels across three outputs and 14 held-out evaluation sets.The agent received only the final aggregate score, not sequence-level labels, individual outputs, or the hidden scoring rule.Source: Dry oracles with invented rules in… · p. 29

Experimental conditions

• Claude Opus 4.7 was evaluated independently on every oracle for 30 rounds under MPRA-framed, unframed, and symbolic task conditions.The framings changed the stated context or alphabet, while the hidden scoring rule remained identical.Source: Dry oracles with invented rules in… · p. 29

CodonBox implementation

• CodonBox exposes Claude Opus 4.7 to hidden genetic systems through one candidate sequence and a single fitness score per round, separating rule reconstruction from collecting high-scoring sequences.The oracle hides both translation and folding, so the agent must infer each sequence-to-fitness mapping experimentally.Source: CodonBox implementation · p. 30
• Each translated product is a 16-residue H/P string evaluated by Dill’s HP lattice model, with exhaustive precomputation fixing the folding physics and a maximum fitness of 9.All 65,536 possible strings were enumerated in advance, and fitness counts favorable non-consecutive H–H contacts in the lowest-energy fold.Source: CodonBox implementation · p. 30
• Eight deterministic worlds vary codon length, alphabet size, silent positions, and additive versus interacting effects, including a familiar control and a hardest multi-feature world; each run lasted 500 design rounds.Codon-to-residue tables were balanced so half of possible codons mapped to H and half to P.Source: CodonBox implementation · p. 30

Data and code availability

• All code used to produce the presented sandboxes is publicly available in the science-sandbox repository.https://github.com/asr2210/science-sandboxSource: Data and code availability · p. 31

Competing Interests

• The authors disclose extensive financial, advisory, board, and patent interests across biotechnology and pharmaceutical organizations, with some interests reviewed under MIT policies.The disclosed relationships include equity holdings, company founding, board memberships, advisory roles, and patents.Source: Competing Interests · p. 31

Supplementary Figures and Tables

• Supplementary figures cover performance versus library size, held-out evaluation sets, activity metrics, long-horizon design, and token cost.These visuals compare 14 human-curated strategies, model-specific one-shot performance, eight activity metrics, blind and informed conditions, and one-shot versus long-horizon runs.Source: Figure S1 · p. 34, Figure S2 · p. 35, Figure S3 · p. 36

MPRAbox one-shot instructions: without prior knowledge

• One-shot agents design exactly 50,000 200bp sequences, commit once, and optimize performance across 14 anonymous evaluation sets with mean_r, k562_r, hepg2_r, and sknsh_r metrics.The without-prior-knowledge condition treats prepare.py as a black-box assay and prohibits further iterations after submission.Source: MPRAbox one-shot instructions: without… · pp. 39–40

MPRAbox one-shot instructions: with prior knowledge

• The informed one-shot condition permits one final commit and provides a systematic baseline-strategy reference to guide library design across 14 anonymous evaluation sets.Participants are instructed to read the reference before designing their library and outperform its best strategy while optimizing across all evaluation sets.Source: MPRAbox one-shot instructions: with… · pp. 41–42

MPRAbox informed-condition strategy reference

• The reference reports Pearson r (mean_r) averaged across five random seeds for library-design strategies using exactly 50,000 sequences unless noted; evaluation sets are anonymous, with contents not disclosed.Documented strategies include DHS topic-weighted, DHS random, DHS stratified, DHS+SEI, and DHS+synthetic sequence sampling.Source: MPRAbox informed-condition strategy… · p. 43
• Across the 14 strategies, mean Pearson r increases with library size on every evaluation set, from 10k through 300k sequences.At 300k sequences, the reported means range from 0.8069 on eval_12 to 0.9424 on eval_09.Source: MPRAbox informed-condition strategy… · p. 46

MPRAbox long-horizon instructions: without prior knowledge

• Long-horizon agents create successive 50,000-sequence libraries, obtain measurements in three cell types, train models from scratch, and receive averaged scores on 14 anonymous evaluation sets.Each experiment uses three random seeds and evaluates models on held-out sequences with real MPRA measurements.Source: MPRAbox long-horizon instructions:… · p. 47
• After each experiment, the process rereads the full notebook and results.tsv, selects, designs, and runs the single most informative next experiment, evaluates a library on 14 anonymous sets, and updates a theory of generalization.Each set returns mean_r, k562_r, hepg2_r, and sknsh_r averaged across three sequence files; eval_01 is the primary metric.Source: MPRAbox long-horizon instructions:… · pp. 48–50

MPRAbox long-horizon instructions: with prior knowledge

• With prior knowledge, the task explicitly targets libraries informative for models evaluated in unmeasured cell types, while retaining three-seed MPRA evaluation and theory refinement.The measured cell types are K562, HepG2, and SK-N-SH; prepare.py trains models from each library and returns averaged scores across 14 anonymous evaluation sets.Source: MPRAbox long-horizon instructions:… · p. 51

MPRAbox dry-oracle instructions: MPRA-framed

• The MPRA-framed dry-oracle setting preserves the goal of designing libraries that generalize beyond measured cell types, using one 50,000-sequence library evaluated on 14 anonymous sets.Agents are instructed to treat prepare.py as a black box, maintain a theory through contradictory evidence, and choose each next experiment based on accumulated results.Source: MPRAbox dry-oracle instructions:… · pp. 55–56

MPRAbox dry-oracle instructions: unframed

• The MPRAbox loop requires planning, generating and scoring each submission, recording results, updating a running theory, and committing every experiment separately.Agents must treat prepare.py as an opaque scoring endpoint, maintain append-only notebooks and reusable skills, and stop after 30 experiments.Source: MPRAbox dry-oracle instructions:… · pp. 59–60

MPRAbox dry-oracle instructions: symbolic

• The black-box task evaluates exactly 50,000 candidate strings, each 200 characters long and drawn from the alphabet {0, 1, 2, 3}, returning scores to maximize.Submissions are evaluated across 14 anonymous test sets; each returns mean_r, k562_r, hepg2_r, and sknsh_r. eval_01 is the primary metric, and the column names are arbitrary labels.Source: MPRAbox dry-oracle instructions:… · pp. 62–63

CodonBox instructions

• CodonBox frames an independently evolved biology as a discovery problem: agents submit fixed-length sequences, observe one functional score, and infer the rules governing performance.The instructions require using all experiments, probing edge cases after forming a theory, and revising that theory when evidence contradicts it.Source: CodonBox instructions · p. 65

CodonBox harness

• The CodonBox harness exposes only the alphabet, query tool, rationale field, and append-only notebook while dynamically setting sequence length and experiment budget for each world.It supports model calls, checkpointed resumption, query logs, transcripts, and hidden ground-truth storage for post-hoc analysis.Source: CodonBox harness · pp. 66, 68, 69
• The harness repeatedly calls the model until the oracle’s experiment budget is exhausted, requiring remaining experiments even when no query is issued.It prompts agents to test edge cases and refine their understanding rather than stop early.Source: CodonBox harness · pp. 70–71
• Only one query is permitted per turn; extra tool-use blocks receive refusal results without consuming experiment budget.The harness still returns results for every tool-use block so the API conversation remains valid.Source: CodonBox harness · pp. 70–71
• After the budget is exhausted, the harness requests a final account and writes run metadata including queries used, best fitness, and best sequence.When a reference table is loaded, it also records true-optimum fitness and the best run’s fraction of optimum.Source: CodonBox harness · pp. 71–72