PaperlayerView exact revision on arXiv

Source-linked AI summary

DrugTargetWorld: A Synthetic Biobank for Training and Benchmarking AI Scientists

Samuel Margolis, Paul Schmiedmayer, Alan Huang, Ethan Chen, Ishan Bhattacharjee, Atman Shah, Ben Viggiano, Fang Cao, Shriya Reddy, Roger Xia, Jack O'Sullivan, Daniel Katz, Matthew Wheeler, Euan Ashley, Bruna Gomes

arXiv:2610.09558v1cs.AI

TL;DR

Real biobanks lack known causal ground truth and restrict access to participant-level data, making end-to-end agent evaluation difficult. DrugTargetWorld procedurally generates synthetic biobanks with hidden causal structure; leading agents recovered nearly two-thirds of causal drivers but rarely integrated evidence into well-supported conclusions.

  • Problem

    Real biobanks lack known causal ground truth and participant-level data are access-controlled, while existing causal benchmarks use predefined variables rather than raw-data phenotype construction and broad protein search.

    Source: §1 · pp. 2–3

  • Method

    DrugTargetWorld procedurally generates cardiovascular biobanks from hidden causal models, requiring agents to construct phenotypes, identify drivers and directions, and optionally select virtual experiments.

    Source: §1 · p. 3

  • Results

    Leading agents recovered nearly two-thirds of causal drivers, but rarely integrated their analyses into correct, well-supported conclusions.

    Source: §5 · p. 25

  • Takeaways & Limitations

    Known ground truth makes judgments in end-to-end biobank target discovery measurable and provides a verifiable reward for agent training.

    Source: §5 · p. 27

    Worlds are synthetic and calibrated to TOPMed/MESA only on selected summaries, with mostly linear mechanisms and effects designed rather than estimated from real disease.

    Source: §5 · pp. 26–27

Abstract

from arXiv · show

Drug target discovery requires distinguishing molecules that causally drive disease from those that are merely associated with it. Training and evaluating AI agents to perform this workflow end-to-end is difficult because real world biobanks lack known causal ground truth and participant-level data is access controlled. We introduce DrugTargetWorld, a framework that procedurally generates simulated biobanks, or "worlds," with known but concealed causal structure. Each world contains genotypes, proteins, health records, outcomes, and synthetic magnetic resonance imaging (MRI) for 54,000 participants. Agents must construct a disease phenotype, identify causal driver proteins, infer the beneficial direction of modulation, and optionally conduct virtual 'wet lab' experiments. We evaluated nine agents in 540 episodes across 20 cardiovascular worlds and three experimental budgets. Opus 5 and GPT-5.6 Sol achieved the highest mean composite scores, 39.98 and 35.38 of 100, respectively, and both recovered 64% of causal drivers on average. However, no agent reliably distinguished misleading non-causal proteins, and performance remained limited by the integrative judgments required to connect phenotype construction, causal evidence, and intervention decisions. By making each world's causal structure known to the evaluator but hidden from the agent, DrugTargetWorld turns end-to-end drug target discovery into a scalable training and evaluation problem with verifiable reward.

1. Introduction

Drug target discovery needs causal evidence, but biobank data and benchmark designs leave key parts of end-to-end discovery difficult to evaluate. DrugTargetWorld creates synthetic worlds with hidden causal answers and tests agents on phenotype construction, target discovery, and intervention decisions.

1. Introduction

  • Phenotype construction means selecting and combining measurements into a participant-level disease representation, a decision that can change what genetic analyses discover.This work remains heavily reliant on human investigators alongside the causal analyses that follow.Source: §1 · p. 2
  • Real biobanks lack verifiable causal ground truth, and observational evidence can reflect confounding, reverse causation, selection bias, pleiotropy, or measurement artifacts.Participant-level data are controlled-access, while existing causal benchmarks use predefined variables rather than requiring phenotype construction and search among thousands of proteins.Source: §1 · pp. 2–3
  • DrugTargetWorld procedurally generates worlds from hidden causal models, requiring agents to construct disease phenotypes, identify causal drivers and intervention directions, and choose optional follow-up experiments.The models include zero to six drivers, misleading non-causal proteins, and a harmful surrogate; cardiovascular archetypes are the initial test cases.Source: §1 · p. 3
  • No agent averaged more than 40 of 100 points across nine agents, 540 episodes, 20 worlds, and three budgets; performance was limited mainly by integrative judgments.The leading agents built phenotypes from raw data and applied genetic instruments, but struggled to connect measurements, causal evidence, and intervention decisions.Source: §1 · p. 3
  • Each world contains linked data for 54,000 participants, and the 20-world evaluation panel spans DCM, HCM, HFpEF, and ischemic disease.Agents receive no disease phenotype, causal labels, or prescribed analysis pipeline.Source: Figure 1 · p. 4

2. Related work

Prior work establishes methods for genetic target discovery, computational phenotyping, and agent research workflows, but agents typically receive a specified question or task. DrugTargetWorld combines these lines by requiring agents to choose and integrate a full discovery strategy against hidden causal ground truth.

Biobank-based drug target discovery

  • Population-biobank target studies use Mendelian randomization, colocalization, and related genetic approaches, but the analysis is still typically carried out by human investigators.DrugTargetWorld instead makes analysis and experiment choices sequential decisions under finite resources.Source: §2 · p. 6

Computational phenotyping in population biobanks

  • Computational phenotyping derives quantitative disease measures from imaging and longitudinal data, and those choices can change which biological associations are discovered.DrugTargetWorld makes phenotype construction part of the task because latent disease severity is never observed.Source: §2 · p. 6

Scientific agents and benchmarks

  • Scientific agent benchmarks span workflow execution and bioinformatics, but often specify the research question or analysis methods rather than leaving the full strategy to the agent.DrugTargetWorld requires agents to construct phenotypes, select analyses and experimental evidence, and integrate conflicting evidence.Source: §2 · p. 6

Related work comparison

  • Table 1 compares benchmark design dimensions; for example, BixBench3 largely specifies the research question and methods, unlike DrugTargetWorld’s open workflow.The table's scope is task structure rather than benchmark difficulty.Source: §2 · p. 6, Table 1 · p. 7

Interactive environments and agent learning

  • Interactive environments support agent training and evaluation through repeated actions, while DrugTargetWorld applies this approach to population-scale target discovery across distinct hidden causal models.Agents choose which evidence to collect and which experiments to pursue.Source: §2 · p. 7

Causal discovery benchmarks

  • Causal discovery benchmarks test recovery of known causal systems, whereas DrugTargetWorld also requires phenotype construction and separation of drivers from non-causal proteins among thousands of candidates.The comparison table characterizes benchmark task structure, not difficulty.Source: §2 · p. 7, Table 1 · p. 7

3. Methods

DrugTargetWorld generates synthetic biobanks with concealed causal structure and asks agents to construct phenotypes, identify drivers, choose interventions and experiments, and submit claims scored against hidden ground truth.

3.1 World generation

  • Each world fixes a structural causal model and parameters across agents, with disease severity driven by zero to six molecular proteins and expressed through archetype-specific phenotypes.The evaluated panel spans DCM, HCM, HFpEF, and ischemic archetypes.Source: §3.1 · pp. 8–9
  • Nine planted biases create misleading evidence, including confounding, reverse causation, selection, pleiotropy, imaging artifacts, and harmful-surrogate discordance.T5, T6, and T8 occur in all 20 evaluated worlds; T4 and delayed-effect designs create observationally ambiguous protein pairs.Source: §3.1 · p. 9, Figure 3 · p. 10

3.2 Synthetic biobank

  • Each generated biobank links 54,000 participants’ molecular, genetic, imaging, clinical, and outcome data, while latent disease severity and imaging-derived measurements remain unreleased.MRI is available for 10,800 participants, and the data include noise, missing modalities, and variable transcript–protein agreement.Source: §3.2 · p. 11

3.3 Agent environment and harness

  • Agents analyze released files by running Python in a minimal sandbox, with instructions but no analysis plan or correctness feedback.Episodes allow up to 30 turns, and their code and interaction histories are recorded.Source: §3.3 · pp. 11–12

3.4 Action space and experiments

  • Agents can choose among no experiments, a limited budget, or an expanded budget; cell perturbations measure first-visit effects, while knockdowns capture five-year effects.The limited budget forces a choice between one knockdown and three cell perturbations.Source: §3.4 · p. 12

3.5 Scoring

  • Scores reward target identification, evidence-matched causal confidence, intervention direction, bias identification, and phenotype construction, while harmful-surrogate recommendations can incur a 30-point penalty.Target identification and causal confidence together account for more than half of the 100 points.Source: §3.5 · p. 13
  • Target identification multiplies recall by precision, while ambiguous pairs reward either experimentally supported resolution or appropriately calibrated uncertainty.A correct intervention direction earns full credit, an explicit unknown earns half, and an incorrect or missing direction earns none.Source: §3.5 · pp. 13–14

3.6 Evaluation

  • Nine agents completed 540 episodes across 20 worlds and three budgets, with total score as the primary outcome and exploratory analyses of recorded research trajectories.Model comparisons average scores across budgets within each world; trajectory analyses describe computations and accessed data, not their validity or interpretation.Source: §3.6 · pp. 15–16

4. Results

The leading agents recovered many causal drivers but remained unreliable across the full workflow, with differences in phenotype construction and persistent difficulty rejecting misleading proteins. Failures also arose in evidence use and code execution, and correct final answers did not always reflect sound supporting analysis.

4.1 Leading agents recovered many causal drivers but remained unreliable across the full workflow

  • Opus 5 and GPT-5.6 Sol scored 39.98 and 35.38 of 100, respectively, with 0.64 target recall each; across episodes, the mean score was 14.1.Fewer than half of episodes scored above zero (258, 47.8%).Source: §4.1 · p. 16
  • Opus 5 exceeded GPT-5.6 Sol by 4.6 paired points (95% CI −0.26 to 9.45), but the difference was not statistically conclusive and run-to-run variation was unmeasured.Opus scored higher in 16 of 20 worlds; the paired t-test gave P = 0.062 and the exact sign test P = 0.012.Source: §4.1 · p. 17, Figure 4 · p. 18

4.2 Phenotype construction separated the two leading agents

  • Opus 5 earned positive phenotype credit in 56 of 60 episodes, compared with 34 of 60 for GPT-5.6 Sol.Phenotypes combining imaging features and checked against clinical endpoints more often correlated strongly with latent disease severity.Source: §4.2 · p. 19
  • Mean phenotype correlations with latent disease severity were 0.785 for Opus 5, 0.637 for GPT-5.6 Sol, and 0.502 for Sonnet 5; validation episodes averaged 0.663 versus 0.402 without validation.Comparisons pooling episodes across models are confounded with model identity.Source: §4.2 · p. 19

4.3 The leading agents based nominations on genetic instruments, but no agent reliably rejected non-causal proteins

  • Opus 5 and GPT-5.6 Sol based most nominations on cis Mendelian randomization, yet no agent averaged above 2.87 of 15 points for bias identification.The leading agents checked instrument strength in 59 and 57 episodes, respectively, but a single cis instrument per protein could not test the exclusion restriction across independent instruments.Source: §4.3 · p. 20
  • The expanded budget did not detectably change leading agents’ scores, and no experiment targeted delayed-effect pairs; causal members of ambiguous shared-instrument pairs were tested in 14 of 24 eligible episodes.Of 413 experiments, 171 (41.4%) targeted a causal driver, 52 (12.6%) the T8 harmful surrogate, and 49 (11.9%) the T2 reverse-causation protein.Source: §4.3 · p. 21

4.5 Failures arose at several workflow steps, and correct answers did not always reflect supporting analysis

  • Phenotype credit was zero in 416 of 540 episodes; other failures included unused analyses, code execution problems, and exact driver sets sometimes receiving full credit despite hard-coded labels.Across episodes, the number of core workflow milestones correlated with score (Spearman ρ = 0.649), but this relationship was confounded by model identity and partly mechanical.Source: §4.5 · p. 23

5. Discussion

DrugTargetWorld makes end-to-end target discovery evaluable against concealed causal ground truth, and the leading agents performed individual analyses but struggled to connect them into well-supported causal conclusions. The discussion identifies phenotype construction, distinguishing causal from non-causal proteins, and deciding when evidence is sufficient as key challenges.

5. Discussion

  • DrugTargetWorld varies hidden causal structures while retaining known drivers, misleading evidence processes, and intervention effects for evaluating each stage of discovery.Its synthetic worlds jointly specify genetic variation, molecular measurements, latent disease severity, multimodal phenotypes, longitudinal outcomes, and intervention responses.Source: §5 · p. 25
  • Leading agents recovered nearly two-thirds of causal drivers but rarely integrated their analyses into correct, well-supported conclusions; errors could be attributed to specific workflow stages.The judgments included how to measure disease, which evidence separates causal drivers from non-causal proteins, and when a claim is sufficiently supported.Source: §5 · p. 25
  • Opus 5 and GPT-5.6 Sol had similar causal-driver recall of 0.64, while removing phenotype credit narrowed their total-score gap from 4.6 to 0.9 points.The strongest phenotypes combined imaging features, sometimes weighted by clinical outcomes, and were almost always checked against other disease signals.Source: §5 · p. 25
  • Leading agents relied mainly on cis genetic instruments, but no analysis family was associated with target recall or incorrect nominations after multiple-testing correction.They often nominated the harmful surrogate—44 and 25 of 60 episodes for the two leading agents—without concluding it was not a causal driver.Source: §5 · p. 26
  • Experiments helped resolve candidates sharing a genetic instrument: agents tested the causal member in 14 of 24 eligible episodes, and tested drivers had the knockdown-implied direction.Pairs whose causal member acted mainly over follow-up were never tested, and expanded budgets did not detectably change leading-agent scores.Source: §5 · p. 26
  • The worlds are synthetic and calibrated to selected TOPMed MESA summaries; their mostly linear mechanisms, effect sizes, and biases were designed rather than estimated from real disease.Evaluation also lacked run-to-run variation, and the minimal harness may underestimate optimized agents; some scoring and world-generation features created exploitable cues.Source: §5 · pp. 26–27
  • Future iterations propose repeated agent and human episodes, greater biological realism, audits for exploitable cues, and planted effects in real biobank data.The authors also propose annotating trajectories and testing frozen candidate methods on unseen worlds and real cohorts.Source: §5 · p. 27

Funding

The study received computational resources and model-use credits from Anthropic’s AI for Science program and an American Heart Association award.

Funding

  • The study received computational resources and model-use credits from Anthropic’s AI for Science program and an American Heart Association Career Development Award to B.G.The award number was 26CDA1589055.Source: Funding · p. 28

Ethics statement

DrugTargetWorld used group-level MESA summaries and anatomical templates for calibration, without redistributing participant-level data or resampling real participants.

Ethics statement

  • Calibration used group-level MESA statistics and ACDC MRI anatomical templates; no participant rows, identifiers, genotypes, assay values, or resampled participants were included.The two author-led human-guided reference episodes involved no research participants, and generated protein identifiers do not map to real proteins.Source: Ethics statement · p. 29

AI use statement

The authors used speech-to-text and AI tools for manuscript dictation, language and formatting assistance, editing, and research-support tasks.

AI use statement

  • Wispr speech-to-text software was used for direct manuscript dictation, while ChatGPT and Claude assisted with minor language and formatting and Grammarly with editing.All authors reviewed and approved the manuscript and take responsibility for its content.Source: AI use statement · p. 29
  • AI assistants also helped edit the manuscript, inspect code and trajectories, and develop audit and sensitivity scripts; the inspected records do not fully document AI assistance during original generator and harness development.AI-assisted static-code classifications were treated as measurements that may contain error, not as independent expert judgments.Source: AI use statement · p. 29

Supplementary Figures:

The supplementary material maps model performance across worlds and documents world dimensions, phenotype validation, experiment use, and workflow attrition. Its annotations distinguish recorded research behavior from evidence of valid analysis.

Supplementary Figures:

  • Figure S1 reports mean scores across three budget regimes for nine agent systems in 20 worlds, labeled by cardiac archetype and design class.The caption states that scores range from 0 to 100, with higher scores indicating better benchmark performance.Source: Figure S1 · p. 35

Panel B. Dimensions, causal non-identifiability, delayed effects, and phenotype baseline

  • The evaluation panel comprised 20 worlds with 54,000 participants each, including 10,800 with MRI, 2,941 proteins, and 8,192 variants.T4 worlds contain a causal driver and noncausal alternative that cannot be separated using observational genetic evidence alone; delayed-effect worlds feature consequences emerging over longer follow-up.Source: Panel B. Dimensions, causal… · p. 38
  • Each model had 60 episodes, and reported score variation reflects worlds and budget conditions rather than run-to-run replication.Episodes covered each of 20 worlds under three budget conditions; scores use a 0–100 scale.Source: Panel B. Dimensions, causal… · pp. 39–40

Panel D. Clinical or imaging endpoints used for internal validation

  • Phenotype analyses include 349 scorable correlations, while internal-validation counts can include multiple endpoints per episode and do not establish statistical validity.The phenotype measure is the absolute correlation with latent disease severity; panel D covers annotated episodes that performed internal validation.Source: Panel D. Clinical or imaging endpoints… · p. 42

Panel B. Experimental planning and process

  • The strategy annotations describe recorded behavior, not whether an analysis was statistically valid or correctly interpreted.“Cis MR primary route” means the annotated strategy supporting final target prioritization, distinct from the stricter operation-based MR/IV calculation audit.Source: Panel B. Experimental planning and… · p. 43

Panel C. Use of returned experimental results

  • Of 360 budget-eligible episodes, 143 received at least one experiment, with 413 experiments delivered, including 36 cell perturbations and 377 in vivo knockdowns.Thirty-nine experiment requests were refused in budget-eligible episodes, and 11 additional requests were refused in observational episodes.Source: Panel C. Use of returned experimental… · p. 44

Panel B. Additional failure and attrition indicators

  • Workflow indicators use 60 episodes per model unless noted, and include submission, genotype-use, nomination-cap, and harness-level events.“No parseable submission at exit” includes episodes without a valid final submission when scored.Source: Panel B. Additional failure and… · p. 45

Panel C. Fixed-label sensitivity for the two leading models

The supplementary analyses specify how world variation and calibration were constructed and describe sensitivity checks for the leading-model comparison. The calibration follows aggregate targets, but does not establish independent real-cohort validity.

Panel D. Sensitivity to the follow-up MRI generation error

  • The sensitivity analysis excludes leading-model world cells with detected follow-up MRI reads, leaving 37 cells across 17 worlds; fixed-label ranges are identification bounds, not confidence intervals.Other panels remove phenotype credit or replace direction and outcome-alignment labels while holding other episode elements fixed.Source: Panel D. Sensitivity to the follow-up… · p. 46

S1. World generation

  • Each world is sampled from a structural causal model whose parameter set is fixed across agents evaluated in that world.The seed sets the world-level parameters, and participant observations are sampled conditionally from the resulting model.Source: S1. World generation · p. 47

S1.1 Population and molecular generation

  • The evaluated profile specifies 54,000 participants, 10,800 imaged participants, 2,941 proteins, and 8,192 variants as benchmark design choices, not estimates fitted to UK Biobank data.The panel shares demographic, MRI, genetic, and outcome target blocks with the mesa-topmed profile.Source: S1.1 Population and molecular generation · p. 47
  • Calibration uses selected MESA/TOPMed aggregate summaries, while causal graphs, molecular effects, disease loadings, and planted biases remain design-specified.The cardiovascular-event proportion anchors prevalence rather than validating the synthetic five-year endpoint.Source: S1.1 Population and molecular generation · p. 47

Genetic variation and linkage disequilibrium

  • Proteins are generated from cis and selected trans genetic effects, shared latent factors, covariates, and assay residuals; transcripts follow a similar structure without a fixed transcript-to-protein causal relationship.The initial panel has 2,941 proteins, each assigned one cis instrument, and approximately 20% receive a trans-genetic effect.Source: Genetic variation and linkage… · p. 49

S1.2 Latent disease generation

  • Latent disease severity depends on hidden molecular drivers, participant risk factors, optional direct genetic effects, and residual variation, with zero to six drivers across worlds.Driver identity, direction, effect magnitude, and instrument strength vary between worlds.Source: S1.2 Latent disease generation · p. 49

Hard-world disease architecture

  • Hard worlds add nonlinear effects, interactions between the strongest drivers, polygenic background, and greater residual variation.The nonlinear transformation saturates molecular effects at large absolute protein values, while the product term adds epistasis-like interaction.Source: Hard-world disease architecture · pp. 49–50

S1.3 Phenotype and longitudinal outcome generation

Worlds translate hidden disease severity into observable phenotypes and longitudinal outcomes, with archetype-specific expression. Planted biases create distinct routes by which noncausal signals can appear disease-related.

S1.3 Phenotype and longitudinal outcome generation

  • Observable phenotypes arise from latent disease severity, with archetype-specific loadings and released imaging, physiologic, clinical, EHR, repeated-visit, and survival measures.The cardiac archetypes express disease differently, including chamber dilation in DCM, wall thickening in HCM, impaired regional motion in ischemic disease, and tissue or diastolic signals in HFpEF.Source: S1.3 Phenotype and longitudinal… · p. 50

S1.4 Planted biases

T1. Confounding

  • BMI can generate a noncausal protein while also contributing directly to disease severity, creating an association without a causal molecular pathway.Source: T1. Confounding · p. 51

T2. Reverse causation

  • A reactive protein is generated downstream of disease, so its disease association does not arise from a causal effect of the protein on disease.Source: T2. Reverse causation · p. 51

T3. Imaging selection

  • Imaging is observed only for the top 20% of a selection score, and conditioning on the imaged population can induce collider bias.Both disease severity and the selected protein can influence imaging availability.Source: T3. Imaging selection · p. 51

T4. Shared genetic instrument / causal non-identifiability

T5. Imaging site batch effect

  • A site-specific spatial distortion can make acquisition site spuriously predictive when site tracks disease prevalence or participant composition.Source: T5. Imaging site batch effect · p. 52

T6. Benign exercise remodeling

The appendix specifies how planted remodeling, genetic pleiotropy, surrogate discordance, and assay shifts challenge interpretation, then documents intervention and scoring rules. Sensitivity analyses show that removing phenotype credit preserves model ranks, while fixing labels substantially reduces leading agents’ scores.

T6. Benign exercise remodeling; T7. Pleiotropic genetic instrument; T9. Assay unit mixture

T8. Harmful surrogate

  • A planted surrogate can predict observed cardiovascular outcomes without representing latent disease or belonging to the causal-driver set.Its designated feature contributes directly to event risk in addition to latent severity and conventional risk factors.Source: T8. Harmful surrogate · p. 53

S1.5 Intervention and longitudinal model

  • The intervention oracle compares a protein’s clamped abundance with a control while sharing other drivers and residuals; cell perturbation and knockdown use different horizons and sample sizes.Cell perturbation uses baseline weights and n = 120; knockdown uses late weights and n = 800. Cell deltas receive noise, whereas knockdown adds no assay noise.Source: S1.5 Intervention and longitudinal model · p. 53

S2.1 Claim intake; S2.2 Pair logic and false positives; S2.3 Null world and pair-free worlds; S2.4 Exit paths

S2.5 Phenotype validity and edge cases; S2.6 Sensitivity to phenotype scoring

S2.6 Sensitivity to phenotype scoring

  • Removing phenotype points reduced the Opus–GPT difference from 4.60 to 0.91 points (95% CI −3.64 to 5.46; P = 0.680), while all nine model ranks remained unchanged.The original means were Opus 5, 39.98/100, and GPT-5.6 Sol, 35.38/100; no-phenotype means were 33.24/90 and 32.33/90, respectively.Source: S2.6 Sensitivity to phenotype scoring · pp. 55–56

S2.7 Statistical software; S2.8 Fixed-label sensitivity

  • Statistical checks reproduced the reported tests and intervals; fixed-label analysis found direction credit fell for both leading agents, while always-aligned labels increased penalized episodes from 22 to 89.The Opus and GPT-5.6 Sol direction-credit ranges were 5.82–6.48 and 5.77–6.43, respectively.Source: S2.8 Fixed-label sensitivity · p. 56, S2.7 Statistical software · p. 56

S3. Agent prompt and harness protocol

The agent prompt places agents in a code-execution sandbox with synthetic biobank tables and MRI, requiring them to construct a phenotype and make target and direction claims. It also specifies experiment options, budget, submission format, and the costs of unsupported nominations and labels.

S3.1 System prompt and opening message

  • Agents write Python in a persistent working directory, execute code in turns, and inspect printed output; a submission ends the episode.The prompt requires one Python code block per turn and specifies the submission path and available tools.Source: S3.1 System prompt and opening message · pp. 57, 61, 62
  • Agents receive paths to synthetic population-biobank data, including covariates, proteins, metabolites, ECG, CT, records, outcomes, genotypes, and cardiac MRI.Repeat imaging and molecular data are available for subsets, and data dictionaries describe covariates and sentinel codes.Source: S3.1 System prompt and opening message · p. 58
  • The task provides no phenotype column: agents must derive a quantitative cardiac MRI phenotype, identify causal drivers, and state the direction of drug modulation.Phenotype construction is scored in its own right and is intended to inform subsequent analyses.Source: S3.1 System prompt and opening message · p. 59
  • Phenotype credit is based on beating an unseen held-out baseline; a phenotype at or below that baseline earns no credit.The prompt states that phenotype construction itself is measured, not merely a preliminary step.Source: S3.1 System prompt and opening message · p. 60
  • The prompt warns that observational genetic direction may fail under pleiotropy or reverse causation, and that a marker can improve measurement while harming patients.It frames experiments as the means to resolve those cases and offers cell perturbation for $150,000 or knockdown for $400,000.Source: S3.1 System prompt and opening message · p. 60
  • Submissions can nominate proteins and directions, label outcome alignment, reject candidates with a bias mechanism, or abstain on proteins that cannot be distinguished.The prompt says extra nominations and unsupported rejections or abstentions can lower score, and mislabeling outcome alignment is especially costly.Source: S3.1 System prompt and opening message · pp. 61–62

S3.2 Execution protocol

The supplement presents a human-guided episode in which an investigator directed an iterative analysis using a language model to write and run code. The episode used the full-program experimental budget, and the archived trace omits large generated outputs for readability.

S3.2 Execution protocol

  • The reported human-guided episode was the highest-scoring one and used the full-program budget of $2,000,000.The investigator directed research strategy while using a language model to write and execute analyses.Source: Supplementary Note S1. Turn-by-turn… · p. 63
  • The trace records each submitted analysis and the environment’s principal output, but truncates large generated tables and file listings.No hidden causal information or benchmark feedback was available during the episode.Source: Supplementary Note S1. Turn-by-turn… · p. 63

Initial task presented to the investigator

The investigator was tasked with constructing a participant-level phenotype and identifying causal proteins and intervention directions from a synthetic biobank. The guided episode combined MRI phenotyping, clinical outcomes, genetics, and longitudinal checks, recovering three targets but misattributing one decoy’s mechanism.

Initial task presented to the investigator

Turn 1 — Map the available data

  • The investigator mapped data availability and identified three complementary routes: MRI phenotype construction, molecular screening against outcomes, and genetic or experimental discrimination of misleading associations.No experimental budget was spent while mapping the data.Source: Turn 1 — Map the available data · pp. 63, 65

Turn 2 — Build reusable MRI features and inspect clinical outcomes

  • MRI features covered cine texture, myocardial geometry, and slice variability; the investigator treated them as intermediate features to test against independently observed disease.The same extraction was performed for repeat MRI, and clinical and longitudinal outcomes were inspected.Source: Turn 2 — Build reusable MRI features… · p. 65

Turn 5 — Expand genetically anchored discovery

  • The genetic screen expanded from an observational shortlist to the proteome, but shared loci required conditional analysis because one variant could implicate multiple proteins.The strongest pQTL alone was not considered sufficient to identify the causal molecule.Source: Turn 5 — Expand genetically anchored… · p. 68

Turn 6 — Broad longitudinal and source-of-bias analysis

  • Longitudinal checks surfaced a harmful surrogate, a delayed causal driver, and a protein that tracked later disease without causing it; a data-type error interrupted one diagnostic table.PROT_0852 improved measured morphology while worsening cardiac mortality; PROT_0531’s effect emerged mainly over five years, while PROT_0371 tracked later disease without causation.Source: Turn 6 — Broad longitudinal and… · pp. 68, 89
  • The investigator recovered all three causal targets and their therapeutic directions and avoided the harmful-surrogate safety penalty, but misclassified PROT_1858’s mechanism and received 0/15 for discrimination.PROT_1858 was noncausal, but the investigator attributed it to pleiotropy rather than the sealed model’s reverse-causation mechanism.Source: Turn 6 — Broad longitudinal and… · p. 89

Supplementary Note S2. Turn-by-turn trace of an observational human-guided DrugTargetWorld episode

The supplementary results trace an observational human-guided episode and report broad variation in model scores, analysis choices, and experiment use. Even strong analyses and exact driver recovery did not guarantee high composite scores, which also reflected phenotype and bias-identification credit.

Supplementary Note S2. Turn-by-turn trace of an observational human-guided DrugTargetWorld episode

  • The human-guided observational-only episode iteratively used imaging phenotyping, molecular association, genetics, MR-style analyses, and survival analyses before selecting targets.It had no simulated intervention experiments; the investigator instead directed the sequence of analyses before target selection.Source: Supplementary Note S2. Turn-by-turn… · p. 89

Interpretation carried forward

  • Higher protein aligned with higher T1 and mortality in one carried-forward pattern, but with lower T1 and mortality in another, leading to proposed inhibition and activation, respectively.The intervention proposals assumed that lower T1 represented therapeutic improvement.Source: Interpretation carried forward · p. 114

Selected returned output

  • The investigator retained PROT_1534, PROT_2336, and PROT_0318 despite PROT_0330 ranking first by protein–T1 association, because genetic analysis lowered its rank.The analyses included all-cause and cardiac mortality and proportional-hazards diagnostics before human selection of three targets. The final claims were observational causal inferences, not experimentally resolved target effects; lead-region proxy MR was not verified cis-MR.Source: Selected returned output · p. 124

S3.1 Score distributions

  • 39.98 and 35.38 were the mean scores of Opus 5 and GPT-5.6 Sol, respectively; only six of 540 episodes scored above 80.All six episodes above 80 had precision and recall of 1.0 and full target-identification, causal-confidence, and intervention-direction credit; they differed only in phenotype and bias-identification credit.Source: S3.1 Score distributions · p. 124

S3.2 Scores by world design

  • Two-driver worlds averaged 23.4 points, while worlds with three to six drivers averaged 11.2–12.2; difficult worlds averaged 10.3 versus 17.3 for other non-null worlds.The null world averaged 7.9, and 16 of its 27 model episodes nominated a protein.Source: S3.2 Scores by world design · p. 125

S3.3 Phenotype construction

  • Only 21 episodes earned full phenotype credit, and the 124 with positive credit averaged 6.33 of 10.Outcome-weighted composites learned feature weights from clinical outcomes and observed disease signals, whereas hand-weighted composites used agent-chosen weights; examples included refined cine-image measurements and a submitted fivefold out-of-fold ridge prediction that earned full credit.Source: S3.3 Phenotype construction · p. 125

S3.4 Genetic analyses and candidate sets

  • In characterized traces, some agents screened candidates by correlation, while Opus 5 and GPT-5.6 Sol checked instrumental-variable assumptions in 33 and 37 of 60 episodes, respectively.Cross-molecular operations appeared in 125 of 540 episodes and were more frequent for GPT-5.6 Sol than Opus 5 (57 versus 38 of 60 episodes; paired difference 31.7 percentage points, 95% CI 20.0–41.7, Holm-adjusted P = 0.0013). Candidate-set size was not monotonically related to score.Source: S3.4 Genetic analyses and candidate sets · pp. 125–126

S3.5 Use of experiments

  • Of 360 episodes with an experimental budget, 145 (40.3%) requested an experiment and 143 (39.7%) received at least one.There were 413 deliveries, mostly in vivo knockdowns (377; 91.3%); agents misread returned results in 16 of the 143 episodes that received experiments, and expanded-budget score changes varied by model.Source: S3.5 Use of experiments · p. 126
Loading 2610.09558v1…