PaperlayerView exact revision on arXiv

Source-linked AI summary

POSEIDON: Scaling Training Environments for Scientific Agents from Code Repositories

Qifan Zhang, Yuhan Li, Nuo Chen, Bowen Liu, Xiao Hu, Miao Peng, Chenyi Zi, Lixin Liu, Lydell Huang, Jing Tang, Jia Li

arXiv:2610.32448v1cs.AI

TL;DR

Scientific-agent training environments are scarce and difficult to scale. Poseidon mines scientific repositories, derives complementary tasks from executable workflows, and refines candidates through validation. Pearl-built Qwen3.5 models improve across six challenging scientific-agent benchmarks.

  • Problem

    Executable, verifiable training environments for scientific agents remain scarce, and constructing them at scale is difficult.

    Source: §1 · p. 2

  • Method

    Poseidon mines scientific repositories, derives four complementary task types from executable workflows, and recursively revises candidates that fail verification.

    Source: §1 · pp. 2–3

  • Results

    Pearl-trained Qwen3.5 models improve consistently across six scientific-agent benchmarks, including a 10.4-percentage-point gain for Qwen3.5-27B on BioCoder.

    Source: §4.2 · p. 8, §5 · p. 11

  • Takeaways & Limitations

    Pearl’s scale and disciplinary diversity offer opportunities for training scientific discovery agents.

    Source: §1 · p. 3

    The formulation requires each environment to be data-dependent, executable, verifiable, and diverse across repositories, disciplines, and task types.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Training environments remain a major bottleneck for agents in data-driven scientific discovery. Existing collections of executable and verifiable training environments are small, and their construction remains difficult to scale. To address this limitation, we introduce Poseidon, an automated pipeline for generating diverse, verifiable training environments from scientific code repositories at scale. From the same repository budget, Poseidon extracts nearly 10 times as many training environments as a prior approach by decomposing scientific workflows into four complementary task types: Reproduction, Subflow, Completion, and Correction, where each type targets a distinct capability for scientific research. A recursive improvement loop further refines failed candidates and recovers otherwise discarded environments. Using Poseidon, we construct Pearl, a collection of 97,659 executable and verifiable environments from 7,110 public repositories spanning 10 scientific disciplines. To the best of our knowledge, Pearl is the largest and most broadly multidisciplinary collection of scientific-agent training environments to date. Qwen3.5 models (9B, 27B, and 35B-A3B) trained on Pearl achieve consistent gains across six challenging scientific-agent benchmarks. Overall, the scale and diversity of Pearl offer opportunities for training scientific discovery agents.

1. Introduction

Scientific-agent training environments are scarce, while repository-based workflows offer executable, verifiable research tasks. Poseidon derives complementary tasks from those workflows and uses recursive refinement to increase the yield of environments.

1. Introduction

  • Scientific workflows require agents to combine reasoning, coding, and tool use, yet training environments for these workflows remain severely limited.D3-Gym yields 565 environments from 239 repositories by converting each extracted workflow into an end-to-end task, leaving more varied supervision underused.Source: §1 · p. 2
  • Under a matched budget of 306 repositories, Poseidon produces 5,370 environments, 9.3× as many as D3-Gym’s 575.The pipeline mines scientific repositories through source discovery, task synthesis, and verification.Source: §1 · p. 2
  • Among environments rejected in the first verification round, 49.4% pass after the validator provides diagnostic feedback and revision suggestions for the generator.The recursive loop recovers flawed but fixable candidates that would otherwise be discarded.Source: §1 · p. 3
  • Pearl contains 97,659 verified environments from 7,110 repositories across 10 disciplines, and Qwen3.5 models improve across six challenging scientific-agent benchmarks.Controlled ablations show that broader repository coverage and complementary task types contribute to downstream performance.Source: §1 · p. 3

2. Related Works

Prior work scales code-based supervision by transforming repository artifacts into varied executable tasks. Poseidon applies this strategy to scientific workflows, where existing environment collections remain scarce.

2. Related Works

  • Code-repository methods derive completion, executable-specification, software-repair, issue-resolution, and capability-targeted tasks from existing artifacts.These transformations increase the quantity and diversity of supervision available from a repository.Source: §2 · p. 3
  • Poseidon adapts repository transformation to scientific discovery by decomposing executable workflows into complementary, diverse, verifiable training environments.This extends prior code-task synthesis toward training agents for data-driven scientific workflows.Source: §2 · p. 3
  • D3-Gym builds verifiable scientific environments from repositories but treats extracted workflows as end-to-end tasks, yielding relatively few environments per repository.Poseidon instead derives multiple complementary tasks from each workflow.Source: §2 · p. 3

3. Method

Poseidon formulates scientific-agent environments as executable, verifiable task packages and builds them from scientific repositories. Its pipeline discovers repositories, derives tasks from executable workflows, and verifies candidate environments.

3. Method

  • Poseidon’s three-stage pipeline comprises repository discovery, task synthesis, and verification.The method aims to increase scientific-agent problem-solving capability through the scale and diversity of training environments.Source: §3 · p. 4

3.1. Problem Formulation

  • A training environment packages a task description, input dataset, executable sandbox, and evaluation script; the task requires an agent to analyze data, use tools, and produce an answer.The formulation calls for environments to be data-dependent, executable, verifiable, and diverse across repositories, disciplines, and task types.Source: §3.1 · pp. 4–5

3.2.1. Repository Discovery

  • Repository discovery uses expanded domain keywords to find scientific code and data, then builds sandboxes to support later task synthesis.Repository selection requires at least 20 stars, scientific data, a published paper or preprint, and a core scientific discipline; the resulting pool contains 7,110 repositories, with biological repositories intentionally oversampled.Source: §3.2.1 · pp. 5–6

3.2.2. Task Synthesis

  • Task synthesis extracts executable seed programs, represents their workflows as dependency graphs, and derives Reproduction, Subflow, Completion, and Correction tasks.Each type targets a different capability, from full-workflow execution to local analysis, missing-step reconstruction, and scientific error repair; task-specific reference programs generate answers and evaluation scripts.Source: §3.2.2 · p. 6

3.2.3. Verification

  • Verification checks environment reliability and scientific validity, then uses validator feedback and up to three revision rounds to retain candidates that pass.Reliability covers completeness, execution, reproducibility, and grading; scientific validity covers meaningfulness, correct implementation, and internal consistency.Source: §3.2.3 · pp. 6–7
  • Independent review of 600 sampled tasks yielded a 95.5% acceptance rate, and the final Pearl collection contains 97,659 environments.Three PhD researchers reviewed each sampled task, and all three had to approve it for acceptance.Source: §3.2.3 · p. 7

4. Experiment

Pearl fine-tuning consistently improves scientific-discovery performance, and controlled ablations find benefits from broader repository coverage and complementary task types, but no consistent benefit from increasing training-environment difficulty.

4.1. Experimental Setup

  • The evaluation covers six scientific-discovery benchmarks spanning ten disciplines and compares Qwen3.5 models with frontier and baseline models.The three Qwen3.5 backbones were fine-tuned using 64,477 successful trajectories collected from 97,659 environments.Source: §4.1 · pp. 7–8

4.2. Main Results

  • Pearl training consistently improves performance across model scales and scientific domains, including well-specified tasks and open-ended workflows.Gains are particularly pronounced for larger models.Source: §4.2 · p. 8
  • Qwen3.5-27B reaches 67.5% on BioCoder, matching GLM-5.2, while Qwen3.5-35B-A3B reaches 20.0% on SciCode (main), matching GPT-5.5.Qwen3.5-27B also scores 79.0% on MedAgentBench and 69.8% on BixBench-MCQ, exceeding Hy4-Preview (69.0%) and GLM-5.2 (69.3%), respectively.Source: Table 2 · p. 8, §4.2 · p. 9
  • At a matched budget of 2,000 trajectories, Pearl improves seven of eight settings by 0.7–5.2 percentage points, with a 2.1-point average improvement.The comparison uses fewer than 10% of Pearl’s full training data; the two baselines show selective gains and declines.Source: §4.2 · p. 9, Table 4 · p. 9

4.3. Ablation Study

  • With trajectory budget, difficulty, and task-type proportions fixed, broader repository coverage improves performance across all four evaluated benchmarks.At 20% coverage, performance on MedAgentBench and BixBench-MCQ is near or below the untrained backbone; broader coverage yields clear improvement.Source: §4.3 · p. 10, Figure 5 · p. 10
  • DeepSeek-V4-Pro success falls from 94.0% at D1 to 41.0% at D4, but training subsets with difficulty scores of 2.4–3.2 show no consistent downstream trend.Average solution length rises from 6.2 to 16.4 steps across D1–D4; the training subsets each contain 30,000 trajectories.Source: Table 3 · p. 9, §4.3 · p. 10, Figure 5 · p. 10
  • Removing any one of Reproduction, Subflow, Completion, or Correction degrades performance on every benchmark, indicating non-redundant supervision from each task type.The ablation controls total training-data size and source-repository distribution.Source: §4.3 · pp. 10–11, Table 5 · p. 11

5. Conclusion

Poseidon derives complementary tasks from executable scientific workflows and refines failed candidates to construct Pearl at scale. Pearl fine-tuning improves Qwen3.5 models across six benchmarks, with controlled ablations supporting the value of repository diversity and complementary task types.

5. Conclusion

  • Poseidon generates diverse, verifiable training environments from scientific code repositories by deriving four task types and recursively refining failed candidates.The pipeline’s task types are Reproduction, Subflow, Completion, and Correction.Source: §5 · p. 11
  • Pearl contains 97,659 environments from 7,110 repositories across ten scientific disciplines.The environments are executable and verifiable.Source: §5 · p. 11
  • Supervised fine-tuning on Pearl improves Qwen3.5 models across six benchmarks, while ablations find contributions from repository diversity and complementary task types.The results support extracting supervision beyond end-to-end workflow execution.Source: §5 · p. 11

A. Data Distribution

This section reports the task distributions and describes the criteria and independent audit procedures used to assess environment quality.

A. Data Distribution

B. Verification Criteria Details

  • Verification checks data integrity, executability, answer reproducibility, gradability, scientific correctness, program correctness, and repair consistency for correction tasks.Gradability includes accepting the reference answer and rejecting representative incorrect answers.Source: B. Verification Criteria Details · p. 15

C. Independent Human Audit of Environment Quality

  • Because automatic-verifier judgments alone do not establish scientific validity, retained environments receive an independent human audit.The audit samples environments after automatic verification.Source: C. Independent Human Audit of… · p. 15

C.1. Sampling and Stratification

  • The audit randomly samples 600 environments, stratifying proportionally by discipline, task type, and difficulty while requiring at least 10 per non-empty discipline–task-type stratum.Audited cases are disjoint from verifier-development and prompt examples, and auditors do not see verifier outputs or revision history.Source: §C.1 · pp. 15–16

C.2. Auditors and Blinding

  • Each environment is independently reviewed by three qualified auditors, including at least one with matching disciplinary expertise, who initially work blinded to verifier decisions.Auditors record independent decisions before discussing disagreements; unresolved cases go to a senior auditor.Source: §C.2 · p. 16

C.3. Audit Rubric

  • The rubric assesses seven criteria, and an environment is fully valid only when every criterion receives an adjudicated pass.Uncertain judgments do not count as passes; Completion and Correction tasks also receive checks of the missing or faulty computation, symptom, and repair.Source: §C.3 · p. 16

C.4. Execution Checks

  • Execution checks separately assess successful runs, input integrity, output reproduction across two executions, reference-answer acceptance, and rejection of at least two representative incorrect outputs.Infrastructure failures are marked unverified rather than scientifically valid.Source: §C.4 · pp. 16–17

C.5. Audit Outcome

The independent audit found that most sampled environments satisfied all listed quality criteria, while the remainder failed a criterion or could not be conclusively verified.

C.5. Audit Outcome

  • 573 of 600 audited environments (95.5%) satisfied all audit criteria; the other 27 failed at least one criterion or could not be conclusively verified.Criteria included scientific relevance, task identifiability, scientific and program correctness, artifact consistency, execution reproducibility, and evaluation validity.Source: §C.5 · p. 17

D. Benchmark Details

This section describes six scientific-agent benchmarks and the evaluated splits, task sets, and grading procedures used for each.

D. Benchmark Details

  • SciCode evaluates scientific code generation across 80 main problems decomposed into 338 subproblems; this study uses the test split of 65 main problems and 288 gradeable subproblems.The benchmark covers physics, mathematics, materials science, chemistry, and biology, with expert-written tests and numerical-tolerance checks.Source: D. Benchmark Details · p. 17
  • BioCoder tests functionally correct code derived from bioinformatics repositories, and evaluation retains 154 of 207 public problems after exclusions for the Python-only harness and non-executing references.A reference-solution sweep passes all 154 retained problems; grading uses differential testing.Source: D. Benchmark Details · p. 17
  • MedAgentBench contains 300 simulated electronic-health-record tasks across ten clinical workflow categories, graded by rule-based verification of the resulting record state.Agents query and update patient records through FHIR interfaces.Source: D. Benchmark Details · p. 17
  • BixBench poses 205 open-ended bioinformatics questions requiring multi-step analysis of realistic datasets and computational environments, evaluated with an LLM-as-Judge protocol.Source: D. Benchmark Details · p. 17
  • BioMysteryBench evaluates 90 open-ended biological research tasks involving modalities including genomics, transcriptomics, proteomics, and metabolomics, using an LLM-as-Judge protocol.Agents analyze raw data and report conclusions supported by computational evidence.Source: D. Benchmark Details · p. 17
  • ScienceAgentBench contains 102 workflow tasks from 44 publications, and results are reported for both the full set and a 38-task rule-based subset.The other 64 tasks additionally use an LLM judge to score generated figures against reference images at a threshold of 60.Source: D. Benchmark Details · p. 18

E.1. Data Example

The examples illustrate Reproduction, Subflow, and Completion tasks: reproducing a workflow, continuing a specified intermediate analysis, and restoring a removed computation.

Problem Description E.1: Reproduction

  • For each MHV protein, select the viral family with the greatest overlap of 6-mer sets, then estimate human-infectivity probability using the specified naive Bayes classifier with uniform 0.5 priors.Coverage is the fraction of sequence positions within matching 6-mers; predict human infection only when probability is > 0.5 and coverage is > 0.30.Source: Problem Description E.1: Reproduction · pp. 18–19

Problem Description E.2: Subflow

  • The Subflow task continues from provided per-line bivariate samples to fit zero-mean two-component Gaussian mixtures and select covariance families by BIC.It compares spherical and tied_shape families using 10 repeated fits per family; a pooled single-Gaussian estimate is explicitly disallowed.Source: Problem Description E.2: Subflow · p. 19

Problem Description E.3: Completion

  • The Completion task restores the missing detector-pixel conversion to reciprocal-space coordinates, then counts predicted grating peaks inside the measured qy-qz window for two azimuths.The task specifies full trigonometric formulas without the small-angle approximation and requires counts for −0.04° and 5.0° azimuths.Source: Problem Description E.3: Completion · pp. 20–21

Problem Description E.4: Correction

The section presents a correction task alongside prompts for repository discovery, file and field classification, and task-quality review. The correction audit fails only at the potentiated state, while the review prompt emphasizes evidence-based judgments.

Problem Description E.4: Correction

  • At s = +1, the potentiated-state audit fails despite normal convergence of both perturbed SCF solves; the other two states pass the 1 × 10−5 threshold.The task asks the solver to diagnose and minimally correct the checker, rerun the full audit, and report its pass flag and worst relative error.Source: Problem Description E.4: Correction · p. 22

Prompts E.1: LLM Prompt for Expanding Keywords

  • Keyword expansion requests exactly 40 real, specific artifacts across software, formats, databases, and pipelines, excluding generic terms and invented names.The output must be a single JSON object containing a keywords array.Source: Prompts E.1: LLM Prompt for Expanding… · p. 22

Prompts E.2: LLM Prompt for Scientific File Judgment

  • A scientific-file judgment requires workflow functionality, dataset input, and evaluable numerical or visual output; utility-only files and files over 5,000 lines are rejected.The prompt requires one JSON object as output.Source: Prompts E.2: LLM Prompt for Scientific… · p. 23

Prompts E.3–E.4: LLM Prompts for Scientific Field Judgment

Prompts E.6: LLM Prompt for Task Quality Verification

  • The quality validator assesses well-defined outputs, scientific logic, artifact consistency, and natural, self-contained instructions on a 1–5 scale.It must judge only supplied evidence, distinguish static inspection from execution evidence, and avoid penalizing task difficulty, novelty, or complexity.Source: Prompts E.6: LLM Prompt for Task… · p. 25

0. Domain Relevance

  • Domain relevance requires a meaningful scientific objective supported by data analysis, computation, simulation, or interpretation, with rejection reserved for clear evidence of out-of-scope work.Routine computation is in scope when it directly supports a scientific objective; unfamiliarity alone is not grounds for rejection.Source: §0 · p. 25

1. Task Definition and Answer Determinacy

The rubric evaluates whether a task is determinate, scientifically sound, correctly implemented, and consistently specified and graded. It also distinguishes supported defects from unverified runtime properties and treats language quality as a separate check.

1.1 Identifiability

  • A task is identifiable when its requested quantities and conventions determine an answer or a clearly specified set of acceptable answers; equivalent implementations need not use a unique program.Flag missing choices only when a reasonable alternative could change the answer beyond permitted acceptance criteria.Source: §1.1 · pp. 25–26

1.2 Operational Completeness

  • Operational completeness checks consequential choices that affect final outputs, while avoiding demands for arbitrary implementation details or speculation about unseen data behavior.Trace backward from final Gold fields and flag omissions only when alternatives could change the answer beyond acceptance criteria.Source: §1.2 · p. 26

1.3 Multiple-Solution Risk

  • Multiple-solution risk classifies supported variation as none, tolerance-resolvable, sign-resolvable, or method ambiguity, with severity based on its effect on answer validity and grading.No defect exists when the task permits alternatives and the evaluator handles them correctly.Source: §1.3 · p. 26

2.1 Scientific Correctness

  • Scientific correctness checks whether the computation and interpretation are meaningful under stated assumptions, including whether causal or mechanistic claims exceed the specified analysis.Source: §2.1 · p. 26

2.2 Gold Program Correctness; 2.3 Declared Environment Compatibility

  • Gold Program review checks consequential implementation behavior, while environment review compares required tools and libraries with declared dependencies.Without execution evidence, runtime success and file availability remain unverified rather than confirmed defects.Source: §2.2 · p. 26, §2.3 · p. 27

3.1 Prompt–Gold Consistency; 3.2 Eval Script Correctness

  • Prompt–Gold consistency checks answer fields, formats, and visible mathematical constraints; evaluator review checks acceptance of valid answers and rejection of materially incorrect ones.Static inspection must not be described as an executed test, and data-derived numerical correctness requires sufficient data or execution evidence.Source: §3.1 · p. 27, §3.2 · p. 27

4. Language Quality

  • Language review checks whether the prompt is fluent and self-contained, flagging broken syntax, unresolved placeholders, inaccessible references, or terminology that obscures meaning.Awkward but understandable phrasing is minor; length and necessary technical terminology are not grounds for penalty.Source: §4 · p. 27

5. Correction and Completion Tasks

Correction and completion reviews apply only when faulty or incomplete initial code is supplied, and judge the scientific behavior rather than similarity to a reference implementation. The rubric separately classifies issue severity, repairability, task quality, and solving difficulty.

5. Correction and Completion Tasks

  • Correction and completion checks apply only when initial code is faulty or incomplete, assessing symptom consistency, defect presence, repair target validity, and completion scope.Intentional omission of a diagnosis or implementation is not a specification defect when behavior and acceptable outputs are sufficiently defined.Source: §5 · p. 27

6. Severity and Repairability

  • Issues receive fatal, major, or minor severity, while repairability is determined separately and requires a concrete fix supported by the evidence.The same underlying defect should not be counted multiple times across prompt, program, and evaluator.Source: §6 · pp. 27–28

7. Quality Score

  • The 1–5 quality score maps fatal issues to 1, two or more independent major issues to 2, one major issue to 3, minor-only issues to 4, and no supported issues to 5.Source: §7 · p. 28

7. Quality Score; 8. Output Format

  • Acceptance is true exactly when the score is at least 4, but final retention also requires separate execution checks to pass.The prescribed output includes score, accept status, issues, repair suggestions, multiple-solution risk, and unverified checks.Source: §7 · p. 28, §8 · pp. 28–29

Task Inputs

  • The validator reviews agent-visible materials alongside the Gold Answer, Gold Program, Eval Script, environment, initial code, and any execution reports.Source: Task Inputs · p. 29

Prompts E.7: LLM Prompt for Task Revision via Validator Feedback

  • Revision may correct task specification, prompt–program alignment, evaluation logic, dependencies, tolerances, or sign conventions, while preserving the scientific goal and leaving the Gold Program unchanged.A scientifically incorrect Gold Program or fundamental unfixable issue requires UNREPAIRABLE rather than a prompt rewrite.Source: Prompts E.7: LLM Prompt for Task… · pp. 29–30

Prompts E.8: LLM Prompt for Difficulty Annotation

  • Difficulty is rated from the answer-relevant dependency slice, considering coordinated steps, method choice, parameter judgment, and required domain knowledge rather than code volume or prompt length.Correction and completion tasks are rated for diagnosing and fixing the defect; when levels are equally defensible, choose the lower one.Source: Prompts E.8: LLM Prompt for Difficulty… · pp. 31–32

F. Comparison of Executable Scientific-Agent Training Datasets

The comparison evaluates Pearl, MedAgentGym, and D3Gym using a unified extraction, tokenization, and sampling protocol. It measures length, semantic diversity, and instruction difficulty, with the supplied table identified as a length-and-diversity comparison.

F. Comparison of Executable Scientific-Agent Training Datasets

  • Pearl, MedAgentGym, and D3Gym are compared under a unified protocol across instruction and response or trajectory length, semantic diversity, and instruction difficulty.Diversity uses equally sized random subsets averaged over multiple seeds; difficulty is summarized by its mean and distribution.Source: F. Comparison of Executable… · p. 33
  • Semantic diversity is measured with Vendi Score and mean cosine distance to the embedding centroid, using sampled instruction embeddings.Source: F. Comparison of Executable… · p. 33
  • Table 6 compares dataset length and semantic diversity; the supplied passages do not report its cell values or a dataset winner.Source: Table 6 · p. 33

Related papers

Loading 2610.32448v1…