PaperlayerView exact revision on arXiv

Source-linked AI summary

ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

Hejia Geng, Zesen Huang, Haoyang Li, Wenbin Li, Koutian Wu, Zihan Zhou, Yuanbo Pang, Weihao Liu, Zigong Xu, Zhiping Li, Zongzheng Zhang, Chuanfei Dong, Jiankai Sun, Tianzhe Zheng, Fengyu Xie, Yue Ma, Yueheng Shi, Tong Xie, Zonglin Di, Xianrong Liu, Qucheng Gao, Yimin Liu, Jiaming Pan, Sheng Huang, Xiao-Han Ma, Lanqing Yuan, Zhenlin Zhu, Ziang Liu, Ziyang Xu, Junkai Wang, Kangkai Liang, Jiayi Xian, Zehong Zhao, Liuwei Xu, Jingxu Xie, Peijin Zhang, Qiang Gao, Chengyi Xing, Zhe Zhao, Xi Wang, Yaopeng Xing, Xing Meng, Zhenfei Yin, Yingcheng Wu, Ling Yang

arXiv:2609.19134v1cs.CLcs.CY

TL;DR

Scientific repositories are difficult to convert into reliable learning experience because toolchains, conventions, and correctness criteria are fragmented. ScienceIDE packages expert-defined scientific contracts into executable environments with validated task factories and shared learning interfaces. Experiments report held-out repair gains and selected broader benchmark gains, while the evidence remains mainly about reference-verifiable software tasks.

  • Problem

    Scientific repositories are difficult to convert into reliable learning experience because heterogeneous toolchains, implicit conventions, and specialized scientific verification complicate reproducible task construction.

  • Method

    ScienceIDE packages expert-defined scientific modules, acceptance criteria, runtimes, checks, and private verifiers into reusable environments whose factories generate validated tasks and episodes.

  • Results

    SFT raises repair reward across scientific environments, including 4B on PLUTO-Particles-Dust from 0.0000 to 0.3333 (+33.33 points) and 9B on PLUTO-RMHD/ResRMHD from 0.0000 to 0.2857 (+28.57 points).

  • Takeaways & Limitations

    ScienceIDE makes expert scientific judgments reusable across task generation, execution, verification, evaluation, SFT, RL, and subsequent scientific work.

  • Takeaways & Limitations

    The evidence concerns predominantly reference-verifiable repair and implementation tasks in computational physics and geoscience, not open-ended discovery.

Abstract

from arXiv · show

Scientific code repositories encode decades of human knowledge in executable models, methods, and tools. Yet fragmented toolchains, implicit domain conventions, and specialized correctness criteria make this knowledge difficult to convert into reliable learning experience-a challenge we call the scientific experience bottleneck. We introduce ScienceIDE, infrastructure for turning the world's scientific code into programmable environments for scientific agents. Guided by expert-defined scientific cases and acceptance criteria, agents transform repositories into executable environments that support task generation, execution, and scientific verification. These environments provide a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation. Using verified interaction trajectories, we train PhAI-IDE-72B, PhAI-IDE-9B, and PhAI-IDE-4B. The model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities. ScienceIDE lays the foundation for an integrated workspace for agent learning and scientific practice, making humanity's scientific software a shared substrate for developing scientific intelligence. Code: https://github.com/aitofound/ScienceIDE

1. Introduction

ScienceIDE addresses the scientific experience bottleneck: converting heterogeneous scientific repositories and implicit correctness conventions into reliable training experience. It uses expert-defined executable environments, validated task factories, and shared interfaces for learning and evaluation.

  • 1. Introduction: ScienceIDE targets the scientific experience bottleneck caused by heterogeneous toolchains, reproducibility challenges, specialized verification, and the need to turn runnable code into validated learning tasks.Scientific discovery also requires sustained interaction, hypothesis testing, and transfer from evidence beyond ordinary coding specifications.
  • 1. Introduction: The infrastructure turns expert judgments into reusable executable environments and scientific checks, separating scientific contracts from repeated task authoring.This lets tasks ranging from defect injection to expensive-path porting be graded against the code’s scientific tests and tolerances.
  • 1. Introduction: Environment-specific factories generate candidate tasks, while execution and scientific verification admit only valid tasks whose interactions produce graded trajectories.The same infrastructure supports repair, implementation, reproduction, and acceleration across environments.
  • 1. Introduction: Common interfaces convert verified interactions into supervised fine-tuning data, reinforcement-learning rewards, and held-out evaluation tasks or environments.Expert effort concentrates on scientific boundaries and objectives rather than reconstructing every generated task.

2. ScienceIDE

ScienceIDE constructs versioned scientific environments from expert-defined modules, executable checks, and reusable task factories. Validated episodes then connect scientific interaction to evaluation, supervised fine-tuning, reinforcement learning, and further scientific work.

  • 2. ScienceIDE: A ScienceIDE environment packages a versioned repository module with its runtime and scientific checks, while factories propose candidates that validation turns into learning tasks.The construction unit is a module with a coherent scientific responsibility and executable coverage.
  • 2.1. Compiling scientific expertise into executable environments: Experts define module boundaries and scientific equivalence; approved modules retain editable workspaces, calibrated checks, private verifiers, source pins, dependencies, and observed hazards.These assets are reused across task families rather than rebuilt for each task.
  • 2.2. From official tests to scientific checks: Scientific checks survey official tests, regressions, and examples, pairing fixed inputs and graded outputs with pass policies grounded in observables or physical invariants.Pointwise policies use absolute-plus-relative tolerances, while invariant policies handle run-to-run variation such as random streams or diverging flows.
  • 2.3. Specializing reusable methods into task factories: Factories adapt reusable procedures to local paths, cases, build recipes, and meaningful transformations, while curators retain decisions about observables, equivalence, and acceptance.The taxonomy includes Acceleration, Repair, Discovery, Reproduction, Integration, Calibration, and Implementation; candidates become tasks only after validation.
  • 2.4. Manufacturing and validating scientific tasks: Factory proposals require executable evidence that they are observable, solvable, and trustworthy before entering the registry as validated tasks.Validation distinguishes scientific disagreement, incomplete delivery, and infrastructure failure, and accepted episodes support evaluation and learning.
  • 2.5. Connecting scientific interaction to model learning: A shared episode interface records artifacts, trajectories, check rewards, execution status, and resource use for evaluation, SFT, and RL.The interface also lets interactions generate proposals for new objectives that re-enter scientific validation.

3. Experiments

Experiments evaluate ScienceIDE agents on hard scientific coding tasks and test whether verified scientific trajectories improve held-out repair and broader capabilities. Results show strong but variable task performance, resource–correctness trade-offs, transfer across benchmarks, and benefits from verifier-guided interaction.

  • 3.2.1. Overall scientific task performance: 67.1% success makes Fable 5.1 the highest observed performer on ScienceIDE-Hard, ahead of Opus 5 at 64.6% and Astra at 63.1%.The comparison covers 85 scientific repair and implementation tasks under a one-hour budget.
  • 3.2.1. Overall scientific task performance: Top point estimates do not establish a statistically determined ordering because Fable has a single-attempt estimate and Opus–Astra repeat intervals overlap.The benchmark uses strict success on a fixed 85-task set with within-task repeat variability described by intervals.
  • 3.2.2. Budget and resource efficiency: At ten minutes, Astra reaches 49.6% success versus Fable’s 25.9%, but Fable overtakes it at approximately 31 minutes.Between 20 and 60 minutes, Astra gains 2.0 points, compared with 11.8 for Fable, 16.0 for Opus, and 30.2 for Qwen3.8 Max.
  • 3.2.2. Budget and resource efficiency: Fable reaches 67.1% success at $7.90 per task, whereas Astra reaches 63.1% at $3.56, showing that resource use and scientific correctness are distinct dimensions.Across fifteen profiles, success correlates descriptively with runtime at −0.22 and output volume at 0.01.
  • 3.3. Learning from Scientific Trajectories: SFT raises scientific repair reward across environments, including 4B PLUTO-Particles-Dust from 0.0000 to 0.3333 (+33.33 points) and 9B LAPS from 0.3125 to 0.5000 (+18.75 points).These gains use the environments’ original numerical checks under the same repair protocol.
  • 3.3. Learning from Scientific Trajectories: Scientific-interaction training also improves selected code, reasoning, and knowledge benchmarks, including 9B BBH Word Sorting from 0.240 to 0.576 (+33.60 points).Reported code gains include repair, execution prediction, defect detection, and generation; the results provide evidence of positive transfer without establishing uniform improvement.
  • 3.4. Learning through Scientific Interaction: Outcome-only rewards can penalize budget-truncated long-horizon repairs, creating a shortcut in which the policy improves by shortening trajectories instead of solving tasks.An unmasked run shows reward collapse alongside tokens per turn falling by more than threefold, motivating truncation masking.
  • 3.4. Learning through Scientific Interaction: After 30 masked training steps, held-out reward rises 2.4× on LAPS and 2.0× on MITgcm-biogeo over the base Qwen3.5-4B checkpoint.Training reward increases while budget truncation falls, without shorter responses: LAPS truncation drops from 39.5% to 6.6%, while mean tokens per turn rise from 1103 to 1135.

4. Related Work

Related work spans scientific models, scientific agents, evaluation suites, executable environments, and scientific foundations for supervision. ScienceIDE positions itself around reusable expert-approved modules and checks that feed evaluation, SFT, and RL.

  • Specialist scientific models: Scientific models develop domain capabilities through specialized objectives, corpora, or architectures across areas including proteins, PDEs, weather, materials, symbolic regression, and scientific reasoning.
  • Scientific agents and discovery: Scientific agents have been evaluated through computer-checked mathematics, program search, and research-oriented contributions spanning mathematics, physics, and biology.
  • Measuring scientific ability: Existing evaluation suites vary in task source and verification, covering authored problems, publication-derived tasks, research-level questions, engineering, and reproduction of existing results.
  • Scientific foundations and supervision: Science contributes theoretical ideas, scientific databases, corpora, and graduate- or research-level problems that support models, pretraining, and reasoning evaluation.
  • Executable environments and scientific interaction: Executable environments separate interaction, action interfaces, and optimization, while scientific settings expose agents to controls, molecular rewards, simulators, and broader workflows.
  • Positioning ScienceIDE: ScienceIDE makes an expert-approved scientific module and its checks a reusable unit for producing task instances, unlike benchmark instances as evaluation units.Environment-specific factories adapt authoring procedures, while evaluation, SFT, and RL consume the resulting interactions.

5. Limitations

The evidence is concentrated on reference-verifiable repair and implementation tasks, while verification, benchmarking, contamination control, and transfer remain bounded or unresolved.

  • Scope: Evidence covers predominantly repair and implementation in computational physics and geoscience rather than open-ended discovery, with broader task-family coverage still a development target.The hard set probes only up to a one-hour budget and two coupled edit sites.
  • Verification: Agreement with selected observables and calibrated tolerances does not establish correctness outside those checks, and independent audits and adversarial reward-hacking evaluations remain outstanding.Factories amortize but do not eliminate expert curation, and repository, dependency, or hardware changes require revalidation.
  • Evaluation: Scores compare model–harness systems under fixed budgets, while serving speed, unequal repeat coverage, legacy timestamp cues, and incomplete retrieval isolation limit fine-grained and contamination-free ranking.
  • Transfer: SFT gains concern selected public benchmarks and RL gains concern hinted held-out tasks within training environments, neither establishing transfer to unseen codebases.SFT separation uses task identifiers, and RL comparisons do not estimate between-seed variance.

6. Conclusion

ScienceIDE turns expert judgments into reusable scientific-agent experience while exposing where verified execution remains difficult. The conclusion calls for broader coverage, transfer tests, and verification suited to open-ended research.

  • 6. Conclusion: ScienceIDE separates scientific judgment from repeated task production by making expert decisions reusable contracts while requiring executable evidence for candidate tasks.
  • 6. Conclusion: Experiments reveal a gap between plausible code and verified scientific execution, widest for coordinated edits in large codebases or unstated numerical conventions.The work also reports selected public-benchmark SFT gains and hinted within-environment RL gains.
  • 6. Conclusion: The next step is to broaden scientific coverage, test transfer across codebases and task families, and support checks for competing hypotheses and novel outcomes beyond fixed references.The stated aim is an exchange in which science supplies grounded learning experience and better agents extend scientific work.

S1. Organizations

This section lists the organizations represented in the paper’s affiliations. It includes universities, a research institute, and an international theoretical-physics center.

  • Organizations: Listed organizations include UCLA, UIUC, NJIT, UC Irvine, Georgia Tech, and UT Austin.
  • Organizations: The affiliations also include UC Berkeley, Caltech, UC San Diego, and UC Santa Cruz.
  • Organizations: Additional listed organizations are The Ohio State University, Asia Pacific Center for Theoretical Physics, and Washington University in St. Louis.

S2. Scientific contracts and experience inventory

Scientific contracts make executable checks reusable across task families while preserving provenance and cohort-specific accounting. Calibration and expert review define finite, scientifically meaningful acceptance boundaries rather than universal tolerances.

  • Pointwise checks, invariants, identity alignment, and explicit warrants define graded outputs and the scientifically relevant deviations each bound must distinguish.Timings, adaptive-step counts, rank layout, and random draws are excluded as observables when they may vary legitimately.
  • Calibration runs and optional alternative builds provide finite evidence about numerical spread and cross-build separation, not universal floors or automatic tolerances.The curator and domain expert retain responsibility for choosing the acceptance boundary.
  • Once fixed, the same check suite supports repair, implementation, acceleration, and reproduction tasks without silently redefining scientific equivalence.Factories vary initial state, deliverable, and objective while reusing executable acceptance criteria.
  • Cohorts remain separate because evaluation, scientific-contract, and task-supply populations support different claims and must not be pooled.Task supply and check coverage do not change the ScienceIDE-Hard evaluation denominator.
  • Policy-era probing used 2,722 attempts instead of 4,845 under uniform three-probe evaluation, a 43.8% reduction, while 3,195 graded trajectories cost an estimated $703.These accounting comparisons are descriptive, and the probe cost is subscription-metered rather than an end-to-end construction cost.

S3. Evaluation protocol and integrity

ScienceIDE-Hard fixes a shared 85-task evaluation denominator, but reported scores remain model–harness measurements subject to validity, retrieval, and provider-channel integrity constraints. Historical audits exposed contamination paths that required reruns or invalidate simple leaderboard interpretations.

  • ScienceIDE-Hard contains 85 tasks from 18 environments and five repository families, with strict success requiring private scientific checks rather than compilation or execution alone.The panel includes 52 repair and 33 implementation tasks, and every leaderboard model receives the same task identifiers.
  • The full analysis uses validity-filtered attempts and task-balanced estimates, while repeat-instability statistics use only the first three valid attempts per task.Fable uses one latest valid attempt per task; other models average observed valid repeats within each task.
  • Infrastructure errors are excluded from the denominator, whereas valid AgentTimeoutError counts as budget exhaustion and failure.The historical audit found 1,705 effective tasks, 136 without headroom, and 322 with unverified floors, but it does not certify the current registry.
  • Historical container audits found 455 upstream-directed commands, 21 confirmed successful fetches, and 13 scored solutions among 5,314 trials, with staging scripts rewriting agent-side network declarations.The corrected path uses a live-container allowlist and voids confirmed fetch trials.
  • Provider-side channels bypassed container isolation: Gemini received substantive tool content on 71 of 85 tasks, while a DeepSeek aggregator solved 13 of 20 tasks before reruns under an allowlist.The DeepSeek correction moved a historical sample score from .365 to .294, not a timeless leaderboard value.

S4. Agent error analysis and scientific coverage

ScienceIDE-Hard reveals that failure behavior, scientific coverage, and repeat outcomes vary substantially across agents. Error reviews further show that scientific repair requires correct localization, codebase-specific conventions, and complete delivery beyond passing local tests.

  • Failure behavior differs sharply: Qwen exhausts its budget on 37.3% of selected attempts, Haiku has no recorded timeouts but returns at or below baseline on 74.9%, and MiniMax combines 31.6% exhaustion with 34.1-minute unsuccessful episodes.Termination status alone cannot establish why an agent stopped.
  • 28.0% of model–task pairs show both success and failure within three valid attempts, with Gemini changing outcomes on 33 of 85 tasks versus eight for Astra.Aggregate success rates can therefore conceal unreliable task-level behavior.
  • Scientific coverage is uneven: Astra and Sol both reach 47.3% on PLUTO, while Athena++ favors Astra 96.3% to 38.5% and LAPS favors Gemini 29.5% to Sol 20.8%.These reversals do not isolate a domain effect because repository families differ in language and task composition.
  • In a common single-attempt sample, Fable solves 57 of 85 tasks while the union of fifteen agents solves 72, including 11 uniquely solved tasks and 13 solved by none.This indicates observed complementarity, not a tested ensemble performance guarantee.
  • Reference-convention mismatches account for 71.4% of Fable’s and 64.9% of Astra’s task-balanced failures, showing that runnable code may still violate numerical precision or state conventions.Examples include omitted timestep retention and a truncated coefficient that causes tracer divergence.
  • Wrong-target edits account for 10.7% and 17.0% of the two models’ failures, while local tests can pass even when private-reference grading rejects the repair or delivery omits required outputs.The review labels are evidence-supported rather than experimentally isolated causes, and 17 attempts remained unresolved.

S5. Learning settings and supplementary results

The supplementary study specifies how verified scientific trajectories train models and evaluates transfer across code, reasoning, and knowledge benchmarks. It also documents reinforcement-learning execution choices, including masking budget-truncated trajectories from policy loss while retaining their rewards for baselines.

  • Qwen3.5-4B, Qwen3.5-9B, and Qwen2.5-72B-Instruct train on 4,567 segments from 564 tasks, with 544 validation segments from 81 tasks and masked failed or repetitive actions.The task-identifier-disjoint corpus supervises new assistant actions and tool calls selected by numerical-equivalence verification.
  • Screening gains extend across code, reasoning, and knowledge benchmarks, including 9B BBH Word Sorting at +33.6 points and 4B BBH multistep arithmetic at +4.8 points.Other reported gains include 72B APPS Introductory +6.25 points and 4B MMLU-Pro Computer Science +4.69 points.
  • On confirmation comparisons, 9B Word Sorting rises from 27.2% to 63.2% and 4B multistep arithmetic from 92.8% to 98.4%; the 72B HumanEvalFix C++ screening gain does not persist.Full splits combine screening and confirmation rather than constituting independent replications.
  • Each reinforcement-learning iteration rolls out and grades episodes with the native verifier reward, then performs one optimizer step using concurrent rollout and training pools.Episodes run in the same Terminus-2 containers and verifiers used for evaluation.
  • Budget-truncated trajectories retain reward contributions to the group baseline but receive no token-loss contribution, preventing unusable cutoff tokens from receiving policy gradients.The method addresses a collapse in which penalizing long hard-repair trajectories reduced tokens per turn and increased further truncation.
  • Across reported reinforcement-learning runs, reward rises from 0.427 to 0.828 on LAPS and from 0.381 to 0.597 on MITgcm-biogeo while budget truncation falls from 39.5% to 6.6% and 34.1% to 23.8%.LAPS turns per episode shorten as completion occurs earlier, while tokens per turn remain near 1100.

S6. Environment sources

The frozen registry comprises 27 upstream scientific codebases mapped to 64 derived environments, with source citations, pinned repositories, and recorded licenses. The sources span diverse scientific domains and software licenses.

  • S6. Environment sources: 27 upstream codebases map to 64 derived environments in the frozen registry, with each source linked to its citation, pinned repository, and recorded license.The release ledger provides the complete environment-to-source map.
  • S6. Environment sources: The registry includes astrophysical fluid dynamics, electromagnetics, inverse dynamics, plasma physics, quantum simulation, and single-cell immune-receptor analysis.Representative sources include Athena++, Meep, TSID, Gkeyll, QuTiP, and Scirpy.
  • S6. Environment sources: The source collection spans permissive and copyleft licensing, including MIT, BSD, Apache-2.0, GPL, ISC, NASA-1.3, and mixed licensing terms.Several entries record license scope or bundled-license conditions alongside the repository name.
  • S6. Environment sources: Individual entries identify the upstream project, its primary citation, and the repository or package name used in the environment registry.Examples include PyAMG, pymatgen, EPOCH, G4CMP, and Basilisk.
Loading 2609.19134v1…