Source-linked AI summary
ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyond
Mingming Zhao, Jiqian Dong, Kangping Xu, Zadid Hasan, Chengrui Fan, Shan Jiang, Shuai Mao, Ting Lingya, Linyi Zou, Tailin Zhou, Yun Hin Chan, Wenkai Zhang, Zhanhong Zhou, Guowei Huang, Hongliang Li, Wenjing Cun, Zhitang Chen, Mingxuan Yuan, Yanhui Geng
TL;DR
Long-horizon research agents lack robust mechanisms for continuity, recovery, and value-driven compute allocation. ScienceFlow addresses this with recoverable executable states, ESTRA-governed transitions, and evidence-aware execution control, achieving 70.22 ± 1.18% Any-Medal on full MLE-bench, 4.92 percentage points above the strongest reported baseline.
Problem
Long-horizon research agents still face limitations in reproducibility, provenance, confidence calibration, novelty validation, transfer, and causally linked experiment histories.
Method
ScienceFlow organizes research into executable workspace segments with recoverable states, ESTRA-governed transitions, and evidence-aware execution control.
Results
70.22 ± 1.18% Any-Medal on the full 75-task MLE-bench exceeded the strongest reported baseline by 4.92 percentage points, alongside strong performance across scientific modeling and mathematical optimization.
Takeaways & Limitations
ScienceFlow demonstrates strong and consistent long-horizon research performance across machine learning, scientific modeling, and mathematical optimization.
Takeaways & Limitations
Backbone comparisons are descriptive rather than statistically conclusive because they use only three runs on one task.
Abstract
from arXiv · showhide
Enabling LLM agents to sustain productive, stable, and goal-aligned research over extended horizons is a central challenge for autonomous machine learning and scientific discovery, as progress hinges on continuously managing evolving state, exploration decisions, and computational resources. Pioneering autoresearch agents, despite great success, still lack mechanisms for continuity, recovery from dead ends, and value-driven compute allocation, which inherently undermines overall search efficiency, wastes computational resources, and lowers the chance of ultimate success. To bridge this gap, we introduce ScienceFlow, an end-to-end autoresearch agent framework that organizes long-horizon research work into research segments grounded in executable workspaces. It represents research progress as recoverable executable states, enabling efficient exploration, revision, and execution. Transitions between research segments are governed by Executable-State Transition through Re-Anchoring (ESTRA), which selects either the live state or an archived state as the next anchor and determines whether to continue or redirect the research trajectory. An evidence-aware execution controller allocates resources to physical jobs based on resource availability, remaining budget, and validated progress. We evaluate ScienceFlow on tasks spanning machine learning, scientific modeling, and mathematical optimization. Results on diverse long-horizon benchmarks demonstrate its ability to sustain effective research processes, highlighted by a SOTA 70.22 percent Any-Medal score on the full MLE-bench within a 24-hour budget, outperforming prior reported results by 4.92 percentage points. The efficacy of ScienceFlow further demonstrates that efficient state management, adaptive exploration, and objective-aligned execution are critical for scaling autonomous research beyond short-horizon interactions.
1 Introduction
ScienceFlow addresses the challenges of long-horizon autonomous research by preserving recoverable executable state, adapting research trajectories, and aligning execution with validated progress. It evaluates these abstractions across machine learning, scientific modeling, and mathematical optimization, achieving strong benchmark performance.
- ScienceFlow: ScienceFlow organizes research around recoverable executable workspaces that preserve concrete trajectory state for continuation, branching, and recovery.Validated workspace checkpoints are produced from task-specific result signals, while text-only responses or context limits do not serve as equivalent executable anchors.
- Evaluation: ScienceFlow evaluates a common set of long-horizon research abstractions across machine learning, scientific modeling, and mathematical optimization.The evaluation examines whether the abstractions transfer across substantially different executable tasks.
- Evaluation: 4.92 percentage points above the strongest reported baseline, ScienceFlow performs strongly on the 75-task MLE-bench under a 24-hour budget, while achieving the best group-balanced score on SciModelingBench.It also reaches or improves published frontiers on multiple mathematical optimization problems.
- Motivation: Long-horizon research requires jointly maintaining executable state, adapting trajectories, and aligning execution with validated progress rather than merely extending interaction time or compute.These requirements arise because research state accumulates code, datasets, checkpoints, solver states, and evaluation outputs that textual summaries cannot faithfully reconstruct.
- ScienceFlow: ESTRA governs transitions between research segments, enabling the system to continue, redirect, or recover without losing useful progress.The framework selects between live and archived executable states as research anchors.
2 Method
ScienceFlow structures long-horizon research as recoverable executable states organized into segments, with ESTRA selecting anchors and directions at segment boundaries. A separate evidence-aware controller manages physical execution using resource constraints and validated progress.
- Long-Horizon Research Process: ScienceFlow represents research progress as recoverable executable states that persist across iterations for resumption, trajectory adaptation, and resource coordination.Each state binds an executable workspace with compact memory, validation evidence, and resource records.
- Research State: Each research state contains a workspace snapshot, structured long-horizon memory, validation evidence, and associated archival information.The workspace snapshot can include code, data scripts, checkpoints, cached features, outputs, metadata, and references to large artifacts.
- Research Segments: Result signals trigger stage-gate evaluation, compact progress summarization, workspace snapshotting, and archival of the resulting state with evidence and resource records.Segments also transfer control to ESTRA after a text-only response or when accumulated context reaches the configured capacity threshold.
- ESTRA: At segment boundaries, ESTRA independently chooses a current or archived execution anchor and whether to extend the trajectory or redirect to a new branch.These two binary choices yield four possible outcomes for initializing the next research segment.
- Execution Control: ScienceFlow separates research-route selection from physical execution control: workers propose executable jobs, while an evidence-aware controller decides their execution using availability, remaining budget, and validated progress.This separation lets route deliberation and runtime resource management operate as distinct decisions.
3 Experiment
ScienceFlow sustains long-horizon research through recoverable trajectory management, evidence-guided execution control, and adaptive resource allocation. Experiments across ML, scientific modeling, and optimization show strong benchmark performance, mechanism-specific benefits, and continued improvement over extended campaigns.
- MLE-bench results: 70.22 ± 1.18% Any-Medal rate on full 75-task MLE-bench exceeds the strongest reported baseline by 4.92 percentage points.ScienceFlow also reaches 74.56 ± 0.88% on the Medium tier, 10.52 percentage points above the best reported baseline.
- Trajectory management: ScienceFlow restores archived execution anchors and selectively reuses workspace artifacts when exploratory branches fail to improve the incumbent.Examples include returning to S59 for tree-model exploration over saved CNN features and returning to S112 after an unproductive stacking branch.
- Evidence-guided search: Validation-guided decisions reduce the best validated loss from 0.11 to 0.1025 and finally 0.0597 while preserving unsuccessful trials for later comparison or reuse.Retrospective test scores remain hidden from the agent, while their improvement alongside validation supports the informativeness of the search signal.
- Mechanism ablation: 80.30 ± 2.62% Any-Medal rate with the full system falls to 66.67 ± 2.62% without ESTRA, while task trajectories associate ESTRA with delayed medal discovery.Removing execution control increases median first-medal time from 5.18 to 8.05 hours on APTOS 2019, whereas removing ESTRA increases it from 2.12 to 5.17 hours on Jigsaw.
- Backbone trade-offs: GLM-5.1 achieves the highest mean test AUROC of 0.9947 at $8.11, while DeepSeek-V4-Flash-Preview reaches 0.9882 at $0.096 as the strongest lower-cost trade-off.DeepSeek-V4-Pro-Preview is cheapest at $0.074 but has the lowest mean AUROC, 0.9847; comparisons are descriptive because they use three runs on one task.
- Long-horizon optimization: 393.229 on KTTSP-hard ranks ScienceFlow third, following a 79.6% objective reduction from 1928.39 mission days at S01 to 393.229 at S112.Across the campaign, 188 persisted snapshots support memory folding, while peer-worker validation redirects search toward route ordering and route swaps.
4 Related Work
Related work spans autonomous research systems organized around search, multi-agent collaboration, memory, and long-horizon control across machine learning, natural sciences, and optimization. Despite rapid progress, persistent weaknesses in reproducibility, validation, transfer, and causal experiment histories motivate ScienceFlow’s combination of executable-state management and adaptive execution.
- Landscape: Surveys organize autonomous research agents by methodological stages, historical evolution, task taxonomies, and foundational capabilities such as reasoning, tools, memory, collaboration, and optimization.These complementary frameworks systematize a rapidly fragmenting landscape.
- End-to-end systems: End-to-end systems now automate research loops across disciplines, including idea generation, code execution, paper writing, agentic tree search, and LLM-guided wet-lab experimentation.The cited examples include The AI Scientist and its successor in machine learning, alongside natural-science agents coupling reasoning with laboratory hardware.
- Open limitations: Persistent limitations include brittle reproducibility and provenance, weak novelty validation, uncalibrated confidence, poor transfer from monolithic architectures, and memories unable to sustain causally linked experiment histories.These limitations are identified as convergent concerns across surveys of the field.
- MLE-bench: 16.9% medal rate is attained by the o1-preview-powered AIDE baseline on full MLE-bench, whose 75 Kaggle competitions require end-to-end solutions within 24 hours on one GPU.MLE-bench evaluates submissions against original Kaggle medal thresholds and screens for plagiarism with automated detectors.
- MLE-agent directions: Search-driven agents, hierarchical multi-agent systems, and memory architectures represent three major MLE-agent directions, while ML-Master 2.0’s three-tier cognitive cache reaches a 56.4% full-benchmark success rate.Search work highlights operator-set bottlenecks; multi-agent work decomposes workflows into collaborating specialists; memory work restructures execution histories.
- Closest mechanisms and distinction: Adjacent systems separately provide typed research memory, execution-state restoration, trajectory verification, or resource-aware scheduling, whereas ScienceFlow integrates these mechanisms through its own state-transition and execution framework.The cited systems include AutoSci, MAGE, PIVOT, and agentic schedulers using telemetry or runtime monitoring.
5 Conclusion
ScienceFlow is presented as a workspace-grounded autonomous research system for long-horizon executable work. It combines recoverable states, ESTRA-governed segment transitions, evidence-aware execution control, and isolated worker workspaces synchronized at research-segment boundaries.
- System design: ScienceFlow organizes long-horizon executable research around recoverable workspace states and ESTRA-governed transitions between research segments.These components support autonomous research across evolving workspaces and segment boundaries.
- System design: Evidence-aware execution control manages research execution alongside configurable homogeneous workers operating in isolated workspaces.Workers synchronize at research-segment boundaries.
- Evaluation: ScienceFlow was evaluated across machine learning engineering, mathematical optimization, and engineering optimization.The evaluation spans multiple autonomous research domains.
Contributions and Acknowledgments
The paper lists its authors and identifies one internship contribution and three team leaders through author footnotes.
- Contributions and Acknowledgments: The authors are Mingming Zhao, Jiqian Dong, Kangping Xu, Zadid Hasan, Chengrui Fan, Shan Jiang, Shuai Mao, Ting Lingya, Linyi Zou, Tailin Zhou, Yun Hin Chan, Wenkai Zhang, Zhanhong Zhou, Guowei Huang, Hongliang Li, Wenjing Cun, Zhitang Chen, Mingxuan Yuan, and Yanhui Geng.
- Contributions and Acknowledgments: Zhitang Chen, Mingxuan Yuan, and Yanhui Geng are identified as team leaders.
- Contributions and Acknowledgments: Kangping Xu’s contribution was completed during an internship.
A Implementation Details … B MLE-bench Supplementary Results
ScienceFlow implements research as segmented reasoning–action–observation loops with stable-prefix context assembly and recoverable workspace states. At segment boundaries, ESTRA uses evidence and resources to select an execution anchor and extend or redirect the research trajectory.
- A.1 Context Construction and Interaction Pipeline: ScienceFlow assembles each segment’s model-facing context from a stable prefix, a selected anchor state, ESTRA direction, and segment-local interaction history.The stable prefix contains worker, runtime, tool, and task components; anchor-specific and local content is updated at segment boundaries.
- B MLE-bench Supplementary Results: The supplied passages describe implementation details for context construction and ESTRA transitions but provide no substantive MLE-bench supplementary result.No B-section result passage is included in the input.
- A.1 Context Construction and Interaction Pipeline: The initial segment combines stable context with a compact initial-workspace view specifying the objective, evaluator, budget, artifact contract, and task-specific guidance.MLE-bench guidance emphasizes inspection, leakage-safe validation, and bounded baselines before expensive training.
- A.1 Context Construction and Interaction Pipeline: During forward research, a stage gate records result cards, workspace snapshots, validation evidence, and resource records before archiving the resulting state.The worker then resumes the current segment without including the stage-gate exchange in segment-local history.
- A.2 ESTRA Decision and Re-Anchoring Pipeline: When a segment closes, ESTRA selects either the live workspace or an archived executable state and chooses whether to extend or redirect the trajectory.Archived-state selection restores the corresponding workspace and folds the post-anchor branch into completed evidence while preserving indexed result cards.
- A.2 ESTRA Decision and Re-Anchoring Pipeline: ESTRA evaluates candidate anchors using validation evidence, workspace contents, recent progress, failure history, peer evidence, and remaining resource budget.Validation is treated as one signal among several, allowing preservation of promising routes or redirection from saturated ones.
- A.2 ESTRA Decision and Re-Anchoring Pipeline: Each ESTRA deliberation returns one structured decision containing the selected anchor and direction, current bottleneck, supporting evidence, decision rationale, and next search focus.Tool use is disabled during this deliberation.
- A.2 ESTRA Decision and Re-Anchoring Pipeline: Fold summarizes completed exploration as historical evidence, retaining the active workspace for current-state anchors or restoring an archived snapshot when re-anchoring.Exact workspace restoration is required only for archived anchors, and the anchor-direction axes yield four ESTRA outcomes.
B.1 Evaluation on the Full MLE-bench Set … C.1 Formal Task Definitions
ScienceFlow’s full MLE-bench evaluation reports descriptive medal outcomes across all 75 competitions and analyzes operational telemetry where complete records exist. The paper also formalizes three mathematical optimization benchmarks, including feasible circle packing, ratio minimization, and a verified Hermite-construction bound for the Fourier sign-uncertainty constant.
- B.1 Evaluation on the Full MLE-bench Set: Per-task Scores are mean task-specific benchmark scores over three independent runs, and raw scores should not be compared across competitions because evaluation metrics differ.Results are grouped by the official Lite, Medium, and High complexity splits.
- B.1 Evaluation on the Full MLE-bench Set: 54 of 75 tasks (72.0%) had at least one medal, comprising 22 gold, 17 silver, and 15 bronze.The per-task Any-Medal indicator is descriptive and differs from the three-run medal-rate calculation in Table 1.
- B.1 Evaluation on the Full MLE-bench Set: 35 of 75 tasks (46.67%) had a documented medal-producing result by four hours, while the final archive contained at least one medal for 54 tasks.The four-hour figure is a descriptive best-observed cutoff reconstructed from task logs, not the mean of three independent four-hour evaluations.
- B.2 Operational Telemetry Coverage: All 75 tasks were evaluated under the same ScienceFlow protocol, but mechanism-level analysis covers 54 tasks because both ESTRA-event records and checkpoint/snapshot telemetry were available only there.The remaining 21 tasks retain benchmark outcomes but lack complete mechanism-level records, so the smaller denominator reflects telemetry availability rather than task selection or configuration.
- C Mathematical Optimization Details: For circle packing with n = 26, candidates place centers pi = (xi, yi) in the unit square with nonnegative radii, subject to containment and non-overlap constraints.The score is the sum of radii of a feasible configuration, so larger values are better.
- C.1 Formal Task Definitions: Ratio minimization uses n = 16 distinct points in R2, with an objective invariant to translation, rotation, and uniform scaling; the reported ρ(P) is better when lower.Published ρ(P)2 values are square-rooted to match Table 3 reporting.
- C.1 Formal Task Definitions: The uncertainty benchmark follows the first Hermite-polynomial formulation and evaluates a verified upper bound on C4 from the largest positive sign-changing root after removing z2.The construction enforces Qc(0) = 0, orients the polynomial to be positive at infinity, and excludes the separate Laguerre-polynomial refinement.
C.2 Evaluation and Feasibility Audit · D SpOC4-KTTSP Supplementary Details
ScienceFlow’s feasibility audit re-evaluates saved candidates from immutable workspace artifacts, verifies task-specific constraints, and recomputes objectives rather than relying on rounded ledger metrics. Audited results include conservative circle-packing repair, direct ratio evaluation, independently reconstructed uncertainty verification, and explicitly documented baseline provenance.
- C.2 Evaluation and Feasibility Audit: Candidates are re-evaluated from decimal literals in immutable workspace snapshots instead of rounded stage-ledger metrics.The audit parses saved artifacts before comparison.
- C.2 Evaluation and Feasibility Audit: Malformed, unreadable, or infeasible candidates are excluded after structural and task-specific feasibility checks.The audit also recomputes each displayed objective.
- C.2 Evaluation and Feasibility Audit: 2.63598308491745569 is the strictly feasible repaired circle-packing score after δ = 9.26 × 10−15, preserving the precision shown in Table 3.Repair is used only for a conservative feasible score and never to improve a candidate.
- C.2 Evaluation and Feasibility Audit: 3.590157365310609 is the best audited DeepSeek-V4-Flash-Preview ratio, versus 3.590157365325370478 for openPangu-2.0-Pro.The audit uses 60-digit coordinates, exactly 16 two-dimensional points, all 120 pairwise distances, and direct dmax/dmin evaluation.
- C.2 Evaluation and Feasibility Audit: 0.348200107555 and 0.343293122432 are reported ScienceFlow uncertainty bounds that both pass the reconstructed public AlphaEvolve B.4 Hermite verifier.Verification uses symbolic real roots and 200-digit root approximations.
- C.2 Evaluation and Feasibility Audit: Published Table 3 entries retain source-reported constructions and precision rather than representing independent reruns.Sources are cited papers or released repositories.
- C.2 Evaluation and Feasibility Audit: AlphaEvolve’s uncertainty entry is independently reproduced, while OpenEvolve is run locally and scored with its local task evaluator.The passage identifies these as distinct provenance procedures for the compared entries.
D.1 Competition, Tracks, and Evaluation Protocol
The evaluation uses ESA’s SpOC4 Keplerian Tomato Traveling Salesperson Problem across three target-count tracks, with submissions validated through constrained Lambert-transfer trajectory solving. Results use CPU-only evaluation and are reported as public leaderboard outcomes without compute normalization across tracks.
- Competition and tracks: SpOC4 evaluates spacecraft routes that collect all lunar-orbit targets in minimum mission time under orbital-transfer and maneuverability constraints.The study evaluates all three official instances.
- Competition and tracks: 50, 182, and 1,052 targets define the easy, medium, and hard tracks, respectively.These are the official SpOC4 track sizes.
- Evaluation protocol: Each submission specifies a target permutation, N −1 departure epochs, and N −1 transfer durations, which are checked for validity and chronological consistency.Each transfer is then solved with a Lambert solver in both directions, allowing at most 20 revolutions.
- Evaluation protocol: 100 m s−1 nominal-leg ∆V, at most five exception legs above that threshold, and a 600 m s−1 per-leg maximum constrain trajectories, scored by final-target arrival time.Trajectory evaluation is CPU-only; easy and medium use staged continuation runs, while KTTSP-hard uses a separate ten-day campaign.
- Evaluation protocol: Public leaderboard outcomes are reported without compute-normalized comparison because resumed stages and worker counts differed across tracks.The hard-track campaign used a separate ten-day setup, and the reported model set included DeepSeek-V4-Pro-Preview, DeepSeek-V4-Flash-Preview, and GLM-5.1.
D.2 Context-Folding Trace on KTTSP-hard
On the KTTSP-hard trace, ScienceFlow folds an oversized append-only stage ledger into an addressable historical summary while retaining decision-critical cards for verification and continuation. On-demand unfolding restores selected archived cards for active comparison without changing the underlying ledger.
- Context folding: At S115, Fold compressed cards S01–S112 from 73,964 to 4,661 characters, reducing the memory view by 93.7%.The folded view replaced historical cards with the addressable summary L_S01_S112.
- Active unfolding: S18 remained an invalid_submission with metric=1761.93330934, whereas S112 recorded metric=393.229 with high validity.The S18 card had no submission artifact, so its timing refinement was not an eligible verified comparable entry.
- Context folding: The budgeted view retained S01, S18, and S112 as verification cards alongside raw current-segment cards S113–S115.The summary exposed S112 as the best validated available stage and L_S01_S112 as the expansion identifier.
- Context folding: The historical summary preserved S01’s metric=1928.39350319 and S112’s metric=393.229 while summarizing 107 intermediate stages.It also recorded validity and brief descriptions for representative stages across the folded range.
- Active unfolding: Unfold(L_S01_S112, stages={S18, S112}) returned the original S18 and S112 cards for active comparison.The runtime resolved the addressable summary back to source cards in .run results.md through indexed workspace reads.
D.3 ESTRA-Triggered Folding Trace on KTTSP-hard
The recorded ESTRA transition responds to a context-limit boundary by restoring archived state S23, folding the abandoned S24–S27 branch into a compact summary and state packet. It preserves negative evidence and the persistent research record while redirecting the active executable trajectory.
- ESTRA transition: At 17:17 UTC on June 21, ESTRA switched from terminal stage S27 to archived anchor S23 at a context-limit research-segment boundary.The fold covered abandoned branch S24–S27 and moved the worker to the restored trajectory.
- State compression: 845 characters summarized the folded branch, while the restored trajectory received a 3,583-character state packet.The packet preserved raw cards, snapshots, logs, and the terminal archive.
- Abandoned-tail evidence: 1746.89350787 remained the S26 and S27 metric, with timing-only refinement exhausted and route ordering identified as the dominant bottleneck.The next route was directed toward proxy-prefiltered construction and bounded checks rather than further timing polish.
- Recovery semantics: After restoration, the worker resumed from S23 while folded S24–S27 evidence remained archived and addressable through Unfold without altering the persistent research record.The transition changed the active route and agent-facing context, not the underlying research record.
D.4 Peer-Guided Search Trace on KTTSP-hard
The KTTSP-hard trace shows ESTRA using compact peer evidence to redirect W01’s search without transferring artifacts or copying W00’s procedure. This adaptation reduced W01’s duration from 1746.8935 to 393.229 days, ultimately outperforming W00’s final validated result.
- Peer-guided coordination: W00’s validated 521.9054-day solution entered W01’s ESTRA context as compact peer evidence, while both workers retained isolated executable workspaces.W01 used the evidence to redirect its trajectory rather than receiving transferred artifacts or workspace state.
- First redirect: At S25, W01 diagnosed route ordering as the bottleneck and redirected toward proxy-prefiltered nearest-neighbor construction with PyKEP verification.The proposed approach ranked by orbital distance and verified only the top candidates rather than evaluating the full grid.
- Search limitations: 1746.89 days was W01’s timing-refinement result, while greedy nearest-neighbor rewriting was too slow at 47 seconds per 10 legs.W01 later reached 1321.8824 days but remained approximately 802 days behind W00, motivating a second redirect.
- Validated overtaking: 75.9% improvement reduced W01’s incumbent from 1746.8935 to 420.2742 days, and W01 reached 393.229 days versus W00’s final validated 519.6255 days.W01 produced the 420.2742-day result within 12 hours and 17 minutes of the S79 redirect and ultimately outperformed W00 by 24.32%.
- Independent adaptation: W01’s lead came from independently combining a 10-degree inclination-bin route, phase-proximal construction, and finer timing grids; applying windowed SLSQP alone improved the objective by only 0.02 days.The trace therefore attributes the improvement to peer-guided operator selection rather than direct solution reuse.
E SciModelingBench Provenance and Evaluation Metadata
This section documents SciModelingBench’s provenance, task metadata, reconstruction procedures, evaluation protocol, and runtime leakage boundaries. It emphasizes that evaluation uses audited source data and controlled workspaces, without claiming contamination-free testing.
- Task metadata: SciModelingBench evaluation covers 12 tasks from the public 0.10.0 release, with submission size, diagnostic summary size, and query limits defined per system–task pair.Table A8 defines N as candidates per submission, K as shared-diagnostic summary size, and Q as the total query limit; ScienceFlow’s two workers split Q.
- Data reconstruction: The benchmark reconstructs retained settings from original experiments or simulators, separating candidate identity, visible observations, hidden reference data, and trusted objectives.Predictive models may rank candidates, but their predictions are not treated as ground truth outside the agent’s permitted observations.
- Task reconstruction: Task-specific reconstruction resolves duplicated, conflicting, or context-poor legacy records before defining finite measured evaluation pools and ranking targets.Examples include TFBind sequence deduplication, replicate-aware TFBind10 affinity, grouped superconducting compositions, protein-identity splits for GFP, stochastic Hopper evaluation, and context-preserving DrugMatrix reconstruction.
- Protocol and metrics: Before agent runs, audits fixed score thresholds, structured holdouts, submission sizes, and primary metrics to distinguish submissions while keeping queried fractions small.The choices were based on source-data analyses and task-specific random, simple, and cross-validated audits rather than agent results.
- Leakage controls: Runtime controls isolate agents from hidden labels, trusted objectives, evaluator caches, full datasets, submission records, and external network access.The evaluation harness returns one full-batch score and separately records query use and the best valid artifact.
- Contamination scope: The evaluation claims runtime leakage controls and audited provenance, not contamination-free testing, because public papers or upstream data may have appeared in model pretraining.General scientific knowledge is allowed, but distinguishing inference from memorized candidate labels can be difficult.