Self-Supervised Scaling of Terminal Environments for Scientific Domains

• Science and engineering domains remain less covered by current training data, while task-specific reference solutions and verifiers are costly to author and difficult to scale.Scientific outputs can include arrays, meshes, waveforms, sequences, and tables whose validity cannot be inferred from artifact existence alone.Source: §1 · p. 2
• SWR uses an existing workflow as both reference solution and source of expected outputs, releasing some input–output pairs while withholding others for evaluation.Agents construct a new program without access to the workflow code.Source: §1 · p. 2, Figure 2 · p. 3
• 838 tasks solved across three attempts yielded 1,422 verified trajectories from a release spanning 500 workflows and 46 software families.The verified trajectories were oversampled for reconstruction-only supervised fine-tuning.Source: §1 · p. 2
• The paper formulates workflow-derived tasks and a hierarchical semantic verifier, then characterizes reconstruction and evaluates verified trajectories as SFT data against matched-token controls.The verifier and release checks are assessed, including through an independent audit after the verifier is frozen.Source: §1 · p. 3

• Executable benchmarks extend from generated-program execution to interactive digital environments and terminal agents, while expected behavior is still usually specified separately for each task.Later pipelines increase the supply of environments, tasks, and trajectories for terminal-agent training.Source: §2 · p. 3
• Execution feedback supports agent revision, and scientific verification may require numerical tolerance, geometric equivalence, or domain-specific invariants.Here, public comparisons guide revision, while acceptance requires agreement on hidden inputs, structural compliance, and anti-shortcut checks.Source: §2 · pp. 3–4
• Unlike program induction and system-identification work that may seek reusable abstractions or predictive models, this task retains editable programs only when they match workflow outputs on hidden inputs.Agreement on a finite held-out set is not recovery of the original implementation or underlying mechanism.Source: §2 · p. 4

3.1 Task Definition

• Each task asks an agent to build an editable program from public examples while the workflow source and hidden inputs remain withheld for behavioral evaluation.A recorded reconstruction passed all six public URDF scenarios but failed on an unseen categorical value; another model's reconstruction passed it.Source: §3.1 · p. 4

3.2 Task Generation from Workflows

• Each workflow yields six variants with typed, disjoint public and hidden scenarios, and release checks require deterministic, stable, input-sensitive outputs plus failed replay controls.Fixed checks allow new workflows and configurations to extend the collection; the experiments use 500 workflows.Source: §3.2 · pp. 4–5

3.3 Hierarchical Semantic Verification

• The verifier averages task-specific comparisons on each split; public scores guide revision, but hidden score 1 and structural and anti-shortcut checks are required for acceptance.Structural checks can accept invalid constant output, while byte-identical comparison can reject behaviorally equivalent output; semantic verification targets both failure modes.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Reconstruction SFT

• Only trajectories whose final programs pass strict acceptance enter the eligible SFT pool, and trajectories for evaluation tasks are excluded before sampling.Sampling creates a multiset, so repeated trajectories reweight examples; training predicts each recorded action from its preceding interaction history.Source: §3.4 · pp. 5–6

4.1 Experimental Setup

• The evaluation measures strict hidden success, requiring a valid program, complete agreement on hidden inputs, and anti-shortcut checks; SWR100 tests held-out instances, not unseen workflows.TB2, TB4, and LHTB measure terminal-agent skill transfer rather than end-to-end science problem performance.Source: §4.1 · p. 6

4.2 Behavioral Validity of SWR

• SWR comprises 500 workflows across 46 software families and six domains, and outputs change across every compared pair for 2,984 tasks.On the remaining 16 tasks, 35 of 36 compared pairs change; overall, 75.1% of assessed output fields vary with configuration.Source: §4.2 · p. 7
• The three public-output replay controls reach 34.3%–44.3% agreement on hidden output fields but meet strict acceptance on no released task.Release inclusion requires all three controls to fail, so this result does not establish resistance to untested shortcuts.Source: §4.2 · p. 8

4.3 Verifier Reliability

• The frozen verifier’s post-freeze audit finds 2.0% false acceptance and 2.0% false rejection against adjudicated labels across 200 reviewed cases.The audit includes 100 valid and 100 invalid cases.Source: §4.3 · p. 8

4.4 Agent Performance

• Of 2,781 executable candidate workspaces, 1,591 emit the same output across different hidden scenarios and 75 replay a public output.Successful runs end at a median of 58 turns versus 85 turns for failures attributed to the model; the comparison does not establish causation.Source: §4.4 · p. 8
• On SWR100, GPT-5.6 Sol reaches 20.0% Pass@1, while the other seven models reach 2.0%–6.0%; 43 of 44 successes fall in three domains.No success occurs in Data & Document Workflows or Earth & Space Sciences.Source: Figure 6 · p. 9, §4.4 · p. 9

4.5 Reconstruction SFT

• At 3,000 entries, Qwen3.8-27B improves mean TB2 from 47.94% to 53.56% and has the highest mean among four matched-token controls on all four evaluations.Each training-entry budget is evaluated across three seeds; the 3,000-entry budget includes repeated samples.Source: §4.5 · p. 10, Table 4 · p. 10, Table 5 · p. 11

5 Conclusion

• Reconstruction SFT achieves the highest mean among compared training corpora on all four evaluations under a matched assistant-loss-token budget.Mean downstream terminal-agent performance improves across three seeds; the evaluations measure terminal-agent transfer, not autonomous scientific discovery.Source: §5 · p. 11
• A workflow supplies the reference solution and public examples plus hidden outputs, keeping reference generation and verification targets on one executable source.The release covers 500 workflows and 46 software families.Source: §5 · p. 11
• The conclusion identifies finite workflow coverage, related task patterns, and uneven difficulty across domains and families as dataset limitations.Future work can test generalization to unseen workflows and separate additional unique trajectories from repeated sampling.Source: §5 · p. 11

Appendix

• The appendix documents benchmark specification and dataset audit, including verifier stress tests, repeated-attempt and full-suite diagnostics, and SFT corpus construction and controls.It also covers SWR100 model and domain results, interaction and statistical diagnostics, and failure signatures across model–task pairs.Source: Appendix · p. 14

A Benchmark Specification

• The wrapped source workflow supplies the reference behavior; a task is added only when it passes strict acceptance and modal, nearest, and best-public replay controls fail.The best-public control uses hidden references to select among public outputs.Source: A Benchmark Specification · p. 15
• Each candidate maps one JSON scenario to one JSON object, receives semantic feedback on public scenarios, and is evaluated on withheld hidden scenarios.Any editable implementation satisfying the common interface is admissible.Source: A Benchmark Specification · p. 15
• A task is retained only if its workflow executes successfully, is deterministic and schema-stable, responds to scenario changes, and has disjoint public and hidden scenarios.The wrapped workflow must pass the verifier, while predefined replay strategies must fail.Source: A Benchmark Specification · p. 15
• The held-out SWR100 tasks are excluded from the final reconstruction-SFT corpus, but this does not establish generalization to unseen workflows.TB2, TB4, and LHTB are external evaluations with no shared training source.Source: A Benchmark Specification · p. 15
• Final success requires an executable reconstruction, schema-compatible outputs, matching hidden behavior, and anti-shortcut passage; partial semantic agreement is diagnostic only.Most tasks use recursive comparison of mandatory fields; 60 geometry tasks use a domain-specific comparator.Source: A Benchmark Specification · p. 15
• Each of 500 workflows yields six task variants, retained only when the source program passes and public-output replay controls fail.The algorithm constructs the reported 3,000-task release.Source: A Benchmark Specification · p. 16

B Dataset Composition and Audit

• SWR covers 46 software families across six domains, and no family exceeds 2.4% of the benchmark.Software & Formal Systems has 11 families and 660 tasks; Figure 10 shows the domain-to-family hierarchy.Source: B Dataset Composition and Audit · p. 17, Figure 10 · p. 20
• Tables occur in 2,028 tasks and logs or text in 1,776; the 46 families also span JSON, images, audio, scientific data, code, sequences, geometry, and other artifacts.Figure 8 maps output types by family; Figure 9 presents released artifact examples from each domain.Source: B Dataset Composition and Audit · p. 17, Figure 8 · p. 18, Figure 9 · p. 19
• 86.7% of tasks use five scenario parameters, while numeric values are the most common parameter type at 66.1%.The audit reports 66.1% numeric, 26.7% strings, 5.2% Booleans, 1.9% arrays, and 0.1% mixed types.Source: B Dataset Composition and Audit · p. 17
• The median task has 11 checkable output fields and the 90th percentile is 95.3; strict grading inspects every required field.The field-count audit excludes digest-like profiling fields; longer outputs include nested tables, geometry descriptors, or scientific arrays.Source: B Dataset Composition and Audit · p. 20, Figure 11 · p. 20
• 44.3% hidden-field agreement for oracle-selected best-public replay, 38.0% for nearest-scenario replay, and 34.3% for constant-output replay; none strictly passes a task.Every evaluated task changes under at least one scenario comparison, and 2,984 change for every observed scenario pair.Source: B Dataset Composition and Audit · p. 21, Table 7 · p. 21
• Six variants of a workflow share its underlying mechanism, while their hidden scenario sets and behavioral targets differ; coverage is finite and difficulty uneven.Related tasks can share instruction patterns and program structure, and software coverage is not exhaustive.Source: B Dataset Composition and Audit · p. 21

C Repeated-Attempt Pass@3 Analysis

• Pass@3 counts a task as solved when at least one of three Qwen3.8-Max attempts succeeds, and successful attempts are retained as verified trajectories.The three evaluations use the same finalized verifier.Source: C Repeated-Attempt Pass@3 Analysis · p. 21
• 838 tasks are solved across three attempts, yielding 27.9% Pass@3 and 1,422 verified trajectories for reconstruction SFT.Of the solved tasks, 383 succeed once, 326 twice, and 129 on all three attempts.Source: C Repeated-Attempt Pass@3 Analysis · p. 21
• 20 of 46 software families still contain no solved task after three attempts.The result indicates that additional sampling leaves a substantial part of the release unresolved.Source: C Repeated-Attempt Pass@3 Analysis · p. 21

D Full-Suite Diagnostics

• Of 2,781 evaluated workspaces with executable candidates, 1,591 return the same output across hidden scenarios and 75 replay released outputs; failed runs average 75.4% diagnostic agreement.Many executable submissions fail to reproduce how workflow outputs depend on inputs.Source: D Full-Suite Diagnostics · pp. 21–22
• Successful reconstructions end at a median of 58 turns, versus 85 turns for model-attributed failures, and family pass rate correlates negatively with median interaction length (r = −0.647).The longer interactions characterize failed runs rather than a few extreme trajectories.Source: D Full-Suite Diagnostics · p. 22, Figure 12 · p. 22

E SFT Corpus Construction

• Uniform re-verification yields 1,422 successful trajectories from 838 reconstruction tasks, which are oversampled into 3,000 reconstruction-only training entries.Repeated entries reuse the same interaction and do not add independent trajectories.Source: E SFT Corpus Construction · p. 22

F Reconstruction Fine-Tuning Scaling and Controls

• At 88.48M assistant-loss tokens, reconstruction SFT attains the highest mean among the evaluated corpora.The comparison holds loss-bearing text fixed but leaves task composition, trajectory diversity, and verification entangled.Source: F Reconstruction Fine-Tuning Scaling… · p. 23, Table 9 · p. 23
• The 3,000-entry checkpoint exceeds Base for every seed on all four benchmarks, with non-decreasing means as the sampled-entry budget rises.The reported budgets include repeated samples and do not isolate the effect of adding independent trajectories.Source: F Reconstruction Fine-Tuning Scaling… · pp. 22–23, Table 8 · p. 23

G SWR100 Model and Domain Results

• Of 800 scheduled trials, 797 produce records, 785 receive verifier scores, and 44 pass; missing and unscored runs remain in the denominator.Runs with execution records but no final score also remain in the Pass@1 denominator.Source: G SWR100 Model and Domain Results · p. 24, H SWR100 Interaction and Statistical… · p. 25, Table 10 · p. 25
• Across models, 27 successes occur in Software & Formal Systems, eight each in Physical & Engineering Simulation and Spatial, Visual & 3D, and one in Life & Molecular Sciences; none occur in Data & Document Workflows or Earth & Space Sciences.GPT-5.6 Sol accounts for 20 successes, followed by Kimi-K3 with six and Grok 4.6 with four.Source: G SWR100 Model and Domain Results · p. 24

H SWR100 Interaction and Statistical Diagnostics

• Under a three-hour-per-task upper bound, median interaction length ranges from 69.5 to 356.5 turns and median token use from 4.49M to 18.59M.These measures describe interaction-budget use; task success is reported separately.Source: H SWR100 Interaction and Statistical… · p. 24, Figure 13 · p. 24
• SWR100 uses one trial per model–task pair: the 14-point gap between GPT-5.6 Sol and the next-best model is large, but differences among models with two to six successes are unresolved.The task identities are disjoint from the processed SFT corpus, and the evaluation is in-domain.Source: H SWR100 Interaction and Statistical… · p. 24

I Verifier Mutation Stress Tests

• All numeric perturbations up to 0.99× tolerance are accepted, and all from 1.01× onward are rejected; the full stress-test suite reports no incorrect decisions.The verifier also rejects deeper-field changes, coherent numeric shifts, wrong-scenario outputs, and all 9,901 cases that ignore an input axis.Source: Table 11 · p. 25, I Verifier Mutation Stress Tests · p. 26, Figure 14 · p. 26
• The held-out audit samples 100 tasks disjoint from the pilot and covers 400 equivalence-preserving controls and 800 invalid mutations.Two annotators achieve 96.0% raw agreement, and 98.0% of predefined labels agree with human judgments after adjudication.Source: I Verifier Mutation Stress Tests · pp. 26–27, Table 13 · p. 27
• On the held-out 1,200-case audit, the verifier disagrees with 9 predefined mutation labels: it accepts 3 of 800 invalid cases and rejects 6 of 400 valid cases.These provisional rates are 0.38% false acceptance and 1.50% false rejection, respectively.Source: I Verifier Mutation Stress Tests · p. 26, Table 12 · p. 26
• After adjudication, the verifier makes two false accepts and two false rejects on 200 blinded cases, yielding 2.0% rates for each error type and 98.0% agreement.The remaining disagreements center on numerical tolerance and geometric equivalence; rejected valid cases are alternative surface tessellations that fail sampled-point coverage.Source: I Verifier Mutation Stress Tests · p. 27, Table 14 · p. 27

J Failure Signatures and Matched Trajectory Cases

• Among 756 failed trials, numeric mismatch is the primary label for 460 (60.8%), while 130 (17.2%) lack an executable candidate.Multi-label analysis finds 173 trials with both numeric and categorical mismatches and 19 with both numeric and structural mismatches.Source: J Failure Signatures and Matched… · p. 27, Figure 15 · p. 28
• Failure bottlenecks differ by model: some runs stop before a gradeable reconstruction exists, while others produce executable artifacts that fail to recover hidden behavior.Numeric mismatch dominates failures for GPT-5.6 Sol, Kimi-K3, Qwen3.8-Max, and Doubao Seed 2.1.Source: J Failure Signatures and Matched… · pp. 27–28
• Across four qualitative same-task contrasts, extensive observations and longer interaction do not ensure success; recorded successes turn observations into executable hypotheses tested beyond public examples.The selected cases diagnose mechanisms rather than estimate how often they occur.Source: J Failure Signatures and Matched… · pp. 28–29, Table 15 · p. 28
• In the color, SBOM, and video cases, unsuccessful runs spend interaction on inspection and perform fewer candidate tests than their matched successful runs.The comparisons are descriptive and do not establish test frequency as a cause of success.Source: J Failure Signatures and Matched… · p. 29
• Public checks can miss hidden-input failures: Qwen3.8-Max passes six robotics checks before an integer cast rejects a hidden categorical value, and 45 color-task feedback calls still leave the coupled XYZ transform outside tolerance.Informative testing for these cases also covers unseen valid types, unknown categories, and dependencies among output fields.Source: J Failure Signatures and Matched… · p. 29, Figure 16 · p. 30, Figure 18 · p. 32
• The SBOM failure leaves no executable candidate after 16 evidence-gathering commands, one reconstruction edit, and one candidate test, so semantic accuracy is never evaluated.The paper identifies early artifact creation, testing after new evidence, hidden-input checks, and retaining a runnable candidate as observable trajectory properties.Source: J Failure Signatures and Matched… · p. 29, Figure 17 · p. 31