PaperlayerView exact revision on arXiv

Source-linked AI summary

State Representation Shapes Long-Horizon Agent Learning and Transfer: An Empirical Study

Dong Nie, Rui Lan, Yixin Nie, Lingzi Hong

arXiv:2610.15407v1cs.AI

TL;DR

End-to-end success alone does not distinguish learning an action from applying it as task state changes. The paper compares supervised learning across workflow horizons and state views in deterministic tool workflows. Compact-to-full transfer fails in the controlled artifact tasks, but the authors do not isolate content reduction from other observation changes.

  • Problem

    End-to-end success alone cannot distinguish failure to learn an action from failure to apply it as task state changes.

    Source: §1 · p. 1

  • Method

    The study varies supervised training horizon and state representation in deterministic tool workflows, evaluating agents on held-out tasks under crossed training and evaluation views.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    Across three paired 1.7B seed pairs, compact-trained adapters completed 50/50 held-out H16 workflows in compact view and 0/50 in full view; paired content-dependent choices showed similar view dependence.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The controlled workflow results identify representation-sensitive transfer as a bottleneck, while the ScienceWorld study shows a different cross-view pattern.

    Source: §5 · p. 11, §6 · p. 12

    The observation intervention changes both displayed content and relevance information, so the experiments do not isolate content reduction from position, salience, or prompt length.

    Source: §3.2 · p. 4, §5 · p. 11

Abstract

from arXiv · show

An end-to-end success rate alone cannot distinguish failure to learn an action from failure to apply it as task state changes. We study this distinction in deterministic tool workflows that provide the current artifact state and candidate action recipes. Using 1.7B, 8B, and 27B Qwen models, we compare supervised learning across workflow horizons and cross the state representation used for training with the one used at evaluation. Horizon-matched supervision yields high completion, but an H4-trained 1.7B model transfers poorly to longer workflows under full-state observations. In an H16 crossover, compact-trained 1.7B adapters complete 50/50 held-out tasks in compact view and 0/50 in full view for each of three paired training seeds; full-trained outcomes vary across seeds. A separate 8B adapter pair shows a similar asymmetry. The view dependence also appears in paired-choice tasks where both candidate calls are executable and the correct choice depends on table contents; here, the compact mask does not consult the target action. An explicit relevance cue improves full-view execution but does not close the gap for the tested checkpoint. In a separate 27B ScienceWorld study, compact-state training improves local accuracy under both views, indicating that the transfer pattern depends on the task and observation construction. Together, these experiments identify representation-sensitive transfer as a bottleneck in the controlled workflows, without isolating content reduction from other changes to the observation.

1 Introduction

The study asks whether supervised behavior transfers across workflow horizons and state views when agents must select and execute actions using supplied artifact states and recipes. Results distinguish horizon-matched learning from transfer: compact-trained agents can fail under full-state evaluation, while controls and a separate task family delimit that pattern.

1 Introduction

  • The study tests whether supervised learning transfers across workflow horizons and state representations, separating action learning from applying actions as task state changes.The controlled workflows provide the current artifact state and candidate recipes, focusing on use of available state rather than memory-based reconstruction.Source: §1 · p. 1
  • The experiments vary training and evaluation views and use action-level supervision, which does not independently test exploration or temporal credit assignment.Evaluations include longer workflows, representation crossover, correct-prefix probes, and free-running rollouts.Source: §1 · p. 2
  • Across three paired 1.7B seed pairs, compact-trained adapters completed 50/50 held-out H16 workflows in compact view and 0/50 in full view, while full-trained outcomes varied across seeds.A separate 8B adapter pair showed a similar asymmetry; the full-view learning-curve decline did not recur uniformly across seeds.Source: §1 · p. 2
  • Content-dependent paired choices and relevance-cue controls show that view dependence is not limited to target-selected masking, while ScienceWorld exhibits a different cross-view pattern.The compact view retains inputs selected by the evaluator, so the benefit cannot be attributed solely to hiding other artifact contents.Source: §1 · p. 2

2 Related Work

Related work addresses agent learning and horizon transfer, tool interfaces and observation relevance, and observation management and execution reliability. This study evaluates supervised transfer across workflow horizons and state representations in deterministic tool workflows.

Agent learning and horizon transfer

  • AgentTuning improves agent capabilities through supervised interaction-trajectory training mixed with general instruction data (Zeng et al., 2024), while this study tests supervised transfer across horizons and state views.Related work also covers expert corrections after student interactions (Ross et al., 2011; Lauffer et al., 2025) and horizon generalization with macro-actions and subgoals (Kim et al., 2026).Source: §2 · p. 2

Tool interfaces and observation relevance

  • Prior work studies tool-interface design and observation sensitivity, including position effects (Liu et al., 2024) and distractor training or selection (Yang et al., 2025b; Luo et al., 2026).This study changes visible artifact contents while keeping task recipes and action targets fixed.Source: §2 · p. 2

Observation management and representation transfer

Execution reliability and recovery

3 Experimental Design

The design uses deterministic artifact workflows to test action selection, tool-call generation, and completion across horizons and state views. It compares full and compact observations, then evaluates horizon-specific training, crossover transfer, local decisions, autonomous rollouts, and recovery.

3.1 Deterministic artifact workflows

  • Workflows are directed acyclic graphs of required operations; the main chain tasks use H ∈{4, 8, 16}, and agents select between two recipes and return a JSON tool call.Tools perform table operations, while exact verification checks required artifacts and their dependency ancestry; rejected calls consume budget and leave artifacts unchanged.Source: §3.1 · p. 3
  • The separate H16 paired-choice family presents two executable recipes, with a table-cell change flipping which action meets the goal; its independent stage inputs test content-dependent choice, not dependency composition.Compact observations retain the active input and artifact names while hiding other contents, without consulting the correct action.Source: §3.1 · p. 4

3.2 Full and compact observations

  • FULL displays all available artifact contents, whereas COMPACT displays contents for the reference action’s inputs; both views preserve artifact names and unchanged task mechanics.The reference is the gold action during training and the evaluator-selected next action at evaluation, so compact view both reduces displayed content and supplies relevance information.Source: §3.2 · p. 4
  • The intervention does not isolate content reduction from position, salience, or prompt length, and attributing gains solely to reduced distraction would require a relevance-matched control.Inference-time controls match an explicit input-name relevance cue across full and compact views; a recent2 mask instead uses recency without consulting the reference action.Source: §3.2 · p. 4

3.3 Learning and evaluation protocol

  • The study evaluates untuned and horizon-specific Qwen3 adapters, but differences in generation, adaptation, data exposure, and inference paths preclude isolating parameter-count effects.Training uses gold-prefix observations and JSON actions; the 1.7B and 8B horizon datasets each contain 800 action examples, but equal action counts do not match workflow counts, token exposure, or compute.Source: §3.3 · pp. 4–5
  • Transfer tests use common held-out H16 tasks across three paired 1.7B seeds, while probes distinguish canonical action agreement from autonomous completion and measure recovery after rejection.Correct-prefix probes restore gold states; free-run success requires verified completion under the policy’s own calls, and a controlled 8B comparison injects the same rejected call across train/test-view conditions.Source: §3.3 · p. 5

4 Results

Horizon-matched supervision supports high completion, but short-to-long and cross-view transfer can fail. Across controlled workflows, compact-view gains and transfer depend on training and evaluation views, while local accuracy, completion, and recovery provide distinct diagnostics; a separate ScienceWorld study shows a different pattern.

4.1 Learning and horizon transfer

  • Horizon-matched 27B adapters completed 20/20 tasks at H4, H8, and H16 under both views, but H4-trained 1.7B transfer reached only 5/20 H8 and 0/20 H16 full-view completions.The 27B results are horizon-matched; the H4-trained row alone tests short-to-long transfer.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 Test-time state representation

  • For fixed checkpoints, compact observations raised H16 completion from 0/20 to 18/20 for the H4-trained 1.7B adapter, but the view change also supplies relevance information.The same 1.7B contrast held on twenty fresh tasks; untuned 8B likewise rose from 0/20 to 16/20 at H16.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Representation-dependent learning

  • Across three 1.7B seed pairs, compact-trained adapters completed 50/50 H16 tasks in compact view and 0/50 in full view, while full-trained outcomes varied.The paired train-view by test-view completion interaction averaged 106 percentage points; these are descriptive seed-level estimates, not a population-level test.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Content-dependent selection

  • In the paired-choice control, compact-trained adapters chose both table-content-dependent variants correctly on 50/50 pairs in compact view but only 9/50 in full view; full-trained accuracy was 49–52/100 in full view.Both candidate calls were executable, and the compact mask was independent of the correct recipe; full-trained adapters’ near-chance matching-view accuracy prevents a symmetric transfer claim.Source: §4.4 · p. 8, Table 5 · p. 9

4.5 Mixed-view training as a transfer control

  • The mixed-view pilot retained 10/10 compact completions but reached only 1/10 under full observations; its comparisons do not establish a reliable advantage for random masking.The pilot used one seed and ten development tasks, and none of six paired success contrasts against the two baselines was conclusive.Source: §4.5 · p. 9, Table 6 · p. 9

4.6 Local accuracy, completion, and recovery

  • Selection and execution diagnostics diverge: 1.7B supplied-recipe calls were correct on 167/168 cases, while fresh-seed selection accuracy was 40.3% with full tables and 70.0% with names only.On branching tasks, 8B post-stage-four canonical accuracy rose from 38.8% to 77.5% under compaction, but completion changed from 1/20 to 3/20 (p = .625); 65 audited mismatches were legal progress-making actions.Source: §4.6 · p. 9

4.7 External evaluation on ScienceWorld

  • In ScienceWorld, compact-state training reached 78.64% local action agreement on compact and 76.89% on full DEV observations, using a different model, task structure, and observation construction.This differs from the artifact crossover; natural workflow length covaries with task semantics, so the study does not isolate a horizon effect.Source: §4.7 · p. 10

5 Interpretation and Limitations

The interpretation centers on a replicated compact-to-full transfer failure while qualifying what the controls establish and what the tested settings leave unresolved.

5 Interpretation and Limitations

  • At H16, all three compact-trained 1.7B adapters completed 50/50 held-out workflows in compact view and 0/50 in full view; full-trained outcomes varied across seeds.The full-view learning-curve decline was not consistent across seeds, and full-view training remained near chance on pivotal paired choices under the tested budget.Source: §5 · p. 11
  • Matched relevance cues improve full-view execution without closing the gap, while reference-independent masking and content-dependent paired choices show the pattern is not limited to target-selected masks.The paired-choice task uses independent stage inputs, so it tests content-dependent selection rather than dependency composition.Source: §5 · p. 11
  • The evidence is bounded by differing 27B settings, a single-pair 8B crossover and error probe, and workflows that exclude unrestricted tool discovery and destructive-error repair.ScienceWorld extends evaluation to a public simulator, but its natural horizon covaries with task semantics and its state views differ from the artifact intervention.Source: §5 · p. 11

6 Conclusion

The conclusion reports representation-sensitive transfer in controlled workflows, while the appendix defines the workflow and observation interventions that frame that finding.

6 Conclusion

  • Supervised agents can complete tasks at their training horizon yet fail when the same task state is presented in another view.The H16 compact-to-full failure replicated across three 1.7B seed pairs and also appeared in a content-dependent choice task without target-selected masking.Source: §6 · p. 12

A.1 Workflow construction and interaction

  • Tasks are directed acyclic graphs of required operations, using deterministic table transformations and supplied recipes to test state-conditioned recipe selection and tool invocation.The verifier requires completion of all nodes with verified dependencies; rejected calls consume budget but leave artifacts unchanged.Source: §A.1 · p. 14

A.2 Observation construction

  • FULL displays all available artifact contents; COMPACT retains contents for reference-action inputs and shows other available artifacts without their contents.The paired views preserve candidate recipes, action requirements, transition rules, and verification, but the intervention also changes content amount and salience.Source: §A.2 · p. 14

A.3 Training

The appendix documents the training data and exposure, comparison design, and evaluation partitions and procedures for the controlled studies.

A.3 Training

  • The 1.7B and 8B horizon studies use 800 action examples per dataset, while 27B training uses separate horizon-specific data and exposure settings.The 27B source files contain 200 workflows each; all 3,200 H16 rows were retained, and its H16 batch differs from the smaller-model runs.Source: §A.3 · p. 15, Table 12 · p. 16
  • Equal optimizer steps do not ensure equal token exposure or compute, and the 27B runs differ in adaptation, generation, and effective data exposure.The study reports verified training exposure without asserting unrecoverable active compute replication details.Source: §A.3 · pp. 15–16, §A.4 · p. 17

A.4 Comparison design

  • Horizon-transfer evaluations hold checkpoint and state view fixed while varying evaluation horizon; the H16 crossover pairs the same tasks across training and evaluation views.The full-trained/full-evaluated H16 arm reuses the horizon-matched experiment rather than providing an independent replication.Source: §A.4 · p. 16
  • The 27B checkpoint curves use full-state oracle prefixes, and comparisons across model sizes do not isolate parameter count.The configurations differ in generation, adaptation method, and effective data exposure.Source: §A.4 · p. 17

A.5 Evaluation and data partitions

  • Local 1.7B and 8B evaluation uses greedy decoding, while untuned and fine-tuned 27B evaluations use different inference paths.Full/compact comparisons within a run pair the same tasks, whereas untuned/tuned comparisons do not; exact task fingerprints still require consolidation with cluster manifests.Source: §A.5 · p. 17

A.6 Metrics and uncertainty

The section defines metrics that separate correct-state action decisions from autonomous completion and describes how task pairing and observation controls support the analyses. It also specifies the horizon and paired-choice controls.

A.6 Metrics and uncertainty

  • Oracle-prefix accuracy measures agreement with the canonical next action on restored gold-prefix states; clean oracle-path rate requires agreement at every probed decision, not autonomous success.Free-run success instead measures verified completion under the model's own actions; final progress, attempted calls, rejected-call incidence, and observed recovery capture additional outcomes.Source: §A.6 · p. 17
  • Fixed-checkpoint comparisons use tasks as the sampling unit and preserve pairing; crossover intervals condition on trained adapters, while comparisons with separate 27B runs are descriptive.The original crossover uses 20,000 paired task-bootstrap resamples; exploratory sign-flip tests are unadjusted, and development studies are not fresh-seed confirmations.Source: §A.6 · pp. 17–18

A.7 Observation masking and execution

  • Rejected calls consume an action-budget unit, leave artifacts unchanged, and return an error; observation masking changes displayed contents while preserving underlying artifacts.A recipe whose output already exists is rejected, and the environment does not execute the gold action after an invalid choice.Source: §A.7 · p. 18

A.8 Horizon and accumulated state

  • The horizon control compares H4, H8, and H16 prefixes of shared H16 chains; Recent-2 selects artifact contents by successful creation history without consulting the reference action.It retains all artifact names and both candidate recipes; the caption distinguishes correct-prefix accuracy from free-run completion, which uses 2H calls.Source: §A.8 · p. 18, Table 14 · p. 19

A.9 Paired content-dependent choices

  • The paired-choice study flips a table cell to make the correct choice content-dependent while keeping the active input shared by both executable candidates.Its 50 held-out test pairs use equal-length candidate completions, and the compact view retains artifact names while showing only active-input contents.Source: §A.9 · p. 19, Table 15 · p. 19

B Statistical Analyses and Crossover Results

The H16 crossover shows a large asymmetry in transfer between compact and full views, while controls and other task settings qualify how broadly to interpret it. Additional interface, padding, horizon, and model comparisons distinguish local action selection from workflow completion.

B Statistical Analyses and Crossover Results

  • The H16 crossover interaction is 1.10: compact-trained success is 1.00 compact versus 0.00 full, while full-trained success is 0.75 compact versus 0.85 full.The paired task-bootstrap 95% interval is [0.85, 1.35]; the evidence supports asymmetric cross-view transfer, not general full-state robustness.Source: B Statistical Analyses and Crossover… · p. 19, Table 18 · p. 21

B.1 Untuned 27B observation contrasts

  • At H16, untuned 27B clean oracle paths improve from 0/20 full to 18/20 compact, while free-run completion changes from 16/20 to 20/20.At H8, clean paths change from 9/20 to 19/20, while completion changes from 19/20 to 20/20; contrasts are exploratory and unadjusted.Source: §B.1 · p. 21

C.1 Interface controls

  • Names-only exceeds full-table accuracy by 29.69 percentage points in the local interface study, while untuned 8B reaches 160/160 in both conditions in an earlier replication.The later horizon protocol changes the state distribution and requires many decisions, unlike a single local probe.Source: §C.1 · p. 22, Table 21 · p. 22
  • With three distractors, 1.7B accuracy is 107/240 versus 173/240 with one, while 8B accuracy is 240/240 versus 230/240.In the subsequent factorial study, aggregate accuracy is 73.54% for 1.7B and 96.04% for 8B.Source: §C.1 · p. 22

C.2 Padding and development studies

  • Inert padding preserves 100% accuracy through approximately 800 tokens and yields 96.25% at 1,200 tokens; the largest-tier paired comparison is inconclusive.An earlier 75–80% result also changed artifact identity and binding cues, so it is not a clean length control.Source: §C.2 · p. 22

C.3 Chain observation contrasts

  • On chains, compact-state success is 15/20 versus 1/20 full at H8 and 16/20 versus 0/20 full at H16 for untuned 8B.Mean calls also fall from 15.75 to 10.30 at H8 and 32.00 to 20.35 at H16; the compact view identifies relevant inputs as well as hiding contents.Source: §C.3 · p. 23
  • For untuned 27B, H16 clean oracle paths rise from 0/20 full to 18/20 compact, while free-run completion rises from 16/20 to 20/20.At H8, oracle accuracy changes from 92.5% to 100%; completion changes from 19/20 to 20/20, with inconclusive completion contrasts at these sample sizes.Source: §C.3 · p. 23

C.4 Horizon-matched learning curves

  • Horizon-matched supervision yields high endpoint completion; on the full-state grid, 1.7B H16 accuracy rises from 71.9% at step 20 to 99.4% at step 100.By contrast, the H4-trained 1.7B adapter completes 25% of H8 tasks and none at H16 under full-state evaluation; the grid does not establish a controlled model-size or sample-efficiency ranking.Source: §C.4 · p. 23, Table 22 · p. 23

C.5 27B step-100 full/compact evaluation

The study evaluates separately trained, full-state 27B adapters under full and compact views at step 100. All observed runs complete in both views, but the evaluation does not test compact-training transfer or establish equivalence beyond these tasks.

C.5 27B step-100 full/compact evaluation

  • At step 100, every evaluated free run completes in exactly H calls without an invalid action, with 100% verified final progress in both views.The table reports full and compact results for separately trained full-state adapters on twenty paired test tasks per horizon.Source: §C.5 · p. 24, Table 23 · p. 24
  • These all-success endpoints are an observed ceiling on 20 task seeds, not evidence of equivalent behavior on unseen distributions or compact-trained/full-tested transfer.Because no episodes contain rejected calls, conditional recovery has no error-containing denominator.Source: §C.5 · p. 24

D.1 Branching task metrics

On shallow branching tasks, compact observations improve untuned 8B local accuracy but do not reliably rescue task completion. Chain and 27B results distinguish invalid-action incidence, recovery, and completion, with protocol differences limiting direct comparisons.

D.1 Branching task metrics

  • 83.8% versus 36.3% at H8 and 77.5% versus 38.8% at H16 post-stage-four canonical oracle accuracy favor compact view, while autonomous success tests are inconclusive.H8 success changes from 1/20 to 7/20 (p = 0.070), and H16 from 1/20 to 3/20 (p = 0.625); mean verified H16 final progress decreases from 39.7% to 34.1%, also inconclusively.Source: §D.1 · p. 24, Table 24 · p. 25
  • Across shallow branching tasks, 65 noncanonical actions were legal and increased verified progress, so canonical action disagreement is not itself an execution error.Invalid-action status is used to count execution errors.Source: §D.1 · p. 24
  • Matched-error evaluation injects the same rejected call after the same verified four-action prefix and fixes the remaining budget to standardize post-error behavior.Its no-injection baseline follows model-generated states and does not estimate the causal effect of rejection; it covers one error type, adapter pair, and task family.Source: §D.1 · p. 24

D.2 Untuned 8B chain reliability

  • In untuned 8B chain runs, every full-state episode contains invalid actions, with only one H8 and no H16 episodes succeeding.Under compact evaluation, four of nine error-containing H8 episodes and eight of twelve error-containing H16 episodes succeed.Source: §D.2 · p. 24

D.3 Untuned 27B chain reliability

  • At H16, all twenty untuned 27B full-state runs contain an invalid call, though sixteen succeed; only two compact-state runs contain an invalid call.Full-state legal-action rate is 68.13% versus 99.38% under compact observations; every full-state disagreement with the canonical target is illegal.Source: §D.3 · p. 25
  • The horizon-matched H16 adapter records 20/20 free-run successes without observed errors in either view, but uses different task seeds and an NF4 plus LoRA inference path.These before/after values are different observed execution patterns, not a paired estimate of fine-tuning effects.Source: §D.3 · p. 25

E Training-Intervention Pilot Details

The pilot compares four view-training conditions while retaining the same action targets and recipes. Its small, separately evaluated paired contrasts and unmatched token counts limit what can be inferred about training strategies.

E Training-Intervention Pilot Details

  • The pilot compares full-only, compact-only, full/compact, and full/random training while keeping artifact names, recipes, and target actions unchanged.Each 400-example mixture half presents source examples once rather than duplicating them; random masking can hide required artifact contents, though identifiers and recipes suffice for these tasks.Source: E Training-Intervention Pilot Details · p. 25
  • The pilot uses the first 800 action rows from a 3,200-row H16 full-state manifest, covering 50 complete workflows with identical gold prefixes and completions.Mixture assignment uses a deterministic hash ordering; among the 400 random-view examples, masking probabilities are 0.25, 0.50, and 0.75 for 134, 133, and 133 examples.Source: E Training-Intervention Pilot Details · pp. 25–26
  • All four runs use Qwen3-1.7B BF16 LoRA action-level SFT on correct prefixes, with 800 distinct rows presented once and no on-policy or rejection-recovery examples.Training examples are retained without truncation, but matching examples and order does not match input-token counts or compute.Source: E Training-Intervention Pilot Details · pp. 25–26
  • Evaluation repeatedly tests ten H16 development tasks across four adapters, three views, and two modes; the 580 episodes are not 580 independent tasks.Endpoint and checkpoint probes use distinct episode sets, and free-run states can diverge across policies even though matched states receive matched block masks.Source: E Training-Intervention Pilot Details · p. 26
  • The pilot’s paired success contrasts are exploratory, unadjusted for six comparisons, and condition on the realized adapters.Small-sample tests do not establish equivalence or a general strategy ranking; the added adapter pair uses a different evaluation set and is reported separately.Source: Table 26 · p. 26, E Training-Intervention Pilot Details · p. 26

F ScienceWorld Protocol and Additional Results

The ScienceWorld study reports local agreement and autonomous outcomes across representation and natural-horizon evaluations. Compact-trained models improve local accuracy under both views, while short-only results vary by workflow length and seed.

F ScienceWorld Protocol and Additional Results

  • The initial ScienceWorld fine-tuning data contain 39 trajectories and 1,170 action examples, with 13 DEV trajectories and 440 action examples; DEV supports checkpoint selection, not untouched generalization.Full and compact renderings share target actions.Source: F ScienceWorld Protocol and Additional… · p. 26
  • The representation pilot evaluates ten official-test variations under three seeds and four train/test-view cells, totaling 120 rollouts without accessing official-test gold paths.Its local DEV estimates and autonomous official-test estimates use different task sets.Source: F ScienceWorld Protocol and Additional… · p. 27
  • The natural-horizon study evaluates 30 workflows unused by initial train/DEV selection, ten per reference-path-length bin, across six model/seed configurations.Natural length covaries with task semantics; bins use reference-path lengths, not proven minimal solutions or the agent’s actual calls.Source: F ScienceWorld Protocol and Additional… · p. 27
  • The compact-trained model’s compact advantage is 3.48 percentage points at step 80 and 1.74 points at step 100, while accuracy improves under both evaluation views.The unadapted checkpoint has 155/440 correct full-view predictions and 154/440 compact-view predictions.Source: F ScienceWorld Protocol and Additional… · p. 27
  • Short-only short-bin successes are 9/10, 7/10, and 10/10 by seed, versus long-bin successes of 2/10, 0/10, and 3/10.Corresponding local-accuracy decreases are 23.24, 20.65, and 20.26 percentage points; these seed-wise patterns describe this sample, not a randomized length effect.Source: F ScienceWorld Protocol and Additional… · p. 27, Table 29 · p. 28
  • Local agreement of at least 80% accounts for 27/37 successful short-only rollouts versus 19/53 below that threshold, and 17/56 versus 10/34 for mixed-horizon training.Reported Spearman associations with binary success are 0.489 and 0.128, respectively; these are descriptive seed-by-workflow diagnostics.Source: F ScienceWorld Protocol and Additional… · pp. 27–28
  • Only 3/90 short-only and 2/90 mixed-horizon evaluations have clean oracle paths, and all correspond to successful autonomous runs.Perfect agreement in separate reference-state probes is therefore not necessary for observed autonomous success; trajectories are not inspected to determine deviations or recovery.Source: F ScienceWorld Protocol and Additional… · p. 28
  • ScienceWorld results rely on aggregate run summaries whose raw trajectories and rendering rules are not consolidated for independent verification, and the planned 58-task official-test extension was not run.The 120-episode pilot counts parser rejections and recovery, but differing error cases and remaining budgets prevent causal attribution to representation.Source: F ScienceWorld Protocol and Additional… · p. 28, Table 30 · p. 28

G Evidence Provenance

The provenance account distinguishes audited records from unresolved verification gaps. Controlled workflow records are retained, while ScienceWorld and repository evidence have specific limits.

G Evidence Provenance

  • Controlled workflow experiments retain task fingerprints, split manifests, protocol files, checkpoint identifiers, and per-action trajectories.Later H16 pairs, 8B crossover, matched-error, content-choice, and horizon–state controls have separate frozen task sets and outcome records; superseded runs are not independent replications.Source: G Evidence Provenance · p. 28
  • The 27B training audit establishes completed step-100 checkpoints and reported exposure, but does not independently verify all held-out evaluation trajectories.Some H8 indices and token lengths were reconstructed, and exact launch arguments and active H16 compute-replica counts could not be recovered.Source: G Evidence Provenance · p. 29
  • ScienceWorld aggregate summaries pass arithmetic checks, but raw trajectories and the cluster renderer needed for independent trajectory-level verification are not consolidated.The planned larger official-test confirmation set contributes no results.Source: G Evidence Provenance · p. 29
  • Exploratory repository runs reached 0/5 exact successes in V1 and V2 and 0/1 in V3.1 per condition, so they are excluded from main claims.Those failures do not distinguish model limitations from interface calibration.Source: G Evidence Provenance · p. 29
Loading 2610.15407v1…