PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research

Marjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva, Harold Chaput, Mohammad Reza Taesiri

arXiv:2608.11216v1cs.AI

TL;DR

Existing AI-research benchmarks often specify the data, metrics, and improvement direction, leaving open-ended research difficult to evaluate reproducibly. AutoWorldModel-Bench tests coding agents improving world models across eight structured-state game environments under fixed compute, and agents improved their starter in 63 of 64 sessions, mainly through non-trivial research edits.

  • Problem

    Most AI-research benchmarks target well-scoped problems with predefined datasets, metrics, and improvement directions, while controlled evaluations of open-ended research remain scarce.

  • Method

    AutoWorldModel-Bench evaluates coding agents in a six-hour closed loop, improving starter world models across eight games represented with unified structured state.

  • Results

    Agents improved their starter on 63 of 64 sessions, with gains concentrated in long-horizon rollout and 91% of winning edits being non-trivial research modifications.

  • Takeaways & Limitations

    AutoWorldModel-Bench provides a fixed-budget setting for measuring whether coding agents can steer research-style world-model exploration rather than engineering-to-spec improvement.

  • Takeaways & Limitations

    Because the benchmark uses explicit structured state from game engines, its findings are limited to transition modeling over known entities and may require adaptation for pixel-based settings.

Abstract

from arXiv · show

World modeling is an unsettled field: architectures, training objectives, and state representations interact in complex ways, and no single recipe dominates across environments. This makes it an ideal testbed for AI coding agents acting as autonomous researchers--a setting in which the improvement direction is not specified in advance, unlike the engineering-to-spec tasks that dominate current agent benchmarks. We introduce AutoWorldModel-Bench, a closed-loop benchmark in which frontier coding agents autonomously improve a provided world-model starter under a fixed compute budget. The benchmark spans eight game environments under a unified structured-state representation--ground-truth entity state extracted from each game and consumed through a shared tensor format--which isolates dynamics modeling from perception and enables minutes-per-run iteration. Across 64 sessions, Codex-5.4 and Claude Opus 4.6 improve their starter on 63; in 91% of sessions the winning edit is a non-trivial research-style modification--a new objective, representation, rollout procedure, or architectural change--rather than a hyperparameter tweak. Our benchmark offers a setting in which frontier coding agents can be evaluated on open-ended research rather than engineering-to-spec problems.

1 Introduction

The introduction frames world modeling as a rapidly advancing field with a broad design space and presents autonomous research agents as a promising but rigorously under-evaluated paradigm. AutoWorldModel-Bench connects these trajectories through a structured-state benchmark and a six-hour protocol for autonomous world-model improvement.

  • Motivation: World models have advanced from latent dynamics predictors to systems that imagine long-horizon trajectories, plan in learned state spaces, and simulate interactive video games.This progress has created a rich design space for researchers.
  • Motivation: Autonomous AI research agents can read codebases, formulate hypotheses, write training pipelines, run experiments, and iterate with minimal human intervention.Their potential to accelerate scientific progress motivates rigorous evaluation.
  • Motivation: Current evaluations span machine-learning engineering, broader ML research and replication, and early scientific-discovery settings, but remain uneven.The introduction identifies a need for stronger evaluation of autonomous research systems.
  • Contributions: AutoWorldModel-Bench covers eight game environments with a unified structured-state representation that sidesteps perception and supports minutes-per-run iteration.The benchmark includes pipelines for data extraction, preparation, and model evaluation.
  • Contributions: The benchmark defines a six-hour compute-bounded closed-loop protocol in which an agent autonomously improves a provided starter world-model architecture.The evaluation covers Codex-5.4 and Claude Opus 4.6 across 64 (game, starter) sessions.

2 Related Work

Prior world-model research spans pixel- and latent-based dynamics, object- or entity-centric representations, and evaluations beyond one-step prediction. Automated AI research systems motivate a distinct benchmark setting in which coding agents iteratively improve state-centric world models under fixed compute budgets.

  • World models: World-model research spans action-conditioned video prediction, latent dynamics, visually grounded game simulators, and large-scale generative models.Much of this work learns from pixels or pixel-inferred latent variables, intertwining perceptual and dynamics errors.
  • Object-centric and entity-centric representations: Object-centric approaches represent dynamics over objects, interactions, or slots rather than monolithic frame embeddings.This line of work argues that factored representations make dynamics more compositional, across both perception and dynamics modeling.
  • Automated AI research agents: Recent LLM-based systems automate code synthesis, experiment design, execution, analysis, and iterative refinement across expanding research evaluations.Existing evaluation suites cover ML engineering, open-ended research workflows, replication, and scientific-discovery settings.
  • World-model evaluation: AutoWorldModel-Bench is state-centric and closed-loop, evaluating autonomous coding agents that produce improved models within a fixed compute budget.This contrasts with prior benchmarks that evaluate pre-trained models, including reward-free exploration and video-generation world-model tasks.

3 AutoWorldModel-Bench

AutoWorldModel-Bench standardizes eight game environments through a structured ECS-based state representation and evaluates world models with complementary single-step, rollout, and scenario tests. Its closed-loop harness lets agents iteratively modify, train, and evaluate starter models using persistent experiment logs and fixed task infrastructure.

  • State representation: Each frame is serialized as a JSON envelope containing the player action, global state, and ordered entity slots with typed components.Components include Transform, Physics, Collider, Material, and Gameplay fields.
  • Benchmark environments: Eight game environments use adapters and manifests to map engine state into a shared canonical frame envelope.Manifests specify entity kinds, mutability, action fields, and coordinate bounds.
  • Dataset: The dataset contains 152,000 episodes totaling over 158 million frames across the eight benchmark games.Trajectories come from heuristic, random, and RL agents, with RL agents added for six games after manual review.
  • Evaluation: World models are evaluated with teacher-forced prediction, open-loop autoregressive rollouts, and per-game scenario tests.Open-loop evaluation measures error compounding over horizons h ∈ {10, 20}.
  • Closed-loop harness: Agents iteratively analyze prior runs, design changes, train, and evaluate models within self-contained tasks backed by persistent experiment logs.Each run is capped at 10 minutes, and an independent verifier scores the best checkpoint.

4 Experiments

The experiments evaluate autonomous world-model improvement across four structured-state starter families, two coding agents, and eight games under a fixed compute budget. Models are assessed using held-out test and scenario scores that emphasize long-horizon rollout accuracy while combining position and alive-status quality.

  • Starter models: Four starter families—RSSM/Dreamer, AR-Transformer, D3PM, and MaskGIT—operate on tensorized structured state and predict entity dynamics plus game-specific fields.Continuous models predict relative position deltas, while discrete models quantize positions and predict absolute bins; all use velocity consistency loss λ=0.1.
  • Agents and compute budget: 64 sessions evaluate Claude Opus 4.6 and Codex-5.4 across 8 games and 4 starters, with each agent receiving one H100 for 6 hours and a 10-minute training-run cap.Every game–starter pair is evaluated exactly once per agent.
  • Data splits: Agents access only training and validation trajectories during sessions; test and scenario splits remain held out for final post-session evaluation.Each game’s trajectories are partitioned into training, validation, test, and scenario splits.
  • Evaluation metrics: At each horizon, the composite combines normalized Position L1 and Alive F1 as 0.9 · (1 − Position L1) + 0.1 · Alive F1.Position L1 measures mutable-entity position error, while Alive F1 evaluates alive/dead prediction and discourages predicting every entity as alive.
  • Evaluation metrics: The final score weights horizons h1, h10, and h20 by 0.1, 0.2, and 0.7, respectively, and is recomputed on held-out test and scenario splits.The weighting favors long-horizon rollout accuracy; validation scores guide in-session search.

5 Results

Across 64 sessions, agents improved held-out test scores in 63 and scenario scores in 56, with gains concentrated at long horizons and on weaker starters. Codex-5.4 led Claude Opus 4.6 on mean test score and token efficiency, while winning edits were usually non-trivial research-style changes.

  • Overall performance: 63 of 64 sessions beat the starter on held-out test, with mean lift +0.196 and median +0.115; Codex-5.4 improved every task.The sole regression was Claude Opus 4.6 on BREAKOUT/D3PM, at -0.001 test score.
  • Overall performance: 56 of 64 sessions beat the starter on the scenario suite, with mean lift +0.170 and median +0.149.Scenario and test lifts agreed in direction on 55 of 64 sessions.
  • Agent comparison: Codex-5.4 won 19 of 32 tasks versus Claude Opus 4.6’s 13, with mean best test scores of 0.772 and 0.736, respectively.The +0.036 gap matches the head-to-head margin, though differing experiment counts make the comparison subject to a best-of-k caveat.
  • Agent comparison: Claude Opus 4.6 used 1.44× more tokens than Codex-5.4, while Codex-5.4 achieved cumulative ∆test scores of +6.71 versus +5.85.Median total-token use was 37.2M for Claude Opus 4.6 and 25.9M for Codex-5.4.
  • Edit characterization: 58 of 64 session-winning experiments (91%) were non-trivial, and seven of the eight largest per-game lifts involved structural changes rather than hyperparameter-only modifications.Non-trivial changes included new objectives, representation changes, rollout procedures, and architectural edits.
  • Horizon decomposition: At h10 and h20, mean lifts reached +0.205 and +0.215, improving 62 and 63 of 64 sessions, versus only +0.056 at h1.This horizon asymmetry indicates gains primarily in long-horizon rollout behavior rather than one-step fit.

6 Limitations

The benchmark’s findings are bounded by harness-dependent agent comparisons, horizon-weighted evaluation, and structured-state inputs. Its exact-ground-truth evaluation isolates world-modeling quality but limits conclusions about perception, planners, policies, rewards, and downstream controllers.

  • Agent-system confounds: Agent comparisons conflate model quality with harness effects, including context management, restart behavior, tool use, and experiment scheduling.Separating these effects would require swapping harnesses or evaluating both models under a shared orchestration layer.
  • Evaluation scope: Aggregate-score gains emphasize action-conditioned rollout fidelity at h10 and h20, so they should not be interpreted as uniform improvements at every horizon.One-step prediction is already strong for many starters, while most gains occur in longer open-loop rollouts.
  • Representation scope: Structured game-engine state removes perception and enables fast experiments but restricts the benchmark to transition modeling over known entities.Pixel-based settings, visual attributes, or omitted physics variables may require adapting architectures or extending the tensor representation.
  • Evaluation scope: Exact ground truth targets action-conditioned prediction, long-horizon rollout consistency, and scenario-level state evolution, separating world-model quality from downstream design choices.The evaluation does not directly assess planners, policies, rewards, or downstream controllers.

7 Conclusion … A.1 Frame Envelope Example

AutoWorldModel-Bench evaluates coding agents as autonomous world-model researchers across diverse games and fixed-budget settings, emphasizing long-horizon rollout improvement. Its structured-state schema represents frames through game metadata, actions, global state, and role-specific entity components.

  • 7 Conclusion: AutoWorldModel-Bench evaluates AI coding agents across eight games, four starter architectures, and a fixed compute budget.The benchmark is presented as a closed-loop evaluation of automated world-model research.
  • 7 Conclusion: Both Codex-5.4 and Claude Opus 4.6 reliably improve their provided starter.The supplied conclusion passage states that both agents improve their starters, without specifying a numerical success rate here.
  • 7 Conclusion: Improvement is concentrated in long-horizon rollout rather than one-step fit.This identifies the benchmark’s observed improvement pattern in the conclusion.
  • 7 Conclusion: Unlike engineering-to-spec benchmarks, AutoWorldModel-Bench does not specify the direction of improvement in advance.It instead measures whether agents can conduct open-ended world-model research, as stated in the supplied conclusion passage.
  • A Data Schema and Tensor Representation Details: The structured-state appendix specifies a frame envelope and data tensor representation for the benchmark.It provides concrete examples and full specifications for the representations introduced in Sections 3.1 and 3.2.
  • A.1 Frame Envelope Example: A SNAKE frame envelope records the game identifier, frame index, action, global state, and occupied entity slots.The example at frame 12 includes a mutable head entity in slot 0 and an immutable wall segment in slot 4.
  • A.1 Frame Envelope Example: Mutable entities are prediction targets, while immutable entities provide conditioning context.The SNAKE example distinguishes head, body, and food from the wall in this way.
  • A.1 Frame Envelope Example: Components attach optionally according to entity role, so an immobile wall need not carry Physics or Gameplay components.The frame example shows role-specific component selection rather than a fixed component set for every entity.

A.2 Data Tensor Layout … A.6 Dataset Statistics

The appendix defines a unified structured-state dataset spanning eight games, with shared registry, state, and action-tensor formats plus game-specific entity and control details. It also documents RL trajectory collection and dataset scale, totaling 152,000 episodes and over 158 million frames.

  • A.2 Data Tensor Layout: Each episode is represented by three tensors: a registry, dynamic state sequence, and action-related data.Table A.1 summarizes the common tensor layout using entity-slot budget N and episode length T.
  • A.2 Data Tensor Layout: The representation uses fixed entity slots with immutable registry features and dynamic per-frame state, illustrated by a five-slot PONG episode.PONG assigns slots to two paddles, a ball, and two immutable walls; all five slots remain alive across 200 frames.
  • A.3 Cross-Game Registry Structure: All games share a 34-dimensional registry schema, aligning structurally similar entities while preserving game-specific values such as geometry and scale.PONG and BREAKOUT balls share circle shape, positive radius, and mutability features but differ in exact radius and arena-dependent scale.
  • A.4 Per-Game Entity and Action Details: Per-game entity rosters specify kinds, mutability, and action fields, which map into a shared 7-dimensional action vector through game-specific masks.The action schema enumerates populated slots, value types, and continuous ranges or discrete class lists; unused slots are zero and masked.
  • A.4 Per-Game Entity and Action Details: Action semantics vary by game: discrete controller classes, analog jump magnitudes, and signed jump values occupy the unified action representation.Frogger uses five classes, Breakout three, Asteroids eight, Platformer jump lies in [0, 1], and Kong jump lies in [−1, +1].
  • A.5 Reward Shaping for RL Data Collection: Six games include RL-generated trajectories alongside heuristic and random data, using DQN or PPO with potential-based reward shaping during training only.Shaping magnitudes are kept 5–30× smaller than task rewards, and shaped rewards are excluded from the collected dataset.
  • A.6 Dataset Statistics: The dataset contains 152,000 episodes and over 158 million frames across eight games, with average episode lengths ranging from roughly 300 to over 1,700 frames.Each game contributes 19,000 episodes; all games use Ds=23 and registry dimensionality 34, while entity counts range from 5 to 52 and affect transformer memory.

A.7 Starter Model Architecture Details … A.11 Scenario-Based Test Suite

The appendix specifies four structured-state starter architectures, the agent’s constrained iterative research workflow, and matched session persistence. It also defines open-loop evaluation and scenario-based probes of game rules across eight environments.

  • A.7 Starter Model Architecture Details: All four starters consume W=8 historical frames and predict next-frame entity positions, alive status, terminal flag, and gameplay fields.Dreamer uses a 256-dimensional GRU-based RSSM, while AR-Transformer, D3PM, and MaskGIT use structured entity tokens and shared temporal components.
  • A.8 Agent Task Instructions: Agents optimize final evaluator score through iterative experiments, while editing only train.py, config.json, configs/*, and experiment outputs.The instructions require baseline establishment, fair comparisons, and prioritization of h10/h20 rollout fidelity over proxy metrics such as val_loss.
  • A.8 Agent Task Instructions: The scoring formula weights horizons as 0.1 * h1 + 0.2 * h10 + 0.7 * h20, with each horizon combining position_l1 and alive_f1 as 0.9 and 0.1.Thus, long-horizon rollout stability and position accuracy dominate the stated objective.
  • A.9 Session Persistence: Native vs. Externally-Managed: Both agents receive equal compute time and experiment logs, differing only in persistence: Claude Code persists natively, whereas Codex restarts through an external loop.The restart wrapper supplies prior scores and prompts Codex to propose a new hypothesis rather than repeat experiments.
  • A.10 State-Transition Contract: At h=1 inputs are ground-truth, but at h>1 predicted fields are fed back while actions and globals remain ground-truth and velocity is zeroed.This design tests whether models infer motion from position history rather than using velocity as a shortcut.
  • A.11 Scenario-Based Test Suite: Scenario tests isolate game rules with deterministic actions and randomized initial states, comparing model predictions against ground-truth outcomes across controlled replicate episodes.They are designed to distinguish causal rule learning from statistical trajectory fitting.
  • A.11 Scenario-Based Test Suite: PONG and ASTEROIDS provide the densest suites, with 15 groups and 460 episodes and 14 groups and 420 episodes, respectively.Their probes cover paddle collisions and misses, asteroid destruction and splitting, terminal conditions, cooldowns, lifetimes, wrapping, and action sensitivity.
  • A.11 Scenario-Based Test Suite: BREAKOUT, SNAKE, PLATFORMER, FROGGER, KONG, and RACER test interactions including collisions, rewards, movement, completion, survival, and action-conditioned predictions.Their suites contain 160, 160, 160, 180, 180, and 140 episodes, respectively.

B Change-Type Classifier Methodology · C Additional Results

The appendices define a deterministic, zero-shot change-type classifier that compares each experiment with its session starter and assigns one of nine labels. They also collect supporting analyses covering per-game performance, score trajectories, token usage, experiment counts, winning-experiment position, and additional session results.

  • B Change-Type Classifier Methodology: Each experiment is compared against its session starter and assigned one primary label from a nine-label taxonomy.The classifier operates per experiment using structured diffs.
  • B Change-Type Classifier Methodology: The classifier uses Gemini 3.1 Pro at temperature 0.0 with one schema-constrained call per experiment.The procedure made 1,335 calls and required fields including labels, rationales, confidence, novelty, tuning changes, and structural changes.
  • B Change-Type Classifier Methodology: Its payload combines config and training-code diffs, optional agent writeups, composite scores, and a deterministically parsed list of changed configuration keys.The changed keys are partitioned into tuning and structural categories before classification.
  • B Change-Type Classifier Methodology: The decision procedure first prioritizes the author’s stated primary intent, assigning other detected changes as secondary labels.This narrative check overrides broad diffs when the writeup names one specific change or a related set within one category.
  • B Change-Type Classifier Methodology: Tuning-only edits receive HYPERPARAM, while substantive objectives, architectures, rollouts, augmentations, inference paths, and bug fixes enter structural classification.The gate counts structural changes and their distinct categories, with MULTIPLE reserved for at least three changes across at least two categories without stated intent.
  • B Change-Type Classifier Methodology: The output records the primary label, concise rationale, secondary labels, confidence, novelty, tuning changes, and structural changes.Novelty is true for non-obvious domain-specific ideas and false for standard machine-learning levers.
  • C Additional Results: The additional-results appendix reports held-out per-game breakdowns, best-score trajectories, token usage, completed-experiment counts, winning-experiment positions, and further supporting results.These analyses are referenced from Section 5 and organized across Sections C.1–C.6.

C.1 Per-game scenario-suite breakdowns · C.2 Session progress: best test score over time · C.3 Token efficiency

The scenario-suite analysis shows that agent gains are concentrated at longer horizons, with a mean full-rollout improvement of +0.181 on 55/64 sessions, while session progress tracks only best-so-far validation scores. Claude Opus 4.6 consumes more tokens, but Codex-5.4 achieves larger cumulative improvement and approximately 1.8× greater token efficiency under the benchmark’s token-budget definition.

  • C.1 Per-game scenario-suite breakdowns: Long-horizon lift is strongest on KONG, PLATFORMER, BREAKOUT, and FROGGER, while PONG and RACER remain close and ASTEROIDS collapses past h10.ASTEROIDS nevertheless has the widest gap at h10 before both curves fall toward zero.
  • C.1 Per-game scenario-suite breakdowns: terminal_correct is excluded from the scenario composite because its advantage is localized to h1 and usually flattens or reverses past h10.The retained composite instead blends horizon terms as 0.1 · c1 + 0.2 · c10 + 0.7 · chend.
  • C.1 Per-game scenario-suite breakdowns: +0.181 mean scenario-score improvement occurs on 55/64 sessions at hend, showing gains persist beyond a truncated 20-step horizon.The scenario score uses the full curated rollout length, which varies by episode and can reach a 53-frame cap.
  • C.2 Session progress: best test score over time: Best-so-far validation curves are running maxima over experiment indices within each session, with held-out test scores computed once at session end.Figure C.13 overlays both agents across an 8 × 4 grid of games and starter architectures.
  • C.3 Token efficiency: Claude Opus 4.6 uses median 36.9M prompt tokens per session versus 25.7M for Codex-5.4, a 1.44× ratio.Opus also uses median 354k output tokens versus 203k and 33.5M cache-read tokens versus 23.4M.
  • C.3 Token efficiency: +6.71 cumulative ∆test score for Codex-5.4 exceeds Opus’s +5.85 across 32 shared tasks despite lower token consumption.Codex-5.4 consumes median 1.44× fewer total tokens per session and is approximately 1.8× more token-efficient per unit of score gain.
  • C.3 Token efficiency: The efficiency comparison measures harness-reported token budgets rather than monetary cost and conflates model differences with harness behavior.Disentangling these factors would require running Claude Opus 4.6 under the same harness conditions.

C.4 Experiment budget per session · C.5 Time-to-best within a session

Under the same 6-hour wall clock, Claude Opus 4.6 completes more experiments per session than Codex-5.4, while winning experiments typically occur late in sessions. Median winners appear at fractions 0.77 and 0.81, with roughly half of sessions placing the winner in the final 20% of attempts.

  • C.4 Experiment budget per session: C.4 Experiment budget per session: 23.5 experiments per session is Claude Opus 4.6’s mean, versus 18.2 for Codex-5.4.Both agents operate under the same 6-hour wall clock.
  • C.4 Experiment budget per session: C.4 Experiment budget per session: 25 experiments is Claude Opus 4.6’s median, versus 19.5 for Codex-5.4.The per-session distribution is summarized by a boxplot with every session shown as a jittered point.
  • C.4 Experiment budget per session: C.4 Experiment budget per session: 32 sessions per agent are represented by jittered points in Figure C.15.The box shows the interquartile range, and whiskers show the 5th and 95th percentiles.
  • C.5 Time-to-best within a session: C.5 Time-to-best within a session: 0.77 is the median winning fraction for Claude Opus 4.6, versus 0.81 for Codex-5.4.These correspond to position 18 of 25 and position 11 of 20, respectively.
  • C.5 Time-to-best within a session: C.5 Time-to-best within a session: 47% of Claude Opus 4.6 sessions place their winner in the final 20% of attempts, versus 53% for Codex-5.4.Approximately half the sessions place their winner in the final 20% of attempts.
  • C.5 Time-to-best within a session: C.5 Time-to-best within a session: winning-experiment timing is consistent with continued progress through the session and with a null of “more draws yield”.The passage frames these as two possible explanations for the observed timing pattern.

C.6 Per-experiment wall-clock distribution

Experiments were tightly constrained by a 600 s per-experiment wall-clock cap, with nearly all runs reaching the limit and only a thin early-termination tail. Across sessions, training consumed 222 GPU-hours, or roughly 58% of the nominal session budget, while the remainder supported agent activity and I/O.

  • Per-experiment wall-clock: 1,335 non-template experiments had median harness-measured wall clock 600.1 s, 95th percentile 603.1 s, and maximum 611.5 s.The sub-2 s tail overshoots reflect process-shutdown latency rather than genuine overruns.
  • Per-experiment wall-clock: 98.4% of experiments terminated via the 600 s time limit, leaving a thin lower tail of early-terminating runs.The early exits correspond to errors or maximum-step termination, while the hard cap keeps distributions tight across starters and games.
  • Session-level accounting: 222 GPU-hours of training experiments occupied roughly 58% of the 384 wall-clock-hour session budget across 64 sessions.The remaining ∼42% covered agent deliberation, tool use, code editing, and I/O between experiments.
  • Session-level accounting: Opus-4.6 averaged 23.5 experiments per session versus 18.2 for Codex-5.4.Despite running more experiments, Opus-4.6's individual experiments consumed only marginally different wall-clock time according to the supplied passage.

C.7 Starter-run reproducibility across agent sessions … C.11 Supplementary figures

Across supplementary analyses, identical starter configurations show substantial session-level variability, while agent improvements are concentrated in selective research-style changes and generally arise from nontrivial mechanisms. Additional figures document training trajectories, per-game lifts, experiment-category distributions, and the limited relationship between novelty and expected score improvement.

  • C.7 Starter-run reproducibility across agent sessions: Identical code, hyperparameters, and seed produced starter differences of up to 2,707 steps and 0.26 held-out test composite across 64 sessions.Sessions shared a 600 s wall-clock cap, despite using the same starter configuration and fixed seed 42.
  • C.7 Starter-run reproducibility across agent sessions: ∼15% per-step wall-clock variation from shared CPU, memory, PCIe, and disk resources changes how much training fits within the fixed time budget.All sessions ran on one 8-GPU node with one session per GPU and co-scheduled Docker containers.
  • C.7 Starter-run reproducibility across agent sessions: Long-horizon rollout drives the Kong/Dreamer starter gap: Codex-5.4 and Opus-4.6 differ by 0.181 at h10 and 0.315 at h20 despite nearly identical h1 composites.The reported h1 composites are 0.956/0.960, while h10 and h20 are 0.842/0.661 and 0.729/0.414, respectively.
  • C.7 Starter-run reproducibility across agent sessions: Per-agent starter baselines are therefore used for paired ∆ test-score comparisons rather than a single canonical starter.The passage identifies this as the only fair paired comparison and proposes a one-time frozen starter as a future remedy.
  • C.8 Per-experiment validation-loss trajectories by starter: Validation-loss figures overlay every agent experiment with its starter curve, exposing optimization-trajectory variation by game and starter architecture.Figures C.20–C.23 cover AR-Transformer, D3PM, Dreamer, and MaskGIT experiments.
  • C.9 Change-category selectivity: LOSS, ARCHITECTURE, ROLLOUT, and INFERENCE together account for 75% of session-best experiments, while BUGFIX and INFRA produce none.MULTIPLE is under-represented, falling from 13.8% of experiments to 6.2% of session-winners, a 0.45× ratio.
  • C.10 Case studies: the largest held-out lift per game: Seven of eight per-game winners use new objectives, representations, rollout procedures, or architectural edits; only PONG/MaskGIT uses a learning-rate and step-budget schedule.The case-study winners are selected by the largest held-out ∆test score for each game.
  • C.11 Supplementary figures: Supplementary figures show the largest lifts for ASTEROIDS and SNAKE, a maximum +0.74 lift for Codex-5.4 on SNAKE/AR-Transformer, and one −0.006 regression.Novel-idea experiments comprise 248/1,335 = 18.6%, with a larger upper tail but comparable medians to standard-lever experiments.
Loading 2608.11216v1…