PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentGarten: Code Worlds for Evolving Agents

Jiawei Chi, Shangchen Miao, Zhiyuan Shi, Kailu Wu, Hanyang Wang, Weiliang Chen, Qiyu Dai, Jinshan Ren, Jun Gao, Mingsheng Long, Yueqi Duan, Jiangran Lyu, Jialong Wu, Fangfu Liu

arXiv:2610.12374v1cs.CV

TL;DR

Interactive learning needs environments with faithful state and rules as well as realistic, diverse observations, but existing approaches make it difficult to achieve both. AgentGarten combines programmable engines with a shared neural renderer and trains agents to refine inherited playbooks through interaction. Agents improve across hide-and-seek and four further worlds, with shelters appearing by round 4 and ramp crossings by round 10 in hide-and-seek.

  • Problem

    Existing simulators and game engines provide explicit state and rules but make visual diversity costly, while video world models provide rich observations with implicit state and transition rules.

    Source: §1 · p. 2

  • Method

    AgentGarten pairs engines that maintain persistent state and execute explicit rules with a shared real-time neural renderer that generates observations from structured conditions.

    Source: §1 · p. 2

  • Results

    In hide-and-seek, shelters appeared by round 4 and ramp crossings by round 10, and the same reflective procedure improved outcomes in four further worlds.

    Source: §8 · p. 17

  • Takeaways & Limitations

    Compatible engines can share a renderer, while environments can be authored and extended in code without bespoke visual assets for every scene.

    Source: §1 · p. 2

    A geometry-only renderer cannot distinguish a spinning wheel from a still one, and the reported worlds were built one at a time.

    Source: §7 · p. 16

Abstract

from arXiv · show

Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.

1 Introduction

AgentGarten addresses the trade-off between programmable environments with costly visual asset pipelines and visually rich video world models with implicit state and rules. It combines persistent, programmable engines with a shared real-time neural renderer, then studies agents that refine playbooks across rounds of interaction.

1 Introduction

  • The framework targets a trade-off: simulators and game engines provide explicit state and rules but make visual diversity costly, while video world models offer rich observations with implicit state and transitions.The latter makes environment behavior difficult to inspect, edit, and test directly.Source: §1 · p. 2
  • AgentGarten couples persistent, programmable engine state and interaction rules to a shared real-time neural renderer that generates visual observations from structured conditions.Compatible engines share a renderer, and scenes can be authored or extended in code without bespoke visual assets for every scene.Source: §1 · p. 2, Figure 2 · p. 3
  • Adversarial Forcing distills a pretrained video model into a block-causal renderer, using exact replay for history gradients and real-data adversarial supervision for visual quality.Exact replay enables bitwise-identical rollout recomputation; the adversarial objective addresses visual degradation over long rollouts.Source: §1 · p. 2
  • Agents perceive rendered observations, review failures, and pass written playbooks to later rounds; this loop improves outcomes across hide-and-seek and several other environments.The introduction reports ramp use and shelter construction emerging within a handful of hide-and-seek games.Source: §1 · p. 2

2 Code Worlds

Code worlds separate persistent scene state and interaction rules from visual rendering: engines export structured conditions, and the renderer combines them with appearance references and visual history. Coding agents can construct scenes from images or text, then refine them through rollout and engine checks.

2.1 Formulation

  • A code world pairs a scene program and executing engine with a renderer: actions update persistent state and yield structured conditions, while observations are generated from those conditions and visual history.Because rendered observations do not feed the state transition, rendering errors cannot accumulate in world state; recorded trajectories can be rendered with a different appearance reference or camera.Source: §2.1 · p. 3

2.2 Structured conditions

  • Colorized depth and surface normals provide three-channel structured conditions that can come from estimated video or directly from simulated scene geometry.Simulated scenes supply the maps without detailed materials or textures, and training samples randomly choose one modality.Source: §2.2 · p. 3

2.3 Building code worlds

  • Coding agents build executable scene programs from images or text, using an appearance reference and coarse geometry rather than detailed materials or leaf-level assets.For image inputs, agents fit extracted assets to estimated geometry and extend unobserved regions; for both modalities, they inspect probe-camera rollouts and engine checks, then repair faults before deployment.Source: §2.3 · pp. 3–4, Figure 3 · p. 4

3 A Real-Time Neural Renderer

The renderer adapts a pretrained video model for geometry-conditioned, blockwise autoregressive generation. It is distilled through Adversarial Forcing and served with bounded caching and streaming decoding.

3.1 Backbone

  • A frozen pretrained video backbone separates caption understanding from generation, while a frozen video autoencoder maps RGB and geometry videos to latents.The reference uses the backbone’s clean-frame representation and is neither noised nor denoised.Source: §3.1 · p. 4

3.2 Geometry conditioning

  • Geometry tokens share Transformer layers and temporal coordinates with RGB tokens, while condition dropout and perturbations reduce reliance on estimator-specific cues.Depth and normals are encoded as conditions; fine texture is suppressed, alignment is warped, and condition latents are noised.Source: §3.2 · p. 5

3.3 Teacher-forcing adaptation

  • Fixed-length target blocks are generated from aligned geometry and completed earlier blocks, without access to future blocks.Parallel teacher forcing uses clean copies for history and noisy copies for denoising; each noisy copy cannot see its own clean target.Source: §3.3 · p. 5, Figure 4 · p. 6

3.4.1 Distribution matching

  • Distribution matching trains few-step student rollouts against a frozen bidirectional teacher, with a fake-score model learning the student’s distribution.The student conditions each generated block on its own preceding outputs.Source: §3.4 · p. 5, §3.4.1 · p. 6

3.4.2 Exact replay

  • Exact replay recomputes history encodings with gradients, allowing later DMD and adversarial losses to update history-writing parameters without differentiating through sampling.The rollout records block inputs and clean outputs without gradients; replay uses block-by-block calls to preserve execution structure and numerical agreement.Source: §3.4.2 · p. 6, Figure 5 · p. 7

3.4.3 Adversarial training

  • A real-data adversarial objective supplements distribution matching, and exact R1/R2 regularization computes head-parameter gradients without double-backward through fused attention.The discriminator uses a trainable head over a frozen teacher backbone; the head’s directional derivative is computed with ordinary differentiable operations.Source: §3.4.3 · pp. 7–8

3.5 Streaming inference

  • Streaming inference keeps a permanent sink prefix and a sliding window of recent frames in a bounded cache, evicting older frames as needed.The current block attends to cached frames and text, then joins the recent window after denoising.Source: §3.5 · p. 8, Figure 7 · p. 8
  • Custom Triton kernels fuse operations around attention, CUDA graph replay reduces launch overhead, and an optional tiny decoder reconstructs pixel frames in under 10 ms.Decoded frames are streamed to agents or browsers over WebRTC with bounded queuing latency.Source: §3.5 · p. 8, Table 2 · p. 9

4 Experiments

The experiments assess rollout appearance, replay accuracy, and inference speed. Adversarial Forcing preserves visual detail in long rollouts, while blockwise replay is bitwise exact and rendering exceeds 35 frames per second on one H100 GPU.

4.1 Self Forcing vs. Adversarial Forcing

  • Adversarial Forcing retains natural textures and fine detail throughout 30-second rollouts, unlike Self Forcing, whose rollouts develop repetitive patterns and lose scene detail.Figure 8 compares both renderers on the same condition trajectories.Source: §4.1 · p. 9

4.2 Replay accuracy

  • Block-by-block SDPA replay is bitwise identical to the cached rollout; full-sequence FlexAttention replay has 3.99% relative L2 error.Forward time decreases by 5.4%, while total peak memory changes by 0.2%.Source: §4.2 · p. 9

4.3 Inference throughput

  • The renderer generates 480 × 832 video at over 35 frames per second on one NVIDIA H100 GPU, including condition encoding, four denoising steps, decoding, and host transfer.Measurements use BF16 computation and the four-step sampler.Source: §4.3 · p. 9, §4 · p. 9

5 Agents in Code Worlds

AgentGarten combines varied, replayable code worlds with first-person agent practice. Agents refine tool-use strategies through playbooks, and the round-based procedure is also tested in four further worlds.

5.1 A library of environments

  • Different scene programs and compatible engines form code worlds with a shared renderer, and recorded rollouts can be re-rendered from another camera or visual style without changing events.Examples span navigation, manipulation, tool use, and multi-agent interaction.Source: §5.1 · p. 11, Figure 9 · p. 12

5.2 Hide-and-seek, revisited

  • In hide-and-seek, agents act through short Python programs using rendered first-person views, while each role reviews failures and successes and adds skills to its own playbook.The run begins with empty playbooks and uses several rounds of five games over sampled layouts and seeds.Source: §5.2 · p. 11
  • Agents built cover with a barrier, carried a ramp over a wall, and repositioned it after a failed leap, refining multi-step tool-use strategies from rendered first-person observations.Figure 10 shows the three characteristic behaviors.Source: §5.2 · p. 11, Figure 10 · p. 13
  • Hiders built shelters by round 4 and seekers crossed walls with ramps by round 10, versus self-play RL milestones at roughly 25 million and 100 million episodes; these are different learning paradigms, not a direct sample-efficiency ratio.The pretrained agents ground prior knowledge into closed-loop action, while the RL agents learned from scratch with privileged object state.Source: §5.2 · p. 11

5.3 Rounds of practice

  • Practice runs from task files that state goals, actions, and limits but no solutions; agents play within a fixed budget, then archive playbooks for fresh-conversation agents in later rounds.Agents see only camera frames during practice, without coordinates, a map, or scores until the episode ends.Source: §5.3 · p. 13, Figure 11 · p. 14

5.4 Beyond hide-and-seek

  • Across four worlds, companion-dog engagement rose from 13 to 19, bridge time fell from 71 to 41 seconds, herding reached four sheep after round 1, and quarry round 4 finished in 329 of 360 seconds.The loader scored nothing in round 2 after clearing rocks in round 1; in round 4 it cleared rocks, delivered one, and parked.Source: §5.4 · pp. 13–14

6 Related Work

Related work spans executable environments, video world models, and agents that retain written experience. AgentGarten combines explicit world evolution with learned rendering and applies experience-sharing agents to camera-based worlds.

Executable environments and code worlds

  • Executable environments provide inspectable, revisable representations, but their observations inherit the fidelity limits of conventional graphics assets.Prior work includes simulated benchmarks, procedural environments, and code-based environment construction and dynamics recovery.Source: §6 · p. 14

Explicit state behind a video model

  • AgentGarten separates world evolution from appearance, using depth or surface normals, short-block causal rendering, and complete executable environments for agents.Unlike related systems, agents act in the executable environment and improve over repeated rounds.Source: §6 · p. 15

Interactive video world models

  • Video world models support interactive visual environments, but represent the world through generated histories and learned memory rather than an executable environment.The renderer remains responsible for appearance, and generated frames need not perfectly realize supplied state.Source: §6 · p. 15

Autoregressive video generation and distillation

  • Autoregressive distillation methods train students on their own rollouts; related work adds real-data GAN losses and differentiable history passes.Adversarial Forcing uses exact replay to avoid numerical divergence and match rollouts bitwise.Source: §6 · pp. 15–16

Agents that learn from written experience

  • Written-experience agents retain reflections, reusable insights, or executable skills; AgentGarten passes playbooks between independent agents in fresh conversations.In hide-and-seek, agents use camera frames and receive no solution in the task file.Source: §6 · p. 16

Gradient penalties and higher-order derivatives

  • Exact gradient penalties matter because discriminator parameter gradients require differentiating an input gradient, unsupported by fused attention kernels such as FlashAttention.The cited work combines forward-over-reverse products and fused attention JVP kernels for an exact penalty with a frozen discriminator backbone.Source: §6 · p. 16

7 Future Work

Future work targets more complete rendering conditions, scalable creation and evaluation of worlds, and ways to manage experience across many worlds.

More compact and complete conditions

  • Depth and normals omit attributes such as material, identity, and geometry-invariant rotation; more compact, complete conditions remain an open problem.Structured text or latent features could describe state that geometry cannot express, but aligning them with code-world state and adapting video models remain open problems.Source: §7 · p. 16

Scaling environments

  • Geometry-only rendering excludes tasks that depend on hidden state such as wheel rotation, and worlds in this report were built one at a time.Future work proposes coding agents to write and revise worlds and automatic checks for observability, meaningful failure, and transferable learning.Source: §7 · p. 16

Scaling experience

  • Across many worlds, agents must select, merge, or discard lessons and retrieve those relevant to the current scene.Code worlds can be reset and replayed exactly to test lessons before passing them on; lessons shared across worlds could become general knowledge about physical scenes.Source: §7 · pp. 16–17

8 Conclusion

AgentGarten combines persistent executable world state and rules with a shared real-time neural renderer, then has agents distill experience into playbooks. Agents improved across rounds in hide-and-seek and four further worlds.

8 Conclusion

  • A simulator or game engine maintains persistent state and executes world rules, while a shared neural renderer turns exported geometry into real-time observations.This division is presented as supporting worlds that are both faithful and realistic.Source: §8 · p. 17
  • Shelters appeared by round 4 and ramp crossings by round 10 in hide-and-seek; the same procedure improved outcomes in four further worlds.Agents acted through rendered observations and later agents inherited and refined written playbooks.Source: §8 · p. 17
  • Because new worlds can be written as code and rendered through the same interface, environments can grow in number and difficulty alongside agents.The conclusion anticipates agents continuing to evolve through interaction.Source: §8 · p. 17

A.1 Backbone, data, and training stages

The appendix describes the renderer backbone, training data, and staged training setup, including compute configuration and frozen model components.

A.1 Backbone, data, and training stages

  • 390 latent-frame tokens are produced by a 2 × 2 patch embedding at 480 × 832; the renderer has 36 Transformer layers, width 4096, 32 query heads, and 8 key–value heads.Text encoding and video encoding are computed online, while the text stream and video autoencoder stay frozen.Source: §A.1 · p. 21
  • Training clips combine scene captures, driving videos, internet videos of gameplay, navigation, and robot interaction, plus rendered synthetic scenes.Synthetic-scene conditions are exported by the renderer rather than estimated.Source: §A.1 · p. 21
  • All stages use 32 NVIDIA H100 GPUs, one clip per GPU, and two accumulation steps, for 64 clips per optimizer update.Computation uses BF16 with FP32 gradient reductions; optimizers are AdamW with zero weight decay.Source: §A.1 · p. 21
  • Stages 1 and 2 train on five-second then fifteen-second windows; each distillation run uses a teacher and fake score matching its window length.The stage-1 teacher stays frozen, as do the text stream and video autoencoder.Source: Table 5 · p. 21

A.2 Geometry preprocessing

Geometry preprocessing normalizes depth and normals, then reduces and augments geometry conditions before rendering. The procedures include depth-scale handling, normal canonicalization, and randomized condition perturbations.

A.2 Geometry preprocessing

  • Metric depth uses α=1; unknown-scale depth is scaled using the clip’s median valid depth to place that median at u=0.5.Training can jitter the scale for non-metric clips.Source: §A.2 · p. 21
  • Normals are encoded from [−1, 1] for RGB display, and estimated normals are canonicalized by negating the x component of the NormalCrafter prediction.Rendered normals are converted to the same camera-frame orientation.Source: §A.2 · p. 21
  • Geometry conditions are pooled to 28 tokens per latent frame at 480 × 832 resolution, then augmented with texture suppression, spatial warps, modality dropout, and latent noise.Texture suppression and spatial augmentation each occur with probability 0.5; both depth and normals are dropped with probability 0.1.Source: §A.2 · pp. 21–22

A.3 Causal blocks and history perturbation

The renderer uses clean references, blockwise history perturbations, and distillation with replay so later prediction losses can train history encoding. The appendix also details discriminator-gradient computation and bounded streaming-cache behavior.

A.3 Causal blocks and history perturbation

  • Each target block contains four latent frames; visible history frames receive one of four perturbations, while the clean reference remains unperturbed.The reference is a standalone clean prefix, excluded from diffusion timesteps and loss.Source: §A.3 · p. 22

A.4 Distillation

  • Distillation recomputes the final rollout step and uses a DMD loss with a second-pass prediction; the adversarial gradient also acts through replay.The teacher uses guidance scale 6, and the fake-score model takes five updates per student update.Source: §A.4 · p. 22
  • Exact replay recomputes clean-history keys and values in the differentiable graph, matching rollout attention and projection grouping rather than using detached cache values.The grouping preserves the same context, key–value order, and tensor shapes as the cached rollout.Source: §A.4 · p. 22

B Exact R1/R2

  • The R1/R2 construction relies on a frozen backbone so its features and Jacobian are independent of discriminator-head parameters; updating it would require derivatives the detached construction omits.Freezing does not remove the head’s own second-order derivatives.Source: B Exact R1/R2 · pp. 22–23
  • JVPs propagate attention tangents through the forward computation, avoiding differentiation of a FlashAttention backward kernel; the surrogate matches the penalty and its first head-parameter derivative.The surrogate is not a replacement graph for arbitrary higher-order derivatives.Source: B Exact R1/R2 · p. 23

C Streaming Cache

  • The served streaming cache retains five sink and 44 recent latent frames, while rotary positions remap recent history and preserve the sink prefix’s original positions.RGB and condition keys and values are evicted together.Source: C Streaming Cache · p. 23

D Task Files

The task files specify goals, available actions, and limits for worlds in which agents act from rendered observations. They cover companion-dog engagement, bridge cooperation, herding, and quarry work.

D Task Files

  • In the companion-dog task, the agent has 60 seconds to keep one dog willingly engaged using actions such as offering, petting, playing, and withdrawing.Small steps toward the agent are a positive engagement cue; turning aside or stepping away indicates refusal or discomfort.Source: D Task Files · p. 24
  • Task files advise agents to record tested cues, failures, and uncertainties in playbooks, while treating inherited claims as unverified rather than ground truth.The companion-dog task also cautions that it is a synthetic game, not real animal-care advice.Source: D Task Files · p. 24
  • Bridge agents must simultaneously exchange ends of a one-car-wide bridge within 120 simulation seconds, coordinating through first-person views and fixed-duration actions.Cars can pass only in bank pullouts; both moves are blocked if either would collide with or cross the other car.Source: D Task Files · pp. 24–25
  • In herding, two dogs guide four sheep into a fenced pen and keep them there for five continuous seconds, using first-person views without numeric coordinates or the other dog’s action.The sheep can leave the pen again, and the terminal score reflects the fraction currently inside, with a time bonus after the hold.Source: D Task Files · p. 25
  • The quarry task requires a wheel loader to stage two rocks, deliver one to a bunker, and park within 360 simulation seconds without resetting.Rock movement depends on contact, friction, and gravity; starting poses and the opening side vary across seeds.Source: D Task Files · p. 26

Related papers

Loading 2610.12374v1…