Source-linked AI summary
Dreaming in Code for Curriculum Learning in Open-Ended Worlds
Konstantinos Mitsides, Maxence Faldor, Antoine Cully
TL;DR
Open-ended agents need sustained sequences of novel, learnable experiences, but existing methods often generate isolated environments rather than curricula. DiCode uses foundation models to synthesize executable environment code conditioned on competence, and in Craftax it improves mean return while enabling late-game skills. The approach is bounded by the fixed engine and incurs foundation-model inference latency.
Problem
Open-ended learning lacks reliable mechanisms for orchestrating sequences of novel, learnable experiences that sustain progress in complex worlds.
Method
DiCode conditions a foundation model on the agent’s capabilities to synthesize executable environment variations that serve as stepping stones toward a fixed target environment.
Results
16% mean-return improvement is reported, with non-zero success on late-game combat tasks where prior methods achieve 0% success.
Takeaways & Limitations
Executable environment generation provides practical curriculum control for scaffolding increasingly complex behaviors in open-ended worlds.
Takeaways & Limitations
The fixed game engine bounds invention to configurable worlds, while large-foundation-model inference introduces latency absent from simpler UED methods.
Abstract
from arXiv · showhide
Open-ended learning frames intelligence as emerging from continual interaction with an ever-expanding space of environments. While recent advances have utilized foundation models to programmatically generate diverse environments, these approaches often focus on discovering isolated behaviors rather than orchestrating sustained progression. In complex open-ended worlds, the large combinatorial space of possible challenges makes it difficult for agents to discover sequences of experiences that remain consistently learnable. To address this, we propose Dreaming in Code (DiCode), a framework in which foundation models synthesize executable environment code to scaffold learning toward increasing competence. In DiCode, "dreaming" takes the form of materializing code-level variations of the world. We instantiate DiCode in Craftax, a challenging open-ended benchmark characterized by rich mechanics and long-horizon progression. Empirically, DiCode enables agents to acquire long-horizon skills, achieving a $16\%$ improvement in mean return over the strongest baseline and non-zero success on late-game combat tasks where prior methods fail. Our results suggest that code-level environment design provides a practical mechanism for curriculum control, enabling the construction of intermediate environments that bridge competence gaps in open-ended worlds. Project page and source code are available at https://konstantinosmitsides.github.io/dreaming-in-code and https://github.com/konstantinosmitsides/dreaming-in-code.
1. Introduction
Open-ended agents often plateau because suitable experiences are scarce, while existing environment-generation methods discover isolated challenges rather than sustained curricula. DiCode addresses this by synthesizing executable environment code conditioned on competence, enabling progressive learning in Craftax.
- Open-ended agents require continual experiences that remain both novel and learnable to sustain improvement beyond early gains.
- DiCode interleaves training with generation: parent levels and competence guide new descriptions and code, which are compilation-checked and added to training.
- Existing programmatic environment-design methods generate diverse environments but do not coordinate sequences that progressively build on prior capabilities.
- DiCode uses a foundation model to synthesize executable environment logic conditioned on the agent’s current capabilities, while a fixed engine preserves valid physics and consistent mechanics.
- 16% mean-return improvement is reported for DiCode, alongside non-zero success on late-game combat tasks where prior methods achieve 0% success.
- An ablation finds that removing curriculum guidance causes the foundation model alone to fail to sustain progress.
2. Background
The background formulates RL over a distribution of level-specific partially observable decision processes and presents UED as a teacher–student curriculum game. It then describes PLR as a score- and staleness-weighted level replay method.
- 2.1. Problem Setting: The problem setting models each level as a distinct POMDP whose rewards, transitions, and initial-state distribution depend on the level.
- 2.1. Problem Setting: The agent maximizes expected discounted return over a distribution of levels, using policies that can incorporate hidden state and task goals.
- 2.2. Unsupervised Environment Design: UED frames curriculum generation as a game in which a teacher generates levels while a student maximizes standard RL return.
- 2.2. Unsupervised Environment Design: Domain Randomization is a UED instance with constant teacher utility, reducing curriculum generation to random sampling.
- 2.3. Prioritized Level Replay: PLR alternates between generating parameterized levels and replaying stored levels, retaining high-scoring levels in a fixed-size buffer.
- 2.3. Prioritized Level Replay: PLR samples stored levels using a mixture of score-based prioritization and staleness-based replay, controlled by τ.
3. Dreaming in Code
DiCode uses foundation models to synthesize executable level code that adapts curriculum environments to an agent’s evolving capabilities. Generated levels alter world generation and interaction rules, then enter a closed-loop training process with target and archived environments.
- Dreaming in Code: DiCode’s “dreaming” synthesizes executable levels conditioned on the agent’s current skill frontier, creating stepping stones toward a fixed target environment.The generated levels are trained alongside the target environment, allowing evolving skills to guide subsequent generation.
- Environment Search Space: Each generated level is executable code that defines both procedural world generation and interaction rules, producing a distinct POMDP rather than merely a fixed layout seed.The code specifies level-specific transition dynamics and a stochastic initial-state distribution.
- Environment Search Space: In Craftax, the interface can specify topology, entities, resources, inventories, starting conditions, combat formulas, and progression rules.These controls span both initial states and transition dynamics.
- Generation Cycle: Each generation cycle selects an archive parent, generates a level description and executable program, then validates the program through compilation and runtime checks.Failed code is filtered before levels enter training.
- Generation Cycle: The archive stores levels as a directed graph with parent–offspring relationships and performance profiles based on recent achievement success rates.Status categories range from A for SRλ ≥0.75 to D for SRλ <0.25.
- Training: Training batches combine target-environment trajectories, newly generated levels, and archived replay, reserving 20% of simulation budget for the target environment.New levels arrive every v = 2 iterations, while archived replay uses PLR-based learnability scores.
4. Experiments
DiCode is evaluated on Craftax against standardized RL and UED baselines, with results covering aggregate return, hierarchical achievements, curriculum dynamics, and closed-loop ablations. It improves long-horizon performance while generating progressively harder levels that remain learnable.
- Performance on Craftax: DiCode achieves a final mean return of 48.33 versus 41.54 for the strongest baseline, a relative improvement of approximately 16%.The lead is statistically significant early in training and persists throughout the training budget.
- Achievement Breakdown: DiCode reaches 45% success on Make Iron Armour versus 14% for the best baseline, and enters the Gnomish Mines in 30% versus 9% of episodes.These instrumental milestones support survival and deeper progression.
- Achievement Breakdown: DiCode attains 11% and 9% success on Defeat Gnome Warrior and Defeat Gnome Archer, while baseline performance is 0% on both tasks.It also achieves 6% versus 3% on Make Diamond Sword.
- Qualitative Analysis: Generated levels progress from generous scaffolding to added combat pressure and deep exploration objectives such as descending to floor 2.Representative levels 112, 287, and 532 illustrate this global progression.
- Qualitative Analysis: Local mutations remove resource and workstation scaffolding as competence rises, while average success remains approximately 0.5 across training levels.The reported curriculum maintains levels near the agent’s zone of proximal development.
- Importance of Closed-Loop Grounding: Removing parent-level and performance feedback reduces DiCode-OL to 40.91, approximately 15% below DiCode’s 48.33 and comparable to PPO-GTrXL’s 41.54.The ablation indicates that executable environment generation without closed-loop grounding is insufficient.
5. Related Work
Automatic curriculum learning adapts training distributions, while foundation-model environment design enables expressive programmatic generation. DiCode differs by organizing generated environments into sequences that sustain progression rather than isolated behaviors.
- Automatic Curriculum Learning algorithmically adapts the training distribution to reduce reliance on manual design in complex domains.
- Foundation models have been used to curate tasks and synthesize diverse environment programs, expanding environment design beyond fixed parameters.
- DiCode uses code-level generation to evolve high-level task semantics and progression logic, bridging strategic competence gaps rather than targeting isolated behaviors or low-level motor control.
6. Discussion and Conclusion
DiCode scaffolds complex behaviors by generating executable environments that make otherwise unreachable long-horizon skills learnable. Its scope is bounded by fixed game-engine mechanics and foundation-model inference costs, while useful stepping-stone selection remains an open challenge.
- DiCode constructs intermediate training worlds that make otherwise unreachable behaviors learnable in open-ended environments.
- Fixed game engines preserve physical validity but bound invention to configuring existing mechanics rather than creating new physical laws.
- Large foundation-model inference introduces latency compared with simpler UED methods, although faster inference and more efficient models may narrow the gap.
- Foundation models can guide exploration by generating and sequencing environments according to agents’ learning signals, reducing the effective search space.
- Reliably distinguishing useful stepping-stone environments from uninformative or distracting ones remains an open challenge.
Impact Statement
The work contributes a reproducible framework for automatic curriculum generation in open-ended reinforcement learning and supports transparent research using open-weight models. Its broader impacts are indirect and remain limited to simulated environments.
- DiCode advances automatic curriculum generation by enabling agents to learn from environments synthesized as executable code.
- The framework provides a scientific basis for studying how learning progress can be sustained where standard training fails.
- Using open-weight models and reproducible tools supports transparent and accessible research.
- Potential applications include reducing manual task design and extending simulation-based training to robotics or game AI, but the work does not involve real-world deployment.
- More capable open-ended agents raise concerns about unintended behaviors, requiring continued attention to evaluation, safety, and alignment as methods scale.
A.1. MiniCraftax API Interface
The MiniCraftax API standardizes generated tasks through a parameter dataclass and an abstract base-class contract. Generated tasks define mechanics, world construction, termination, and success conditions within the simulator interface.
- The standardized interface requires generated classes to inherit from BaseTask and define task parameters and a world-generation function.
- TaskParams stores mechanics that the language model can modify to adjust game dynamics.
- BaseTask provides shared task state, including static parameters, tunable parameters, relevant achievements, completed achievements, and a descriptive label.
- Generated worlds must return a valid EnvState, while episodes terminate on time limits, player death, or completion of all relevant achievements.
- Task success is a binary condition determined by whether all task-relevant achievements are completed.
A.2. Seed Tasks
Seed tasks bootstrap the curriculum around survival, combat, crafting, and resource gathering. Their generated worlds configure starting inventories, nearby resources or mobs, mechanics parameters, and achievement-based objectives.
- Four predefined seed tasks cover Craftax’s fundamental mechanics: survival, combat, crafting, and resource gathering.
- Resource gathering: The coal-collection task gives the player a wooden pickaxe and sword, places five coal blocks nearby, and requires the COLLECT_COAL achievement.
- Combat: The zombie-defeat task starts the player with a wooden sword, places one zombie and three cows nearby, and targets DEFEAT_ZOMBIE.
- Crafting: The wooden-pickaxe task requires collecting wood, placing a table, and crafting the pickaxe, while providing a sword and nearby cows.
- Survival: The survival task begins with an empty inventory and three nearby cows, requiring EAT_COW, COLLECT_DRINK, and WAKE_UP achievements.
A.3. Hyperparameters
The appendix reports shared and method-specific hyperparameters for the experiments, including the common PPO-GTrXL solver and distinct settings for SFL, PLR/DR, DiCode, and the foundation model.
- All methods share the underlying PPO-GTrXL architecture and parameters, isolating curriculum differences from the optimization process.
- SFL has a separate set of method-specific hyperparameters documented independently from the shared experiment configuration.
- DR disables replay prioritization by setting the replay probability to 0.0 while using the same buffer infrastructure as PLR.
- DiCode uses method-specific worker-distribution settings that vary when newly generated environments join replayed and target environments in training.
- The foundation-model configuration uses Qwen3-235B through the Hugging Face API with standard text-generation sampling parameters.
A.4. Infrastructure and Computational Cost
DiCode’s training cost is substantially higher than baselines because foundation-model inference dominates runtime. Its curriculum design also emphasizes incremental, targeted tasks and asynchronous generation to manage computational and learning demands.
- Computational cost: ∼48 hours: DiCode’s approximate training time for 2 billion timesteps, versus approximately 10.5 hours for PPO-GTrXL, DR, and PLR.SFL is slightly faster at ∼8.5 hours.
- Computational cost: API latency, rather than inherent algorithmic complexity, dominates DiCode’s longer training duration.The foundation model runs on separate, lower-priority infrastructure with demand-dependent response times.
- Curriculum design: Generated tasks may be discarded after brief training if they underperform, motivating focused tasks instead of bundled, fragile requirements.The prompt explicitly links excessive difficulty or requirement bundling to likely task failure and discard.
- Curriculum design: DiCode’s task-generation guidance favors small, incremental changes that apply focused pressure to learnable bottleneck capabilities.The stated objective is to improve downstream competence on the original Craftax environment rather than optimize local task success alone.
B.3. Open-Loop Ablation (DiCode-OL)
The open-loop ablation removes DiCode’s feedback loop, generating tasks without parent-level or agent-performance information. Its task prompt instead specifies targeted, incremental capability training with explicit scaffolding and achievement conditions.
- Open-loop design: The Open-Loop ablation generates tasks from the static environment description without access to the parent level or the agent’s performance profile.This removes the curriculum feedback signals used by the full method.
- Scaffolding: Initial states compress already-mastered prerequisites so training focuses on the new target capability and avoids unnecessary backtracking.Later-context tasks should provide consistent inventory, tools, and resources while marking prerequisites completed.
- Task targeting: The prompt favors thin-slice tasks with one primary bottleneck capability and at most one supporting sub-skill.It also distinguishes relevant achievements from completed achievements satisfied by the initial state.
- Task targeting: Open-loop task generation targets a specific Craftax mechanic, transition, or survival loop and isolates it for practice.The prompt frames targeted capability sampling as a way to select a meaningful gameplay slice.
- Task specification: Task descriptions must specify implementation-relevant details such as coordinates, quantities, block types, and mob identifiers.This requirement is intended to make generated tasks directly implementable without guessing.
C.1. Curriculum Case Studies
The curriculum case studies illustrate progressive scaffolding from simplified resource and armor tasks toward realistic survival and combat transitions. Later tasks remove guarantees, add natural resource acquisition, and increase combat requirements.
- Progression: Four representative generated levels illustrate DiCode’s progression from basic survival to deep exploration.The examples are presented as Python-class docstrings describing the generated curriculum.
- Simplified scaffolding: A simplified level pre-equips tools and resources, places workstations and iron, and pre-unlocks descent to focus on armor crafting.Its mechanics reduce melee spawning while retaining moderate environmental structure.
- Resource acquisition: A more realistic task requires natural collection of coal and iron, workstation placement, iron-armor crafting, and descent to Floor 1.It starts with wood, stone, and a wood pickaxe, without guaranteed deposits or pre-placed workstations.
- Task specification: The case studies distinguish relevant achievements from completed achievements so initial inventories and prerequisites are not redundantly retrained.Task definitions explicitly list both categories alongside world and mechanic settings.
- Survival progression: A survival-oriented task requires iron-armor crafting followed by eight hostile-mob kills to unlock descent to the dungeon.Its design increases melee and ranged spawn multipliers and teaches the sequence of gathering, crafting, clearing, and descending.
D.1. Full Achievement Breakdown
The achievement breakdown evaluates performance across Craftax’s achievement hierarchy using aggregate success rates and learning curves. Results are summarized across five seeds after 2 × 10^9 steps with standard-error reporting.
- Final success rates: Aggregate success rates compare DiCode with baselines across all defined Craftax achievements.Figure 5 reports the final achievement success-rate comparison.
- Evaluation protocol: 2 × 10^9 steps: results report means and standard errors across 5 random seeds.The learning-curve plots use shaded regions to denote standard error across the five seeds.
- Learning curves: Achievement learning curves show time-series performance for individual achievements and their acquisition rates.Figure 6 focuses on how performance develops over time rather than only final success.
D.2. Ablation Analysis
The ablation compares DiCode’s closed-loop curriculum with an open-loop variant. Their performance gap underscores the role of conditioning generation on the agent’s capabilities and parent levels.
- The evaluation reports final mean return together with standard error.Table 7 summarizes final performance using mean return and standard error.
- The ablation compares DiCode (Closed-Loop) against DiCode-OL (Open-Loop).This comparison is used to validate the importance of the closed-loop curriculum.
- The performance gap highlights the necessity of conditioning generation on the agent’s current capabilities and parent levels to sustain progress.