PaperlayerView exact revision on arXiv

Source-linked AI summary

SPADE: Self-Play in Adaptive Synthetic Executable Environments

Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques

arXiv:2608.19197v1cs.CLcs.AI

TL;DR

Fixed training environments do not expand with increasingly capable language agents, limiting the supply of adaptive, verifiable goals. SPADE makes environment design learnable through self-play, improving performance over fixed-environment baselines across games and tool-use settings, including +13.9 on ACEBench-Agent at 30B-A3B.

  • Problem

    Existing hand-curated, synthetic, and frozen-verifier environment pools keep goal distributions fixed, while generators and ungrounded self-play remain limited in breadth and adaptability.

  • Method

    SPADE trains an Environment Designer and Reasoning Agent through self-play, using corpus grounding, environment memory, and hint-based regret to generate executable adaptive environments.

  • Results

    Across games and tool-use settings, SPADE outperforms fixed-environment baselines, including +13.9 on ACEBench-Agent at 30B-A3B.

  • Takeaways & Limitations

    SPADE shows that a single model can design training environments and improve from them, moving beyond fixed benchmarks toward open-ended continual self-improvement.

  • Takeaways & Limitations

    Environment complexity remains bounded by the designer’s base model and generation budget, while fixed-task benchmarks do not formally establish open-ended reasoning growth.

Abstract

from arXiv · show

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

1 Introduction

SPADE co-evolves an executable environment designer and reasoning agent through self-play, addressing fixed or human-limited training pools with adaptive, corpus-grounded environments. It improves games and tool-use performance at 30B+ scale while supporting continual self-improvement.

  • Motivation: Existing approaches remain limited because harnesses do not update model weights, human-curated pools scale with authoring capacity, and frozen synthetic generators do not expand with the agent.Ungrounded self-play is likewise bounded by the model’s own prior knowledge.
  • Framework: SPADE uses one LLM as an Environment Designer that writes complete executable Python environments and a Reasoning Agent that learns by acting in them.The roles co-evolve: improving agents receive increasingly capable environments, making environment design a learnable component of post-training.
  • Framework: Each environment exposes a Gym-style reset()/step() interface and represents a complete MDP, unifying single-turn reasoning with multi-turn agentic interaction.Because computable MDPs can be expressed as programs, SPADE is not restricted to a hand-designed environment parameterization.
  • Training signal: A hint-based regret reward trains the Environment Designer on the performance gap between the Reasoning Agent with and without privileged hints.This targets solvable environments near the agent’s capability frontier, avoiding both unsolvable adversarial tasks and reward-inflating cooperation.
  • Design pipeline: SPADE grounds environment design in pretraining-corpus knowledge and accumulated experience, spanning cognitive-skill games and tool-use tasks rather than a single domain.The design also includes environment validation, reward-hacking avoidance, and curriculum design for scaling beyond small models.
  • Results: +5.3 points on average over the strongest fixed-environment baseline at 30B-A3B, with up to +7.5 on individual games benchmarks; tool-use gains reach +5.7 on BFCL v4 multi-turn and +13.9 on ACEBench-Agent.Games gains hold across all three backbones, while BFCL v4 also improves by +10.3 at 4B.

2 Related Work

Prior work spans self-play for LLMs, unsupervised environment design, and synthetic environments for agentic RL, but commonly depends on curated or fixed environment-generation processes. SPADE instead jointly trains an Environment Designer and Reasoning Agent online, using hint-based regret to co-evolve executable multi-turn environments with the agent’s capabilities.

  • Self-Play for LLMs: Earlier LLM self-play methods rely on curated seed data or evaluation sets, whereas a newer line seeks to remove that dependency while training proposers and solvers from minimal seed.The cited approaches include self-distillation (Chen et al., 2024; Yuan et al., 2024; Singh et al., 2023), adversarial language games (Cheng et al., 2024), and learned-proposer bootstrapping (Fang et al., 2025b; Sundaram et al., 2026).
  • Unsupervised Environment Design and Open-Endedness: Unsupervised environment design builds on curriculum learning and learnability-driven adaptation, with POET (Wang et al., 2019) introducing paired environment-agent co-evolution and PAIRED (Dennis et al., 2020) introducing adversarial design with minimax regret.These foundations motivate adaptive environment generation beyond static task pools.
  • Environment Synthesis and Scaling: Recent environment-synthesis systems address the agentic-RL environment-pool bottleneck by generating worlds and toolsets, but their generators are typically frozen, hand-engineered, or trained on separate signals.This work follows the shift from single-turn RL with verifiable rewards to multi-turn agentic RL.
  • Environment Synthesis and Scaling: SPADE jointly trains an Environment Designer and Reasoning Agent online with hint-based regret, producing executable Python multi-turn MDP environments whose distribution co-evolves with the agent’s capability frontier.This contrasts with generators that are typically frozen, hand-engineered, or trained on a separate signal.

3 Preliminaries

SPADE represents training environments as Gym-style Markov decision processes and trains its LLM policy with verifiable rewards using GRPO. The environment interface exposes reset() for initialization and step(a) for state transitions and rewards.

  • Markov decision processes: SPADE models environments as MDPs with state, action, transition, reward, and initial-state components exposed through Gym-style reset()/step() functions.reset() samples an initial observation, while step(a) advances the environment and returns the resulting interaction outputs.
  • Gym-style environment example: A generated Wordle-style environment illustrates stateful multi-turn execution, with termination on a correct guess or truncation after six turns and rewards of 1 or 0, respectively.Each guess produces per-letter feedback to guide deduction, while the environment tracks the hidden target and remaining turns.
  • Reinforcement learning from verifiable rewards: The policy πθ is trained with reinforcement learning from verifiable rewards using Group Relative Policy Optimization (GRPO) (Shao et al., 2024).For each prompt, GRPO samples a group of G responses and computes group-normalized advantages before applying a clipped policy-gradient update with KL regularization.

4 SPADE: Self-Play in Adaptive Synthetic Executable Environments

SPADE uses one LLM in alternating Environment Designer and Reasoning Agent roles to generate executable environments and learn within them. Hint-based regret, corpus grounding, and environment memory make the training curriculum adaptive and diverse.

  • Framework: SPADE alternates one LLM between designing executable Gym-style environments and solving them as a Reasoning Agent, unifying single-turn reasoning with multi-turn tool use.Generated environments implement reset()/step() and are validated for syntax and executability before training.
  • Adaptive curriculum: Hint-based regret steers the Environment Designer toward environments at the Reasoning Agent’s learning frontier by contrasting hinted and unhinted performance.Paired trajectories show hints exposing task patterns or narrowing searches that otherwise require repeated failed attempts.
  • Optimization: The framework stabilizes joint two-role training by normalizing role-specific advantages, centering rewards, and upweighting less frequent Environment Designer trajectories.These techniques help the coupled objectives contribute comparably during optimization.
  • Results: Training on diverse synthetic games improves held-out competition-math, science, code-generation, and procedural-reasoning benchmarks across backbone scales.Table 1 compares SPADE with retrained fixed-environment baselines using eight held-out benchmarks spanning four capability families.
  • Curriculum sources: Corpus grounding supplies novelty while cross-episode environment memory provides regret-scored, skill-tagged seeds that help avoid repetition and environments that are too easy or too hard.The memory lets each round begin from tasks the Reasoning Agent currently finds difficult rather than starting from scratch.

5 Experimental Setup

The experiments train SPADE across three Qwen3 backbones and evaluate executable game and multi-turn tool-use environments against fixed-environment or synthetic-environment baselines. All benchmarks are held out from training, with environments grounded in large math, science, or code corpora and regenerated at task-specific intervals.

  • Training configuration: SPADE trains Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507, using GRPO for 400 rollouts of 24 environments with unchanged stabilization procedures.The 30B model is the primary model; the 8B run enables thinking for both roles, while the 4B and 30B runs use instruct models.
  • Environments: The game setting uses self-contained Python environments with verifiable rewards, rotating three of six cognitive-skill categories across 24 games, grounded in a 15k-document math-and-science corpus with k=4.Games must pass syntax and execution checks, and the Reasoning Agent win rate is kept within the target band before entering the pool.
  • Baselines and evaluation: Fixed-environment comparisons retrain RLVE and static GPT-5.5-generated GRPO baselines from each backbone for 400 iterations, with RLVE stronger throughout Section 6.Evaluation covers hard procedural reasoning and out-of-distribution AIME 2025/2026, GPQA-Diamond, and LiveCodeBench-v6, with all eight benchmarks reported for every backbone.
  • Environments: The tool-use setting generates simulated function-calling tools, mutable backend state, and sequential natural-language instructions with state checks, grounded in a 15k-document code corpus and regenerated with k=8.The Reasoning Agent completes these environments through multi-turn tool calls.
  • Baselines and evaluation: Tool-use evaluation compares SPADE with AgentScaler, Agent-World, Agent World Model, and EnvScaler on BFCL v4 multi-turn, τ 2-bench, and ACEBench-Agent.The cited reference results may differ in training data, budget, base model, and evaluation protocol.

6 Experimental Results

SPADE improves held-out performance across synthetic game and tool-use environment design, with gains increasing at larger scales and transferring beyond the generated tasks. Training also produces increasingly learnable, diverse environments and shifts the Reasoning Agent toward evidence-driven interaction.

  • 6.1 Held-out benchmark performance: +5.3 suite-average improvement over Fixed-env RLVE at 30B-A3B reaches 58.3, with gains transferring to science, code, and procedural reasoning while preserving competition math.SPADE trains only on synthetic games, yet the gains hold late in the 400-step run.
  • 6.1 Held-out benchmark performance: +13.9 on ACEBench-Agent is the largest 30B-A3B tool-use gain, followed by +5.7 on BFCL v4 multi-turn and +3.6 on τ 2-bench.The gain tracks how closely each benchmark’s task structure matches the generated stateful, multi-step environments; BFCL v4 multi-turn gains +10.3 at 4B.
  • 6.2 Training dynamics: Full SPADE raises the learnable share of generated environments to roughly a third late in the 400-step games run, whereas component ablations decline or collapse.Learnable environments are those where the Reasoning Agent wins between 20% and 80% of the time; the corpus, memory, and designer training jointly sustain this supply.
  • 6.2 Training dynamics: Corpus grounding sustains environment diversity, with Vendi/n reaching 0.68 with the corpus versus 0.04 without it.The comparison uses calibrated Vendi scores over SBERT-embedded environments; the t-SNE projection is illustrative.
  • 6.2 Training dynamics: By step 300, the Reasoning Agent gathers evidence before deriving, replacing upfront reasoning that failed when the interface rejected its answer.At step 200 it tests short hypotheses and revises them as results return; this shift appears where the task rewards inference.

7 Ablations

The ablations show that SPADE’s gains depend on co-adapting the Environment Designer and Reasoning Agent, while hint-based regret substantially outperforms a cheaper EMA-based reward. Removing designer training and memory together falls below the untrained base, whereas the EMA alternative recovers much of the hint-based reward’s improvement.

  • 7.1 Environment Designer adaptation: Freezing the Environment Designer and dropping its memory lowers the eight-benchmark average to 40.5, 9.7 points below the untrained base.Figure 10 shows that removing training and memory together drops self-play below base, while removing either alone can peak early and later fall below base.
  • 7.1 Environment Designer adaptation: Co-adaptation drives the curriculum and downstream improvements: freezing the self-play Environment Designer loses the gains, while the ablations remove corpus grounding, environment memory, or designer adaptation.The ablation variants also include a fixed GPT-5.5 designer retaining corpus grounding and memory, contrasting self-play adaptation with a frozen control.
  • 7.2 Reward design: EMA-based learning potential scores environments by deviation from a skill’s recent average success, but its unsigned signal can reward both consistently solved and consistently failed environments.It requires per-skill history and cannot distinguish environment-design effects from agent drift or sampling noise; the fast moving average is logged only diagnostically.
  • 7.2 Reward design: Hint-based regret raises the eight-benchmark average from 50.2 to 58.3 (+8.1), while EMA-based learning potential reaches 55.9 (+5.7).The EMA reward achieves around 70% of the hint-based gain, climbs more slowly, and the signals separate after the first ∼50 steps.

8 Scaling Results

SPADE’s gains over base models increase with model size because its adaptive Environment Designer continues generating frontier-level training environments, unlike fixed environments. Broader cognitive-skill curricula account for most of the improvement, while regret estimates remain noisy for smaller models.

  • Scaling curriculum diversity: The six-skill curriculum outperforms the two-skill curriculum, with the narrower version capturing only about half of the Reasoning-Gym gains.Six skills are targeted through a round-robin schedule with three active per regeneration; the broader curriculum’s gains reflect diversity rather than any single game family.
  • Scaling model size: +8.1 average gain at 30B-A3B, up from +5.2 at 4B and +5.7 at 8B, while matched-budget Fixed-env GRPO remains near +1.2.The adaptive curriculum keeps generating environments near the Reasoning Agent’s capability frontier, whereas static environments become quickly saturated by larger models.
  • Scaling model size: Only the 30B-A3B designer-hint regret estimate stays positive throughout training; smaller models still gain +5.2 and +5.7 despite prolonged negative estimates.The negative finite-sample estimates reflect noisy regret estimation, although regret is non-negative at the optimum.
  • Scaling curriculum diversity: The eight-benchmark suite reports 53.7 versus 58.3 at the best checkpoint, with gains increasing alongside curriculum diversity.The supplied passage identifies these as the best-checkpoint suite values but does not specify the compared candidates beyond the surrounding curriculum-scaling discussion.

9 Discussion

SPADE’s discussion attributes its gains to adaptive environment design and a shared executable interface that supports transfer across settings. It also identifies scale, optimizer, evaluation, and designer-improvement limitations that motivate future work.

  • Discussion: Adaptive environments outperform fixed pools and frozen designers, including a stronger GPT-5.5-based designer, because continual adaptation sustains improvement after fixed pools plateau.The comparison is reported for Table 1 and Table 3; the discussion attributes the advantage to adaptivity.
  • Discussion: A single code-as-environment interface spans games and reasoning tasks, while game-trained planning, constraint satisfaction, and strategic thinking generalize to held-out mathematics, science, and code.
  • Limitations: SPADE’s environment complexity is bounded by the designer’s model scale and generation budget, its learning rule remains human-authored GRPO, and its regret objective lacks formal optimality under fixed-task evaluation.The discussion also describes this complexity constraint as an invisible leash (Chae et al., 2025).
  • Future directions: Future work may compare gradient-updated designers with in-context designers that accumulate and refine strategies without weight updates, while combining SPADE with additional post-training stages.Which approach produces the better designer, and at what scale, remains open.

10 Conclusion … A Notation

SPADE makes environment design a learnable component of LLM post-training through self-play, enabling a single model to design training environments and improve from them. Its framework combines hint-based regret, corpus- and memory-grounded code environments, and a practical 30B+ recipe.

  • 10 Conclusion: SPADE makes environment design a learnable component of LLM post-training through self-play.The framework moves beyond fixed benchmarks toward open-ended, continual self-improvement.
  • 10 Conclusion: Its hint-based regret reward trains the Environment Designer to target the Reasoning Agent’s learning frontier using minimax regret theory.This reward is one of three contributions identified as enabling SPADE.
  • 10 Conclusion: SPADE grounds environment design in a pretraining corpus and accumulated environment memory, while its code-as-environment interface unifies single-turn and multi-turn settings.These design choices are presented as core components of the framework.
  • 10 Conclusion: The paper provides a practical SPADE recipe at 30B+ scale on Qwen3 models.The recipe is listed as the third contribution enabling the framework.
  • 10 Conclusion: SPADE retains its gains after fixed-envi…The supplied passage truncates the condition following “fixed-envi,” so no further claim is added.
  • A Notation: Table 4 consolidates the symbols used throughout the paper.The notation appendix includes a table of symbols used throughout the paper.

B Theoretical Analysis · B.1 Setup and Assumptions

SPADE’s hint-regret game models an Environment Designer choosing executable-environment distributions against a Reasoning Agent choosing policies, with payoffs defined by expected verifier returns. Under sound generation, articulated hints, and internalizability, any pure Nash equilibrium has zero designer regret and a hint-free optimal agent.

  • B Theoretical Analysis: The analysis represents self-play as a two-player game in which the Environment Designer selects D ∈∆(M), the Reasoning Agent selects π ∈Π, and payoffs use expected verifier returns.The designer’s payoff is expected hint-based regret, while the agent’s payoff is unhinted expected return.
  • B Theoretical Analysis: At any pure Nash equilibrium, the Environment Designer’s expected regret is zero, the Reasoning Agent is hint-free optimal on every environment in M, and hints are vacuous.If positive hint-regret remains anywhere, the unhinted agent is suboptimal there and the designer can profitably concentrate on it.
  • B.1 Setup and Assumptions: The valid environment class M contains executable, internally consistent environments whose attempted solutions can be evaluated by an external verifier.Implementation approximates M through syntax checks, executability checks, and tool-use solvability filtering.
  • B.1 Setup and Assumptions: The analysis fixes an inference budget Binf defining a finite trajectory space YBinf and a fine-tuning budget F defining a finite attainable policy set Π.These budgets include evaluation-time choices such as model and tool access, and training-time choices such as optimizer, objective, updates, and regularization.
  • B.1 Setup and Assumptions: For each environment, a fixed designer-emitted hint induces hinted and unhinted policy returns, normalized to [0, 1] for analysis while training uses shaped rewards in [−1, 1].The hint map is fixed, so strategic hint choice is outside the idealization; stochastic inference is handled through expected returns.
  • B.1 Setup and Assumptions: Assumption B.1 restricts the designer to mathematically valid executable environments, ensuring the game is defined over distributions in ∆(M).The operational realization uses syntax, executability, and tool-use solvability checks.
  • B.1 Setup and Assumptions: Assumptions B.2 and B.3 require articulated hints to attain the optimal unhinted value and require hinted behavior to be attainable by some policy without the hint.Together, these assumptions support the equilibrium conclusion that privileged hints become unnecessary at zero regret.

B.2 Main Results · C Method and Experiment Details · C.1 System Prompts and Templates

SPADE’s hint-regret game theoretically forces pure Nash equilibria toward hint-free optimality on every environment, while its executable environment-generation templates enforce solvable single-turn tasks, stateful multi-turn games, and grounded tool-use interactions. The prompts specify interaction structure, reward and termination behavior, robustness requirements, and corpus-based grounding for generated environments.

  • B.2 Main Results: Hint-regret equals hint-free regret, vanishing on solved or unsolvable environments and remaining positive only when a hint closes the agent’s performance gap.This characterization lets the Environment Designer target environments the unaided Reasoning Agent fails but can solve with assistance.
  • B.2 Main Results: Every pure Nash equilibrium drives hint-regret to zero and makes the Reasoning Agent optimal on every environment in M.Hint-regret is nonnegative and is positive exactly when the agent’s reward falls below the optimal reward; equilibria exist under the stated assumptions.
  • C.1 System Prompts and Templates: Single-turn generation prompts require deterministic, multi-step reasoning tasks whose answers are derived from visible observations, hidden from the initial prompt, and verified through a Gym-style reset()/step() interface.Episodes keep one task fixed across attempts, terminate on a correct boxed answer, truncate at a turn limit, and provide specific feedback or reveal the solution after failure.
  • C.1 System Prompts and Templates: Generic multi-turn game prompts require evolving state, at least two meaningful actions per turn, strategic branching, roughly 10–15-turn solutions, and explicit win or lose conditions.Observations expose current state, action results, available actions, and progress, while rewards distinguish wins, losses, intermediate turns, and truncation.
  • C.1 System Prompts and Templates: Corpus-grounded multi-turn prompts turn sampled documents into standalone interactive environments requiring hidden-state discovery, sequential actions, changing observations, and partial rewards for progress.The prompts explicitly prohibit collapsing the task into one-shot question answering and require the interaction to embody a document concept or technique without mentioning the source document.
  • C.1 System Prompts and Templates: Across templates, implementation contracts require complete executable Python, consistent 5-tuple step returns, graceful handling of malformed actions, initialized state, safe arithmetic, and standard-library-compatible imports.Self-verification checks that action sequences change observations, boxed inputs represent per-turn commands, and partial rewards remain within [0, 1].
  • C.1 System Prompts and Templates: Tool-use templates model real workflows as 3–5 atomic user instructions revealed one at a time, with tool calls mutating state and advancing only when each criterion is satisfied.The generated environment subclasses ToolUseBaseEnv, defines tools and message criteria, appends the next instruction after completion, and checks a final done response.

C.2 Implementation Details … E Extended Related Work

SPADE uses executable-environment validation, controlled pool management, and privileged-hint generation to support reliable training. The paper also documents reproducibility resources and qualification limits for cross-system comparisons and backbone transfer.

  • C.2 Implementation Details: Full SPADE runs disable EMA learning-potential, frontier, and variance bonuses, except for the standalone EMA replacement evaluated in Section 7.2.The frontier bonus uses the fast-versus-slow EMA gap, while no separate mixed-outcome variance reward is used.
  • C.2 Implementation Details: The environment pool is fully replaced at regeneration, memory evicts oldest records at 200 entries, rejected candidates persist, and generation retries up to five times.These lifecycle rules define how environment candidates and accumulated memory are maintained during training.
  • C.2 Implementation Details: Privileged hints are generated by a separate designer-side call conditioned on environment source code, which remains hidden from the Reasoning Agent.The hint writer sees the generated code, while the acting agent does not.
  • C.2 Implementation Details: Every candidate environment must pass parsing, reset, and probe-step smoke tests, with tool-use environments additionally screened for reset-gate failures and impossibility.The semantic checks reject environments whose success criteria fail on freshly reset states across tested seeds or whose tasks are unreachable.
  • C.3 Reproducibility: Training and evaluation code, every run configuration, evaluation JSON, and figure-generation scripts are released.These materials are included alongside the released code to support reproduction.
  • C.3.1 Comparability Notes for Table 2: Table 2 comparisons reprint authors’ published scores, while BFCL version and data-snapshot differences limit direct comparability across reference systems.The paper distinguishes its BFCL v4 multi-turn results from reference rows labeled v3 and records protocol differences.
  • C.3.1 Comparability Notes for Table 2: Curriculum breadth drives larger held-out benchmark gains: the full six-skill curriculum outperforms the restricted two-skill variant.This comparison is shown for Qwen3-30B-A3B-Instruct-2507 in the games setting across AIME, GPQA-Diamond, LiveCodeBench-v6, and Reasoning-Gym.
  • C.3.1 Comparability Notes for Table 2: Nemotron transfer raises all four Reasoning-Gym categories above its untrained base, with gains of +9.6, +9.3, +2.6, and +3.2.The gains correspond to RG-Cognition, RG-Algorithmic, RG-Math, and RG-Logic, respectively; other listed benchmarks were not evaluated on this backbone.

E.1 Self-Play for LLMs … E.5 Agentic Memory Design and Self-Improving Code Systems

SPADE extends self-play and unsupervised environment design by training an LLM Environment Designer with hint-based regret to generate executable, full-MDP environments for an improving Reasoning Agent. Across related work, it addresses fixed or heuristic environment generation through open-ended code-based environments, gradient-trained adaptation, and adaptive curricula.

  • E.1 Self-Play for LLMs: Classical self-play progressed from TD-Gammon and AlphaGo to AlphaZero and large-scale multi-agent systems, establishing self-play as a foundation for increasingly capable agents.These systems span backgammon, Go, chess, shogi, Dota 2, and StarCraft.
  • E.1 Self-Play for LLMs: Recent LLM self-play methods improve models by self-generated supervision, but they differ in whether they rely on human demonstrations, self-judgment, minimal seeds, or curated evaluation sets.SPIN distinguishes model outputs from human demonstrations, while Self-Rewarding Language Models generate preference data by judging their own outputs.
  • E.1 Self-Play for LLMs: SPADE generates full MDP environments as executable Python and trains the Environment Designer with hint-based regret grounded in Reasoning Agent returns rather than proxy statistics.This contrasts with data-free self-play methods that generate task statements with sparse terminal rewards and use frozen generators or heuristic rewards.
  • E.2 Unsupervised Environment Design and Open-Endedness: Open-endedness research motivates unbounded environment spaces because systems can plateau when environment parameterizations are exhausted, while rich competitive settings can produce increasingly sophisticated tool use.This provides the broader rationale for moving beyond fixed environment vocabularies.
  • E.2 Unsupervised Environment Design and Open-Endedness: SPADE brings regret-based UED to an unbounded code environment, using an LLM Designer and hint-based co-evolution instead of parameterized spaces or a trained antagonist.Classical UED methods operate over small, fixed design vocabularies such as maze dimensions, terrain friction, and grid layouts.
  • E.3 Synthetic Environment Generation: LLM environment-generation systems produce useful training material and sometimes adapt distributions to agent progress, but their generators generally lack RL-gradient-based joint optimization.SPADE instead trains its Environment Designer through reinforcement learning.
  • E.4 Environment Scaling: Environment-scaling research identifies diverse, abundant environments and adaptive curricula as central to generalization, while SPADE combines gradient-trained design with code-as-environment representation.Related systems adjust difficulty or sampling policies using performance statistics, accuracy, or problem categories.
  • E.5 Agentic Memory Design and Self-Improving Code Systems: Agentic memory research explores runtime episodic memory and learned Python memory architectures, providing a related direction for self-improving code systems.MemRL improves decisions through stored interaction experiences, while ALMA searches over update-and-retrieve memory programs.

F Extended Quantitative Analysis · F.1 Games · F.1.1 Environment Quality

SPADE’s games environments become more learnable during training while remaining well-posed, verifiable, and structurally rich. Extended analyses also examine reward granularity, per-skill learning, embedding robustness, and matched 30B training dynamics.

  • F Extended Quantitative Analysis: Embedding-based conclusions are identical with SBERT and TF-IDF with LSA-128 under the stated multi-embedding robustness protocol.Step-level dynamics use matched 30B-A3B runs, with full SPADE covering steps 0–399 and ablations ending earlier.
  • F.1 Games: The games-setting analyses extend the main text’s environment-quality and diversity analyses.This block focuses on environment quality and related diversity signals in games.
  • F.1.1 Environment Quality: Environment quality is assessed with five signals: four tracked over training and executability measured once on raw generations.The tracked signals include learnability, well-posedness, verifiability, and structural richness.
  • F.1.1 Environment Quality: 0.16→0.31 of environments enter the learnable band while Reasoning Agent win rate rises 0.30→0.62, indicating sharper frontier targeting during learning.Well-posedness remains 97 to 98%, while verifiability rises 90 to 93%; Table 7 attributes quality gains to difficulty targeting rather than simpler environments.
  • F.1.1 Environment Quality: Reward granularity increases over training, measured by mean distinct reward levels per environment and early-to-late changes overall and by domain.The analysis includes strictly partial reward levels in the canonical 30B games run.
  • F.1.1 Environment Quality: Logical Deduction and Spatial Reasoning improve most, while Causal Inference declines slightly despite starting high.Figure 17 reports per-skill win rates and gains over training for the canonical SPADE-30B run.

F.1.2 Environment Diversity

SPADE maintains near-ceiling, stationary environment diversity throughout training, spanning all 13 domains and sustaining high novelty. Corpus grounding is the key driver: removing it collapses diversity, whereas memory and Environment Designer training do not.

  • Diversity dynamics: 20.8 versus 21.0 Vendi Score across early and late training, with 0.94 mean pairwise cosine distance and 97 of 100 steps entirely novel.Per-step diversity remains at the mixed-population ceiling throughout the run.
  • Domain coverage: All 13 taxonomy domains appear from step 0, averaging 8.4 of 24 environments per batch, with Mathematics at 30% and Physics at 20%.The remaining shares are Medicine 11%, Chemistry 10%, CS 7%, Engineering 6%, and other 16%.
  • Distribution stationarity: A 5-fold AUC of 0.551 ± 0.024 indicates early- and late-half environments are difficult to distinguish after seed-document reuse is removed.This supports a stationary environment distribution over training.
  • Corpus ablations: 0.70 versus 0.04 Vendi Score in matched ablations shows corpus grounding preserves diversity, while removing it collapses the environment distribution.The comparison holds the skill set and other specified conditions matched; the table also reproduces the corpus/no-corpus separation under TF-IDF/LSA embeddings.
  • Scope and limitations: Diversity is evaluated with batch size 24, while limitations include step-level rather than per-environment learnability, rubric-based difficulty, and raw-generation executability measurement.Larger boot-step batches are subsampled for comparability.

G Extended Qualitative Analysis · G.1 Games · G.1.1 Additional Privileged-Hint Examples

The appendix qualitatively analyzes all 3,310 environments from the canonical SPADE-30B run, finding distinct executable programs grounded in diverse documents and examining games through privileged-hint examples. These examples span positive-regret regimes from frontier gaps to near-mastery gaps, making hint utility interpretable in task context.

  • G Extended Qualitative Analysis: The canonical SPADE-30B run generated 3,310 environments, with the learnable-band share nearly doubling from 0.16 to 0.31 while other listed quality properties held constant.The analysis covers all generated environments and reports stable well-posedness, verifiability, and program richness.
  • G Extended Qualitative Analysis: Semantic diversity remained at the mixed-population ceiling throughout all 400 training steps, according to the appendix summary.This finding is part of the appendix’s four reported results for the canonical run.
  • G Extended Qualitative Analysis: All 3,310 generated environments were distinct programs, comprising 2,388 distinct initial states and 1,513 distinct seed documents.The analysis therefore operates on environment content rather than surface identifiers such as class names.
  • G Extended Qualitative Analysis: The qualitative analysis examines each environment’s seed document, generated program, and opening observation rather than relying on surface identifiers.Class names were not treated as content evidence because 58% retained the scaffold default despite the naming instruction.
  • G.1 Games: The games appendix adds privileged-hint examples and traces how one environment evolves from early to late training.This extends the games analysis beyond aggregate results to qualitative environment trajectories.
  • G.1.1 Additional Privileged-Hint Examples: Figure 19 presents four same-record positive-regret game task–hint pairs spanning large frontier gaps to a small mastery-regime gap.Task summaries preserve goals, hidden information, and usable interaction, while hint excerpts retain substantive guidance; headers report mean return without hint / with hint.

G.1.2 Environment Evolution over Training … G.3.3 Full Source of a Generated Tool-Use Environment

SPADE generates executable, stateful environments whose diversity expands with corpus grounding, while covering reasoning, games, and multi-turn tool use through a shared reset()/step() interface. The gallery illustrates hidden state, intermediate feedback, programmatic verification, and terminal rewards in complete generated programs.

  • G.1.2 Environment Evolution over Training: Across training, generated games evolve from a 370-line car-ownership environment with 29 state variables to a 376-line thermodynamics environment with 19 state variables, both grounded in seed documents.The examples use web-scraped documents, including an off-topic personal-finance document and a thermodynamics problem, and expose hidden state through Gym-style programs.
  • G.1.3 Matched-Step Generation Contrast: SPADE’s corpus-grounded designer generated diverse domains, whereas the no-corpus run repeated the same rotating-maze family 41 times across matched training steps.At steps 290–312, the corpus run spanned probability theory, quantum tomography, volcanology, hematology, radar processing, and hypoelliptic operators; the no-corpus run produced 41 rotating-maze environments.
  • G.2 Tool Use: Tool-use environments simulate banking, retail, support, telecom, and smart-home APIs, advancing one atomic instruction only after hidden-state criteria programmatically verify completion.Examples include banking transfers and invoice payment, plus ticket assignment, note addition, status updates, and resolution checks.
  • G.3.3 Full Source of a Generated Tool-Use Environment: The tool-use source exemplars operationalize long-horizon workflows through per-step message criteria over hidden state, including account transfers, invoice payment, ticket updates, and final resolution.These criteria enforce exact account identifiers, transfer amounts, invoice IDs, ticket assignments, notes, and status transitions before advancing.
  • G.2 Tool Use: The 30B tool-use run remains structurally stationary: program length falls from 247 to 226 lines and tools from 7.6 to 5.9, while instructions remain 4.7 per environment.Per-step criterion complexity stays near 1.9–2.0 conditions, and roughly one environment in five includes a guarded failure case.
  • G.3 Generated Environment Gallery: The generated-environment gallery demonstrates that one executable interface spans stateful deduction, mathematical reasoning, and multi-turn tool-use environments with verifiable episode outcomes.The complete source examples are reproduced in the emitted format, including hidden state, action handling, feedback, and terminal verification.
  • G.3.1 A Minimal Generated Environment: A minimal generated environment is a single Python class with Gym-style reset()/step(), stateful multi-turn feedback, and verifiable terminal reward, exemplified by a six-turn Wordle-like game.Correct guesses terminate with reward 1, while exhausting six turns truncates the episode with reward 0; the same pipeline supports answer grading and tool use.
  • G.3.2 Full Source of Two Exemplar Generated Environments: The full exemplar programs initialize hidden state in reset(), process multi-turn actions with intermediate feedback, and compute clipped terminal returns, enabling multi-step goals such as securing a car title.The car environment verifies a terminal state in which the car is solely titled to the agent and reports reward 1.0; the thermodynamics environment hides system parameters and targets zero net entropy change.
Loading 2608.19197v1…