PaperlayerView exact revision on arXiv

Source-linked AI summary

SPADE: Self-Play in Adaptive Synthetic Executable Environments

Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques

arXiv:2608.19197v1cs.CLcs.AI

TL;DR

Existing training environments remain fixed or exhaustible as language agents scale, limiting the supply of diverse adaptive goals. SPADE makes environment design learnable through self-play, improving held-out reasoning benchmarks by +5.3 points on average and lifting multi-turn tool-use benchmarks by up to +13.9 points.

  • Problem

    Existing synthetic and self-play environment methods do not provide adaptive, diverse, complete multi-turn environments that grow with the agent.

    Source: §1 · p. 2

  • Method

    SPADE trains an Environment Designer and Reasoning Agent through self-play, using hint-based regret, corpus grounding, executable environments, and accumulated environment memory.

    Source: §4.3 · p. 9, §10 · p. 18

  • Results

    +5.3 points on average across eight held-out math, science, code, and reasoning benchmarks, with tool-use gains of +5.7 on BFCL v4 multi-turn and +13.9 on ACEBench-Agent at 30B-A3B.

    Source: §1 · p. 3, §9 · p. 17

  • Takeaways & Limitations

    SPADE shows that a single model can design its own training environments and improve from them, moving beyond fixed benchmarks toward open-ended continual self-improvement.

    Source: §10 · p. 18

    Environment complexity remains bounded by the designer model’s context and generation budget, while hint-based regret lacks a proof of optimal curriculum generation and evaluation remains fixed-task.

    Source: §9 · p. 17

Abstract

from arXiv · show

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

1 Introduction

SPADE addresses the fixed, non-adaptive supply of training environments by having one LLM co-evolve executable environments and a reasoning agent through self-play. It improves games and tool-use performance at 30B+ scale while adapting task complexity to the learner.

Motivation

  • Existing hand-built, synthetic, and ungrounded self-play environments remain fixed, scale with human authoring, or cannot reliably exceed the model’s own knowledge.SPADE grounds design in pretraining-corpus knowledge and accumulated experience, addressing the bottleneck of scarce adaptive environments.Source: §1 · pp. 2–3

Framework

  • A single LLM acts as an Environment Designer that writes executable Python environments and a Reasoning Agent that learns to solve them through self-play.Each environment implements Gym-style reset()/step() interactions and a complete MDP, unifying single-turn reasoning with multi-turn agentic tasks.Source: §1 · p. 2

Method

  • SPADE uses hint-based regret—the performance gap with and without privileged hints—to target environments that are solvable yet at the Reasoning Agent’s capability frontier.This avoids the failure modes of unconstrained adversarial or purely cooperative environment design.Source: §1 · p. 2

Framework

  • Unlike fixed generators, SPADE trains the Environment Designer with reinforcement learning, so the environment distribution co-evolves as the agent improves.Because computable MDPs can be represented as programs, the designer is not restricted to a hand-designed environment parameterization.Source: §1 · pp. 2–3

Results

  • SPADE’s adaptive curriculum shifts from simple single-skill tasks toward state-gated, multi-turn environments as the Reasoning Agent improves.Figure 2 shows four generated environments across one 30B-A3B run, contrasting SPADE’s moving curriculum with fixed human-curated or frozen synthetic pools.Source: Figure 2 · p. 3, §1 · p. 3
  • +5.3 points on average over the strongest fixed-environment baseline at 30B-A3B, with tool-use gains of +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent.On games, gains reach +7.5 on individual benchmarks; BFCL-v4 also improves by +10.3 at 4B.Source: §1 · p. 3

2 Related Work

SPADE extends self-play and unsupervised environment design to LLMs by having an Environment Designer generate executable, full multi-turn MDPs that co-evolve online with a Reasoning Agent. Unlike prior fixed or separately trained environment-generation approaches, it uses hint-based regret in a single RL loop to adapt the environment distribution to the agent’s capability frontier.

Self-Play for LLMs

  • Earlier LLM self-play methods use curated seeds, fixed evaluations, shared or separate proposer-solver models, and often single-terminal tasks; SPADE instead co-evolves executable environments with embedded verification across games and tool use.SPIRAL and SPICE are closest predecessors, but their generated tasks have terminal rewards; SPADE’s verifier is part of each environment rather than a fixed Guide or judge.Source: §2 · pp. 3–4

Unsupervised Environment Design and Open-Endedness

  • Unlike classical UED and curriculum-based LLM training, SPADE writes complete MDPs in an unbounded code-defined space instead of selecting from fixed parameterizations or task pools.Prior UED methods search spaces such as maze dimensions and layouts, while SPADE makes environment design itself the learning agent in online co-evolution.Source: §2 · p. 4

Environment Synthesis and Scaling

  • SPADE trains the Environment Designer online with the Reasoning Agent via hint-based regret, generating full multi-turn MDPs as executable Python and co-evolving the environment distribution with agent capability.This contrasts with generators that are typically frozen, hand-engineered, or trained using a separate signal, and with task generators providing only sparse terminal rewards.Source: §2 · p. 5

3 Preliminaries

SPADE represents training tasks as fully observed Gym-style Markov decision processes and trains its LLM policy with verifiable rewards using GRPO. The framework combines executable environment interaction with policy optimization through group-relative advantages, clipped updates, and KL regularization.

3.1 Markov Decision Processes and Gym-Style Environments

  • SPADE models environments as fully observed MDPs with Gym-style reset() and step() interfaces that return observations, rewards, termination, truncation, and auxiliary information.An MDP is defined by state and action spaces, transitions, rewards, and an initial-state distribution.Source: §3.1 · p. 5, §3 · p. 5
  • A stateful Wordle-style environment illustrates the interface: correct guesses terminate with reward 1, while exhausting six turns truncates the episode with reward 0.Each guess returns per-letter feedback to support multi-turn deduction.Source: §3.1 · p. 5

3.2 RLVR and GRPO

  • SPADE trains its policy with reinforcement learning from verifiable rewards using GRPO, which samples response groups, computes group-normalized advantages, and applies clipped policy gradients with KL regularization.Both the Environment Designer and Reasoning Agent rewards update the shared policy via GRPO.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

4 SPADE: Self-Play in Adaptive Synthetic Executable Environments

SPADE uses one LLM as both an Environment Designer and Reasoning Agent, generating executable Gym-style environments and learning to solve them through shared self-play. Hint-based regret, corpus grounding, and environment memory adapt difficulty and diversity toward the agent’s learning frontier.

4.1 Dual-Role Self-Play and Code-as-Environment

  • SPADE alternates role-specific environment design and reasoning-agent learning with one shared policy, representing tasks as executable Python programs implementing reset()/step().The environment code encodes transitions and rewards, supporting both single-turn reasoning and multi-turn agentic interaction; valid candidates undergo syntax and executability checks.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Hint-Based Regret Reward

  • Hint-based regret is the Reasoning Agent’s return gap between plays with and without privileged, task-specific hints, steering design toward challenging but teachable environments.A difficulty anchor targets a 0.4–0.6 Reasoning Agent win-rate band, while floored regret selects teachable environments within that band.Source: §4.1 · pp. 6–7

4.3 Environment Design

  • Training on diverse synthetic games improves all eight held-out reasoning, science, code, and procedural benchmarks across backbone scales, with competition math preserved.The comparison includes retrained fixed-environment baselines and reports unweighted average performance across eight benchmarks.Source: Table 1 · p. 9, Figure 5 · p. 10

4.2 Hint-Based Regret Reward

  • Privileged hints expose the behavioral source of positive regret by giving agents near-direct probe patterns or narrowing audio-task searches to useful parameter ranges.The paired trajectories show independent plays whose returns differ because hints reduce otherwise repeated failed attempts.Source: §4.2 · pp. 8–9, Figure 4 · p. 8

4.3 Environment Design

  • Corpus grounding supplies novelty while cross-episode environment memory supplies difficulty-aware seeds, respectively controlling what environments concern and how hard they are.Memory stores generated environments with regret scores and skill tags, helping the designer avoid mastered tasks and track the agent’s moving frontier.Source: §4.3 · p. 9

5 Experimental Setup

SPADE is trained with three Qwen3 backbones using delayed, memory-based environment regeneration and regret-driven self-play. Games and tool-use settings vary generation, grounding, agent interaction, and regeneration interval, while evaluations use held-out benchmarks and fixed or synthetic-environment baselines.

5.1 SPADE Training Recipe

  • SPADE trains Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507 with GRPO, delayed environment updates, accumulated environment memory, and hint-based regret.Training uses 400 rollouts of 24 environments; the 30B model is primary, while the 8B model enables thinking for both roles.Source: §5.1 · p. 10

5.2 Domain: Games

  • Games use executable Python environments spanning rotating cognitive-skill categories, grounded in 15k math-and-science documents, with k=4 and syntax, execution, and difficulty validation.Each rollout contains 24 games, and the Environment Designer reward includes a flat-top difficulty anchor alongside floored hint-based regret.Source: §5.2 · pp. 10–11
  • Evaluations hold out all benchmarks and cover procedural reasoning plus out-of-distribution reasoning and code across eight benchmarks, alongside fixed-environment RLVE and GPT-5.5 baselines.Table 1 reports all eight benchmarks for every backbone; component controls such as freezing the Environment Designer or removing grounding are evaluated in Section 7.Source: §5.2 · p. 11

5.3 Domain: Tool Use

  • Tool-use environments provide simulated function-calling tools, mutable backend state, sequential user instructions, step-by-step hints, and multi-stage validation, grounded in 15k code documents with k=8.The Reasoning Agent is rewarded only after completing every instruction; evaluation compares SPADE with AgentScaler, Agent-World, AWM, and EnvScaler on three multi-turn benchmarks.Source: §5.3 · pp. 11–12

6 Experimental Results

SPADE improves held-out performance in both synthetic game and tool-use settings, with gains increasing at larger scales and transferring beyond the generated environments. Training also sustains diverse, increasingly difficult environments while shifting the Reasoning Agent toward evidence-driven interaction.

6.1 Quantitative Analysis

  • +5.3 over Fixed-env RLVE at 30B-A3B, where SPADE reaches a suite average of 58.3 and transfers gains to science, code, and procedural reasoning.The margin grows with model size, while competition math is preserved.Source: §6.1 · p. 12
  • +13.9 on ACEBench-Agent, +5.7 on BFCL v4 multi-turn, and +3.6 on τ 2-bench at 30B-A3B, with gains tracking structural similarity to generated tool-use environments.SPADE leads dedicated data-synthesis systems on BFCL v4 multi-turn and ACEBench-Agent.Source: §6.1 · p. 12

6.2 Qualitative Analysis

  • Corpus grounding sustains diversity at Vendi/n 0.68 versus 0.04 without it, while full SPADE raises the learnable environment share to roughly a third late in training.The corpus provides breadth, whereas Environment Designer training increases difficulty.Source: §6.2 · p. 12, Figure 7 · p. 13, Figure 6 · p. 13
  • Environment Designer training reduces formula reveals from 25% to 5% and increases distinct reward levels from 3.7 to 5.8, including strictly partial levels from 2.2 to 4.0.The no-corpus run instead repeats the same rotating-maze task 41 times consecutively.Source: §6.2 · p. 12, Figure 8 · p. 13
  • By step 300, the Reasoning Agent gathers evidence before deriving, replacing upfront reasoning that failed at step 0 while retaining long derivations when tasks require them.On procedural environments, late-step commands have a median length of 8 tokens, and late-checkpoint benchmark gains confirm retained ability.Source: §6.2 · p. 13
  • The full adaptive configuration outperforms every partial and frozen-designer control in the 30B-A3B games setting.Table 3 reports best-checkpoint suite averages over the same eight benchmarks as Table 1.Source: Table 3 · p. 14

7 Ablations

SPADE’s gains depend on co-adapting the Environment Designer with the Reasoning Agent and on using hint-based regret rather than a cheaper EMA-based learning-potential reward. Removing designer adaptation and memory can underperform the untrained base, while the alternative reward recovers roughly 70% of the full gain but learns more slowly.

7 Ablations

  • The ablations compare corpus grounding, environment memory, designer adaptation, and reward choices using Qwen3-30B-A3B-Instruct-2507 games evaluated on the same eight-benchmark suite.Table 3 reports selected checkpoints, Figure 10 shows full evaluation curves, and Table 6 reports reward-ablation checkpoints.Source: §7 · p. 14, §7.2 · p. 16

7.1 Ablation: Environment Designer Adaptation

  • Freezing the Environment Designer and removing its memory lowers the eight-benchmark average to 40.5, 9.7 points below the untrained base.A fixed GPT-5.5 designer retaining corpus grounding and memory reaches 53.0 versus 50.2, but does not improve LiveCodeBench-v6: 42.6 versus 43.2.Source: §7.1 · p. 15, Figure 10 · p. 15
  • Full co-adaptation remains strongest late in training, while partial and frozen variants peak early and can later fall below the untrained base.The no-corpus run peaks near step 111 and the GPT-5.5-designer run near step 175; the ablation curves cover the eight-benchmark suite.Source: §7.1 · p. 15, Figure 10 · p. 15

7.2 Ablation: Environment Designer Reward

  • EMA-based learning potential scores environments by deviation from a skill’s recent average using fast and slow moving averages, avoiding hinted replay but requiring per-skill history.Its unsigned deviation can score environments that are always solved or never solved similarly, while the fast mean is logged only diagnostically.Source: §7.2 · pp. 15–16
  • The hint-based regret reward raises the eight-benchmark average from 50.2 to 58.3 (+8.1), whereas EMA-based learning potential reaches 55.9 (+5.7).The two trained reward variants isolate the reward choice; their trajectories separate after roughly 50 steps, with EMA-based learning potential climbing more slowly.Source: §7.2 · p. 16

8 Scaling Results

SPADE’s gains over base models increase with scale, reaching +8.1 at 30B-A3B versus about +1.2 for Fixed-env GRPO, while broader curricula account for much of the improvement. Smaller models still benefit despite noisy regret estimates, supporting adaptive environment generation as the scaling mechanism.

Scaling model size

  • Adaptive environment generation keeps producing tasks near the Reasoning Agent’s capability frontier, whereas static environments become quickly fitted as models scale.This explains why larger models benefit more from SPADE than from a fixed training signal.Source: §8 · p. 16

Scaling curriculum diversity

  • The six-skill curriculum outperforms the two-skill version, which captures only about half the Reasoning-Gym gains and smaller GPQA-Diamond and LiveCodeBench-v6 gains.The comparison holds the rest of the training setup fixed; gains track curriculum diversity rather than any single game family.Source: Figure 13 · p. 16, §8 · pp. 16–17

Scaling model size

  • +8.1 average gain at 30B-A3B rises from +5.2 at 4B and +5.7 at 8B, while Fixed-env GRPO remains near +1.2 across sizes.The comparison covers eight benchmarks under a shared training recipe.Source: §8 · p. 16, Figure 12 · p. 17

Regret signal

  • Only the 30B-A3B model maintains a positive designer-hint regret estimate, while 4B and 8B estimates dip below zero for long stretches.Despite the noisy finite-sample signal, the smaller models still improve over base by +5.2 and +5.7.Source: §8 · p. 17

9 Discussion

SPADE’s adaptive environment design outperforms fixed or frozen-designer environments because continual adaptation sustains improvement. Its shared code interface transfers reasoning skills across games and boosts interactive tool-use benchmarks, while limitations and future directions concern scale, optimization, evaluation, and broader automation.

Discussion

  • Adaptive environments outperform fixed pools and frozen designers, including a stronger GPT-5.5 designer, because continual adaptation sustains gains after fixed pools stop improving.The comparison is reported in Table 1 and Table 3, with the explanation attributed to adaptivity.Source: §9 · p. 17
  • +13.9 on ACEBench-Agent and +5.7 on BFCL v4 multi-turn at 30B-A3B show that the shared code-as-environment interface especially benefits interaction-heavy tool use.In games, planning, constraint satisfaction, and strategic thinking generalize to held-out mathematics, science, and code.Source: §9 · p. 17

Limitations

  • SPADE’s complexity is bounded by the Environment Designer’s model scale and generation budget, while GRPO remains a fixed human-designed optimizer.Hint-based regret is motivated by PAIRED but is not proven to produce an optimal curriculum, and evaluation remains fixed-task.Source: §9 · p. 17

Future directions

  • Future work could compare gradient-updated designers with in-context evolution and extend adaptive self-play from environment design across data curation, training, and the learning rule.Whether learned weights or in-context evolution is better, and at what scale, remains open.Source: §9 · p. 17

10 Conclusion … A Notation

SPADE makes environment design a learnable component of LLM post-training through self-play, enabling a single model to generate training environments and improve from them. Its hint-based regret objective, corpus grounding, environment memory, and code-as-environment interface support gains that persist as fixed-environment baselines saturate.

10 Conclusion

  • +8.1 average points over base at 30B-A3B, with gains of +5.2 and +5.7 at 4B and 8B, while the margin over the strongest fixed-environment baseline grows with scale.These results cover held-out reasoning and code benchmarks.Source: §10 · p. 18
  • +13.9 on ACEBench-Agent and +5.7 on BFCL v4 multi-turn demonstrate that the same recipe lifts multi-step tool-use benchmarks.Source: §10 · p. 18
  • Hint-based regret reward trains the Environment Designer to produce environments at the Reasoning Agent’s learning frontier, grounded in minimax regret theory.Source: §10 · p. 18
  • Pretraining-corpus grounding and accumulated environment memory anchor environment design, while a code-as-environment interface unifies single-turn and multi-turn settings.Source: §10 · p. 18
  • A practical recipe at 30B+ scale on Qwen3 models enables SPADE’s reported improvements after fixed-environment baselines begin to saturate.Source: §10 · p. 18
  • SPADE demonstrates that a single model can design its own training environments and improve from them, moving beyond fixed benchmarks toward open-ended, continual self-improvement.Source: §10 · p. 18

A Notation

B Theoretical Analysis · B.1 Setup and Assumptions

The analysis models hint-regret self-play as a two-player game and characterizes pure Nash equilibria: positive regret drives environment concentration until the agent is hint-free optimal everywhere. The setup restricts environments, trajectories, policies, hints, and verifier returns through finite-budget assumptions.

B Theoretical Analysis

  • The framework treats self-play as a two-player game in which the Environment Designer selects an environment distribution and the Reasoning Agent selects a policy, with stochastic payoffs defined by expected verifier returns.The game is defined over ∆(M) × Π.Source: B Theoretical Analysis · p. 29, §B.1 · p. 30
  • At any pure Nash equilibrium, the Environment Designer’s expected regret is zero, the Reasoning Agent is hint-free optimal on every environment in M, and hints are vacuous.If positive hint-regret remains anywhere, the Designer can profitably concentrate on that region because the unhinted agent is suboptimal there.Source: B Theoretical Analysis · p. 29

B.1 Setup and Assumptions

  • The valid environment class M contains mathematically consistent executable environments whose solutions can be externally verified; implementation approximates this through syntax, executability, and solvability checks.The Environment Designer samples distributions D ∈∆(M).Source: §B.1 · pp. 29–30
  • A fixed inference budget Binf induces a finite trajectory space YBinf, covering architecture, context, prompting, generation, sampling, rollouts, tools, memory, and verification choices.These choices determine the possible Reasoning Agent trajectories at evaluation time.Source: §B.1 · p. 29
  • A fixed fine-tuning budget F induces a finite attainable policy set Π, determined by the optimizer, objective, updates, and regularization, so maxima over policies are attained.The policy set formalizes which training procedures can produce.Source: §B.1 · p. 29
  • For each environment, a fixed hint map gives policies hinted or unhinted trajectory distributions, with verifier returns normalized to [0, 1] for analysis.Training instead uses a shaped reward in [−1, 1].Source: §B.1 · p. 29
  • The formal assumptions require sound generation, articulated hints that attain the optimal unhinted value, and internalizability of hinted behavior within the attainable policy class.Sound generation corresponds operationally to syntax, executability, and tool-use solvability filtering.Source: §B.1 · p. 30

B.2 Main Results · C Method and Experiment Details · C.1 System Prompts and Templates

This section presents the theoretical rationale and system design behind hint-regret-driven environment generation, including prompts and templates for creating solvable, deterministic, corpus-grounded executable environments. It details how these environments support progressive multi-turn reasoning and tool-use tasks through hidden state, evolving observations, branching interactions, and standardized agent interfaces.

B.2 Main Results

  • At every pure Nash equilibrium, hint-regret is zero on every environment, so the Reasoning Agent is hint-free optimal throughout the environment space.Hint-regret is nonnegative and positive exactly when the agent underperforms the optimal reward; the Environment Designer can profitably deviate to any environment with positive regret.Source: §B.2 · pp. 30–31
  • The regret signal differs from plain adversarial difficulty because it rewards targeting the learnable frontier rather than merely maximizing the agent’s failure.It vanishes on solved and unsolvable environments and is positive only when hints would close the agent’s performance gap; uniformly optimal policies ensure such equilibria exist.Source: §B.2 · pp. 30–31

C.1 System Prompts and Templates

  • Single-turn prompts require deterministic, multi-step reasoning games with hidden answers, large answer spaces, computed solutions, hinting feedback, and a fixed task across retries.The executable contract uses reset() and step(), verifies boxed answers, terminates on success, and truncates after a maximum turn count while revealing the solution.Source: §C.1 · pp. 32–33
  • Multi-turn games require evolving state, at least two meaningfully different actions per turn, roughly 10–15-turn optimal play, and clear win conditions with ideally a lose condition.Observations include available actions and progress, while initial observations include rules, goals, and starting state.Source: §C.1 · pp. 33–34
  • Corpus-grounded prompts require environments to implement document concepts as genuine multi-turn interactions rather than one-shot quizzes or static question answering.The canonical games use sampled documents and require hidden state, multi-turn structure, and partial reward; generated games must remain standalone without mentioning the source document.Source: §C.1 · p. 37
  • Multi-turn game environments require sequential actions, hidden information, state-changing commands, updated observations, partial rewards in [0,1], and termination only upon reaching the goal.The interface parses each \boxed command, updates state, returns a new observation, and assigns full reward only for success; prompts also demand branching decisions.Source: §C.1 · pp. 37–38
  • Robustness requirements enforce standard-library implementations, initialized state, escaped braces, five-tuple step returns, graceful invalid-command handling, and no crashes or unsafe division.Self-verification traces two action sequences to ensure observations change, commands evolve state, and partial rewards remain bounded.Source: §C.1 · pp. 36, 38
  • Tool-use generation samples one of ten application domains and one of four task variants, with variant weights of 3:2:1:1; each user message is a short, specific, atomic instruction.The environments allow 4–8 tool calls per message.Source: §C.1 · pp. 38–39
  • Tool-use templates model real workflows by revealing 3–5 atomic user instructions sequentially, requiring completion of the current criterion before exposing the next.Each instruction has a state-checkable criterion, and the environment advances after tool calls satisfy that criterion rather than presenting the full workflow upfront.Source: §C.1 · pp. 38–39
  • Solvability checks require every success criterion to be achievable through its instruction and false at reset, including explicit execution records for read or verification steps.This prevents agents from skipping steps, deadlocks, and criteria that advance merely because a reset state already satisfies them.Source: §C.1 · pp. 39–40
  • Criteria must be type-correct, derivable from instructions or readable tool results, reachable through state-setting tools, and matched exactly by each instruction.These requirements prevent deadlocks caused by type mismatches, hidden values, missing state fields, or irrelevant instruction details.Source: §C.1 · p. 40
  • The designer must self-trace reset(seed=0) and the prescribed solution sequence, verifying ordered criterion transitions, message completion, and successful submission without exceptions.A broken criterion at reset, a never-satisfied step, or a KeyError/IndexError requires fixing before the environment is returned.Source: §C.1 · p. 40
  • The environment template constrains implementations to deterministic, self-contained ToolUseBaseEnv classes with 3–8 tools, typed parameters, distractors, graceful errors, and exact schema–method parameter matching.Tools are simulated methods using only standard-library imports, with randomness restricted to reset().Source: §C.1 · pp. 40–41
  • Multi-turn templates reveal instructions progressively through _advance_if_done(), require all messages plus the answer "done", and support execute, read_then_write, missing_value, and with_irrelevant task variants.The variants respectively test ordered tool use, predicate-conditioned writes, information discovery before writing, and explicit handling of unsatisfiable requests via decline_request.Source: §C.1 · pp. 42–43
  • The Reasoning Agent receives each observation in a fixed language-game prompt that requests valid actions, step-by-step reasoning, and a final answer in \boxed{} form.The gameplay interface wraps the observation as the current turn’s user message.Source: §C.1 · p. 43

C.2 Implementation Details … E Extended Related Work

SPADE’s implementation validates executable environments, manages regeneration and memory explicitly, and evaluates transfer and curriculum breadth while documenting reproducibility and comparison-protocol differences. The reported evidence shows broader curricula and a second backbone family improve held-out reasoning performance under the stated evaluation conditions.

C.2 Implementation Details

  • Full-SPADE runs disable the learning-potential, frontier, and variance bonuses, while Table 5 specifies the complete training configuration.The learning-potential bonus appears only as a standalone reward replacement evaluated in Section 7.2.Source: §C.2 · p. 43, Table 5 · p. 44
  • SPADE replaces each environment pool during regeneration, caps memory at 200 records with oldest-first eviction, persists rejected candidates, and retries generation up to five times.Failed candidates are regenerated up to a fixed number of attempts, and rejected candidates remain available for inspection.Source: §C.2 · p. 43
  • Privileged hints are generated by a separate designer-side call conditioned on environment source code, which the Reasoning Agent never sees.This preserves the distinction between the hint-producing process and the agent’s observable interaction with the environment.Source: §C.2 · p. 43
  • Candidate environments undergo programmatic smoke tests, while tool-use environments additionally require deterministic reset validity and LLM screening for hard-but-feasible success criteria.Candidates that fail to parse, crash, error on reset across tested seeds, or have unreachable, pre-satisfied, or underivable criteria are rejected.Source: §C.2 · p. 43

C.3 Reproducibility

  • Training and evaluation code, every run’s configuration files, evaluation JSONs, and figure-generation scripts are released with the paper.These artifacts support reproduction of the reported experiments and visualizations.Source: §C.3 · p. 44

C.3.1 Comparability Notes for Table 2

  • Table 2 comparisons require protocol caution because reference systems use different harnesses, benchmark versions, aggregation rules, base checkpoints, training data, and budgets.The reported differences include BFCL v3 versus v4, τ 2-bench variants, and reference aggregates that are not always unweighted domain means.Source: §C.3.1 · p. 45
  • Curriculum breadth drives larger held-out gains: the full six-skill curriculum outperforms the restricted two-skill variant across the reported benchmarks.This comparison uses Qwen3-30B-A3B-Instruct-2507 in the games setting, with AIME, GPQA-Diamond, LiveCodeBench-v6, and four Reasoning-Gym categories.Source: Figure 14 · p. 45
  • All four Reasoning-Gym categories improve over the untrained Nemotron-30B-A3B-BF16 base, with gains of +9.6 in RG-Cognition and +9.3 in RG-Algorithmic.The remaining gains are +2.6 in RG-Math and +3.2 in RG-Logic; trajectories initially dip before improving.Source: D Extended Ablations · p. 45, Figure 15 · p. 46

E.1 Self-Play for LLMs … E.5 Agentic Memory Design and Self-Improving Code Systems

SPADE extends self-play and unsupervised environment design by training an LLM Environment Designer to generate executable full MDPs, using hint-based regret grounded in Reasoning Agent returns. Unlike prior synthetic-environment and memory-design systems, it co-evolves adaptive environments and agents through RL across reasoning, tool-use, and other settings.

E.1 Self-Play for LLMs

  • Prior self-play research spans classical games, LLM self-training, adversarial language games, and proposer-solver systems, but data-free methods generally generate tasks rather than stateful environments.These approaches also commonly rely on sparse rewards or heuristic measures of difficulty, learnability, uncertainty, or variance.Source: §E.1 · pp. 47–48
  • SPADE generates full executable MDP environments and trains the Environment Designer with hint-based regret grounded in measured Reasoning Agent return rather than proxy statistics.This contrasts with data-free self-play methods that generate tasks with sparse terminal rewards and use frozen generators or heuristic proxy rewards.Source: §E.1 · p. 48

E.2 Unsupervised Environment Design and Open-Endedness

  • SPADE brings regret-based UED to LLM post-training with an RL-trained designer that emits arbitrary Python environments and requires no trained antagonist.Classical UED methods operate in parameterized spaces with small, fixed design vocabularies, whereas SPADE uses hint-based regret for co-evolution.Source: §E.2 · p. 49
  • SPADE’s environment space is unbounded because the designer writes arbitrary executable code, extending open-endedness beyond fixed parameterized environment vocabularies.This code-as-environment representation supports a single interface spanning single-turn reasoning and multi-turn agentic settings.Source: §E.2 · p. 49, §E.3 · p. 51

E.3 Synthetic Environment Generation

  • SPADE differs from synthetic-environment generators by jointly RL-training its designer and agent to produce full MDPs at the Reasoning Agent’s competence frontier.Related systems often adapt through prompting heuristics, filtering, validation, or novelty signals without giving the generator RL gradients.Source: §E.3 · p. 51

E.4 Environment Scaling

  • SPADE combines gradient-trained environment design with code-as-environment representation to produce an adaptive curriculum of executable MDPs that co-evolves with the Reasoning Agent.Prior work shows that environment scale, diversity, and curriculum design improve generalization, motivating this combined approach.Source: §E.4 · pp. 51–52

E.5 Agentic Memory Design and Self-Improving Code Systems

  • SPADE complements agentic memory and self-improving code systems by generating the training environments that produce experience, while those systems optimize memory use or editable scaffolding.Its adaptive environments could populate the experience streams used by MemRL- and ALMA-style systems; HyperAgents leaves co-evolving task distributions as future work.Source: §E.5 · p. 52

F Extended Quantitative Analysis · F.1 Games · F.1.1 Environment Quality

SPADE’s games environments become more learnable and reward-granular over training while remaining well-posed, verifiable, executable, and structurally rich. Extended analyses also confirm robustness across embedding choices and identify which reasoning skills drive learning gains.

F Extended Quantitative Analysis

  • Embedding-based conclusions are identical with SBERT and TF-IDF with LSA-128 under the stated multi-embedding robustness protocol.Step-level dynamics use matched 30B-A3B runs, with full SPADE covering steps 0–399 and ablations ending earlier.Source: F Extended Quantitative Analysis · p. 53

F.1 Games

  • The games analyses extend the environment-quality and diversity signals summarized in the main text.This extended block focuses on quantitative games-setting analysis, including environment quality and related training behavior.Source: §F.1 · p. 53

F.1.1 Environment Quality

  • The analysis tracks five environment-quality signals, with four measured over training and executability evaluated once on raw generations.Learnability, well-posedness, verifiability, and richness are tracked over training, while executability is tested by re-executing generated code in a sandbox.Source: §F.1.1 · p. 53
  • Environment richness stays near 320 lines, 13 hidden state variables, and 8 to 10 interaction turns, while reward levels increase from 3.7 to 5.8.Strictly partial reward levels increase from 2.2 to 4.0, indicating finer-grained feedback rather than trivial programs.Source: §F.1.1 · p. 53, Figure 16 · p. 54
  • Raw generated environments execute successfully in 84.9% of runs, rising to 90.3% after Markdown-fence stripping and 100% after sanitizer filtering.The dominant residual failure is a 7.4% f-string brace-escaping bug in \boxed{} action templates.Source: §F.1.1 · p. 53
  • Learnable environments rise from 0.16 to 0.31 while Reasoning Agent win rate increases from 0.30 to 0.62, showing sharper frontier targeting without simpler environments.Well-posedness remains 97 to 98% and verifiability 90 to 93%; Table 7 attributes quality gains to difficulty targeting rather than simplification.Source: Table 7 · p. 53, §F.1.1 · p. 53
  • Per-skill gains are largest for Logical Deduction and Spatial Reasoning, whereas Causal Inference declines slightly despite starting high.Figure 17 decomposes the canonical SPADE-30B run’s Reasoning Agent win-rate changes over training.Source: Figure 17 · p. 54, §F.1.1 · p. 54

F.1.2 Environment Diversity

SPADE maintains near-ceiling, stationary environment diversity throughout training, with corpus grounding identified as the decisive factor. Removing corpus grounding causes diversity to collapse despite temporary recovery under RL exploration, while memory and Environment Designer training do not materially drive diversity.

Training dynamics

  • Per-step diversity remains near the practical ceiling, with Vendi increasing from 20.8 to 21.0, pairwise cosine distance fixed at 0.94, and novelty present in 97 of 100 logged steps.Mixed batches have a calibrated ceiling of 21.3 for batch size 24.Source: §F.1.2 · p. 54

Domain coverage

  • All 13 taxonomy domains appear from step 0, averaging 8.4 domains per batch, while a temporal linear probe achieves only 0.551 ± 0.024 AUC after seed-document reuse is removed.The batch composition includes Mathematics at 30%, Physics at 20%, Medicine at 11%, Chemistry at 10%, and CS at 7%.Source: §F.1.2 · p. 55

Scope and limitations

  • Diversity is measured at the step level rather than per environment, with limitations from missing scalar rewards, rubric mismatch, raw-generation executability, and batch-size-24 comparability.The LLM rubric measures absolute difficulty and coherence, while executability is assessed before sanitization.Source: §F.1.2 · p. 55

Corpus ablations

  • Corpus grounding preserves diversity at the mixed-population reference level, whereas removing it collapses Vendi to 0.04 and later re-collapses after a temporary peak near step 100.The no-corpus run peaks at Vendi ∼5.2 around step 100, coinciding with its best evaluation checkpoint at step 111; corpus-grounded runs remain at the ceiling for all 400 steps.Source: §F.1.2 · p. 55, Table 8 · p. 55, Figure 18 · p. 56

G Extended Qualitative Analysis · G.1 Games · G.1.1 Additional Privileged-Hint Examples

The qualitative analysis of SPADE-30B finds that generated environments become more learnable and remain diverse through corpus grounding, while games examples illustrate privileged hints across the regret range. It also verifies that the analyzed environments are distinct programs grounded in diverse documents and initial states.

G Extended Qualitative Analysis

  • The learnable-environment share nearly doubles from 0.16 to 0.31 while well-posedness, verifiability, and program richness remain constant.This result comes from analyzing all 3,310 environments generated by the canonical SPADE-30B run.Source: G Extended Qualitative Analysis · p. 57
  • Semantic diversity stays at the mixed-population ceiling throughout 400 training steps, while the stream continues producing never-seen environments through the end.Difficulty targeting improves within an unchanged task distribution, sharpening the curriculum without reducing its breadth.Source: G Extended Qualitative Analysis · p. 57
  • Removing corpus grounding collapses generation to one task family, reducing Vendi/n from 0.68 to 0.04, whereas freezing Designer training or memory leaves diversity intact.These findings identify corpus grounding as the source of diversity while separating it from the effects of Designer training and accumulated memory.Source: G Extended Qualitative Analysis · p. 57
  • All 3,310 generated environments are distinct programs, with 2,388 distinct initial states and 1,513 distinct seed documents.The analysis uses content rather than surface identifiers; 585 class names appear, and 58% retain the scaffold default despite the naming instruction.Source: G Extended Qualitative Analysis · p. 57

G.1 Games

  • The games analysis supplies additional privileged-hint examples and traces one environment’s evolution from early to late training.This extends the qualitative analysis beyond aggregate environment statistics to game-level behavior.Source: §G.1 · p. 57

G.1.1 Additional Privileged-Hint Examples

  • Figure 19 presents four positive-regret task–hint pairs from the canonical 30B games run, spanning large frontier gaps to a small mastery-regime gap.Each pair comes from one logged game record, with the task summarized from the reset observation and the hint excerpted verbatim from substantive guidance.Source: §G.1.1 · p. 57, Figure 19 · p. 58
  • Figure 19 reports mean returns without hint and with hint for each same-record pair, making hint utility legible from task context.Task summaries preserve the goal, hidden information, and usable interaction, while hint excerpts remove only standardized formatting and runtime-wrapper text.Source: Figure 19 · p. 58

G.1.2 Environment Evolution over Training … G.3.3 Full Source of a Generated Tool-Use Environment

This section presents generated environments as executable, multi-turn tasks with structured state transitions, branching actions, explicit verification, and reward-based termination. It illustrates these designs through corpus-grounded examples spanning games, tool-use workflows, inheritance and auto-loan decisions, thermodynamic cycles, and customer-support ticket management.

G.1.2 Environment Evolution over Training

  • Late environments shift toward structured, instrumented laboratory tasks, while program size, functional hidden state, and verifiability hold steady; added difficulty comes from state-gated interaction and finer reward grading.At training step 0, CarOwnershipDisputeEnv has 370 lines and 29 state variables.Source: §G.1.2 · p. 58

G.1.3 Matched-Step Generation Contrast

  • SPADE with corpus grounding generates diverse domains, whereas the no-corpus run repeats the same rotating-maze family across 41 accepted generations.The corpus-grounded batch contains 24 distinct programs, including probability, quantum tomography, volcanology, hematology, radar, and operator tasks.Source: §G.1.3 · pp. 59–60

G.2 Tool Use

  • Tool-use environments simulate everyday API domains through atomic instructions that advance only when hidden-state criteria confirm completion.The criteria check salient state effects, but may leave textual qualifiers—such as which account pays an invoice—unbound.Source: §G.2 · p. 60
  • The tool-use scaffold remains structurally stationary across training, while fresh states, identifiers, and instruction sequences are regenerated across stable API domains.Mean program length changes from 247 to 226 lines and tools per environment from 7.6 to 5.9, while instructions remain 4.7 and criterion complexity stays near 1.9–2.0 conditions.Source: §G.2 · p. 61

G.3 Generated Environment Gallery

  • The generated-environment gallery presents representative outputs in the exact format emitted by the Environment Designer, beginning with a minimal multi-turn example.The gallery demonstrates the same format across generated environment examples.Source: §G.3 · p. 61

G.3.1 A Minimal Generated Environment

  • A minimal Wordle-style environment implements Gym-style reset()/step(), stateful turns, intermediate feedback, and terminal rewards of 1 for success or 0 at truncation.The episode ends after a correct guess or six turns, with the target sampled during reset.Source: §G.3.1 · p. 61

G.3.2 Full Source of Two Exemplar Generated Environments

  • The full exemplar environments are self-contained programs with hidden reset state, multi-turn feedback, and verifiable terminal rewards rather than summed intermediate rewards.The car-dispute example requires multiple actions to verify rights, assess the will, consider refinancing, or transfer title; episode return is clipped to [−1, 1].Source: §G.3.2 · p. 62
  • The environment branches on inheritance and loan conditions, including death clauses, wills, refinancing, title ownership, and whether the grandfather remains available.Different states determine whether the borrower faces immediate repayment, can secure the car, or must rely on a will or state law.Source: §G.3.2 · pp. 63–64
  • The environment exposes state-dependent actions such as consulting a lawyer, transferring title, researching refinancing, checking title status, and verifying liens.Available actions change after dialogue and document-related events, including will confirmation, title transfer, refinancing, and legal consultation.Source: §G.3.2 · pp. 63–65
  • The generated dialogue encodes concrete outcomes, including tax or probate risks, refinancing eligibility limits, and legal advice about joint ownership and lifetime transfer.The environment reports that gifts may trigger tax or probate issues, refinancing can fail because of credit or underwater loans, and a lawyer recommends transferring title while the owner is alive.Source: §G.3.2 · pp. 63–65
  • The car-title environment branches on refinancing, liens, wills, and state ownership laws, requiring sequential actions before awarding reward 1.0 for securing the vehicle.Its solution procedure includes consulting the grandfather or lawyer, checking title status, transferring the title, releasing liens, and confirming sole ownership.Source: §G.3.2 · pp. 65, 71
  • The thermodynamics environment hides system parameters and the target entropy change while exposing multi-turn actions for heating, expansion, cooling, and entropy measurement.It models a monatomic ideal gas through a three-step cycle and requires the agent to discover the zero net entropy change through manipulation.Source: §G.3.2 · pp. 66–67
  • The environment guides the agent through a complete thermodynamic cycle whose three phases heat at constant volume, expand isothermally, and cool at constant pressure.The implementation transitions between phase-specific states and exposes only the next valid actions after each process.Source: §G.3.2 · pp. 67–69
  • Each command updates physical state variables and returns observations containing the current temperature, volume, pressure, entropy change, and available next actions.The environment supports measuring cumulative entropy and scanning hidden parameters without revealing the full solution.Source: §G.3.2 · pp. 67–69
  • The verifier records phase-wise entropy changes using distinct formulas for constant-volume heating, isothermal expansion, and constant-pressure cooling, including a negative cooling contribution.The implementation appends each change to an entropy history and reports the cumulative total during the interaction.Source: §G.3.2 · pp. 67–69
  • The thermodynamic environment assigns partial credit for incomplete progress, including reward = 0.95 when the cycle is nearly complete and reward = 0.7 when cooling finishes before truncation.Incomplete or time-limited episodes explicitly report that the cycle was not fully verified.Source: §G.3.2 · pp. 70–71
  • The thermodynamic exemplar requires heating, expansion, cooling, and entropy measurement in sequence, with success verified when the net entropy change is effectively zero.The environment exposes diagnostic actions, accepts a small floating-point deviation, and returns a reference solution sequence for completing the cycle.Source: §G.3.2 · pp. 70–71

G.3.3 Full Source of a Generated Tool-Use Environment

  • The generated CustomerSupportTicketWorkflowEnv subclasses ToolUseBaseEnv and defines reset(), while the harness dispatches OpenAI-format calls, replays criteria after each call, and terminates after all criteria pass and a final answer is submitted.The excerpt identifies this as a 200-line environment generated at training step 392 of the 30B tool-use run.Source: §G.3.3 · p. 71
  • The CRM environment initializes tickets, agent availability, and mutable state for customer-support workflows.State tracks ticket assignments, statuses, notes, and recent tool-action outcomes.Source: §G.3.3 · p. 71
  • It exposes tools to search tickets, update statuses, assign available agents, add notes, resolve tickets, and list assignments.Assignment rejects missing or unavailable agents, while other tools update matching ticket records.Source: §G.3.3 · pp. 71–72
  • The environment advances through multi-turn instructions and verifies each criterion before requiring the final answer <answer>done</answer>.The prescribed sequence finds high-priority new tickets, assigns TICKET-001, adds a note, changes it to in_progress, and resolves it.Source: §G.3.3 · p. 72

Related papers

Loading 2608.19197v1…