Open-Ended Learning Leads to Generally Capable Agents

• Unlike conventional agents that often catastrophically fail when goals change, the learned policy develops general heuristics such as directed experimentation, information seeking, object-related tagging, and emerging cooperation.These behaviours are applicable across tasks rather than optimized for one specific game.Source: 1| Introduction · p. 2
• The agent scores reward in all humanly solvable held-out tasks and generalizes zero-shot to tasks far outside training distributions, including hide-and-seek and capture-the-flag variants.The evaluation includes games explicitly held out from training and tasks beyond the training distributions.Source: 1| Introduction · p. 2
• XLand procedurally generates rich 3D multiplayer worlds and games spanning competitive, cooperative, mixed, balanced, imbalanced, and strategically deep settings.The resulting tasks require visual understanding, navigation, physical manipulation, memory, logical reasoning, and theory of mind.Source: 1| Introduction · p. 2, Figure 2 · p. 2
• The open-ended training system uses normalized score percentiles, attention-based goal modeling, dynamically generated tasks, and sequential population training with policy distillation.Task generation responds to agent performance, while successive generations redefine the performance frontier and contribute new policies for multiplayer training.Source: 1| Introduction · p. 2
• Finetuning on a new task for 100 million steps, about 30 minutes of compute, produces drastic gains over zero-shot performance and often succeeds where training from scratch fails completely.This demonstrates that the learned general capabilities can support efficient transfer to new tasks.Source: 1| Introduction · p. 2

2| XLand Environment Space

• XLand combines fixed control, observation, gadget, movement, and physical interfaces with smoothly variable layouts, objects, lighting, rewards, and co-player policies in a multi-agent 3D simulation.Players receive first-person pixel and proprioceptive observations, act simultaneously, and receive binary rewards when their goals are satisfied; episodes last T=900 iterations.Source: 2| XLand Environment Space · pp. 3–4

2.1| World Space

• Worlds are procedurally generated initial physical states whose topology, dynamic objects, player positions, lighting, and gadgets create challenges such as navigation, search, interception, and manipulation.Topologies use connected arrangements of predefined tiles; objects vary in location, orientation, shape, colour, and size, while players can use freeze or tagging gadgets.Source: 2.1| World Space · p. 4

2.2| Game Space

• Games assign each player a goal expressed through Boolean combinations of physical predicates, enabling a smooth space from simple object-finding tasks to strategically deep games such as hide and seek.A task combines one world, one game, and co-player policies; their interactions can require different optimal behaviours, making the resulting challenge space larger than the sum of its components.Source: 2.2| Game Space · p. 5

3.1| World Properties

• Worlds are represented by height maps and topology graphs whose directed edges encode straight-line movement, falling, or ramp traversal between neighboring tiles.Navigational complexity is characterized through shortest-path and resistance-distance distributions over graph vertices.Source: 3.1| World Properties · pp. 5–6

3.1.1| World Vastness

• More than 10^16 unique 9×9 world topologies satisfy connected-ramp and 50%-accessibility constraints, establishing a vast space of worlds.The estimate is corrected for eight possible symmetries and obtained through Monte Carlo sampling.Source: 3.1.1| World Vastness · p. 6

3.1.2| World Smoothness

• Small local changes in world topography produce bounded changes in shortest-path and resistance-distance distributions, suggesting L-Lipschitz world smoothness.The relationship is evaluated using Cauchy-Schwarz divergence between the corresponding path distributions.Source: 3.1.2| World Smoothness · p. 6
• Exploration difficulty measures predicate states with no reward, while cooperativeness and competitiveness quantify joint versus partial reward states across players.Goals with κ(g) = 0 or κ(g) = 1 make every policy optimal; balance extends cooperation under permitted goal transformations.Source: 3.1.2| World Smoothness · p. 8

3.2.1| Game Vastness

• More than 10^37 unique two-player games arise from 200 atomic predicates, three options, and three-predicate conjunctions, alongside more than 10^18 goals.This bound assumes that non-negated atomic predicates are independently solvable.Source: 3.2.1| Game Vastness · p. 8

3.2.2| Game Smoothness

• Exploration difficulty is 1-Lipschitz, while cooperativeness and competitiveness have bounded Lipschitz changes when one goal is modified.Figure 9 empirically verifies smooth transitions in these game properties under simple goal edits.Source: 3.2.2| Game Smoothness · pp. 8–9

3.2.3| Game Diversity

• Game diversity is guaranteed by constructing nearby two-player games with any feasible desired change in competitiveness.For target change m, the construction satisfies comp(G′) = comp(G) + m and distance at most κ̂(G)|m|.Source: 3.2.3| Game Diversity · p. 9

Hide and Seek

• The game space spans navigation, fully cooperative coordination, fully competitive hiding and seeking, capture-the-flag competition, and cyclic XRPS strategy.These examples vary in navigation, manipulation, synchronization, opponent modeling, and time-extended counter-strategy demands.Source: 3.2.3| Game Diversity · p. 9, Hide and Seek · pp. 9–10

4| Goal and Metric

• Expected return and simple averaging are inadequate because task complexities and optimal-value scales differ, while infimum performance provides no learning signal for impossible or extremely hard tasks.The goal is therefore a generally capable policy rather than performance on a particular test distribution.Source: 4| Goal and Metric · p. 10

task space.

• The metric normalizes each task using an iteratively improved Nash-equilibrium estimate, then requires Pareto dominance across normalized-score percentiles to certify progress.This addresses incomparable task scales and favors fewer catastrophic failures, broader competence, and broad ability over narrow expertise.Source: 4| Goal and Metric · p. 10, task space · pp. 11–12
• The percentile descriptor represents performance at normalized-score percentiles from 0 through 50, exposing hardest-task performance, median competence, and reward coverage.An agent is better only when it is at least as good at every percentile and strictly better at one, making broader ability preferable to narrow expertise.Source: task space · p. 11

4.2| Evaluation Task Set

• Evaluation tasks uniformly cover representative world topology, game competitiveness and balance, game complexity, and pretrained co-player conditions rather than sampling XLand arbitrarily.The test set contains 1,678 world-game pairs played with seven co-players, totaling 11,746 tasks; validation holds out nearby worlds, games, and co-players.Source: 4.2| Evaluation Task Set · p. 12

4.3| Hand-authored Task Set

• The hand-authored set adds 42 held-out, interpretable and often out-of-distribution tasks, including Capture the Flag, Hide and Seek, King of the Hill, Stop Rolling, and Tool Use.These tasks address the difficulty of interpreting automatically generated successes and test generalization to rare challenges.Source: 4.3| Hand-authored Task Set · pp. 12–13, Figure 11 · p. 13

• The process targets agents that can zero-shot generalise to XLand test-set tasks, measured using normalised percentiles.Source: 5| Learning Process · p. 13
• Training consists of three main components, though only the first is specified here.Source: 5| Learning Process · p. 13
• Deep RL updates a single agent’s neural network to maximise expected return across a distribution of training tasks.Source: 5| Learning Process · p. 13

2. Dynamic task generation

• Task distributions change throughout training and are optimized with population-based training to improve agents’ normalized validation percentiles across generations.Each generation uses different learning objectives and bootstraps behavior from the preceding population.Source: §2 · p. 13

5.1| Deep Reinforcement Learning

• The agent uses recurrent deep reinforcement learning with RGB, proprioceptive, and goal inputs to produce values and policies for each task.The policy is trained with V-MPO, while PopArt normalizes the value function for each task.Source: 5.1| Deep Reinforcement Learning · p. 13
• The GOAT architecture attends to goal options and switches to an option representation when its predicted value exceeds the current full-goal estimate.This enforces consistency with the game space’s value-consistency property, while auxiliary predicate prediction shapes internal representations.Source: 5.1| Deep Reinforcement Learning · p. 14

5.2| Dynamic Task Generation

• Dynamic task generation addresses uninformative tasks by changing the training distribution in response to the agent’s policy and training on an instantaneous task distribution.Tasks are filtered from a proposal distribution using tests of usefulness at the current learning stage.Source: 5.2| Dynamic Task Generation · p. 15
• A proposed task is accepted when the agent is unlikely to score highly yet is likely to outperform a control policy, typically uniform random actions.Comparing against the control policy is intended to ensure that the agent’s actions meaningfully affect task returns.Source: 5.2| Dynamic Task Generation · p. 15

3. The control policy is not performing well

• Dynamic filtering removes tasks that are too easy, too hard, or already sufficiently solved, while population-based training adapts these criteria and selects Pareto-dominant agents.Agents have individual task distributions and hyperparameters; periodic dominance transfers weights, task distributions, and mutated hyperparameters to the weaker agent’s population slot.Source: §3 · p. 15

5.3| Generational training

• Generational training distills each population’s best agent into the next generation, expands the co-player pool, and changes objectives across generations.Early generations use self reward-play to maximize participation, while later generations optimize raw environment reward and normalized percentiles.Source: 5.3| Generational training · p. 16
• Self reward-play significantly increases participation during equal training time by rewarding agents for alternating between satisfying a goal and its negation.It improves exploration but does not make agents competent, leaving normalized percentiles low by construction.Source: 5.3| Generational training · p. 16

5.4| Combined Learning Process

• Deep reinforcement learning, dynamic task generation, and generational training operate on nested timescales to drive open-ended improvement in task coverage and competence.The combined process is limited by the expressivity of the environment and the agent’s neural network rather than a bounded optimization metric.Source: 5.4| Combined Learning Process · pp. 16, 19

6.2| Agent Training

• 95% participation, 82% 10th-percentile and 112% median normalized test scores, plus 585 average return, were achieved by the final generation-5 agent.The 585 average return was at least 65% of the optimal policy value on hand-authored tasks.Source: 6.2| Agent Training · p. 20

6.2.1| Dynamic Task Generation Evolution

• Dynamic task generation emergently shifted training toward stronger opponents, more competitive games, fewer options, and tasks requiring more initial predicate changes.These changes created increasingly difficult training problems without direct control over the task properties.Source: 6.2.1| Dynamic Task Generation Evolution · p. 21

6.2.2| Ablation Studies

• Ablations show that early self reward-play and dynamic task generation are necessary for learning, while later PBT, GOAT, and dynamic task generation each materially improve performance.Removing all early methods prevented meaningful learning; later removals reduced best-agent performance by approximately 20%, 30%, and 65%, respectively.Source: 6.2.2| Ablation Studies · p. 21
• Generation-based training outperformed the previous generation as training progressed, for both best-agent and population-average comparisons.Generation 5 surpassed generation 4 even when generation 4 received the same training duration.Source: 6.2.2| Ablation Studies · p. 21

6.3.1| Coverage

• After excluding physically or procedurally impossible tasks, the final agent participated in every held-out test task.Failures involved making the co-player hold objects, placing it on specific floors, or a rare physics bug spawning an object outside the reachable region.Source: 6.3.1| Coverage · p. 22, Figure 18 · p. 22

6.3.2| Relative Performance

• The agent improved most relative to evaluation policies on highly competitive, multi-predicate, high-atom-change, few-option, and object-interaction games.Overall, improvements were greatest in the most challenging parts of the game space.Source: 6.3.2| Relative Performance · p. 22

6.4| General Capabilities

• On hand-authored tasks, compared with final generation 4, final generation 5 showed better navigation, more consistent hilltop access, continuous pursuit, and strategic countering by stealing the yellow sphere.Source: 6.4| General Capabilities · pp. 23–24
• Probe case studies revealed emergent option switching, tool use, and within-episode experimentation with visual verification despite unseen task configurations.The agent selected newly valuable options, constructed a temporary ramp, and repeatedly reshuffled objects until discovering and maintaining a rewarding arrangement.Source: 6.4| General Capabilities · pp. 25–26
• Across social-dilemma evaluations, agents increasingly avoid conflict and cooperate with equally trained copies, while hide-and-seek performance improves through alternating seeker and hider strategies.These behaviours emerge without direct training against checkpoints or on hide-and-seek itself, suggesting transfer from the broader task distribution.Source: 6.4| General Capabilities · p. 27, Conflict Avoidance · p. 27, learning process · p. 27

6.4.4| Goal Interventions

• After a mid-episode goal change, the agent achieves almost the same normalised score as when starting with a fixed goal, demonstrating robust on-the-fly adaptation.The evaluation used 500 single-option, one-predicate test tasks and compared changed-goal performance with state-reset performance.Source: 6.4.4| Goal Interventions · p. 28, Figure 26 · p. 28

6.4.5| Failed Hand-authored Tasks

• The agent fails hand-authored tasks requiring escape from chasms, multiple ramps, or theory-of-mind-guided co-player manipulation, revealing assumptions that agents cannot become trapped and limited behavioural experimentation.It receives no reward on some such tasks, including following tasks where success requires coaxing a co-player to a target floor.Source: 6.4.5| Failed Hand-authored Tasks · p. 28

6.5| Finetuning for Transfer

• 340% average reward improvement after 100 million finetuning steps raised performance from 39% zero-shot to 77%, while training from scratch failed on most tasks.The 100-million-step budget corresponds to approximately 30 minutes, and finetuning also solved Make Follow Easy, which failed zero-shot.Source: 6.5| Finetuning for Transfer · pp. 28–29, Figure 27 · p. 29

6.6| Representation analysis

• Kohonen analysis shows time and object possession in LSTM and GOAT representations but not goal embeddings, while rewarding states and relevant predicates are represented despite reward being absent at inference.The agent’s world understanding is focused on the currently pursued option; unrelated goal predicates are not represented across the examined modules.Source: 6.6| Representation analysis · pp. 29–30, Figure 28 · p. 30
• Representations also encode the number of missing predicates and exact game structure, identify several concepts with BAC above 75%, and preserve value consistency about 90% of the time.Information about missing predicates is preserved in GOAT but becomes less differentiated, while exact game details remain in goal embeddings and value consistency concerns multi-option goals.Source: 6.6| Representation analysis · pp. 30–31, Figure 29 · p. 31

Multi-agent and progressive learning

• The work differs from prior multi-agent and progressive-learning methods by using generations as self-contained reinforcement-learning units whose objectives can change across a massive space of games.It also relates to autocurricula, multi-task knowledge sharing, contextual games, Boolean goal structure, auxiliary losses, and social-dilemma analysis.Source: 7| Related Work · p. 31

Iterative improvement and percentiles

• Its iterative-improvement framework extends ranking and self-play ideas to worlds, games, and co-players while handling incomparable task rewards through distribution-level percentiles.Percentile-based performance curves are used to characterize general capability, drawing inspiration from robust reinforcement learning and ROC curves.Source: 7| Related Work · p. 32

Procedural environment generation

• Procedural-generation research has evolved environments through game-rule evolution, GANs, evolutionary search, agent-driven editing, and difficulty adaptation, with this work likewise using success measures to influence task generation.Examples include generated platformer and Minecraft levels, CPPN2GAN, and PCGRL.Source: 7| Related Work · p. 32

Curriculum over tasks

• The work’s dynamic task generation and world-agent co-evolution are forms of automated curriculum learning that adapt training distributions using learning progress, performance, replay priorities, information gain, or unsolved-task selection.Unlike many heuristic curricula, its heuristic parameters are adapted with PBT.Source: 7| Related Work · p. 32

Curriculum over goals

• Prior goal-conditioned curricula generate or prioritize goals from achieved observations, learning progress, uncertainty, difficulty, or adversarial setters, while this work similarly uses setters and solver performance to shape goal distributions.Its self reward-play modification is related to asymmetric self-play but uses the same agent for both sequential roles within one episode.Source: 7| Related Work · pp. 32–33

World-agent co-evolution

• World-agent co-evolution resembles POET and PAIRED in dynamically generating high-dimensional worlds, but the proposed procedure is simpler because it trains one agent and filters levels by estimated success probability.The resulting process creates the base training and evaluation distribution and exemplifies AI-generating algorithms.Source: 7| Related Work · p. 33

• XLand provides a vast, diverse, smooth environment space of procedurally generated worlds and multiplayer games for training and evaluating generally capable agents.General capability is characterised using an iteratively revised metric of normalised score percentiles on an evaluation task set.Source: 8| Conclusions · p. 33
• Open-ended learning dynamically generates training tasks from agent behaviour, sequentially distilling each population from the previous generation while revising the improvement metric.The process uses deep reinforcement learning to iteratively improve the frontier of normalised score percentiles.Source: 8| Conclusions · p. 33
• Agents show broad ability across held-out tasks, catastrophically failing on only a small fraction deemed humanly impossible.The agents’ emergent behaviours include experimentation, success recognition, and greater cooperation with other competent agents, generalising to out-of-distribution probes.Source: 8| Conclusions · p. 33
• Analysed internal representations clearly encode the structure and state of the goals agents are tasked to follow.Source: 8| Conclusions · p. 33
• Pretrained agents can be quickly finetuned for complex out-of-distribution tasks, extending their demonstrated zero-shot generalisation.The results suggest that agents can acquire general capabilities across vast task spaces without human demonstrations.Source: 8| Conclusions · p. 33

A.1| Worlds

• Topology generation yields varied coherent convex structures through tile connectivity, random geometric scaling, lighting, and occasional reflections.The playable area is the largest connected component of exposed floor, and generated worlds can include ramps, height variation, and diagonal floor pieces.Source: A.1.1| Procedural World Generation · p. 40

A.1.1| Procedural World Generation

• Objects are sampled within the playable area using nonlinear CPPN probability maps, with random size, colour, shape, and optional clustering by visual attributes.The generator supports black, purple, or yellow objects shaped as cubes, spheres, pyramids, or slabs, placed independently or in spatial groups.Source: A.1.1| Procedural World Generation · p. 40

A| Appendix

• Worlds can be conditioned to resemble an existing world by biasing topology and perturbing object and player latent variables while resampling other categorical features.This enables smoothly varying worlds and supports evolutionary generation from parent worlds.Source: A.1.1| Procedural World Generation · p. 40

A.1| Worlds

• Game-conditioned generation modifies a world so every player, object, and topological element referenced by a specified game is present and exposed in the playable area.The transformation is defined as wG = f(w, G), producing a world suitable for game G.Source: A.1.1| Procedural World Generation · p. 40

A.1.1| Procedural World Generation

• World–agent co-evolution maintains dynamic training and evaluation sets, admitting mutated worlds when the agent succeeds in at least one but fewer than half of 100 evaluation episodes.Worlds that no longer satisfy this criterion across their latest 100 episodes are discarded from training, keeping challenges neither trivial nor impossible.Source: A.1.1| Procedural World Generation · pp. 40–41

A| Appendix

• As navigation improves, the co-evolving training distribution becomes more complex, producing worlds with long paths, forks, narrow routes, and hidden goals.The process begins from an open-room training world and evolves worlds toward increasingly challenging but learnable navigation problems.Source: A.1.1| Procedural World Generation · pp. 40–41

A.1.1| Procedural World Generation

• Procedural generation combines Wave Function Collapse topology with CPPN-based object and player spawn maps in Unity to produce playable 3D worlds.WFC assigns connected geometric tiles under local constraints, while CPPNs determine entity placement from spatial and topological features plus latent variables.Source: A.1.1| Procedural World Generation · p. 40, Figure 32 · p. 41

Counting worlds

• Worlds are counted when they contain a region A occupying at least wh/2 tiles, connecting every pair of points, and having no path leading outside.These conditions prevent irreversible decisions from leaving the region.Source: A.1.2| Counting worlds · p. 41
• The trivial upper bound is 6^(w·h)·(1+4+4)^(w·h), from six flat-tile levels, four ramp orientations, and four diagonal-tile orientations.Source: A.1.2| Counting worlds · p. 41
• A lower bound starts from a ceil(w/2)×ceil(h/2) subgrid whose floor-level graph has one strongly connected component, then embeds it into the full grid.Bidirectional edges become ramps, one-directional edges become flat tiles, and remaining tiles receive the highest floor; the resulting region A occupies at least 75% of the world while preserving connectivity and preventing exits.Source: A.1.2| Counting worlds · p. 41, Figure 34 · p. 42
• Exact counting is approximated by enumerating subgrids and estimating the probability of obtaining exactly one strongly connected component through Monte Carlo sampling of 200,000 worlds.The analysis sets n=w=h and accounts for less-constrained boundary rows or columns when dimensions are even.Source: A.1.2| Counting worlds · p. 42

A.1.3| Worlds linear projection

• The learned projections show checkerboard-like topology patterns that correspond to navigational complexity and shortest-path distributions.The figure displays projection dimensions and their signs, with dotted lines indicating the effective size of learnable parameters.Source: Figure 35 · p. 43, A.1.3| Worlds linear projection · p. 43
• The method learns a 2D linear projection whose distance from a learned bias predicts a target world property, using gradient descent for 30,000 iterations.The projection uses learning rate 0.1 and includes learned world weights, bias, and scalar offset.Source: A.1.3| Worlds linear projection · pp. 42–43
• The analysis applies the projection to the entropy properties H2(ρ(w)) and H2(ρsp(w)), which satisfy the positivity assumption.Entropy is always positive, so these properties meet the method’s stated requirements.Source: A.1.3| Worlds linear projection · p. 43
• Rotational and symmetry invariance reduces the projection’s trainable weights from 81 to 15, significantly reducing overfitting.The 9 by 9 world-shaped parameterization averages over all 8 rotations/symmetries.Source: A.1.3| Worlds linear projection · p. 43

Games

• The current XLand game space is not constrained to its listed relations, predicates, number of players, or options and can be expanded further.These technical details describe the present instance rather than the limits of the overall system.Source: A.2| Games · p. 43

Relations

• near(a,b) is true when object a is at most 1 meter from object b.The threshold is calibrated against a player avatar height of 1.65m.Source: A.2.1| Relations · p. 43
• Uniform random sampling misses extreme competitiveness–balance combinations, whereas local search readily populates the full range, including extremes.The comparison covers 1,000 games with 1, 2, and 3 options and shows a strong correlation between balance and competitiveness under uniform sampling.Source: Figure 36 · p. 43
• Proximity is computed between object surfaces using center-of-mass geometry, rather than merely measuring distances between centers.The procedure finds closest surface points through a midpoint construction and is relatively cheap to compute.Source: A.2.1| Relations · p. 43
• see(a,b) tests unobstructed center-of-mass visibility for nonplayers and field-of-view rendering for players.For players, the relation is true exactly when b would appear in a’s field of view.Source: A.2.1| Relations · p. 43
• on(a,b) denotes contact between a player or object and the upper surface of a specified floor colour.This relation is restricted to floor–player or floor–object pairs.Source: A.2.1| Relations · p. 43
• hold(a,b) denotes a player holding an object with its beam, including the possibility that two agents hold the same object simultaneously.The relation is restricted to player–object pairs.Source: A.2.1| Relations · p. 43

A.2.3| Generating games

• Naively sampling predicates and option matrices produces a highly non-uniform distribution, making fully competitive games almost impossible to obtain.Games are therefore generated with a greedy local search targeting specified balance and competitiveness values.Source: A.2.3| Generating games · p. 44
• The generator repeats local-search construction 10 times and selects the game closest to the target balance and competitiveness, although exact targets may be impossible.For example, no 1-option, 1-predicate game has competitiveness 0.25 and balance 0.75.Source: A.2.3| Generating games · p. 44

A.2.4| Creating alike games

• Bijective object recolourings create alike games with identical option counts, conjunctions, balance, and competitiveness, but the process may fail to produce a distinct game.The procedure retries until a new game is found or 100 unsuccessful attempts occur; games without objects, such as hide and seek, cannot yield new variants this way.Source: A.2.4| Creating alike games · p. 44

A.2.5| Generation of a 3 player game

• Three-player games are formed from a two-player game by mixing options from both goals, independently negating predicates with 50% probability, and randomly permuting agents and goals.Source: A.2.5| Generation of a 3 player game · p. 44

A.2.2| Atomic predicates

• XLand contains 212 unique atomic predicates relating 2 players, 9 movable objects, and 5 floors through 4 relations, after accounting for logical symmetries.The slab object is never used in atomic predicates.Source: A.2.2| Atomic predicates · p. 43, Table 2

A.2.6| PCA projection

• Games are embedded as fixed-length predicate-valuation vectors over goals, enabling kernelised analysis such as Kernel PCA without explicitly constructing the over-6 · 10^65-dimensional game space.The resulting game-space norm corresponds to Euclidean distance in the induced representation.Source: A.2.6| PCA projection · p. 46

A.3| Holding out tasks from training

• Hold-out games and worlds are deduplicated, split into validation and test sets, and used to reject colliding training samples, so neither set shares a game, world, or task with the other or with dynamic training.Source: A.3| Holding out tasks from training · p. 46

A.4| Reinforcement Learning

• Reinforcement learning uses V-MPO with batch size 64 and unroll length 80, plus on-policy distillation masked to non-rewarding states for exploration assistance.The policy observes features and its goal but not reward; only the value head receives reward during training.Source: A.4| Reinforcement Learning · pp. 46–47

A.6| Network architecture

• The network combines a ResNet visual torso, recurrent LSTM state, structured goal embeddings, atom prediction, and GOAT attention before six policy heads.Goal embeddings aggregate predicates and options compositionally; GOAT uses attention over the recurrent state and goal representation, while policy outputs are not conditioned on reward.Source: A.6| Network architecture · p. 47
• The policy factorizes six conditionally independent action groups, efficiently representing all 2100 joint actions.The action space contains six discrete dimensions, and every combination is feasible.Source: Table 5 · p. 47, A.6| Network architecture · p. 48

A.6.1| Auxiliary losses

• Auxiliary losses are added jointly with reinforcement-learning updates without tuned weights, including GOAT consistency, atom prediction, and external-value alignment.The external and internal value functions co-distill through an L2 alignment loss, while atom prediction uses multilabel cross-entropy.Source: A.6.1| Auxiliary losses · p. 48, GOAT · p. 48

A.7| Population Based Training

• Population-based training evolves agents by periodically copying an eligible parent’s weights and independently mutating hyperparameters, while inheriting the parent’s task set.Evolution checks begin after 5e8 steps and recur every 1e8 steps.Source: A.7| Population Based Training · p. 48

A.8| GOAT

• Placing GOAT after recurrence enables counterfactual goal queries without redundant trajectory unrolling and separates goal-invariant recurrent reasoning from goal-conditioned processing.The authors argue this placement supports reacting to changed goals mid-episode, despite no direct training for that behavior.Source: A.8| GOAT · pp. 48–49

A.9| Multi-agent analysis

• The evaluation examines multi-agent behavior with neural-network checkpoints, 1000-world matchup tests, 1000-episode matchup experiments, and Chicken social dilemmas encoded using task predicates.Source: A.9| Multi-agent analysis · p. 49, A.9.1| Hide and seek · p. 49, A.9.2| Conflict Avoidance · p. 49

Properties)

• For goals with κ(g)=0 or κ(g)=1, every policy is optimal because the reward is constant across states.The resulting value is Tz for every policy when κ(g)=z∈{0,1}.Source: Properties) · p. 53
• Changing one predicate state between games moves the game representation by 1/(2N_phi), while the construction preserves the number of rewarding predicate states.The proof toggles the state’s reward status in a copy of the second game, depending on whether it was already rewarding.Source: Properties) · pp. 53–54
• For any two-player game G with kappa-hat(G) = k and feasible competitiveness change m satisfying k|m| in N, there exists G' with comp(G') = comp(G) + m and ||G - G'||_G <= k|m|.The proof constructs G' by changing predicate states while preserving the relevant count of rewarding states.Source: Properties) · p. 54

cess)

• Removing the kth option from g produces g_l with optimal value V*(g_l) <= V*(g).The inequality follows because g_l differs from g by missing the kth option, thereby restricting the available choices.Source: cess) · p. 54
• Adding potentially additional predicates to each option produces g_u with an optimal value bounded analogously relative to g.The supplied passage states the construction and indicates that the corresponding reward-function comparison yields the bound, but does not include its final displayed inequality.Source: cess) · p. 54