Scaling Empowered Agents

1 Introduction

• Empowerment maximization has lacked empirical demonstrations in large, challenging environments where survival and resource collection are difficult.Prior work has argued that such open-ended settings are especially important for empowerment.Source: §1 · pp. 1–2
• Mutual information skill learning estimates state empowerment because the number of executable skills from a state lower bounds its empowerment.A second policy can ascend this estimated empowerment landscape toward states with many skill options.Source: §1 · p. 2
• The paper presents a general recipe converting scalable skill-learning methods into empowerment maximization and reports scaling where baselines cannot.It studies resource gathering, survival, and environment modification, including house-building at night and tool use in Crafter.Source: §1 · p. 2

Intrinsic Motivation Exploration Methods

• Prior work predicts that empowerment maximization can produce environment modification, resource gathering, and survival, and this work presents scalable evidence in large open-ended settings.These behaviors had previously been demonstrated only in simple experiments.Source: §2 · pp. 2–3

Potential and Effective Empowerment

• Potential empowerment measures maximal control from a context, while effective empowerment measures how much a particular distribution over open-loop controls communicates to the future.Potential empowerment is a property of context; effective empowerment also depends on the controls.Source: §2 · p. 3

Prior Deep Learning Methods for Estimating and Maximizing Potential Empowerment

• Closed-loop empowerment measures the channel capacity from a collection of reactive policies to outcomes, aligning it with what a standard reinforcement-learning agent can do.Prior deep-learning approaches such as VIM and LatentGCE estimate open-loop empowerment, with LatentGCE requiring sufficient action-sequence coverage in high-empowerment regions.Source: §2 · p. 3

Unsupervised Skill Discovery

• Mutual information skill-learning methods learn diverse policies by maximizing information between skill latents and policy outcomes, and scale to high-dimensional locomotion and pixel-based tasks.They commonly use a skill classifier on outcomes as a reward and a variational lower bound on mutual information.Source: §2 · p. 3

Skill Learning and Effective Empowerment

• Skills are not potential-empowerment-maximizing policies, but can estimate potential empowerment; this work uses that connection for scalable maximization in large open-ended settings.The paper identifies this as a gap in prior work exploiting the skill-learning connection.Source: §2 · p. 3

3 Defining Empowerment

• Potential empowerment is the maximum mutual information between behavior and future states, conditioned on the starting state.The paper adopts closed-loop policies as behaviors and considers states many steps into the future.Source: §3 · pp. 3–4
• A skill is a context- and state-conditioned policy indexed by a skill vector, with the context specifying its starting point.The context state and the current state play distinct roles: the former sets the skill's start point, while the latter guides its action.Source: §3 · p. 4
• Potential empowerment is the mutual information between a skill vector and a future state sampled from that skill's discounted state occupancy.The skill distribution is assumed uniform; the set of skills comprises policies assigned nonzero probability.Source: §3 · p. 4
• Potential empowerment measures the maximum bits skills can broadcast into the future, while effective empowerment measures how good a policy set is from a particular state.An approximate solution to the inner effective-empowerment optimization provides a lower bound on empowerment.Source: §3 · p. 4

4 Skills Estimate Empowerment (SEE): A Scalable Algorithm for Maximizing Empowerment

• SEE is an online, scalable two-policy algorithm that uses skill learning to estimate empowerment and a potential policy to maximize that estimate.The skills measure controllable futures from a state; the potential policy moves toward states offering greater control.Source: §4 · pp. 4–5

4.1 The Inner Optimization: Learning Skills

• SEE adapts contrastive successor features to estimate effective empowerment with a skill classifier trained on transitions separated by geometrically sampled intervals.The expected classifier reward is a variational lower bound on mutual information, which in turn lower bounds potential empowerment.Source: §4.1 · p. 5

4.2 Practical Considerations

• In continuous benchmarks, joint mutual-information classifiers performed better than conditional classifiers, which otherwise overfit to the starting context.The skill reward still uses the true starting state; SEE uses the joint classifier by default in continuous settings and a conditional classifier in Craftax-Classic.Source: §4.2 · pp. 5–6
• Empowerment is restricted to selected state variables because agents can otherwise gain empowerment by changing controllable but meaningless variables.The restriction determines which state changes are salient to skills without limiting the policy's control information.Source: §4.2 · p. 6
• For large state spaces, SEE alternates potential-policy rollouts with skill-policy rollouts, using skills as local probes of the empowerment landscape.The estimated potential empowerment then serves as the reward for training the policy that navigates toward states with more control.Source: §4.2 · p. 6

5.1 Scaling and Baselines

• SEE scales to manipulation, locomotion, and open-ended Crafter, unlocking 7 −8 of 22 achievements per rollout versus ∼5.5 for prior Crafter exploration methods.The comparison uses the same training budget of 1 × 109 environment steps.Source: §5.1 · pp. 7–8
• In Ant, SEE seeks the arena center; in AntSoccer it reaches the ball, while effective skills spread out or kick the ball in different directions.In BuilderBench, SEE approaches and stays near the blocks while skills move them in different directions; baselines fail to remain close.Source: §5.1 · p. 8, Figure 3 · p. 9

5.2 Ablating Design Decisions

• Switching between potential and effective empowerment policies during data collection yields stable policies, unlike separately trained policies that inconsistently reach target regions.The separate-training ablation misses the center in Ant and does not consistently reach the blocks in BuilderBench.Source: §5.2 · p. 8
• Conditional MI learns empowering behavior in AntSoccer and BuilderBench, but overfits to small perturbations in high-dimensional Ant; in Crafter it performs best among ablations and prevents joint-MI collapse..Source: §5.2 · p. 8
• Replacing the bilinear difference-in-representation classifier with a more expressive logit worsens training stability and learned behavior across environments.The general classifier can fit collected data closely, leading policies to overfit sensitive classifiers.Source: §5.2 · p. 9

5.3 Emergent Behaviors

• Potential policies gather empowerment by reaching useful states, then skills spend it through actions such as spreading out, kicking the ball, or moving blocks.In Crafter, gathering resources and tools precedes skills such as building houses, mining, and crafting tools.Source: §5.3 · p. 9, Figure 4 · p. 10
• In Crafter, agents modify their environment by building houses, including when defending against enemies.The behavior is observed in the scaled-up setting; earlier demonstrations were in simple tabular settings.Source: §5.3 · p. 9, Figure 5 · p. 10
• Crafter agents learn to kill and dodge enemies, while a simple survival reward does not produce the more complex crafting behaviors.The survival-reward comparison uses a reward of 1 per step and 0 upon termination.Source: §5.3 · p. 10

6 Discussion

• The method maximizes potential and effective empowerment through a conceptual link to skill learning, with practical considerations for high-DoF settings.The authors suggest future work investigate simplified parameterizations that address these practical considerations at scale.Source: §6 · p. 10
• Empowerment maximization is sensitive to world representations, and their interplay remains an open question.Source: §6 · p. 11

AI use statement

• Generative AI assisted with experiment-code implementation, figures, minor writing edits, appendix details, and manuscript formatting, but not methods development or theoretical results.Source: AI use statement · p. 11

Reproducibility statement

• The appendix provides algorithm pseudocode and implementation details, with code, videos, and additional figures available on the project website.Source: Reproducibility statement · p. 11

A Algorithm and Implementation Details

• When skills are independent of the starting state, joint MI and conditional MI are equivalent; sampling a trajectory state and geometric future state as a pair breaks that assumption.The conditional MI variant is described separately in Section A.2.Source: A Algorithm and Implementation Details · p. 14

A.1 Training the classifier

• The classifier logit uses a bilinear difference-of-representations form with temperature τ = 0.1.Skills are assumed uniformly distributed on a sphere.Source: §A.1 · p. 14
• InfoNCE trains a classifier to distinguish skill–outcome pairs from the joint distribution versus pairs drawn from the product of marginals, yielding a variational lower bound on mutual information.The bound becomes tight at the optimum.Source: §A.1 · p. 14
• In continuous settings, an unconstrained transition decoder can assign arbitrarily large empowerment estimates even to skills that barely change the state.The section motivates an information bottleneck to constrain this problem.Source: §A.1 · pp. 14–15
• SEE adds Gaussian noise to states during classifier training so skills must be identifiable beyond the noise scale; policy and value optimization use unnoised states.Craftax-Classic uses no noise.Source: §A.1 · p. 15

A.2 Data Collection & Conditional MI

• Joint MI training samples states and discounted future states across trajectories to reduce context overfitting, while CMI training branches fresh skills from copied switch states, making joint and conditional MI equivalent.An encoder trained only on switch-state data can overfit to the context, particularly in continuous settings.Source: §A.2 · p. 15

A.3.1 Inner Optimization: Skill Learning/Effective Empowerment

• SEE anneals the effective-policy reward from unnormalized exploration (λ = 0) to the normalized empowerment-bound form (λ = 1); Craftax-Classic λ = 0 learns >9 achievements but loses the lower-bound interpretation.Using λ = 1 throughout learns far fewer achievements; annealing gives the largest empowerment estimate in Craftax-Classic.Source: §A.3.1 · p. 16
• Hindsight Experience Relabeling resamples a start state and greedily relabels the skill during policy and value training, improving sample efficiency while introducing bias and some skill collapse.The relabeled skill maximizes a Monte Carlo estimate of cost-to-go.Source: §A.3.1 · p. 16

A.3.2 Outer Optimization: Potential Policy and Critic Optimization

• The potential critic averages skill values over eight uniformly sampled skills as its reward estimate, which is an expected lower bound on pointwise potential empowerment under the stated assumptions..Source: §A.3.2 · pp. 16–17

A.4 Empowerment maximization pseudocode

• The algorithm alternates two-phase rollouts, classifier fitting, inner skill updates, empowerment estimation, and outer potential-policy updates; classifier data use Joint MI or CMI branching.SAC uses replay, while PQN uses two-stage rollouts.Source: §A.4 · p. 17

B Experimental setup and evaluation

• Evaluations run the potential policy for 100 steps, then execute a sampled skill for up to 300 steps; continuous results average 128 episodes in 128 new environments.Craftax-Classic achievement counts cover two-phase rollouts in 128 randomized environments.Source: B Experimental setup and evaluation · p. 17

B.1 Continuous Experiment Details

• Continuous-environment visitation counts distinct planar cells seen during training; SEE and its ablations are measured during the skill phase, while baselines are measured over full rollouts.Ant and AntSoccer use 1 m × 1 m tiles; BuilderBench uses 5 cm × 5 cm tiles pooled across blocks.Source: §B.1 · p. 18

B.1.1 Continuous hyperparameters

• Table 1 reports SEE hyperparameters with joint MI in continuous environments; Ant and AntSoccer use different classifier-training encoder observations, while BuilderBench deterministically positions three cubes in a corner and initializes the gripper at the center-edge of the play area, far from the blocks.Source: Table 1 · p. 18, §B.1.1 · p. 18

B.2.1 Craftax-Classic

• Craftax-Classic uses PQN for GPU acceleration, and Figure 6 illustrates a single two-phase episode with achievements in the shared potential phase and skill rollouts from the switch state.The illustrated rollout reports achievements without extrinsic reward and represents one seed.Source: §B.2.1 · p. 19, Figure 6 · p. 19

C Baselines

• VIM estimates open-loop empowerment with a learned action-sequence decoder, whereas LatentGCE learns a generative forward model; both are open-loop baselines.VIM’s partition function is generally intractable, so it regresses an approximation instead.Source: C Baselines · p. 20

C.1 Validating Empowerment Fields in Simple Continuous Benchmarks

• In pendulum and ball-in-box benchmarks, LatentGCE identifies the predicted high-empowerment regions, while VIM does not recover them.Figure 7 compares predicted fields with reference fields; each learned field uses one seed.Source: §C.1 · p. 20

C.2 Implementation in the main evaluation environments

• The authors adapted LatentGCE to scaled continuous settings and implemented VIM from its published algorithm because complete public implementations were unavailable.LatentGCE’s public code did not implement its latent-learning version, and VIM had no publicized codebase.Source: §C.2 · p. 20
• Both baselines optimize potential policies with PPO over open-loop action sequences; VIM uses H = 30 due to instability, while LatentGCE uses H = 100.The methods estimate empowerment over full Ant state, ball position in AntSoccer, and block positions in BuilderBench.Source: §C.2 · pp. 20–21
• LatentGCE recovers the predicted high-empowerment regions in pendulum and ball-in-box fields, whereas VIM does not.The figure shows normalized fields and a linearized reference.Source: Figure 7 · p. 21

D Ablation Details

• Default SEE uses the joint MI classifier in continuous settings and the CMI classifier in Crafter; the switching ablation trains both phases from the same start state without switching.The no-switching variant allocates 25% of rollout time to the potential policy and 75% to the skill phase.Source: D Ablation Details · p. 21, §D.1 · p. 21

D.1 Policy switching.

• Without switching, meaningful potential policies are inconsistent in Ant and BuilderBench, and the policy ends far from targets or accumulates few Crafter-Classic achievements.Across Ant, AntSoccer, and BuilderBench, the potential policy ends far from the center, ball, or blocks, respectively.Source: §D.1 · p. 23

D.2 Conditional versus full MI.

• In Ant, CMI worsens training and barely moves the potential policy; in Craftax-Classic, Joint MI collapses empowerment values despite high achievement scores.CMI and SEE perform similarly in BuilderBench; in AntSoccer, both approach the ball, but SEE kicks it farther during the skill phase.Source: §D.2 · p. 23

D.3 Pair-decoder parameterization.

• The pair decoder worsens visitation, target distance, and skill displacement in Ant and AntSoccer; in AntSoccer it visits 280 cells versus 539 for SEE.In Ant, the pair decoder can show higher learned empowerment despite worse visitation and displacement, underscoring the importance of classifier regularization.Source: §D.3 · pp. 23–24, Figure 9 · p. 24

D.4 Reward normalization

• Normalization has little effect on continuous-setting training metrics, but in Craftax-Classic fixed λ = 1 hurts learned empowerment and the schedule is critical for lower-bound estimates.The figures compare the annealed schedule against fixed λ = 0 and λ = 1.Source: §D.4 · p. 24, Figure 11 · p. 25, Figure 12 · p. 26

E Additional qualitative examples

• Selected rollouts show additional environment modifications across the remaining four Craftax-Classic seeds, with white outlines marking placed tiles.Source: Figure 13 · p. 27, E Additional qualitative examples · p. 27

F Theory

• The theory section provides a justification for the method but identifies empirical tractable empowerment maximization, rather than novel theory, as the primary contribution.It notes that prior work, including VIC, showed versions of the theoretical results.Source: F Theory · p. 27

• Expected cumulative classifier reward estimates a lower bound on potential empowerment, which the potential policy uses to seek states with many options.The classifier identifies skills from states, transitions, or trajectories and is used as a reward.Source: §F.1 · pp. 27–28
• The procedure fits the classifier to the on-policy posterior, holds it fixed for greedy policy improvement, then refits it to the new posterior.Source: §F.1 · p. 29
• With exact on-policy classifier fitting and greedy updates at a fixed start state, each iteration monotonically improves the true MI.The stated improvement result assumes a perfectly fitted classifier at each iteration.Source: §F.1 · pp. 28–29
• The theoretical result assumes exact classifiers and greedy policy updates at fixed start states, unlike practical function approximation.Source: §F.1 · p. 29