Source-linked AI summary
The Geometry of Empowerment
Catherine Ji, Vivek Myers, Sergey Levine, Benjamin Eysenbach
TL;DR
The paper addresses the open question of whether empowerment identifies structurally central states and how those states relate to downstream adaptation. It analyzes empowerment through information and temporal geometries, showing a link to adaptation in tabular settings but a divergence between information and reward geometries in continuous settings or with anisotropic reward priors.
Problem
The relationship between empowerment, structural centrality, and adaptation to downstream rewards has remained an open question.
Source: §1 · p. 2Method
The paper analyzes potential and effective empowerment using information-geometric and temporal-distance interpretations, and studies how skill sets support adaptation to rewards.
Source: §3.2 · p. 5, §4.1 · p. 7, §4.3 · p. 9Results
In tabular settings, empowered agents reach information-geometric central states from which they can adapt, but this intuition fails in continuous-state MDPs or with anisotropic reward priors.
Source: §1 · p. 2Takeaways & Limitations
Scalable formulations that unify empowerment and adaptation require aligning information and reward geometries.
Source: §1 · p. 2Empowerment does not generally guarantee downstream adaptation in continuous-state MDPs or with anisotropic reward priors, where information and reward geometries can misalign.
Source: §4.3 · pp. 10, 12, §5 · p. 12
Abstract
from arXiv · showhide
Empowerment captures the capacity for an agent to actively control its environment. While conceptually appealing as an information-theoretic quantity, the connection between empowerment and structurally central states that provide broad access to future outcomes has remained an open question. In this work, we link empowerment maximization and skill-learning methods to provide new geometries for interpreting and analyzing empowerment. Our analyses answer longstanding open questions on the connections between empowerment and structural centrality. Our analyses also reveal distinctions between information and reward geometries, highlighting important theoretical implications to build scalable empowerment-maximization methods. Website and code can be found at https://empowerment-geometry.github.io/.
1 Introduction
Empowerment is often treated as a task-independent motivation, but its connection to central states and downstream performance has lacked a formal characterization. The paper distinguishes potential empowerment, which seeks states with many options, from effective empowerment, which actively exercises them, and connects these ideas to centrality and adaptation while identifying limits to that connection.
1 Introduction
- Potential empowerment measures the maximum bits actions can convey about a future state, while effective empowerment measures how a policy actively informs that state.The paper treats these as distinct quantities, despite their sometimes being conflated.Source: §1 · p. 1
- The potential policy ascends toward a state with options, whereas the effective policy uses skills to exercise options along different routes.The figure contrasts reaching a mountain peak with descending along distinct trails.Source: §1 · p. 1, Figure 1 · p. 2
- Potential empowerment states are central in information geometry and, in tabular settings, centrality lower-bounds adaptation to downstream tasks.The paper also reports that the connection does not generally hold in continuous-state MDPs or with anisotropic reward priors.Source: §1 · p. 2
- The work gives potential empowerment an information-geometric centrality interpretation and a temporal-distance interpretation, linking effective empowerment to adaptation.The authors identify aligning information and reward geometries as necessary for scalable formulations that unify empowerment and adaptation.Source: §1 · p. 2
2 Related Work
Prior work has connected empowerment with centrality and developed mutual-information skill learning, but has not formally established their relationship across settings. This paper situates its analysis between discrete centrality notions and scalable skill-learning methods, including the distinction between potential and effective empowerment.
2 Related Work
- Earlier centrality and bottleneck notions are appealing but are only well-defined for discrete states and actions.The paper frames its information-geometric centrality results as a rigorous grounding for prior empowerment observations.Source: §2 · p. 3
- Prior work conjectured links between potential empowerment and centrality, but formal connections to centrality notions were lacking.Earlier studies noted reachable-state counts in discrete deterministic settings and correlations with closeness centrality in simple stochastic examples.Source: §2 · p. 3
- Temporal distance extends reachability and closeness ideas to continuous state spaces, and the paper links empowerment to reduced temporal distance to future states.In discrete settings, temporal distance corresponds to a function of state-to-state hitting time.Source: §2 · p. 3
- Mutual-information skill-learning methods scale to high-dimensional continuous-control tasks, but optimize effective rather than potential empowerment.The paper notes it does not present experiments in high-dimensional or continuous settings.Source: §2 · p. 3
3 Preliminaries
The preliminaries define the MDP and skill-policy setting, represent future-state behavior through discounted occupancies, and distinguish the potential empowerment being maximized from the effective empowerment realized by a particular skill policy.
3 Preliminaries
- The analysis uses a fully observed MDP and policies conditioned on both a latent skill and a starting state, with each fixed skill yielding an ordinary policy.The latent skill belongs to a continuous space, including in tabular MDPs.Source: §3.1 · p. 4, §3 · p. 4
3.1 Discounted State Occupancy
- A discounted state occupancy measure is a future-state distribution at a geometrically sampled termination time, and reachable occupancies form a convex polytope within the state simplex.For a fixed starting state, Bellman flow equations constrain the reachable state-occupancy polytope.Source: §3.1 · p. 4
3.2 Empowerment Preliminaries
- Potential empowerment is the maximum attainable channel capacity over skill-conditioned policies, while effective empowerment is the mutual information achieved by a particular policy under a uniform skill source.The paper adapts potential empowerment to reactive skills and measures their influence at a geometrically sampled future state.Source: §3.2 · p. 5
- Mutual-information skill-learning methods maximize effective empowerment rather than potential empowerment.The paper notes that these methods commonly use skills specific to a fixed starting state.Source: §3.2 · p. 5
- Skills probe potential empowerment: easily distinguishable future occupancies indicate that a starting state affords many controllable futures.In the GridWorld illustration, the high-potential center provides access to a maximal set of distinguishable skills.Source: §3.2 · p. 5, Figure 2 · p. 6
4 The Geometry of Empowerment
The section formalizes empowerment as centrality in information and temporal geometries, then shows that its adaptation guarantee depends on reward geometry and state-space setting.
4.1 Information Geometry of Potential and Effective Empowerment Policies.
- In stochastic and continuous MDPs, potential empowerment is the KL 1-center radius of the reachable occupancy polytope, with optimal skills on its boundary.This extends centrality beyond deterministic tabular settings and uses KL divergence rather than Euclidean distance.Source: §4.1 · pp. 6–7
4.2 Empowerment, Bottlenecks, and Temporal Distance
- In discrete MDPs, potential empowerment measures the reduction in expected hitting time when committing to a skill; a contiguous-rollout variant uses temporal distance in continuous settings.Temporal distance reduces to hitting times in the discrete limit.Source: §4.2 · p. 8
- Bottlenecks become high-empowerment states because committing to a skill quickly reaches distinct downstream regions, linking this interpretation to closeness centrality.In the GridWorld, adding narrow openings shifts high empowerment toward the bottleneck openings.Source: §4.2 · p. 8, Figure 4 · p. 9
4.3 Is Empowerment Optimal for Downstream Adaptation?
- For isotropic reward priors, potential empowerment lower-bounds adaptation measured by selecting a learned skill for a sampled reward.The bound is most informative when control is strong over comparatively few distinguishable regions.Source: §4.3 · pp. 9–10
- The lower bound does not rank starting states: Gaussian-width adaptation can favor three reachable states while information radius favors two.Reward adaptation geometry uses Gaussian width, whereas empowerment uses KL divergence.Source: §4.3 · p. 10
- In continuous MDPs, skill returns can converge while empowerment grows without bound, so high empowerment need not imply adaptation to smooth rewards.The construction allows arbitrarily small adaptation to Lipschitz-continuous rewards.Source: §4.3 · p. 10, Figure 6 · p. 11
- With anisotropic reward priors, empowerment can yield arbitrarily large mutual information but near-zero adaptation when reward variation lies outside controllability-selected directions.Empowerment breaks symmetry through controllability, which may not align with reward-preferred directions.Source: §4.3 · p. 12
5 Discussion, Limitations, & Future Work
The paper links empowerment to centrality and adaptation in tabular settings, while identifying limits when information and reward geometries diverge. It argues that addressing those gaps requires methods that better unify the geometries and scale to continuous settings.
5 Discussion, Limitations, & Future Work
- In tabular settings, empowerment-maximizing agents navigate to information-geometric central states and can adapt to unknown rewards, but these guarantees do not generally extend to continuous settings or anisotropic reward priors.The paper describes the adaptation guarantee as a lower bound, not a ranking result.Source: §5 · p. 12
- Information geometry based on KL divergence and reward-adaptation geometry based on Gaussian widths can diverge, so larger mutual information does not necessarily mean better adaptation.For continuous MDPs with Lipschitz-continuous rewards, some skill sets can have unbounded empowerment yet provide no adaptation benefit.Source: §5 · p. 12
- Empowerment provably lower-bounds goal-oriented behavior in tabular settings, but current empowerment- and return-maximization formulations do not make that hypothesis generally true.The paper frames its answer to Salge’s hypothesis [3] as partial.Source: §5 · p. 12
- The authors identify scalable continuous empowerment methods and skill-learning algorithms, and approaches that unify information and reward geometries, as future directions.The analysis can also apply to methods combining extrinsic and information-theoretic intrinsic rewards [39].Source: §5 · pp. 12–13
A Preliminaries for Proofs
The appendix reviews standard information-theoretic results and specifies how they apply to skill priors, skill occupancies, and future states. It also uses noise outsourcing to justify a uniform skill prior without loss of generality.
A Preliminaries for Proofs
- Mutual information is concave in the skill prior for fixed skill occupancies and convex in the occupancies for a fixed skill prior.This applies the standard concavity and convexity result to the skill-to-future-state channel.Source: A Preliminaries for Proofs · p. 16
- The channel-capacity dual form treats each skill occupancy as an output distribution over future states and provides the form used to prove Theorem B.2.When the value is finite, the infimum is attained at a unique center q⋆.Source: A Preliminaries for Proofs · p. 16
- The KL–χ2 comparison is applied to the effective skill posterior and prior, and averaging the resulting bound over s+ gives Theorem 4.3’s χ2 control of mutual information.The distributions used are pπeff(z | s0, s+) and p(z | s0).Source: A Preliminaries for Proofs · p. 16
- Noise outsourcing shows that a measurable transformation of uniform noise can induce any distribution on a standard Borel space, supporting the uniform skill prior used without loss of generality.The paper uses this result in Lemma D.1.Source: A Preliminaries for Proofs · p. 16
B Information Geometry of Empowerment
Potential empowerment is the KL 1-center radius of reachable state occupancies, with optimal skills spanning the boundary of that radius. In discrete deterministic settings, it also equals the logarithm of the number of reachable states.
B Information Geometry of Empowerment
- At a joint optimum, the skill marginal is the unique center, and every skill occupancy lies on the boundary of its optimal KL ball.The optimal skill occupancies are extreme points of the reachable occupancy set.Source: B Information Geometry of Empowerment · pp. 17–18
- Potential empowerment jointly optimizes mutual information by solving the KL 1-center localization problem for the reachable state-occupancy polytope.The information-geometry result assumes policies measurable over the state and skill spaces, so reachable occupancies are those induced by measurable policies.Source: B Information Geometry of Empowerment · pp. 17–18
B.1 Log reachability interpretation of potential empowerment in discrete deterministic settings
- For discrete deterministic MDPs, horizon-H potential empowerment equals log |R_H(s)|, the logarithm of the number of states reachable in H steps.Deterministic dynamics make the future state a function of the action sequence, and a uniform distribution over reachable states attains this entropy.Source: §B.1 · p. 19
C.1 Bottleneck states are high-empowerment states
Potential empowerment can be interpreted through how committing to a skill changes expected rollout hitting times or temporal distances. The contiguous-rollout extension concentrates at bottlenecks, and a continuous uniform skill prior can represent any source distribution.
C.1 Bottleneck states are high-empowerment states
- Potential empowerment is the largest expected reduction in log expected rollout hitting time when committing to a skill.This hitting-time result is stated for discrete MDPs, with the skill-conditioned comparison repeating the same skill on every rollout.Source: §C.1 · pp. 19–20
C.2 Temporal distances over contiguous skill rollouts
- For contiguous skill rollouts, potential empowerment is the maximum expected reduction in temporal distance to future states after choosing an initial skill.The temporal-distance interpretation remains well-defined in continuous settings; in discrete spaces, it has a hitting-time interpretation.Source: §C.2 · pp. 21–22
- In the reported 8 × 8 grid experiments, bottleneck openings have high potential contiguous empowerment, and the estimated field closely matches the original empowerment field.The two grids use center and corner openings.Source: §C.2 · p. 22, Figure 7 · p. 22
D Skill Selection: Fixed Continuous Uniform Priors are General
- A continuous uniform latent prior can represent any discrete or continuous source distribution over skills for the mutual-information objective.The result applies to standard Borel skill alphabets.Source: D Skill Selection: Fixed Continuous… · p. 22
E Continuous versus Discrete Skill Spaces
With finitely many discrete skills, a uniform skill prior can be strictly suboptimal for channel capacity. A continuous uniform prior retains full generality, motivating its use in skill-learning methods.
E Continuous versus Discrete Skill Spaces
- With a finite skill alphabet, fixing the skill prior to be uniform can strictly reduce channel capacity.The proof optimizes the channel at an informative vertex before optimizing the source probability.Source: E Continuous versus Discrete Skill… · p. 23
- A uniform prior over two discrete skills cannot generally achieve channel capacity.The finite-skill example has a capacity-achieving source with p⋆(Z = 1 | s0) = 2/5 when β = 1/2, rather than 1/2.Source: E Continuous versus Discrete Skill… · pp. 23–24
- A fixed continuous uniform prior is fully general, whereas a discrete prior is not; the authors therefore suggest continuous uniform sources for skill-learning methods.VIC optimizes a source distribution and channel, while DIAYN fixes a uniform categorical source; neither claims that fixing the prior preserves the objective [18] [20].Source: E Continuous versus Discrete Skill… · p. 24
F Empowerment Lower Bounds Tabular Adaptation
In tabular settings, empowerment lower-bounds zero-shot adaptation under an uninformed reward prior. The analysis uses occupancy differences and a reward-variance scaling that makes returns invariant to state refinement.
F Empowerment Lower Bounds Tabular Adaptation
- In tabular settings, empowerment lower-bounds zero-shot adaptation under an uninformed reward prior.The theorem considers a full-support reference distribution and Gaussian rewards; the bound is stated for any skill space and skill prior.Source: F Empowerment Lower Bounds Tabular… · p. 24
- The proof relates adaptation to occupancy gaps between each skill’s state occupancy and the skill-marginal occupancy.It bounds adaptation through Gaussian widths of the occupancy-gap vectors and relates mutual information to their norm.Source: F Empowerment Lower Bounds Tabular… · pp. 24, 26
- Scaling reward variance inversely with reference mass leaves the return distribution unchanged when states are split.In the two-state example, splitting halves reference mass and doubles reward variance, preserving the return distribution.Source: F Empowerment Lower Bounds Tabular… · pp. 24–25, Figure 8 · p. 25
- The lower bound tightens as action commitment increases and loosens as the number of absorbing target states grows.The commitment comparison is shown in Figure 9, and the target-state comparison in Figure 10.Source: F Empowerment Lower Bounds Tabular… · p. 26
G When Information Geometry is Insufficient for Adaptation
In continuous state spaces, high empowerment need not imply adaptation to downstream rewards. Skills can encode unbounded information while remaining arbitrarily close in state space and nearly indistinguishable under smooth rewards.
G When Information Geometry is Insufficient for Adaptation
- In the tabular bound examples, greater skill commitment tightens the bound, while more absorbing target states weaken it.These are the trends illustrated by Figures 9 and 10.Source: Figure 9 · p. 27, Figure 10 · p. 27
- A continuum of skills in a continuous state space can yield unbounded potential empowerment.The construction gives at least γ log n bits for every n.Source: G When Information Geometry is… · pp. 27–28
- Even inside an arbitrarily small metric ball, arbitrarily many skills can yield high mutual information while bounded 1-Lipschitz rewards give them nearly the same return.The upper bound on return differences between skills remains unchanged as the number of skills grows.Source: G When Information Geometry is… · p. 28
- For any 1-Lipschitz reward, skills can have high or unbounded mutual information while offering no adaptation advantage beyond a single policy.The authors caution against extrapolating the tabular adaptation bound to continuous spaces with smooth rewards.Source: G When Information Geometry is… · pp. 28–29
H Anisotropic Reward Priors
Anisotropic reward priors weaken the connection between mutual information and adaptation: high empowerment can be directed away from rewarding dimensions. With fixed reward power, the paper constructs skills with high mutual information but zero adaptation.
H Anisotropic Reward Priors
- The adaptation lower bound under an anisotropic reward prior is scaled by λmin(Σ).The authors note that arbitrarily large mutual information can coexist with no adaptation under fixed reward power constraints.Source: H Anisotropic Reward Priors · p. 29
- With an anisotropic reward prior of fixed trace, skills can achieve I(Z; S+ | S0 = s0) = γ log K while JΣ(s0) = 0.The construction holds for every K and a fixed power budget T.Source: H Anisotropic Reward Priors · pp. 29–30
- An anisotropic prior can place reward variance in directions orthogonal to occupancy-gap vectors, leaving skill return gaps at zero while mutual information is unchanged.The construction concentrates all reward variance in a direction orthogonal to the span of the occupancy-gap vectors.Source: H Anisotropic Reward Priors · p. 30
- High control over observations need not mean control over rewarding degrees of freedom under anisotropic priors.The paper warns that an empowerment-maximizing agent could ignore other learnable skills that control those rewarding dimensions.Source: H Anisotropic Reward Priors · pp. 29–30
I.1 Tabular 5 × 5 grid experiments
In grid experiments, empowerment-maximizing policies seek states that provide broad access to future states, with bottlenecks becoming especially empowering when walls constrain movement.
I.1 Tabular 5 × 5 grid experiments
- Random reward vectors and convex optimization identify reachable DSOM-polytope vertices, providing an approximate feasible set for specifying skills.The search does not generally retrieve every candidate vertex.Source: §I.1 · p. 30
- Given a fixed channel encoder, the source distribution is optimized by convex optimization; the resulting empowerment policy is learned with Q-learning.The source-distribution optimization uses the concavity of mutual information in the source distribution for a fixed encoder.Source: §I.1 · pp. 30–31
- From the center of the 5 × 5 grid, latent-conditioned policies cover all states, and the empowerment-maximizing policy moves to the center.The policy maximizes reachability to other states in the MDP.Source: §I.1 · pp. 30–31
I.2 Bottleneck grid experiments
- The bottleneck experiment uses an 8 × 8 grid with impenetrable walls and tests γ values of 0.5 and 0.95.It uses the same vertex search, source-distribution optimization, and Q-learning procedure as the 5 × 5 experiments.Source: §I.2 · p. 31
- With walls, bottleneck openings have high empowerment because skill choice substantially changes temporal distance to future states.In the open grid, the center has maximal potential empowerment.Source: §I.2 · p. 31
I.3 Key-finding experiments
Key-finding and worked-example experiments show how empowerment identifies information-geometric central states, while adaptation geometry distinguishes mutual information from Gaussian width.
I.3 Key-finding experiments
- The key-finding MDP makes a key at (0, 2) necessary for entering the blue-shaded regions.The experiment starts at (4, 2) without the key.Source: §I.3 · pp. 31–32
- Starting without a key, the agent navigates to obtain it and then to the center, identifying the key-acquired state as most central in the information geometry.Skills spread out from the central state, while the potential empowerment policy navigates toward it.Source: §I.3 · p. 32, Figure 11 · p. 33
I.4 Adaptation geometry examples
- In the adaptation examples, each skill transitions in one step to an absorbing target, so its normalized DSOM is the target transition distribution.Example X's antipodal occupancy offsets span one degree of freedom; Example Y's three offsets sum to zero and can be shown in two dimensions.Source: §I.4 · pp. 33–34
- Example X has higher mutual information, while Example Y has larger Gaussian width: skill overlap lowers channel capacity, but target spread increases width.The comparison shows that adaptation and mutual-information orderings can flip.Source: §I.4 · p. 34
I.5 Worked example of centrality in information geometry
- At s2, distinct skill actions induce more spread-out future-state distributions than identical random skill policies, so empowerment favors navigating to and staying at s2.The distinct actions convey more bits about the next state.Source: §I.5 · p. 34