PaperlayerView exact revision on arXiv

Source-linked AI summary

Variational Intrinsic Control

Karol Gregor, Danilo Jimenez Rezende, Daan Wierstra

arXiv:1611.07507v1cs.LGcs.AI

TL;DR

The paper addresses how an agent can discover intrinsic options—policies that reliably achieve different environmental outcomes—without external supervision. It maximizes mutual information between options and termination states through explicit and implicit policy-gradient algorithms, and reports improved intrinsic control and empowerment estimation with applications to extrinsic reward learning. The approach is useful for representing controllable behavior, but the explicit-option algorithm has difficulty with neural or continuous function approximation and with exploring unfamiliar state regions.

  • Problem

    The paper asks how to discover the intrinsic options available to an agent, rather than limiting learning to a small set of task-specific options.

  • Method

    The method maximizes mutual information between options and termination states using closed-loop policies, with explicit and implicit option representations and a variational training objective.

  • Results

    Closed-loop options significantly increase intrinsic control and improve empowerment estimation across tasks, while learned policies also support extrinsic reward maximization.

  • Takeaways & Limitations

    The algorithms represent intrinsic control through executable policies, estimate empowerment in different states, and can be combined with external reward learning.

  • Takeaways & Limitations

    The explicit-option algorithm struggles with neural and continuous function approximation and can fail to explore new state regions because intrinsic reward and policy inference must match.

Abstract

from arXiv · show

In this paper we introduce a new unsupervised reinforcement learning method for discovering the set of intrinsic options available to an agent. This set is learned by maximizing the number of different states an agent can reliably reach, as measured by the mutual information between the set of options and option termination states. To this end, we instantiate two policy gradient based algorithms, one that creates an explicit embedding space of options and one that represents options implicitly. The algorithms also provide an explicit measure of empowerment in a given state that can be used by an empowerment maximizing agent. The algorithm scales well with function approximation and we demonstrate the applicability of the algorithm on a range of tasks.

1 INTRODUCTION

The paper frames intrinsic options as policies that meaningfully affect the world and seeks to discover as many reliably achievable consequences as possible. It develops information-theoretic option learning for representing broad intrinsic control and applying it to both intrinsic and extrinsic objectives.

  • 1 INTRODUCTION: Intrinsic options are policies with termination conditions, distinguished by the environmental states they reach upon termination.The available option set concerns what an agent can achieve independently of its intentions.
  • 1 INTRODUCTION: Unlike traditional option learning, the method targets a larger space of intrinsic options rather than a small task-specific set.The paper argues that options remain more compact than all action sequences because they are distinguished by final states.
  • 1 INTRODUCTION: The approach learns goals or options while preserving an embedded representation that supports control over many goals.This extends goal and state embeddings whose goals were previously assumed to be given.
  • 1 INTRODUCTION: The paper motivates empowerment as reaching states with many options the agent can learn to use, rather than merely many theoretically possible options.It also distinguishes control from model understanding and proposes combining intrinsic control with external reward.
  • 1 INTRODUCTION: The learned intrinsic-control space provides explicit policies, estimates control across states, and can support maximizing extrinsic or intrinsic reward.The estimated control is interpreted as the total number of reliably reachable states.
  • 1 INTRODUCTION: Closed-loop options condition actions on state and are reported to increase intrinsic control and improve empowerment estimation across tasks.This addresses limitations of open-loop action sequences, which cannot properly react to stochastic environment dynamics.

2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE

The paper represents options as policies and uses mutual information between options and termination states to maximize diverse, reliably controllable outcomes. A variational formulation makes the objective trainable by jointly optimizing option selection, inference, and policies.

  • 2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE: An option consists of an element Ω from an option space and a policy π(a|s, Ω) with a termination action producing final state s_f.The option space may be finite, combinatorial, or continuous.
  • 2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE: The controllability distribution p_C(Ω|s_0) groups options by their distinguishable consequences, so options leading to the same state need not be treated separately.In the example, three policy options collapsing onto two final states yield two intrinsic options.
  • 2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE: The objective favors diverse final states while requiring each option to control its resulting states precisely.These goals are combined through mutual information between options and final states.
  • 2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE: Mutual information makes options intrinsically distinct when their executed identities can be inferred from their final states.The formulation balances high option entropy with recoverability of the option from the termination state.
  • 2 INTRINSIC CONTROL AND THE MUTUAL INFORMATION PRINCIPLE: The variational formulation removes the difficult marginal p(s_f|s_0) but introduces an inference distribution q(Ω|s_0,s_f) that predicts options from final states.The paper jointly trains p_C, q, and π to maximize the variational bound I_VB, with I ≥ I_VB.

3 INTRINSIC CONTROL WITH EXPLICIT OPTIONS

The explicit-options algorithm trains closed-loop policies to make option termination states distinguishable, using inferred options to define intrinsic reward and estimate empowerment. In grid worlds, it demonstrates the contrast between adaptive closed-loop control and open-loop action sequences, while exposing training difficulties with function approximation and exploration.

  • Algorithm: The algorithm samples an option, follows its policy to termination, infers the option from the terminal observation, and uses log q(Ω|s0,sf) − log pC(Ω|s0) as intrinsic reward.The policy and option prior are updated with reinforcement learning and policy gradients, respectively.
  • Algorithm: On average, intrinsic reward equals the logarithm of the number of different options available in a state, providing an empowerment estimate.A policy-gradient baseline tracks the expected intrinsic reward, which equals empowerment.
  • 3.1.1 GRID WORLD: In a noisy grid world with 30 discrete options, policies are trained to make the options terminate in as many different states as possible.The environment has five actions and a 0.2 probability of being pushed randomly after each step; the option prior is uniform.
  • 3.1.2 ’DANGEROUS’ GRID WORLD: In the dangerous grid world, open-loop policies lose track of checkerboard sub-lattices and fall toward a low-empowerment trap, whereas observed closed-loop policies navigate the square and learn options there.The agent can respond to its current sub-lattice under closed-loop control but not when committing to an action sequence.
  • 3.2 THE IMPORTANCE OF CLOSED LOOP POLICIES: Open-loop empowerment decreases with option length and favors the top-left corridor, while closed-loop empowerment grows quadratically and favors the grid center.The open-loop reset probability grows exponentially with option length in the dangerous grid world.
  • 3.3 ADVANTAGES AND DISADVANTAGES: The method is simple, model-free, compatible with general function approximation and discrete or continuous option spaces, but training becomes difficult with neural networks and continuous options.The authors report success in simple environments with linear approximation and small finite option spaces, with partial success after alternating fixed-reward and policy updates.

4 INTRINSIC CONTROL WITH IMPLICIT OPTIONS

Algorithm 2 learns implicit options by training a policy to produce diverse, reliably distinguishable terminal states, using an inference policy to assign intrinsic rewards. Experiments show effective control across navigational, visual, object-manipulation, distractor, and noisy environments.

  • Method: Algorithm 2 uses the action space as the option space, trains an inference policy to predict actions from final observations, and reinforces the acting policy with intrinsic rewards.The intrinsic reward is the logarithm of the probability assigned by πq to the action taken, and cumulative intrinsic return is maximized.
  • Four-room grid world: The acting policy implicitly learns to navigate through narrow doors and reach extended trajectories across rooms without receiving the final state explicitly.The final-state-conditioned policy πq provides the conditioning signal, while the followed policy πp learns the corresponding behavior implicitly.
  • Control results: 6.0 nats of average empowerment corresponded to approximately 403 reachable states in the four-room grid world, with endpoint distributions roughly uniform over reachable points.In a three-dimensional visual environment, trajectories of length 12 achieved 5.4 nats, corresponding to 221 different states.
  • Control results: 7.1 nats of average empowerment corresponded to approximately 1200 reachable states when the agent manipulated three pushable blocks.The trajectory demonstrated sequential block pushing together with movement around obstacles.
  • Elements beyond control: The agent achieved nearly maximum intrinsic control over MNIST digit classes despite 600002 total visual states, reaching 99.5 of 100 controllable states.The environment randomly sampled digit instances within each selected class, requiring control-based discrimination of the classes.
  • Robustness and option structure: Closed-loop options performed much better than open-loop options in a noisy grid world, while empowerment remained unchanged when uncontrollable visual distractors were added.The closed-loop policy could correct for environment noise, whereas the open-loop policy could not reliably navigate toward the actual target; distractor robustness was reported separately.

5 CONCLUSION

The paper formalizes intrinsic control maximization for unsupervised option learning and evaluates two algorithms across diverse experiments. It also shows that closed-loop policies matter for empowerment and that intrinsic control can support extrinsic reward maximization.

  • The paper introduces a formalism of intrinsic control maximization for unsupervised option learning.
  • Two algorithms are presented and analyzed across a diverse range of experiments.
  • Closed-loop policies are important for estimating empowerment.
  • Unsupervised learning and intrinsic control are useful for extrinsic reward maximization.

APPENDIX 1: DERIVATION OF THE VARIATIONAL BOUND ON MUTUAL INFORMATION

This appendix derives the variational bound used for the mutual-information objective. The derivation begins by omitting variable indices and the initial state.

  • The derivation presents the variational bound used in the method.

APPENDIX 2: DERIVATION OF ALGORITHM 1

Algorithm 1 estimates the variational-bound gradient by sampling an option and its termination state, then applying likelihood-style policy-gradient updates. A baseline reduces the update using the sampled intrinsic reward.

  • The variational-bound derivation makes parameters θ of pC and φ of q explicit.
  • The φ update samples an option from pC and a termination state by following the policy until termination.
  • The θ update uses a standard policy-gradient procedure based on the sampled option and corrected reward rI − b.The option is sampled from pC, and the resulting log-probability gradient is multiplied by the corrected reward.
  • The baseline b(s0) predicts the sampled empowerment rI and is trained by regressing toward observed rI.
  • The policy can be updated with any reinforcement-learning algorithm to maximize the sampled variational-bound reward rI.

APPENDIX 3: FUNCTIONAL FORMS FOR INTRINSIC CONTROL WITH IMPLICIT OPTIONS

The appendix specifies functional forms for implicit-option policies using observation and action sequences, final-observation embeddings, and recurrent computations. Figure 6 evaluates empowerment with moving visual distractors across environment sizes and option lengths.

  • Functional forms for implicit options: The implicit-option policy computations use an observation-action sequence and an embedding of the final observation.
  • Functional forms for implicit options: The functional forms use linear operations, rectifier nonlinearities, concatenation, and LSTM states.
  • Figure 6: Figure 6 compares empowerment in grid worlds with and without distractors across environment sizes and option lengths.
  • Figure 6: The distractors move and are observed but do not affect the agent, allowing evaluation of whether intrinsic control ignores them.

APPENDIX 4: THE EFFECT OF DISTRACTORS ON INTRINSIC CONTROL

The experiment examines whether the agent can disregard visual distractors that do not affect intrinsic control across grid-world settings.

  • The agent learns to ignore distractors that do not affect its intrinsic control.The grid world includes moving points on a separate feature plane that do not affect the agent, serving only as visual distractors.
  • The distractor experiment varies environment sizes and option lengths.
  • The distractors are moving points that affect the agent only visually, not through the environment’s intrinsic control.
Loading 1611.07507v1…