PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Actionable Representations with Goal-Conditioned Policies

Dibya Ghosh, Abhishek Gupta, Sergey Levine

arXiv:1811.07819v2cs.LGcs.AIstat.ML

TL;DR

Reinforcement-learning representations often rely on generative approaches, while decision-making needs representations that capture actionable factors and environment dynamics. The paper learns such representations from goal-conditioned policies and reports benefits across simulated robotic tasks and applications, including a faster policy-learning result in one task.

  • Problem

    Generative RL representations aim to capture all observation factors, whereas the paper seeks representations focused on factors relevant to decision making and dynamics.

    Source: §1 · p. 1

  • Method

    ARC uses a goal-conditioned policy to identify differences in the actions required to reach goals, then learns a state representation reflecting those functional differences.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    ARC outperforms prior representation-learning methods across simulated robotic domains; in policy-feature learning, it reaches high success by Iteration 100 while the next-best method reaches the goal only 5% of the time.

    Source: §1 · p. 2, §6.6 · p. 9

  • Takeaways & Limitations

    ARC representations are applied to task-specific policies, hierarchical RL, exploration, and reward shaping, including reward shaping based on distance in ARC space.

    Source: §1 · p. 2, §4.2 · p. 4

    ARC is learned by first training a goal-conditioned policy, which the paper identifies as a significant assumption.

    Source: §3 · p. 4

Abstract

from arXiv · show

Representation learning is a central challenge across a range of machine learning areas. In reinforcement learning, effective and functional representations have the potential to tremendously accelerate learning progress and solve more challenging problems. Most prior work on representation learning has focused on generative approaches, learning representations that capture all underlying factors of variation in the observation space in a more disentangled or well-ordered manner. In this paper, we instead aim to learn functionally salient representations: representations that are not necessarily complete in terms of capturing all factors of variation in the observation space, but rather aim to capture those factors of variation that are important for decision making -- that are "actionable." These representations are aware of the dynamics of the environment, and capture only the elements of the observation that are necessary for decision making rather than all factors of variation, without explicit reconstruction of the observation. We show how these representations can be useful to improve exploration for sparse reward problems, to enable long horizon hierarchical reinforcement learning, and as a state representation for learning policies for downstream tasks. We evaluate our method on a number of simulated environments, and compare it to prior methods for representation learning, exploration, and hierarchical reinforcement learning.

1 INTRODUCTION

The paper seeks representations that emphasize state factors relevant to decision making rather than reconstructing every variation in observations. It proposes learning these actionable representations from goal-conditioned policies and applying them to exploration, hierarchical RL, reward shaping, and downstream policies.

1 INTRODUCTION

  • Prior RL representation learning often uses generative approaches that model all state factors, while ARC targets factors relevant to decision making.The paper notes that modeling interactions among all state elements can be challenging.Source: §1 · p. 1
  • ARC compares actions required to reach different goals, representing functionally different goals as far apart even when they are spatially close.Figure 1 illustrates this with nearby houses that require different roads to reach.Source: §1 · p. 2, Figure 1 · p. 2
  • A goal-conditioned policy can be learned through unsupervised exploration without knowledge of downstream tasks, and its representations can support harder tasks than simple goal reaching.The representation-learning process can also be recursive, using one policy’s ARC representations to learn a better goal-conditioned policy.Source: §1 · p. 2
  • ARC is evaluated for task-specific policies, hierarchical RL, and reward shaping, and is reported to outperform prior representation-learning methods across simulated robotic domains.The comparison includes wheeled locomotion, legged locomotion, and robotic manipulation.Source: §1 · p. 2

2 PRELIMINARIES

Goal-conditioned RL trains policies to reach varying goal states, while maximum-entropy RL favors policies that balance reward and action entropy. The paper’s illustration connects differences in goal-conditioned actions to actionable distances and learned representations.

2 PRELIMINARIES

  • A goal-conditioned policy takes both the current state and a goal as input and is trained to reach goals drawn from the state space.The policy maximizes expected goal-reaching return.Source: §2 · p. 2
  • Maximum-entropy RL maximizes reward together with policy entropy, yielding stochastic actions whose randomness reflects how sensitive rewards are to action choice.Actions are more deterministic when the choice is critical and more random when it has minimal effect on future rewards.Source: §2 · p. 2
  • Figure 2 depicts actionable distance as divergence between goal-conditioned action distributions for two states, which is then used to learn a representation..Source: Figure 2 · p. 3

3 LEARNING ACTIONABLE REPRESENTATIONS

The method defines functional differences between goal states through the actions a goal-conditioned policy induces, then learns a representation whose Euclidean distances reflect those differences. Its setup assumes a trained goal-conditioned policy and considers divergence across starting states.

3 LEARNING ACTIONABLE REPRESENTATIONS

  • Actionable distances compare the action distributions induced by two goals at intermediate states: similar distributions suggest functional similarity, while different distributions suggest functional difference.The method assumes an already trained maximum-entropy goal-conditioned policy that can reach arbitrary goal states from arbitrary starting states.Source: §3 · p. 3
  • The actionable distance averages divergence across initial states; restricting the states considered can miss functional differences that arise elsewhere.Under maximum-entropy policies, the distance is well-defined, and goals are similar when they require the same actions and are equally easy to reach.Source: §3 · p. 3
  • The representation is optimized so Euclidean distances between states correspond to actionable distances, emphasizing functionally relevant state elements and suppressing less relevant ones.The resulting distances are meaningful because the representation uses a goal-conditioned policy and captures dynamics and local connectivity.Source: §3 · p. 3
  • A trained goal-conditioned policy is a prerequisite; the paper says it can be learned with existing RL methods using a sparse, task-agnostic reward.The paper argues that functionality-aware representations require environment interaction to observe action outcomes.Source: §3 · p. 4

4 USING ACTIONABLE REPRESENTATIONS FOR DOWNSTREAM TASKS

The section presents ARCs as a way to extend goal-conditioned policies to non-goal-reaching tasks, larger state-space regions, and temporally extended tasks. It describes using ARC as policy input, for reward shaping, and as a hierarchical goal space.

4.1 FEATURES FOR LEARNING POLICIES

  • For non-goal-reaching downstream tasks, policies or value functions can consume φ(s) instead of s, using dynamics-aware features to prioritize decision-relevant state information.Source: §4.1 · p. 4

4.2 REWARD SHAPING

  • ARC shaping adds negative Euclidean distance in ARC space to a sparse goal reward, providing a shaping signal for exploration and policy learning.Source: §4.2 · p. 4
  • ARC typically generalizes better than goal-conditioned policies trained in small regions, including for goals that are difficult for those policies to reach.Source: §4.2 · p. 4

4.3 HIERARCHICAL REINFORCEMENT LEARNING

  • For sequences of behaviors, a high-level policy can issue ARC-space commands that a decoder maps to state goals for the goal-conditioned policy to execute.Using ARC as the goal space de-emphasizes goal components irrelevant to the optimal action and reduces high-level search noise.Source: §4.3 · pp. 4–5
  • ARC clusters can serve as discrete meta-actions: the controller samples a cluster, then a state goal within it, and ARC-induced cluster spaces outperform those from other representations on less-granular room-navigation tasks.Figure 3 contrasts direct ARC-space commands with commands in ARC cluster space.Source: §4.3 · p. 5, Figure 3 · p. 5

5 RELATED WORK

The related work contrasts ARC with generative, contrastive, and dynamics-aware representation learning approaches. It positions ARC as selective for action choice rather than complete input modeling or generic physical-feature representation.

  • Generative representation learning models data or transitions, but must explain and generate the entirety of the input observation.Source: §5 · p. 5
  • Contrastive methods avoid generation but may still model the whole input or rely on user-defined heuristics, whereas ARC directly targets decision-relevant features.Source: §5 · p. 6
  • Unlike dynamics-aware alternatives, ARC represents features relevant to choosing actions, not all physically relevant features, independently controllable features, or only greedy action predictions.Source: §5 · p. 6

6 EXPERIMENTS

The experiments are framed around questions about learning ARCs across continuous-control environments and the properties of the resulting representations.

  • The experimental evaluation is framed around research questions.Source: §6 · p. 6
  • One question is whether ARCs can be learned for multiple continuous-control environments.Source: §6 · p. 6
  • The evaluation also asks what properties the learned representations exhibit.Source: §6 · p. 6

4. Do ARCs provide an effective mechanism for hierarchical RL?

The experiments evaluate actionable representations across simulated navigation, locomotion, and manipulation tasks, using goal-conditioned policies to learn representations for downstream control. ARC supports reward shaping, policy learning, and hierarchical control, with reported advantages over alternative representations and methods.

6.1 DOMAINS

  • The evaluation covers six simulated environments, and ARC and comparison representations are trained on the same goal-conditioned-policy trajectory data.The domains include navigation, wheeled and legged locomotion, and object pushing; comparisons include generative, predictive, inverse-model, and task-specific methods.Source: §6.1 · p. 6, §6.2 · p. 7, §6.3 · p. 7

6.4 ANALYSIS OF LEARNED ACTIONABLE REPRESENTATIONS

  • In navigation, ARC separates states across walls and clusters rooms according to passage bottlenecks, unlike the VAE representation in the four-room comparison.The visualizations show that geometrically nearby states can be far apart in ARC space when they are functionally distinct.Source: §6.4 · pp. 7–8, Figure 5 · p. 8
  • Perturbation analyses show ARC emphasizes CoM, vehicle position, and block position while suppressing joint angles, orientation, and arm movement; VAE and state representations do not capture this pattern.The analysis compares representation changes when decision-relevant factors or secondary factors are varied.Source: Figure 6 · p. 8, §6.4 · pp. 8–9

6.5 LEVERAGING ACTIONABLE REPRESENTATIONS FOR REWARD SHAPING

  • ARC reward shaping learns faster and achieves better asymptotic performance than the compared methods, matching hand-designed CoM-distance shaping.ARC and the predictive model outperform VIME, which uses novelty-based exploration without considering environment dynamics.Source: Figure 7 · p. 9, §6.5 · p. 9

6.6 LEVERAGING ACTIONABLE REPRESENTATIONS AS FEATURES FOR LEARNING POLICIES

  • By iteration 100, the ARC-feature policy solves the reach-while-avoiding task with high success, while the next-best method reaches the goal only 5% of the time.All methods ultimately learn to solve the task, but ARC features learn faster.Source: §6.6 · p. 9, Figure 8 · p. 10

6.7 BUILDING HIERARCHY FROM ACTIONABLE REPRESENTATIONS

  • Hierarchical meta-policies using ARC outperform alternative representations, option-critic, and state-space goal commands on the hierarchical tasks.TRPO and option-critic make no progress; the authors report that ARC-space commands reduce action redundancy and make high-level search easier.Source: Figure 10 · p. 10, §6.7 · p. 10

7 DISCUSSION

ARC represents decision-relevant state features using a goal-conditioned policy, and the paper identifies policy training choices and future directions for scaling this approach.

7 DISCUSSION

  • ARC captures decision-relevant state features, is implicitly dynamics-aware, and supports downstream policy learning, hierarchical reinforcement learning, and exploration.The discussion identifies off-policy goal-conditioned policy learning and interleaving policy and representation learning as promising directions.Source: §7 · p. 11

A.1 TRAINING THE GOAL-CONDITIONED POLICY

  • Goal-conditioned policies are trained with TRPO and entropy regularization, with the goal space matching the state space.Each episode samples start and goal states uniformly and uses a task-specific sparse reward, except Sawyer uses a shaped reward because the sparse formulation is highly sample-inefficient.Source: §A.1 · p. 14
  • The goal-conditioned policy uses a Gaussian action distribution whose neural-network mean receives concatenated state and goal inputs.The mean network has hidden layers of 150, 100, and 50 units, and the learned diagonal covariance is initialized to identity.Source: §A.1 · p. 14

A.2 TRAINING THE REPRESENTATION

The appendices describe how representations are trained and used in reward shaping, downstream policies, and two forms of hierarchical control. They specify data collection and task setups for these evaluations.

A.2 TRAINING THE REPRESENTATION

  • Representations are trained on trajectories from a goal-conditioned policy, with baseline methods spanning reconstruction and one-step prediction objectives.The dataset contains 500 trajectories of 100 timesteps, intended to cover the state space and include meaningful transitions; methods use neural networks, Adam, validation data, and coarse hyperparameter sweeps.Source: §A.2 · p. 14

A.3 REWARD SHAPING

  • Reward shaping is evaluated by learning goal-reaching policies in larger navigation regions using a surrogate reward and a policy initialized from the smaller-region goal-conditioned policy.The study compares representation-based shaping with a hand-specified heuristic and VIME, sweeping the shaping weight α and VIME parameter η.Source: §A.3 · pp. 14–15

A.4 FEATURES FOR POLICIES

  • For downstream policy features, a fixed representation feeds a TRPO policy on an Ant reach-while-avoid task.The task starts at (-1.5, -1.5), targets (1.5, 1.5), and avoids a radius-1 circle centered at the origin; gradients do not update the representation.Source: §A.4 · p. 15

A.5 HIERARCHICAL REINFORCEMENT LEARNING IN LATENT SPACE

  • Latent-space hierarchical control uses a high-level policy to choose latent goals, which a reconstruction network converts into states for the goal-conditioned policy.In the Ant waypoint task, the low-level policy runs for 50 timesteps per high-level goal; the reconstruction network is trained to recover states from representations.Source: §A.5 · p. 15

A.6 HIERARCHICAL REINFORCEMENT LEARNING IN CLUSTER SPACE

  • Cluster-space hierarchical control applies k-means to learned representations, then has a high-level policy choose a cluster from which a latent goal is sampled.The room-navigation task uses 50 ordered checkpoints, with consecutive checkpoints constrained to different rooms; cluster count is swept over values of k.Source: §A.6 · p. 16

B BENCHMARK REPRESENTATIONS

The appendices define benchmark representation objectives, describe the simulated tasks, and detail how hyperparameters are selected for representations and downstream evaluations.

B BENCHMARK REPRESENTATIONS

  • The benchmark compares ARC with VAE, slowness, predictive-model, and inverse-model representations trained on trajectory data.The ARC objective uses precomputed actionable distances; the other descriptions specify reconstruction, forward-prediction, or inverse-dynamics components, with β balancing prediction errors where applicable.Source: B BENCHMARK REPRESENTATIONS · pp. 16–17

C TASK DESCRIPTIONS •

  • The task suite spans 2D navigation, wheeled navigation, Ant locomotion, and Sawyer manipulation, with distinct state and action spaces and downstream evaluations.The descriptions include navigation through rooms or waypoints, reaching while avoiding locations, and block manipulation; Sawyer goal-conditioned training uses a shaped reward because it requires many samples.Source: C TASK DESCRIPTIONS • · p. 17

D HYPERPARAMETER TUNING

  • Hyperparameter tuning covers representation settings, reward-shaping scales, and cluster counts, with selected representations then fixed across downstream applications.ARC requires no tuned parameters beyond latent dimension; the cluster-count sweep found most representations robust to the number of clusters.Source: D HYPERPARAMETER TUNING · pp. 17–18

Related papers

Loading 1811.07819v2…