PaperlayerView exact revision on arXiv

Source-linked AI summary

Planning with Goal-Conditioned Policies

Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, Sergey Levine

arXiv:1911.08453v1cs.LGcs.AIcs.ROstat.ML

TL;DR

The paper asks whether reinforcement learning can automatically provide the abstractions needed for planning in temporally extended tasks, especially with image observations. It introduces LEAP, which plans over latent subgoals using goal-conditioned policies as action abstractions and a latent model as a state abstraction. LEAP significantly outperforms prior work on image-based robot navigation and manipulation tasks requiring non-greedy, multistage behavior.

  • Problem

    Planning needs suitable state and transition abstractions that are often designed by hand, while model-free RL commonly struggles with temporally extended tasks and high-dimensional observations.

  • Method

    LEAP combines short-horizon goal-conditioned policies with model-based planning over latent representations of subgoals and valid states.

  • Results

    LEAP significantly outperforms prior work on image-based robot navigation and manipulation tasks requiring non-greedy, multistage behavior.

  • Takeaways & Limitations

    Using learned policies for temporal abstraction and latent representations for state abstraction enables planning over meaningful subgoals without modeling low-level image observations.

  • Takeaways & Limitations

    TDM reachability predictions are limited to valid goals resembling the states on which the TDM was trained, motivating state abstraction to constrain planning.

Abstract

from arXiv · show

Planning methods can solve temporally extended sequential decision making problems by composing simple behaviors. However, planning requires suitable abstractions for the states and transitions, which typically need to be designed by hand. In contrast, model-free reinforcement learning (RL) can acquire behaviors from low-level inputs directly, but often struggles with temporally extended tasks. Can we utilize reinforcement learning to automatically form the abstractions needed for planning, thus obtaining the best of both approaches? We show that goal-conditioned policies learned with RL can be incorporated into planning, so that a planner can focus on which states to reach, rather than how those states are reached. However, with complex state observations such as images, not all inputs represent valid states. We therefore also propose using a latent variable model to compactly represent the set of valid states for the planner, so that the policies provide an abstraction of actions, and the latent variable model provides an abstraction of states. We compare our method with planning-based and model-free methods and find that our method significantly outperforms prior work when evaluated on image-based robot navigation and manipulation tasks that require non-greedy, multi-staged behavior.

1 Introduction

The paper combines model-free reinforcement learning with planning by using learned policies and latent state representations as abstractions for temporally extended tasks. LEAP plans over subgoals in latent space while low-level policies execute them from image observations.

  • Motivation: Model-free RL acquires behaviors from direct interaction but often lacks the temporal compositionality needed for complex, long-horizon tasks.Planning provides temporal compositionality but typically depends on accurate models and manually designed abstractions.
  • Abstractions: The framework uses abstractions of states and transitions so planning can operate at a higher level than low-level observations.Temporal abstractions plan over higher-level subgoals, while state abstractions use simpler state representations.
  • Method: LEAP learns low-level goal-conditioned policies and uses their value functions as implicit models for planning over intermediate goals.The value function indicates whether a proposed goal is reachable, allowing the planner to compose multiple short-horizon behaviors.
  • Method: LEAP combines model-free short-horizon goal reaching with model-based planning over latent subgoals for image-based, multistage tasks.The latent representation restricts planning to meaningful states, while the policy reaches the selected subgoals.

2 Related Work

Related work has used goal-conditioned policies, latent representations, and hybrid model-free/model-based methods, but LEAP combines these ideas to plan over multiple temporal subgoals in high-dimensional tasks.

  • Goal-conditioned RL: Goal-conditioned methods train policies to reach specified goals but often struggle with long-horizon problems requiring planning.LEAP uses these policies as planning abstractions rather than proposing another goal-conditioned RL method.
  • Latent-space planning: Latent-space planning reduces the difficulty of high-dimensional observations, while LEAP additionally uses goal-conditioned policies for temporal abstraction.This distinguishes LEAP from methods that provide only state abstractions.
  • Temporal difference models: Prior temporal difference models used single-step planning toward one low-dimensional goal, whereas LEAP plans over multiple subgoals.LEAP also learns state abstractions for planning in settings where not all observations are valid states.
  • Hybrid methods: Unlike approaches that use models or planning-inspired architectures to accelerate model-free learning, LEAP uses model-free policies as the abstraction for planning.The higher-level planner can flexibly compose these policies for new tasks.

3 Background

The paper formulates finite-horizon goal-conditioned RL with policies, value functions, and temporal difference models that estimate how well goals can be reached. These reachability estimates support planning, but only for valid goals resembling the training states.

  • Goal-conditioned MDP: The task is a finite-horizon goal-conditioned MDP defined by states, goals, actions, dynamics, rewards, horizon, and initial and goal distributions.The dynamics are time-invariant and unknown, while the policy conditions on the current state, goal, and time.
  • Goal-conditioned MDP: Goal-conditioned RL seeks a policy that maximizes expected cumulative reward while reaching a sampled goal from an initial state distribution.In the considered setting, goals and states occupy the same space.
  • Value functions: The goal-conditioned value function predicts expected future reward for a state, goal, and time.Temporal difference models use a particular time-varying reward to interpret value as goal-reaching performance after a fixed number of steps.
  • Temporal difference models: TDMs measure reachability by estimating how close a goal-conditioned policy can get to another state within a specified horizon.This supplies temporal abstraction for planning over subgoals.
  • Limitation: TDM reachability predictions are reliable only for valid goals resembling the states used during training.This limitation motivates adding state abstractions that restrict planning to valid states.

4 Planning with Goal-Conditioned Policies

LEAP combines short-horizon goal-conditioned policies with planning over latent subgoals, enabling long-horizon tasks while keeping planned states valid in image-based environments.

  • Planning over Subgoals: Goal-conditioned policies reach short-horizon subgoals, while a planner decomposes long-horizon tasks into K sequential subgoals.The policy attempts each subgoal for its allocated time interval before the planner replans.
  • Planning over Subgoals: A goal-conditioned value function measures subgoal reachability, with V(s, g, t) = 0 indicating that the policy reaches goal g after t steps.The resulting feasibility vector evaluates each transition from the current state through intermediate subgoals to the true goal.
  • Planning over Subgoals: Minimizing the norm of the feasibility vector searches for subgoals that make the overall path feasible and terminate at the true goal.The method uses the ℓ∞-norm, which the authors report outperforming the ℓ1-norm.
  • Optimizing over Images: Direct image-space planning is difficult because 64x64 images with three subgoals require over 10,000 optimization dimensions and must remain on an unknown valid-state manifold.The valid image observations form an unknown r-dimensional manifold embedded in a higher-dimensional space R^N, with N ≫ r.
  • Optimizing over Images: LEAP instead optimizes compact latent variables learned by a VAE, decodes them into goal states, and penalizes latent points with low prior likelihood.High-likelihood latent variables correspond to realistic images and help keep planned subgoals within the valid-state manifold.
  • Algorithm: At test time, LEAP trains a goal-conditioned policy and VAE, chooses latent subgoals for a new goal, executes the first decoded goal, and replans after its allotted steps.Algorithm 1 assigns the final subgoal to the true goal and repeatedly optimizes, decodes, and executes subgoals with the policy.

5 Experiments

Experiments evaluate LEAP on vision-based navigation and manipulation, non-vision ant navigation, and ablations of its latent-space and shared-encoder design. LEAP significantly outperforms prior methods on challenging long-horizon tasks, while latent-space planning is critical for valid subgoals and performance.

  • Vision-based comparison: LEAP significantly outperforms prior model-free and model-based methods on vision-based 2D Navigation and Push and Reach tasks.The tasks require non-greedy, multistage behavior from image observations.
  • Vision-based comparison: By 400k environment steps, LEAP achieves a final puck distance under 10 cm, while TDM-100 requires five times as many samples.LEAP composes the short-horizon TDM-25 with three subgoals.
  • Vision-based comparison: In Push and Reach, LEAP selects subgoals that move toward the puck, initiate and complete pushing, and then reach the goal hand position.The decoded latent subgoals correspond to natural task stages.
  • Non-vision comparison: In Ant Navigation, LEAP is the only method that successfully reaches the goal around a long wall despite large negative rewards during the trajectory.The task uses Tmax = 600 and 11 subgoals spaced 50 time steps apart.
  • Ablation study: Planning over latent states achieves dramatically better performance than planning over raw images, whose optimized subgoals are invalid observations outside the valid-state manifold.Reusing the pretrained VAE encoder also accelerates learning, though training from scratch can succeed.
  • Ablation study: LEAP performs well across a wide range of λ, the ℓ∞-norm performs better, and CEM consistently outperforms gradient-based optimizers.These comparisons cover optimizer loss and policy performance.

6 Discussion

LEAP combines goal-reaching policies with latent-state planning to solve temporally extended tasks from high-dimensional observations. The approach avoids both the limitations of purely model-free goal reaching and low-level modeling challenges.

  • LEAP forms temporal abstractions from goal-reaching policies and state abstractions from learned representations for planning.The planner operates over latent states and supplies planned states as subgoals to goal-conditioned policies.
  • Planning over learned latent states enables LEAP to solve tasks that conventional model-free goal-reaching policies cannot solve effectively.The discussion attributes this to composing goal-reaching behavior through planned subgoals.
  • LEAP avoids the challenges of modeling low-level observations associated with fully model-based methods.Its planning operates on learned representations rather than directly modeling high-dimensional observations.
  • The authors identify exploration for goal-conditioned policies as a largely neglected question requiring further study.

A.1 Norm Ablation

The norm ablation compares ℓ∞ and ℓ1 objectives for minimizing the feasibility vector. ℓ∞ performs better because it more consistently drives every feasibility term toward zero.

  • ℓ∞-norm outperforms ℓ1-norm when minimizing the feasibility vector.

A.2 Optimizer Ablation

The optimizer ablation evaluates CEM against tuned L-BFGS, Adam, RMSProp, and SGD on 2D Navigation. CEM performs best in both optimizer loss and resulting task performance.

  • CEM consistently outperforms L-BFGS, Adam, RMSProp, and SGD in optimizer loss and final task performance.The comparison uses optimizers with tuned learning rates.
  • Lower optimizer loss from CEM corresponds to better performance of the policy using its selected plan.

A.3 Likelihood Penalty Ablation

The likelihood-penalty ablation varies λ and measures final RL performance, VAE-prior likelihood, and V values. Increasing λ raises likelihood but lowers V values, producing task-specific optima.

  • Increasing λ raises log-likelihood under the VAE prior while decreasing V values.
  • RL performance is maximized at λ = 0.01 for 2D Navigation and between 0.0001 and 0.01 for Push and Reach.
  • Ant Navigation has an independently verified optimal choice of λ = 0.1.

B.1 2D Navigation

The 2D Navigation experiments use image-based navigation in a constrained room, alongside Push and Reach and Ant Navigation evaluations. The section also identifies the corresponding complete-results figures and TD3 hyperparameter table.

  • 2D Navigation: The 2D Navigation agent moves through an 8 × 8 room containing a central U-shaped wall from 48 × 48 RGB observations.Actions specify 2D velocity, with movement limited to 0.15 units in either dimension per timestep.
  • Results: Complete results are presented separately for 2D Navigation, Push and Reach, and Ant Navigation in Figures 10, 11, and 12.The supplied materials identify these figures by task but do not state their numerical outcomes.
  • Push and Reach: The Push and Reach task uses 84 × 84 RGB top-down images and 2D end-effector position control.Each action moves the robot end effector by at most 2 cm in either dimension, and the task cannot be solved by simple unidirectional pushing.
  • Ant Navigation: Ant Navigation takes place in a 7.5 × 18 room with a narrow central wall and an ant of roughly 0.75-unit radius.The state includes position, orientation, joint angles, and velocities.
  • Training: TD3 hyperparameters are summarized in Table 1.The supplied table passage provides the table title but no individual hyperparameter values.

C Implementation Details

The implementation trains goal-conditioned networks with TD3, samples goals using several replay-buffer and hindsight strategies, and optimizes latent subgoal sequences with CEM. Separate VAE designs represent image-based tasks and the full ant state.

  • Goal-conditioned learning: The Q network and policy concatenate all inputs and use feed-forward networks trained with twin delayed deep deterministic policy gradient.For RIG, the Q network outputs a scalar discounted return; for TDMs, it outputs per-dimension negative final-state distance to the goal.
  • Goal sampling: Goal sampling combines original trajectory goals, randomly sampled replay-buffer states, and future trajectory states selected through hindsight experience replay.The listed proportions are 20% original goals, 40% random replay-buffer states, and 40% future states.
  • Goal sampling: For the Ant Navigation task, replay-buffer sampling is split between 20% replay-buffer states and 20% oracle environment goals.This task-specific adjustment is stated separately from the general sampling scheme.
  • Latent space optimization: CEM optimizes sequences of K r-dimensional latent subgoals by repeatedly sampling candidates, evaluating them with Equation 3, and fitting a diagonal Gaussian to elite samples.The default procedure uses a Gaussian initialization, the top 5% of samples, 15 iterations, and 1000 samples per iteration; Ant Navigation uses 10000 samples and 50 iterations.
  • Latent space optimization: The latent-likelihood weight λ is 0.1 for 2D Navigation and Ant Navigation and 0.001 for Push and Reach.The filtering schedule also changes during optimization, using different elite fractions across the iteration phases.
  • State representation: Separate VAEs are used for 48 × 48 2D Navigation images and 84 × 84 Push and Reach images.The architectures use task-specific convolutional encoder and decoder kernel sizes, strides, and channel counts.
  • State representation: For Ant Navigation, the VAE models the full ant state with multilayer perceptron encoder and decoder networks and an 8-dimensional representation.It uses hidden sizes [64, 128, 64], ReLU activation, and mean squared error loss.
Loading 1911.08453v1…