Source-linked AI summary
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
Suraj Nair, Chelsea Finn
TL;DR
Long-horizon visual manipulation is limited by sparse goal costs and compounding model error. HVF generates goal-conditioned visual subgoals, optimizes them to reduce segment difficulty, and plans between them. It improves success across simulated tasks and extends to real cluttered scenes, while remaining computationally demanding and dependent on visual cost and predictive-model quality.
Problem
Long-horizon visual planning struggles with compounding uncertainty and sparse cost signals when only a distant goal image is provided.
Method
HVF uses self-supervised generative image models and visual model predictive control to generate and optimize subgoals that decompose tasks into shorter planning segments.
Results
HVF provides at least a 20% absolute success-rate improvement across simulated tasks, exceeding 42% success with two subgoals on the longest-horizon task.
Takeaways & Limitations
HVF discovers semantically meaningful visual subgoals and performs challenging long-horizon tasks without rewards or demonstrations.
Takeaways & Limitations
HVF requires expensive nested MPC and video-prediction optimization, while visual cost functions and predictive models remain performance bottlenecks.
Abstract
from arXiv · showhide
Video prediction models combined with planning algorithms have shown promise in enabling robots to learn to perform many vision-based tasks through only self-supervision, reaching novel goals in cluttered scenes with unseen objects. However, due to the compounding uncertainty in long horizon video prediction and poor scalability of sampling-based planning optimizers, one significant limitation of these approaches is the ability to plan over long horizons to reach distant goals. To that end, we propose a framework for subgoal generation and planning, hierarchical visual foresight (HVF), which generates subgoal images conditioned on a goal image, and uses them for planning. The subgoal images are directly optimized to decompose the task into easy to plan segments, and as a result, we observe that the method naturally identifies semantically meaningful states as subgoals. Across three out of four simulated vision-based manipulation tasks, we find that our method achieves nearly a 200% performance improvement over planning without subgoals and model-free RL approaches. Further, our experiments illustrate that our approach extends to real, cluttered visual scenes. Project page: https://sites.google.com/stanford.edu/hvf
1 INTRODUCTION
Long-horizon visual manipulation remains difficult because supervision is sparse, uncertainty compounds with horizon, and generalization requires diverse environments. HVF addresses this by generating and optimizing visual subgoals that decompose tasks into shorter segments.
- Long-horizon visual manipulation must generalize across novel scenes and objectives without practical detailed supervision during data collection.
- Increasing planning horizons compound model error and make visual cost signals noisy or sparse, especially when only the final goal image is provided.
- Subgoals such as grasping a drawer handle, opening the drawer, and reaching for a book provide shorter planning segments with useful intermediate cost signals.
- HVF combines generative image models with model predictive control to decompose long-horizon visual tasks into optimized subgoal sequences.
2 RELATED WORK
Prior work addresses visual manipulation, long-horizon hierarchy, classical planning, and self-supervised goal reaching through distinct model-based, model-free, symbolic, and compositional approaches. HVF is situated among methods that seek novel goals without demonstrations or rewards.
- Visual reinforcement learning and video prediction have enabled complex pixel-based behaviors and generalization to novel objects.
- Prior hierarchical approaches use demonstrations, manually specified primitives, learned temporal abstractions, or model-free reinforcement learning.
- Classical, symbolic, and task-and-motion planning solve long-horizon problems but commonly assume restrictive state-space, reachability, or object knowledge conditions.
- Recent self-supervised methods reach novel goals using model-free or model-based reinforcement learning and identify bottlenecks in long-horizon visual tasks.
3 PRELIMINARIES
The paper formulates visual manipulation as a goal-conditioned MDP whose states are images and whose dynamics, goals, costs, and discount define planning. Experiments use sparse pixel-space distance rather than shaped visual costs.
- The goal-conditioned MDP consists of state space S, action space A, dynamics p, goal set G, cost C, and discount factor λ.
- States correspond to images, actions affect environment dynamics, and goals are subsets of possible states.
- The experiments assume sparse ℓ2 pixel-space distance because accurately reflecting distance between images with a cost function is challenging.
- Visual foresight learns an action-conditioned video prediction model from randomly collected interaction data using supervised maximum likelihood.
4 HIERARCHICAL VISUAL FORESIGHT
HVF samples plausible visual states, evaluates their planning costs, and optimizes latent subgoal sequences so that the worst segment is easy to plan. It then executes visual MPC between the selected subgoals.
- 4 HIERARCHICAL VISUAL FORESIGHT: HVF addresses sparse costs and compounding model error by decomposing long-horizon visual planning into shorter tasks using sampled states.
- Generative model: The method learns a generative model from self-supervised data to sample possible scene states and optimizes subgoals in its low-dimensional latent space.
- Subgoal optimization: HVF optimizes K subgoals by minimizing the maximum planning cost across segments from the current state through the goal.
- MPC and planning: Planning cost between two states is estimated by using the learned dynamics model to plan an action sequence and measure its predicted cost.
- MPC and execution: After optimization, visual MPC executes the task sequentially from each selected subgoal to the next and then to the goal.
- Subgoal optimization: Subgoal optimization uses cross-entropy search in the generative model’s latent space, refining sampled subgoal lists according to their planning effectiveness.
5 EXPERIMENTS
Across simulated navigation and manipulation tasks, HVF improves long-horizon planning by generating useful visual subgoals, including in cluttered real-robot scenes. Its gains are strongest over longer tasks, while performance depends on the number of subgoals and planning configuration.
- Maze Navigation: HVF consistently improves maze-navigation success over TAP and visual foresight without subgoals, especially on longer-horizon settings.Oracle bottleneck subgoals generally perform best, but are not perfect.
- Maze Navigation: HVF-generated maze subgoals correspond to semantically meaningful bottlenecks, such as gaps in walls, for both one- and two-subgoal plans.With two subgoals, the discovered states align with successive gaps required to traverse the maze.
- Simulated Desk Manipulation: HVF with two subgoals yields at least a 20% absolute success-rate improvement across all four simulated desk-manipulation tasks.In the longest task, success rises from approximately 15% with zero or one subgoal to over 42% with two subgoals.
- Simulated Desk Manipulation: HVF discovers actionable desk-manipulation subgoals, including pushing blocks, grasping door handles, sliding doors, and reaching objects.The generated ordering can decompose compound tasks into sequential operations.
- Real Robot Scenes: On the BAIR robot dataset, HVF produces meaningful subgoals in cluttered scenes, including states corresponding to reaching or grasping objects.This evaluates qualitative subgoal generation on real robot imagery.
- Ablations: With a fixed sampling budget, performance drops beyond two subgoals because the subgoal-search problem becomes increasingly difficult.Using the max cost across subsegments is marginally better than using mean cost; HVF still outperforms no-subgoal planning with more visual-MPC samples, while longer horizons favor one over two subgoals.
6 CONCLUSION AND LIMITATIONS
HVF decomposes visual goals into subgoals and learns long-horizon tasks self-supervised, but remains computationally demanding and dependent on better visual models.
- Conclusion: Using more visual MPC samples consistently improves success rates, but HVF still significantly outperforms standard visual foresight.The comparison uses 200 versus 1000 samples at each visual MPC iteration.
- Conclusion: HVF decomposes visual goals into sequences of subgoals and discovers semantically meaningful goals in visual space.The approach performs challenging long-horizon vision-based tasks without rewards or demonstrations.
- Conclusion: HVF learns vision-based tasks entirely through self-supervision without rewards or demonstrations.
- Limitations: HVF requires expensive nested optimization with many iterations of model predictive control and video prediction.The authors suggest policy inference as one possible way to reduce these computational requirements.
- Limitations: Accurate visual cost functions, predictive models, and generative models for cluttered scenes with novel objects remain important bottlenecks.These limitations constrain further performance and applicability in real-world settings.
A METHOD DETAILS
HVF evaluates and executes candidate subgoals with visual MPC, while VAE or CVAE models provide image samples that make subgoal optimization tractable in latent space.
- Visual MPC: Visual MPC samples action trajectories, predicts their outcomes, evaluates image costs against the goal, and refits the action distribution.After five iterations or convergence, it returns the best action.
- Visual MPC: The visual MPC procedure initializes a Gaussian action distribution, sorts sampled actions by cost, and refits the distribution to the best samples.The implementation uses D = 200 samples and retains D* = 40 for refitting.
- Hierarchical execution: Given generated subgoals, the agent plans sequentially toward each one until reaching a cost threshold or a maximum number of steps.Planning then proceeds from the current state to the next subgoal.
- Generative model: HVF uses either a VAE or CVAE to generate candidate images, with the CVAE conditioning generation on the initial image.
- Generative model: The VAE learns an 8-dimensional latent representation using reconstruction likelihood and a KL penalty toward a unit Gaussian.
- Generative model: The CVAE encodes paired current and initial images, combines their representations, and decodes a sampled latent representation into an image.The paired-image conditioning is used to generate states consistent with the episode context.
A.4 DYNAMICS MODEL
HVF uses stochastic variational video prediction as its action-conditioned dynamics model, with a convolutional latent encoder and recurrent convolutional frame predictor.
- Dynamics model: The dynamics model is an action-conditioned stochastic variational video prediction model trained for approximately 300K iterations.
- Architecture: SV2P contains a convolutional network that approximates latent distributions and a recurrent convolutional network that predicts the next frame.The frame predictor uses the previous frame, sampled latent, and action when available.
B EXPERIMENT DETAILS
The experiments evaluate maze, desk-manipulation, and BAIR settings using image-based costs, random interaction data, and task-specific visual MPC horizons.
- Maze navigation: Maze data comprises 10,000 randomly initialized episodes of 100 transitions, with 64x64x3 images and uniformly sampled block actions.
- Maze navigation: Maze planning uses squared ℓ2 pixel distance, action trajectories of length H = 5, and horizons T = 50 and T* = 10.
- Desk manipulation: Desk tasks include closing a sliding door, pushing blocks, and combinations requiring door manipulation and block displacement.The Door + Block task allows 100 timesteps, and Door + 2 Block combines both block pushes with door closure.
- Desk manipulation: Desk data comprises 10,000 randomly initialized episodes of 100 transitions collected through uniformly sampled end-effector actions.
- Desk manipulation: Desk planning uses squared ℓ2 pixel distance and action trajectories of length H = 15, corresponding to five actions repeated three times.The experiment uses T = 50 or 100 and T* = 20 depending on the task.
- BAIR dataset: BAIR evaluation trains the dynamics and generative models on the training split of roughly 15K trajectories and reports qualitative results on the test split.
C.1 ADDITIONAL ABLATIONS
On the hard maze task, increasing the visual MPC planning horizon does not consistently improve performance, while HVF remains better than standard visual foresight; using two subgoals can reduce performance.
- HVF outperforms standard visual foresight even with longer planning horizons on the hard maze task.
- Longer planning horizons do not necessarily improve performance because they create a harder search problem.
- Performance with 2 subgoals is worse than with 1 subgoal in this setting.
C.1.2 VAE LATENT SPACE COST
The latent-space ℓ2 cost provides a weaker planning signal than pixel ℓ2 cost, while subgoals generally improve visual MPC performance across planning horizons. However, using two subgoals can hurt performance at larger horizons.
- Planning Horizon: Subgoals achieve the best success rates across planning horizons of 5, 10, and 15.The comparison uses visual MPC with different planning horizons.
- Planning Horizon: Using two subgoals can reduce performance at larger planning horizons.This exception occurs despite subgoals generally improving performance.
- Latent vs. Pixel Cost: The VAE latent-space ℓ2 cost performs worse than pixel ℓ2 cost across all tested difficulties.It provides close to zero cost when the green blocks are reasonably close together.