PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Planning with Latent World Models

Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, Nicolas Ballas

arXiv:2604.03208v2cs.LG

TL;DR

Existing visual world-model planners struggle with long-horizon tasks because prediction errors compound and search becomes exponential, while prior hierarchical methods rely on task-specific policies or restrictive state and dynamics assumptions. HWM performs hierarchical MPC in a shared latent space, transferring long-horizon predictions as subgoals and compressing primitive chunks into macro-actions. It achieves 70% versus 0% success on real-world Franka pick-and-place and improves long-horizon performance across simulated tasks with up to 3× less planning compute.

  • Problem

    Existing world-model planners struggle on long-horizon, multi-stage tasks, while prior hierarchical approaches require task-specific policies or low-dimensional states and known dynamics.

  • Method

    HWM learns multi-timescale latent world models in a shared space, uses coarse predictions as fine-scale subgoals, and compresses primitive action chunks into latent macro-actions.

  • Results

    70% success on Franka pick-and-place versus 0% for single-level VJEPA2-AC, with gains across three backbones and up to 3× lower planning compute on long-horizon tasks.

  • Takeaways & Limitations

    Hierarchy is a practical mechanism for extending latent world-model MPC beyond short, greedy behaviors while reducing planning cost across task settings.

  • Takeaways & Limitations

    Success rates degrade with task horizon for every evaluated method, and longer or more open-ended environments may benefit from coarser, level-specific abstractions.

Abstract

from arXiv · show

World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control (MPC) directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long-horizon model serve as subgoals for the short-horizon model via latent matching, without task-specific rewards, skill learning, or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.

1 Introduction

HWM addresses the limitations of flat visual world-model planning on long-horizon, multi-stage tasks by combining temporal hierarchy with latent-space subgoal transfer. It demonstrates zero-shot planning improvements across simulated tasks and real-world manipulation.

  • Motivation: Flat world-model planners struggle because prediction errors compound and naive action search grows exponentially with planning horizon.Non-greedy tasks also produce non-monotone goal-matching costs, making direct optimization difficult.
  • Approach: HWM couples long- and short-horizon world models in a shared latent space, using high-level predictions as subgoals for low-level MPC.The high-level planner optimizes macro-actions, while the low-level planner reaches the resulting latent subgoal with primitive actions.
  • Approach: HWM avoids hierarchical policies, skill learning, and task-specific rewards while enabling direct subgoal transfer across temporal scales.Its world models are trained solely with next-latent prediction.
  • Results: 70% success on Franka pick-and-place from a single goal image versus 0% for the single-level VJEPA2-AC planner.This demonstrates zero-shot, non-greedy real-robot manipulation from visual inputs.
  • Results: HWM improves long-horizon performance across three latent world-model backbones, with gains up to +44% and +39% absolute and up to 3× lower planning compute.The evaluated backbones cover manipulation, push manipulation, and maze navigation.

2 Hierarchical Planning with Latent World Models

HWM performs hierarchical MPC in a shared latent space: a high-level planner selects macro-actions and produces latent subgoals, while a low-level planner reaches them with primitive actions. Learned macro-actions compress action chunks, reducing long-horizon rollout and search demands.

  • Hierarchical planning: HWM plans over two latent world models at different temporal resolutions, using high-level predictions as subgoals for low-level MPC.The high-level model uses latent macro-actions, while the low-level model is conditioned on primitive actions.
  • Macro-actions: A learned action encoder summarizes temporally extended primitive-action sequences into variable-duration macro-actions rather than assuming a fixed high-level horizon.The learned representation outperforms concatenated primitives and hand-crafted summaries such as net end-effector displacement.
  • High-level planning: Given an initial and goal observation, high-level MPC optimizes a macro-action sequence whose autoregressive rollout yields intermediate latent subgoals.The planner encodes both observations into latent states before optimizing the macro-action sequence.
  • Low-level planning: The low-level planner optimizes primitive actions toward the first latent subgoal and replans every k interactions in a receding-horizon loop.This connects abstract high-level plans to executable control actions.
  • Macro-actions: Compressing action chunks into latent macro-actions reduces autoregressive rollout steps and shrinks the high-level MPC search space.These design choices target compounding prediction errors and the tractability of long-horizon optimization.
  • World-model architecture: The high-level world model captures long-horizon dynamics with causal latent prediction from macro-actions encoded from low-level action chunks.Training sequences interleave waypoint latents with encoded action chunks.

3 Empirical Evaluation of Hierarchical Planning Across World Models

HWM is evaluated across real-world manipulation, push manipulation, and maze navigation, using multiple latent world-model backbones and harder long-horizon settings. It improves success and planning efficiency over single-level baselines while operating from visual inputs.

  • Evaluation setup: HWM is evaluated with VJEPA2-AC, DINO-WM, and PLDM on Franka manipulation, extended Push-T, and larger unseen mazes.The evaluations use harder settings than the original benchmarks, including non-greedy manipulation and longer horizons.
  • Franka manipulation: 70% Franka pick-&-place success is achieved from a single goal image, compared with 0% for the single-level VJEPA2-AC planner.The Franka tasks include pick-&-place and drawer manipulation requiring non-greedy intermediate motions.
  • Push-T: 61% Push-T success is achieved with DINO-WM, up from 17% for single-level planning on the extended long-horizon task.The evaluation extends start–goal separations to d = 75 timesteps, versus d = 25 in the original DINO-WM setting.
  • Zero-shot control: HWM performs end-to-end visual manipulation without manual decomposition, whereas the flat planner fails on non-greedy pick-&-place and multi-stage drawer tasks.The comparison uses the same VJEPA2-AC architecture and training data for the low-level world model, isolating hierarchical planning.
  • Diverse maze: HWM improves PLDM navigation on larger, unseen maze layouts, with the hardest-setting success rate rising from 44% to 83%.The performance gap widens as task horizon increases, and HWM reaches higher success with 4× less compute in the maze analysis.

4 Analysis of Hierarchical Planning

The analyses attribute HWM’s gains to automatically generated intermediate subgoals, temporal-scale specialization, and learned macro-actions. They also identify a trade-off between macro-action expressiveness and low-level reachability.

  • Manual-subgoal control: 80%/80% flat-planner success with manual subgoals rises from 0%/0% without them, while HWM reaches 70%/60% automatically from one goal image.The result indicates that the underlying VJEPA2-AC world model can solve the tasks when given suitable intermediate targets.
  • Prediction horizons: For horizons ≤1 s, the low-level model is more accurate; for horizons ≥1.5 s, one high-level prediction has lower error than low-level autoregressive rollouts.The result supports assigning long-term guidance to the high-level model and short-term precision to the low-level model.
  • Macro-action representation: Learned latent macro-actions outperform concatenated primitive actions and handcrafted delta-pose summaries for high-level planning.Macro-actions compress action sequences while retaining temporal structure relevant to long-horizon prediction.
  • Macro-action dimensionality: Macro-action spaces with at least 4 dimensions typically produce valid high-level plans, but excessive dimensionality can make subgoals harder for low-level execution.Restricting dimensionality biases proposed subgoals toward greedy-achievable behavior, creating a trade-off between expressiveness and reachability.
  • Representation quality: Lower-dimensional macro-actions increase prediction error and blurrier reconstructions, yet coarse semantic structure can remain sufficient for hierarchical planning.Reconstruction fidelity is not tightly correlated with planning success; contact events and motion direction may be preserved despite reduced visual precision.

5 Related Works

Prior work addresses world-model planning, long-horizon control, and hierarchical MPC, but existing approaches commonly depend on task-specific policies, low-dimensional states, or known dynamics. HWM is presented as a pixel-based next-latent-prediction alternative using latent subgoal transfer.

  • World models and planning: Learned world models enable planning from high-dimensional observations, including pixels, using latent-space dynamics and self-supervised training.Recent work also trains task-agnostic models from unlabeled offline trajectories for zero-shot downstream control.
  • Long-horizon limitations: Long-horizon learned-model planning remains limited by compounding prediction errors and exponentially growing search trees.These issues persist even with a perfect model because the search tree grows with branching factor and horizon.
  • Hierarchical reinforcement learning: Hierarchical reinforcement learning and hierarchical world models often rely on task-specific policy learning, limiting direct zero-shot applicability.Options, skills, and hierarchical policies typically require task-specific rewards or task distributions.
  • Hierarchical MPC: Classical hierarchical MPC is largely restricted to low-dimensional states, hand-engineered representations, or known dynamics rather than raw pixels.Contemporaneous visual methods use related high-level interfaces, including diffusion policies or learned action maps.
  • HWM distinction: HWM uses next-latent-prediction world models at multiple temporal resolutions, with coarse predictions serving as latent-matched subgoals for finer-scale MPC.The comparison identifies HWM as the only listed method supporting zero-shot MPC directly from pixels under this setup.

6 Conclusion

HWM enables zero-shot planning with latent world models trained at multiple temporal scales, transferring high-level predictions as subgoals for low-level planning. Across real-robot and broader task settings, it improves long-horizon success while reducing planning cost.

  • HWM is a hierarchical MPC framework for zero-shot planning with latent world models trained from offline trajectories.
  • Multiple temporal scales let high-level predictions serve as subgoals for short-horizon low-level planning without hierarchical policies, task rewards, or manual decomposition.
  • On a real Franka robot, HWM enables non-greedy pick-and-place from a single goal image, whereas the corresponding single-level planner fails.
  • Across all task settings, HWM improves long-horizon success and reduces planning cost across multiple world-model backbones.

7 Limitations

HWM remains limited by declining success as task horizons increase, and its shared latent space and waypoint-based subgoal training may constrain longer or more open-ended tasks.

  • Success rates degrade with task horizon for every evaluated method, leaving further progress in world modeling and planning necessary.
  • Using one latent space across hierarchy levels enables direct subgoal transfer but may limit abstraction quality at longer horizons and in open-ended environments.
  • Waypoint choices shape subgoal quality, motivating alternatives such as skill discovery, subgoal discovery, or dynamic chunking.
  • Lower-level feedback could help high-level subgoal optimization compensate for detail lost in high-level predictions.

A Training Details

The training pipeline learns low- and high-level latent world models from offline trajectories using next-latent prediction, rollout supervision, temporal waypoints, and macro-action encoding across several environments.

  • Low Level World Model: The low-level world model predicts next latent state representations from interleaved actions and encoded state features.
  • Low Level World Model: Training combines teacher-forcing and multi-step autoregressive rollout losses to address both next-step prediction and compounding errors.
  • Data and preprocessing: Real-robot training uses approximately 96 hours of DROID data and 30 hours of RoboSet data containing RGB observations, proprioception, and end-effector delta-pose actions.
  • Data and preprocessing: For Franka training, trajectories use 256 × 256 images, sampled frame rates, 7-dimensional end-effector states, delta-state actions, and random resize-and-crop augmentation.
  • Franka training: The Franka low-level model uses frozen ViT-g/16 features, 15-timestep trajectories, and a roughly 300M-parameter ViT to predict next-step latents.
  • High-Level World Model: High-level models train on waypoint-conditioned trajectory segments and use latent macro-actions, with architecture and training hyperparameters varying by environment.
  • Diverse Maze: Diverse Maze experiments use a Mujoco PointMaze rendered as 98 × 98 RGB images, with 2,000 episodes per map across 25 training maps.
  • High-Level World Model: Macro-actions compress primitive action chunks, while the high-level model uses waypoint states and a higher-capacity network for long-horizon prediction.

C Computational Analysis Details

Computational analyses compare flat and hierarchical planners through compute–success Pareto frontiers, hyperparameter sweeps, temporal-stride sensitivity, and capacity-controlled evaluations across simulated and visualized tasks.

  • Push-T and Franka: CEM hyperparameter sweeps construct Pareto frontiers relating mean planning time to task success for Franka and Push-T.
  • Diverse Maze: MPPI sweeps vary sample counts and planning horizons for single-level and hierarchical Diverse Maze planners across all difficulty settings.
  • Compute scaling: Figures 8 and 9 plot success rate against test-time compute for Push-T and Diverse Maze under their respective difficulty settings.
  • Temporal stride: Performance improves as the Diverse Maze high-level temporal stride increases from 6 to 12 timesteps, then slightly degrades at larger strides.
  • Capacity control: Capacity-controlled comparisons show that larger single-level models do not improve performance and can degrade, while hierarchical models consistently outperform flat baselines on Push-T and Diverse Maze.
  • Qualitative visualization: Figure 10 visualizes successful high-level subgoal trajectories against single-level trajectories that often struggle with long-horizon search complexity.

F Robot Task Definitions

The Franka evaluation defines drawer and pick-and-place cup tasks through paired initial and goal states. The cup evaluation includes ten subtasks, while the drawer evaluation includes seven.

  • Drawer Tasks: The evaluation includes 7 drawer subtasks involving opening and closing drawers at various positions.
  • Pick-&-Place Cup Tasks: The pick-and-place cup evaluation includes 10 subtasks requiring the robot to move a cup to different target locations.
  • Each task is specified by an initial state and a goal state.

G Per-Configuration Trial Breakdown for Franka

The Franka evaluation reports per-configuration trial outcomes, execution visualizations, offline planning measures, and baseline setup details for manipulation tasks. It also records limitations in comparing policy baselines on drawer tasks and subgoal-based control.

  • Trial Protocol: N = 5 independent trials were run per start–goal configuration, producing 50 pick-and-place trials per object and 35 drawer trials.
  • Per-Configuration Outcomes: 21/27 HWM configurations were fully deterministic, while the remaining 6 matched their modal outcome in 4/5 trials.Overall, 129/135 HWM trials, or 95.6%, matched the modal outcome.
  • Execution Visualization: Execution figures visualize high-level subgoals decoded by the planner and label failed episodes.
  • Offline Planning Evaluation: The offline evaluation compares inferred high-level actions with expert actions using cosine similarity and ℓ1 distance.The expert actions move from the initial observation to a manually defined grasp-point subgoal and then from that subgoal to the goal.
  • Baselines: The baseline setup uses a Franka Panda arm with a bi-finger Robotiq gripper, wrist and side cameras, and chunked action execution for π0.5-DROID and π0-FAST-DROID.The π0.5 and π0-FAST controllers execute 15 and 8 actions per output chunk, respectively.
  • Language Prompts: The π-model prompts were tuned toward short, simple descriptions because the models did not understand colors and complex shape nouns reliably.Tested prompts included references to cups or boxes and surfaces such as planes, platforms, tables, and shelves.
  • Policy Evaluation: The evaluation used the same ten task configurations for π0.5-DROID, with 100 cup trials and 50 box trials.
  • Evaluation Limitations: π-models could not be evaluated on drawer tasks after an Octo evaluation physically damaged the drawer, and their language goals cannot specify subgoals.Octo also performed at 0 without subgoals in the reported setup.

J.2 Goal-Conditioned & Zero-Shot RL Methods

This section identifies goal-conditioned and zero-shot reinforcement-learning baselines and describes their implementations and tuning procedure on Push-T and Diverse Maze.

  • Methods: GCIQL is a goal-conditioned version of Implicit Q-Learning used as a strong offline-RL baseline.
  • Methods: HIQL is a hierarchical goal-conditioned RL method with separate subgoal-generation and subgoal-reaching policies sharing one value function.
  • Methods: HILP learns representations in which distance is proportional to the number of steps between states and directly learns a policy from those representations.
  • Implementation: For Push-T and Diverse Maze, hyperparameters were tuned independently for each difficulty setting using optimal reported configurations.Unlisted hyperparameters used the original repositories' defaults.
Loading 2604.03208v2…