Source-linked AI summary
Compositional Foundation Models for Hierarchical Planning
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, Pulkit Agrawal
TL;DR
Long-horizon planning with separate language, vision, and action experts could avoid the costly collection of paired multimodal robotics data. HiP composes those experts with iterative feedback, and significantly outperforms the baselines on the evaluated long-horizon tasks. Its demonstrations use smaller-scale video and ego-centric action models, and its joint-distribution sampling is approximate.
Problem
Building long-horizon decision systems with paired visual-language-action data is difficult because such data are expensive to collect and hard to scale.
Source: §1 · pp. 1–2Method
HiP composes separately trained language, video, and inverse-dynamics models, using downstream feedback to refine subgoals and generated observation trajectories.
Source: §1 · p. 2Results
HiP significantly outperforms the baselines on the evaluated long-horizon tasks, even though the baselines have access to oracle subgoals.
Source: §3.3 · p. 7Takeaways & Limitations
The compositional approach constructs long-horizon plans without collecting expensive paired decision-making data across modalities.
Source: §1 · p. 2The demonstrations use smaller-scale video and ego-centric action models, and the method approximates sampling from the joint distribution across models.
Source: §5 · p. 11
Abstract
from arXiv · showhide
To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.
1 Introduction
HiP addresses the difficulty of building long-horizon decision systems from costly, hard-to-scale paired language, vision, and action data. It composes separately trained experts for hierarchical planning and uses feedback between them to promote consistent plans.
1 Introduction
- Paired visual-language-action data are expensive and hard to scale, while robotics demonstrations are scarce and costly to collect.The paper also notes that high-performing language models with closed weights cannot currently be fine-tuned.Source: §1 · pp. 1–2
- HiP composes separately trained language, vision, and action experts to plan long-horizon tasks without collecting paired decision-making data across modalities.An LLM proposes sub-tasks, video diffusion generates an observation trajectory, and an inverse model infers actions from images.Source: §1 · p. 2
- Iterative refinement uses feedback from downstream models to promote plans that are jointly consistent, responsive to the goal, and executable.The approach does not require large-model fine-tuning or access to model weights, and can work with input-output API access.Source: §1 · p. 2
- HiP demonstrates promising results on three long-horizon tabletop manipulation environments.Source: §1 · p. 2
2 Compostional Foundation Models for Hierarchical Planning
HiP factorizes long-horizon planning into task, visual, and action components, then composes their outputs into an executable plan. The planners use feedback to account for downstream feasibility and consistency.
2 Compostional Foundation Models for Hierarchical Planning
- HiP factorizes planning into task, visual, and action distributions, composing language subgoals, image trajectories, and action trajectories into a long-horizon plan.The plan is conditioned on the goal and initial image under a Markov assumption.Source: §2 · p. 3
2.1 Task Planning via Large Language Models
- The task planner prompts a pretrained LLM with domain examples, then ranks candidate subgoals using visual feedback to account for downstream plausibility.Direct LLM sampling may select a subgoal that is already completed and has low likelihood under the visual planner.Source: §2.1 · pp. 3–4
2.2 Visual Planning with Video Generation
- The visual planner generates observation trajectories conditioned on the current image and subgoal, and action-feasibility feedback guides diffusion toward trajectories likely to yield feasible actions.The model is pretrained on Ego4D and fine-tuned on task-specific videos; full action-trajectory sampling during denoising is slow, motivating an approximation.Source: §2.2 · pp. 4–5
2.3 Action Planning with Inverse Dynamics
- The action planner uses an inverse dynamics model to infer actions from pairs of observations along the generated trajectory.It is initialized with VC-1 weights pretrained on egocentric images and ImageNet, then fine-tuned on paired observation-action trajectories.Source: §2.3 · p. 5
3 Experimental Evaluations
HiP is evaluated on three tabletop environments with randomized tasks and unseen combinations. It outperforms the baselines on long-horizon tasks, with additional evaluations of generalization, pretraining, refinement, and visual grounding.
3.1 Evaluation Environments
- The three environments vary objects, layouts, distractors, and task completion states; test tasks use novel combinations, with task-completion or subtask-completion rates as metrics.Paint-block and object-arrange tasks have randomized object arrangements; kitchen tasks can include a subtask completed before the start.Source: §3.1 · pp. 6–7
3.2 Baselines
- Baselines include goal-conditioned policies, video and action planners, and an LLM skill manager; SayCan was not run on kitchen-tasks.SayCan requires skills paired with affordance functions, which the authors note can be difficult to define for kitchen tasks.Source: §3.2 · p. 7, §3.3 · p. 8
3.3 Results
- HiP significantly outperforms baselines on seen and unseen long-horizon tasks across paint-block, object-arrange, and kitchen-tasks.The baselines have access to oracle subgoals.Source: §3.3 · p. 7, Table 1 · p. 8
- HiP’s performance remains intact on unseen tasks formed from novel combinations of colors, object categories, and kitchen subtasks.Figure 4 visualizes an execution on an unseen paint-block task.Source: §3.3 · p. 8, Figure 4 · p. 8
- Ego4D-pretrained video diffusion yields higher success rates and lower FVD scores, with its advantage widening as domain-specific training data shrinks.The caption reports the same pattern at reduced dataset sizes.Source: §3.3 · p. 9, Figure 5 · p. 9
- VC-1 initialization matches a randomly initialized inverse dynamics model trained on 10K trajectories with 1K trajectories in paint-block and object-arrange, and 3.5k in kitchen-tasks.ResNet-18 requires 2.5K trajectories in paint-block and object-arrange, and 6k in kitchen-tasks, to approach the VC-1 model’s performance.Source: Figure 6 · p. 9
- Task-plan refinement is critical to HiP’s performance, while visual-plan refinement improves performance by a smaller margin in paint-block.Without task-plan refinement, subgoals are agnostic to the observation and can lead to erroneous visual and action planning.Source: §3.3 · p. 10, Figure 7 · p. 10
- The frozen MiniGPT-4 VLM matches the learned classifier in paint-block, but performs worse in the more visually complex object-arrange environment.The authors note that prompt engineering using task domain knowledge was essential for the VLM classifier.Source: §3.3 · p. 10, Figure 7 · p. 10
4 Related Work
Prior work grounds language models through paired-data fine-tuning or fixed skill functions, while HiP combines language, video, and egocentric action models through joint-distribution sampling and iterative consensus.
4 Related Work
- SayCan [2] sequences language-model instructions through fixed skill-based value functions, whereas HiP represents skills as video-based planners that can generalize to new skills.Other approaches fine-tune language models for embodied tasks, requiring paired vision-language data that is difficult to acquire [11] [44].Source: §4 · p. 10
- Continuous-space diffusion planners typically rely on task-specific datasets [3] [57]; HiP builds on UniPi and combines its video planning with LLMs for longer-horizon plans.UniPi uses internet videos to train image-space planners.Source: §4 · p. 10
- HiP combines language, video, and egocentric action models by sampling from their product distributions with iterative consensus, following a zero-shot integration approach similar to Li et al..Unlike Flamingo [5], this avoids joint finetuning with paired datasets, which are difficult to collect.Source: §4 · pp. 10–11
5 Limitations and Conclusion
HiP combines separately trained modality-specific models and reports strong results across three long-horizon manipulation domains, while acknowledging limits from model scale and approximate joint sampling.
5 Limitations and Conclusion
- The approach relies on smaller-scale, directly trained video and robot-action models because high-quality foundation models for these modalities are not yet available.Sampling from the joint model distribution also uses approximations; the authors identify more efficient and accurate consistency sampling as future work.Source: §5 · p. 11
- HiP significantly outperforms state-of-the-art approaches across novel state-goal compositions in three long-horizon robot manipulation domains.The reported video and egocentric action models are smaller simulation-trained proxies for larger pretrained models.Source: §5 · p. 11
- The conclusion proposes combining additional pretrained modalities, including touch and sound, and describes decision-making from powerful pretrained models with little training data as a future direction.The authors expect such systems could be cheaper to train and more capable, but present this as a belief about future systems.Source: §5 · p. 11
Appendix
- The appendix outlines factorization, component architectures and training, implementation details, ablations, and runtime analysis.Its listed ablations include consistency methods, subgoal granularity, video-planning model choice, and iterative-refinement hyperparameters.Source: Appendix · p. 16
A Factorizing Hierarchical Decision-Making Process
- The appendix factorizes hierarchical decisions into task, visual, and action distributions, using stated conditional-independence assumptions.The task distribution is modeled independently of the observation; the visual trajectory is conditioned on subgoal and observation, and actions are recovered from adjacent observations.Source: A Factorizing Hierarchical… · p. 16
B.1 Task Planning
The appendix describes task planning through visual grounding, video-based visual planning, and inverse-dynamics action planning, with consistency mechanisms across the hierarchy.
B.1 Task Planning
- A multi-class classifier estimates whether a candidate subgoal matches the current observation and task, assigning higher scores to corresponding subgoals.The appendix frames visual grounding as estimating a conditional-to-marginal density ratio and describes classifier-based density-ratio estimation [48, 15, 18, 47].Source: §B.1 · p. 16
B.2 Visual Planning
- Video diffusion generates a visual trajectory conditioned on the language subgoal and current observation, with the observation tiled across time for video encoding.The model diffuses in an autoencoder-learned latent space; its latent video representation captures variation across time, height, and width.Source: §B.2 · p. 17
- A binary classifier scores whether latent observation trajectories support high-likelihood actions, and its gradients bias video-diffusion denoising toward consistency with action planning.It is trained to distinguish dataset trajectories from trajectories with randomly shuffled frames.Source: §B.2 · p. 18
B.3 Action Planning
- The inverse-dynamics model infers robot states from observations and derives actions from consecutive state differences, with the final action dimension encoding gripper state.The state contains six joint-angle dimensions and one gripper dimension.Source: §B.3 · p. 18
C.1 Task Planning
The appendix details task-planning classifiers and a VLM alternative, visual-planning training, inverse-dynamics data and evaluation, and training hyperparameters.
C.1 Task Planning
- The task-planning classifier is trained on observation, goal, candidate-subgoal, and correct-label data, with 58k paint-block, 82k object-arrange, and 50k kitchen examples.The VLM alternative uses LLM-generated candidate subgoals and domain knowledge to remove completed goals and sequence remaining ones without training data.Source: §C.1 · pp. 18–19
- The VLM often misses uncommon objects such as hard drives and hallucinates object presence, leading to incorrect reasoning; direct prompting to generate the next subgoal also failed.The authors hypothesize that this direct-generation failure reflects inability to perform in-context learning.Source: §C.1 · p. 19
C.2 Visual Planning
- Visual models are pretrained on approximately 344k Ego4D clips, then video diffusion is finetuned on approximately 100k subgoal-labeled trajectories; a classifier predicts action-likelihood from trajectories.The video autoencoder uses pixel-level L1 reconstruction and negative perceptual similarity, while the noise model uses latent-space denoising loss.Source: §C.2 · p. 19
C.3 Action Planning
- Inverse dynamics is trained on 1k paired trajectories in paint-block and object-arrange and 3.5k in kitchen, then evaluated by MSE on 100 unseen trajectories.Evaluation uses four seeds, with standard error shown as a shaded area in Figure 6.Source: §C.3 · p. 20
C.4 Hyperparameters
- Task-planning training uses 50 epochs, batch size 256, and learning rate 1e −3; visual and action models use their separately specified training schedules and learning rates.The video model is pretrained and finetuned in separate stages, and the inverse-dynamics models train for 20 epochs.Source: §C.4 · p. 20
D Implementation Details for Gato and SayCan
Gato and SayCan were adapted as baselines using 110k training datapoints, with Gato given oracle subgoals and SayCan grounded through CLIPort primitives. HiP outperforms SayCan on seen and unseen paint-block and object-arrange tasks, while SayCan could not be run on kitchen tasks.
D Implementation Details for Gato and SayCan
- Gato was trained with 110k language, observation-trajectory, and action-trajectory datapoints per domain and given oracle subgoals.HiP’s visual planner and inverse dynamics used 100k robot videos and 10k state-action pairs, respectively; Gato and SayCan received 110k datapoints for fair comparison.Source: D Implementation Details for Gato and… · p. 20
- SayCan uses CLIPort pick-and-place policies and pixel-space logits as affordances to ground the LLM; its hardcoded scoring function does not apply to object-arrange.SayCan was not run on kitchen tasks because that domain has no clear-cut primitive.Source: D Implementation Details for Gato and… · pp. 20–21
- HiP outperforms SayCan on seen and unseen paint-block and object-arrange tasks, but SayCan could not be run on kitchen tasks.SayCan requires tasks to be expressed as primitives paired with affordance functions.Source: D Implementation Details for Gato and… · p. 21
E.1 Consistency between task planning and visual planning
These subsections examine consistency refinement across task, visual, and action planning, then test subgoal granularity. Learned visual consistency classification substantially improves subgoal selection over approximate likelihood scoring, while action-plan refinement gives results comparable to the classifier and overly coarse subgoals reduce success.
E.1 Consistency between task planning and visual planning
- 98.2 ± 1.5% subgoal prediction accuracy for the learned classifier exceeds 54.3 ± 7.2% for variational-lower-bound scoring and 16.67% for random selection with M = 6.The approximation’s weak performance may reflect a coarse diffusion distribution that yields a loose bound and uncalibrated likelihoods; a larger diffusion model could tighten the bound.Source: §E.1 · p. 21
E.2 Consistency between visual planning and action planning
- 72.5±1.9 success rate with inverse-model refinement compares with 72.8 ± 1.7 using the learned classifier and 71.1 ± 1.3 without visual refinement on unseen paint-block tasks.The authors conclude that the classifier approximates whether an observation trajectory leads to a high-likelihood action trajectory while remaining computationally efficient.Source: §E.2 · p. 21
E.3 How granularity of subgoals affects performance of HiP ?
- HiP (standard) and the more granular variant have similar success rates, whereas the less granular variant declines and UniPi has the lowest success rate.The study varies subgoals from one pick-and-place operation to two, with UniPi imagining the entire trajectory from the goal description.Source: §E.3 · p. 22, Table 2 · p. 22
E.4 Ablation on Visual Planning Model
The visual-planner ablation finds video diffusion outperforms RSSM across all three domains, with a wider gap in visually complex domains. The sensitivity study reports best performance at guidance scales 1 or 1.25, with degradation at higher values.
E.4 Ablation on Visual Planning Model
- Video diffusion outperforms RSSM in all three domains, and the gap widens in the more visually complex object-arrange and kitchen-tasks domains.The gap is small in paint-block.Source: Table 3 · p. 22, §E.4 · pp. 22–23
E.5 Analyzing sensitivity of iterative refinement to hyperparameters
- HiP performs best when visual-refinement guidance scale ω′ is 1 or 1.25, and performance degrades at higher values.The study varies ω′ from 0.5 to 2 in increments of 0.25.Source: Table 4 · p. 23, §E.5 · p. 23
F Analyzing Runtime of HiP
Runtime analysis breaks down episode time into planning and execution components. Visual planning accounts for most planning time; planning times are similar across domains, while execution and total episode times vary with simulation properties and average subgoal count.
F Analyzing Runtime of HiP
- Visual planning takes the majority of HiP’s planning time; planning time is similar across domains, but action execution and total runtime vary with simulation properties and average subgoal count.The runtime breakdown covers task planning, visual planning, action planning, and action execution, averaged across 1,000 seen tasks per domain.Source: F Analyzing Runtime of HiP · p. 23, Table 5 · p. 23