PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Language Planning

Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan Tompson

arXiv:2310.10625v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Long-horizon visual planning must account for task semantics and world dynamics, motivating the question of how to plan with information-rich video and Internet-scale data. VLP combines tree search with vision-language and text-to-video models to generate video plans, substantially improving long-horizon task success over prior methods on simulated and real robots.

  • Problem

    Long-horizon physical planning must account for both task semantics and world dynamics, raising the question of how to use information-rich video and Internet-scale data for planning.

    Source: §1 · pp. 1–2

  • Method

    VLP uses tree search with vision-language and video-model building blocks to turn an image observation and language goal into a video plan of image subgoals.

    Source: §2 · p. 3

  • Results

    VLP substantially improves long-horizon task success over prior methods on simulated and real robots, while increased planning horizon and branching factor improve execution success.

    Source: §3.1 · p. 6, §3.2 · pp. 7–8

  • Takeaways & Limitations

    Generated video plans can be translated into robot actions using a goal-conditioned policy conditioned on each intermediate frame.

    Source: §2 · p. 3, §2.3 · p. 4

    The image-based world-state representation can omit full 3D state and latent factors such as physics or mass.

    Source: §5 · p. 10

Abstract

from arXiv · show

We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planning (VLP), an algorithm that consists of a tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able to synthesize long-horizon video plans across different robotics domains: from multi-object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be translated into real robot actions via goal-conditioned policies, conditioned on each intermediate frame of the generated video. Experiments show that VLP substantially improves long-horizon task success rates compared to prior methods on both simulated and real robots (across 3 hardware platforms).

1 INTRODUCTION

Video language planning (VLP) combines vision-language and text-to-video models in tree search to produce long-horizon multimodal plans. The paper reports improved planning and task completion in simulated and real robotics settings.

1 INTRODUCTION

  • VLP addresses complementary limits: text-based plans struggle to ground physical constraints, image-focused VLMs struggle with dynamics, and text-to-video models typically produce only short clips.The approach combines high-level planning with detailed visual dynamics learned from video.Source: §1 · pp. 1–2
  • VLP uses a VLM to propose text actions, a text-to-video model to simulate rollouts, and a VLM heuristic to score task progress before recursively expanding the search.The search constructs long-horizon plans from possible video sequences, and the plans can be converted into robot actions using goal-conditioned policies.Source: §1 · p. 2, Figure 1 · p. 2
  • VLP systems achieve long-horizon task completion far more often than alternatives including PaLM-E (Driess et al., 2023) and RT-2 (Brohan et al., 2023).Experiments span simulated and real settings across three robot hardware platforms; VLP also produces more complete, coherent plans with more consistent scene dynamics than baselines, including end-to-end models generating long videos from long-horizon instructions.Source: §1 · pp. 2–3
  • Expanding the search branching factor lets VLP improve plan quality with more inference compute, while incomplete language-labeled video data can still support training and long-horizon recomposition.Short video snippets can be sequenced into longer plans, and video segments without language labels can still teach dynamics.Source: §1 · pp. 2–3
  • Co-training on Internet-scale data enables VLP to generalize to new objects and configurations across tasks such as stowing objects and rearranging groups of blocks.The paper describes deployment through inverse dynamics models or goal-conditioned policies for multi-step robot tasks.Source: §1 · pp. 2–3

2 VIDEO LANGUAGE PLANNING

VLP takes an initial image and language goal and searches for a long video plan of image sub-goals, which a goal-conditioned policy can translate into robot actions.

2.1 USING VISION-LANGUAGE AND VIDEO MODELS AS PLANNING SUBMODULES

  • VLP uses VLMs to propose goal- and image-conditioned text actions, text-to-video models to synthesize resulting states, and a VLM heuristic to estimate steps remaining to the goal.The policy and heuristic are finetuned on trajectory snippets; the dynamics model is trained on short image trajectories paired with language labels.Source: §2.1 · p. 3

2.2 PLANNING WITH VISION-LANGUAGE MODELS AND VIDEO MODELS

  • Parallel hill-climbing search expands plan beams with VLM-proposed actions and candidate videos, selects by heuristic value, and periodically replaces the lowest-valued beam with the highest-valued one.To limit exploitation of model irregularities, VLP discards generated videos whose heuristic increase exceeds a fixed threshold.Source: §2.2 · p. 4

2.3 ACTION REGRESSION FROM VIDEO THROUGH GOAL-CONDITIONED POLICIES

  • A short-horizon goal-conditioned policy predicts controls toward each planned frame, while fixed-horizon replanning addresses accumulated execution error and plans that do not finish the task.The policy is trained on paired image and low-level control snippets, using current and future states to predict an action.Source: §2.3 · pp. 4–5
  • VLP substantially outperforms UniPi and a directly combined VLM policy in synthesizing complete video plans across simulation and real environments.The supplied table caption reports this qualitative comparison but provides no numerical values.Source: Table 1 · p. 5

3 EXPERIMENTAL RESULTS

VLP synthesizes long-horizon video plans, executes them in simulated and real environments, and generalizes across objects, lighting conditions, and tasks. The experiments also examine how planning budget and action extraction affect performance.

3.1 LONG-HORIZON VIDEO SYNTHESIS

  • Increasing search branching improves video-plan success, and VLP generates plans for 7DoF mobile manipulation and multiview 14DoF bimanual tasks.The bimanual plans use four camera views and can stack bowls, cups, and utensils.Source: §3.1 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6
  • For Language Table synthesis, VLP outperforms direct long-horizon video generation and a policy-plus-video ablation, supporting hierarchical planning and the value function.Success was visually assessed across 50 generated videos per method; increasing video branching, language branching, and beam count also substantially improves synthesis success.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 LONG-HORIZON EXECUTION

  • VLP substantially outperforms baseline methods on long-horizon execution tasks involving block rearrangement, color grouping, and making a line.The tasks are around 1,500 steps; many baselines become stuck and stop acting effectively.Source: Table 2 · p. 7, §3.2 · p. 7
  • Increasing planning horizon and branching improves execution success, while conditioning a goal policy on every intermediate video frame performs best among tested action-extraction methods.The planning improvements come at the cost of inference time.Source: §3.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

3.3 GENERALIZATION

  • VLP executes tasks with new objects and lighting conditions, and generalizes to new tasks on unseen objects using pretrained Internet knowledge.Execution uses video-model visual dynamics to generate goals and a goal-conditioned controller to reach nearby goals.Source: §3.3 · pp. 8–9, Figure 9 · p. 9, Figure 10 · p. 9

4 RELATED WORK

Prior language- and vision-language planning methods rely on text plans and may struggle with world dynamics, while existing video-planning work largely focuses on short horizons. VLP combines VLMs and text-to-video models through forward search to plan across long horizons, unlike HiP’s one-step-at-a-time planning.

4 RELATED WORK

  • LLMs can generate step-by-step language plans, and VLMs can condition plans on visual inputs, but static-image pretraining can limit plans to text and impede reasoning about world dynamics.Examples include LLM-based planning (Huang et al., 2022a; Ahn et al., 2022; Wang et al., 2023a) and visual conditioning through language descriptions or token embeddings (Zeng et al., 2022; Huang et al., 2022b; Driess et al., 2023).Source: §4 · p. 9
  • Video dynamics models capture object motion, and newer text-to-video models generate detailed frames and can drive decision policies, but prior work predominantly targets short-horizon videos.VLP combines text-to-video models with VLMs and tree search to produce long-horizon video plans, potentially predicting hundreds of frames ahead.Source: §4 · p. 9
  • HiP combines language, video, and action models for hierarchical planning, whereas VLP uses forward search to reason about long horizons rather than generating and acting on plans one step at a time (Ajay et al., 2023).The forward-search composition combines a VLM with a video model.Source: §4 · p. 10

5 LIMITATIONS AND CONCLUSION

VLP combines vision-language and text-to-video models for long-horizon decision making, but image-based state representations and video dynamics can be inaccurate. The authors present test-time planning as a way to compose foundation models into more complex behaviors and suggest broader applications and future model compositions.

Limitations

  • Image-based world states omit full 3D state and latent factors such as physics and mass; multi-view videos or heuristics using the full video plan may partly address this limitation.The paper notes that its approach uses images as the world-state representation.Source: §5 · p. 10
  • Synthesized videos sometimes show objects spontaneously appearing or teleporting, and larger video models, more training data, or physics-focused reinforcement-learning feedback could help improve dynamics accuracy (Black et al., 2023).The authors report that the video dynamics model does not always simulate dynamics accurately.Source: §5 · p. 10

Conclusions

  • VLP combines vision-language models with text-to-video models for long-horizon decision making, and the authors argue that planning can generate behaviors more complex than its component models.The experiments largely explore VLP in robotics.Source: §5 · p. 10
  • The authors identify steerable media generation as a possible application beyond robotics and propose further exploration of test-time compositions of different foundation models.They describe such compositions as a potential source of gains for building more complex intelligent systems.Source: §5 · p. 10

A APPENDIX

The appendix collects additional results and details on evaluation, datasets, training, and planning.

A.1

A.2

A.3

A.4

A.5

A.1 ADDITIONAL RESULTS

Additional results show that VLP can produce robust language and video plans, while also revealing occasional video-interpretation errors and inconsistent physics. Its goal-conditioned policy can still execute tasks when generated visual goals are misaligned or contain boundary artifacts.

Failure Cases

  • VLP sometimes misinterprets tasks, such as depicting a manipulator as an octopus, and sometimes generates videos with objects inconsistently appearing or disappearing.These failures are illustrated in Fig. XI and Fig. XII, respectively.Source: §A.1 · p. 14

Robustness of Goal-Conditioned Policy

  • The goal-conditioned policy can execute visual tasks despite generated goals misaligned with the observed board and artifacts in their boundaries.The paper attributes this robustness to the policy discarding most image-goal information and focusing on what is needed for immediate execution; see Fig. XIII.Source: §A.1 · p. 14

Additional Video Plans

  • VLP robustly synthesizes language and video plans for additional block-arrangement tasks.Additional plans are shown in Fig. XIV.Source: §A.1 · p. 14

A.2 EVALUATION DETAILS

The evaluation assessed video-level goal satisfaction and robot execution using task-specific completion measures. Video generation and execution were evaluated across 50 samples or environments per task, with fixed planning and execution budgets.

A.2 EVALUATION DETAILS

  • VLP execution used a horizon-2 plan with beam width 2, branching factor 16, and 16 frames per video; policies were called on 16 simulated or 10 real-world frames.In simulation, the goal-conditioned policy was called four times for each of the first 16 frames; real-world evaluation used the first 10 frames to account for physics differences.Source: §A.2 · p. 14
  • Execution rewards used ground-truth block states, task-specific distance criteria, and a 1,500-timestep limit, with early termination when a task was completed.Tasks measured same-color block proximity within 0.1 units, blocks within 0.2 x and 0.27 y units of a corner, or blocks within 0.05 units of the board center, respectively.Source: §A.2 · p. 14
  • Video evaluation manually checked whether any generated frame satisfied the long-horizon goal, using 50 videos per goal, task, and method.Generating each long-horizon video took approximately 30 minutes.Source: §A.2 · p. 14
  • Execution evaluation used 50 environments per task; VLP took approximately 1 hour per environment, compared with approximately 0.5 hours for RT2 baselines.All methods received the same 1,500-timestep execution budget, with performance measured at the end unless completion occurred earlier.Source: §A.2 · p. 14

A.3 DATASET DETAILS

VLP training data spans simulated and real Language Table trajectories, a broad video-data mix for 7DoF mobile manipulation, and teleoperated kitchen-stacking demonstrations for 14DoF bimanual manipulation.

Language Table

  • Language Table training used approximately 10,000 trajectories across several hundred goals; three goals were selected for experiments, with roughly 20,000 trajectories and 400,000 short-horizon text labels combined.The selected goals enabled easy automated evaluation.Source: §A.3 · p. 14

7DoF Mobile Manipulator

14DoF Bi-Manual Manipulation

  • For 14DoF bimanual manipulation, VLP used approximately 1,200 teleoperated kitchen-stacking demonstrations, each annotated with roughly 20 instructions, totaling roughly 25,000 short-horizon text labels.The demonstrations progressed from stacking bowls, to cups on bowls, to utensils on cups.Source: §A.3 · p. 15

A.4 TRAINING DETAILS

A.4 specifies domain-specific training setups for VLP’s video models, VLMs, and goal-conditioned policies, as well as the settings used for its baselines.

Video Models

  • VLP trains separate text-to-video models per domain, generating 16-frame videos from 24 × 40 resolution and upsampling them through 48 × 80 to 192 × 320.The base model uses 64 TPUv3 pods for 3 days; the higher-resolution superresolution models are trained for 1 day, following Du et al. (2023b).Source: §A.4 · p. 15

VLM Models

  • A single 12B PaLM-E VLM is fine-tuned jointly to predict heuristics and policies, using separate domain models trained for 1 day on 64 TPUv3 pods.The VLM architecture and codebase follow PaLM-E (Driess et al., 2023).Source: §A.4 · p. 15

Goal-Conditioned Policy

  • The goal-conditioned policy uses LAVA architecture with a ResNet goal-image encoder replacing CLIP’s text encoder, and is trained per domain for 1 day on 16 TPUv3 pods.The policy is robust to noise in synthesized goals and can execute control actions in the real environment from those goals (Figure XIII).Source: §A.4 · p. 15, Figure XIII · p. 16

Baselines

  • The LAVA baseline follows its paper’s settings, while UniPi follows VLP’s text-conditioned video-model training approach on long-horizon video elements.LAVA is conditioned on long-horizon goal text and trained for 1 day on 16 TPUv3 pods; RT-2 uses the original architecture and codebase with a 12B RT2-PaLM-E model, trained for 3 days on 64 TPUv3 pods (Lynch et al., 2023; Du et al., 2023b; Brohan et al., 2023).Source: §A.4 · p. 16

A.5 PLANNING DETAILS

VLP planning uses a horizon-16 search with language and video branching, a clipped heuristic, and DDIM video sampling. The implementation varies across robot domains, including model, beam-width, resolution, and heuristic-threshold changes.

Language Table

  • Language Table planning used horizon 16, beam width 2, and branching factors of 4 for both language and video.DDIM sampling used 64 base-resolution and 4 higher-resolution timesteps with classifier-free guidance 5; 16 videos were generated in parallel on a 4-TPU pod, and planning took about 30 minutes. The heuristic clipping threshold was 50, with videos removed when one rollout improved by more than 50.Source: §A.5 · p. 16

7DoF Mobile Manipulator

  • For 7DoF mobile manipulation, VLM planning used PaLM-E scene captions and few-shot PaLM plans, with beam width 3.The video diffusion model was finetuned only on RT-1 data, used base and super-resolution sizes of 64 × 80 and 256 × 320, and the goal-conditioned policy used only each segment’s last frame. PaLM-E (Driess et al. (2023)), PaLM (Chowdhery et al. (2022)), and SayCan (Ahn et al. (2022)).Source: §A.5 · p. 16

14DoF Bi-Manual manipulation

  • 14DoF bi-manual manipulation followed the Language Table planning setup but used a heuristic clipping threshold of 15.The planning setup was otherwise the same as Language Table.Source: §A.5 · p. 16

Related papers

Loading 2310.10625v1…