Source-linked AI summary
Video Language Planning
Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Kaelbling, Andy Zeng, Jonathan Tompson
TL;DR
Long-horizon robotic tasks require both abstract action planning and detailed visual dynamics. VLP combines vision-language policies and heuristics with text-to-video dynamics in tree search, then executes intermediate video goals through goal-conditioned control. Across simulated and real robotic settings, the paper reports improved long-horizon task success compared with prior methods, while noting limitations in visual state representation and video dynamics accuracy.
Problem
Planning physical tasks requires both high-level semantic abstractions and low-level world dynamics, which text-only language planning does not fully represent.
Method
VLP uses vision-language models as action policies and heuristics, text-to-video models as dynamics models, and tree search to synthesize long-horizon video plans.
Results
VLP improves long-horizon manipulation performance across simulated and real robots, with execution success increasing as planning horizon and branching factor increase.
Takeaways & Limitations
Intermediate-frame goal-conditioned policies translate synthesized video plans into low-level robot actions for complex manipulation tasks.
Takeaways & Limitations
Image-based world states omit full 3D information and latent factors such as physics or mass, while video dynamics can produce inaccurate object motion.
Abstract
from arXiv · showhide
We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planning (VLP), an algorithm that consists of a tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able to synthesize long-horizon video plans across different robotics domains: from multi-object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be translated into real robot actions via goal-conditioned policies, conditioned on each intermediate frame of the generated video. Experiments show that VLP substantially improves long-horizon task success rates compared to prior methods on both simulated and real robots (across 3 hardware platforms).
1 INTRODUCTION
Video Language Planning addresses the limits of text-only planning by combining high-level language actions with information-rich visual dynamics. It uses vision-language and text-to-video models in a scalable tree-search algorithm to synthesize long-horizon video plans for complex robotic tasks.
- Physical-world interaction requires planning both semantic task abstractions and low-level state-transition dynamics.
- Text-only language models can generate step-by-step plans but are limited in representing information-rich world dynamics.
- The algorithm iteratively proposes actions, simulates video rollouts, evaluates future states with a VLM heuristic, and expands the best sequences through tree search.
- VLP integrates vision-language models for high-level text actions with text-to-video models for predicting low-level outcomes.
- Experiments cover simulated and real robots performing complex long-horizon manipulation tasks, with task completion rates often exceeding the next best approach.
2 VIDEO LANGUAGE PLANNING
VLP searches over long-horizon video plans by combining VLM policies and heuristics with video-model dynamics, then translates intermediate visual goals into robot actions. Its planning and execution procedure uses parallel search, goal-conditioned control, and replanning.
- VLP maps an initial visual observation and language goal to a video plan whose frames serve as intermediate sub-goals.
- VLM policies propose abstract text actions conditioned on the current image and goal, while video models synthesize possible future world states.
- VLP evaluates and expands long-horizon plans with parallel video-plan beams, branching actions and rollouts, and a VLM heuristic.
- ACTION REGRESSION FROM VIDEO THROUGH GOAL-CONDITIONED POLICIES: Goal-conditioned policies convert each synthesized video frame into low-level control by treating the next frame as a nearby visual goal.
- ACTION REGRESSION FROM VIDEO THROUGH GOAL-CONDITIONED POLICIES: Receding-horizon control addresses accumulated execution error and plans that cannot fully complete very long tasks within a fixed horizon.
3 EXPERIMENTAL RESULTS
Experiments evaluate VLP for synthesizing and executing long-horizon plans, showing benefits from search, intermediate-frame control, and generalization across simulated and real settings.
- Long-Horizon Video Synthesis: VLP substantially outperforms direct long-horizon video synthesis and VLM–video chaining on object-rearrangement video plans.The comparison attributes gains to hierarchical structure, the planning procedure, and the value function.
- Long-Horizon Video Synthesis: Increasing video, language, or beam branching substantially increases long-horizon video-synthesis success.The analysis varies search branching and beams, with success percentages reported for the make line task.
- Long-Horizon Execution: VLP executes long-horizon goals in simulated environments and on real Language Table and 7DoF mobile-manipulator robots.The system also synthesizes consistent four-camera plans for dexterous stacking tasks.
- Long-Horizon Execution: VLP substantially outperforms PaLM-E, UniPi, LAVA, and RT2 on long-horizon simulated execution tasks.Evaluated tasks include moving blocks, grouping blocks by color, and making blocks in a line, with horizons around 1500 steps.
- Long-Horizon Execution: Larger planning horizons and branching factors improve execution success, while increasing inference time.Using a goal-conditioned policy on every intermediate synthesized video frame gives the best overall action-extraction performance.
- Generalization: VLP generalizes execution to three new objects, substantially different lighting, a new building location, and new tasks on unseen objects.The reported task-generalization setting uses VLMs and text-to-video models pretrained on broad Internet data.
4 RELATED WORK
Related work uses language and vision-language models for high-level robot planning, while video models provide future visual states and dynamics reasoning.
- Language and Vision-Language Planning: LLMs and VLMs generate language-based plans conditioned on instructions and, in some cases, visual inputs.These approaches commonly map generated steps to pretrained policies or existing APIs.
- Language and Vision-Language Planning: VLMs may be limited to text-expressible plans and may struggle to reason about world dynamics because many are pretrained on static images.Video models instead offer potential for informative future image sequences and dynamics modeling.
- Positioning VLP: Unlike HiP’s one-step-at-a-time planning, VLP combines a VLM and video model through forward search over long-horizon future actions.The forward-search composition is presented as the mechanism for simulating and reasoning about longer horizons.
5 LIMITATIONS AND CONCLUSION
VLP’s image-based state representation and video dynamics model impose important limitations, although the paper identifies possible remedies and future applications.
- Limitations: Image states may omit full 3D information and latent physical factors such as mass.The paper suggests multi-view videos or heuristics using the full video plan as partial remedies.
- Limitations: The video dynamics model sometimes produces inaccurate simulations, including objects spontaneously appearing or teleporting.The authors suggest larger models, more training data, or explicit physics-oriented reinforcement learning feedback.
- Conclusion: The paper concludes that combining VLMs with text-to-video models generates more complex behaviors than their individual components, with possible applications beyond robotics.Steerable media generation is given as an example downstream area.
A APPENDIX
The appendix provides additional results and methodological details supporting the main paper.
- Appendix: Appendix sections cover additional results, evaluation, datasets, training, and planning details.These materials are organized in Sections A.1 through A.5.
A.1 ADDITIONAL RESULTS
Additional experiments identify failures from incorrect web-knowledge transfer and inconsistent physics, while showing robustness in goal-conditioned execution despite noisy synthesized goals.
- VLP sometimes misinterprets novel tasks when transferring world knowledge from other text-to-video datasets.In one example, the video model interprets the manipulator as an octopus.
- VLP sometimes synthesizes videos with inconsistent physics, causing objects to appear or disappear.
- Goal-conditioned policies can execute visual tasks despite misaligned synthesized goals and artifacts in generated boundaries.The policy can discard most goal-image information and focus on the portion required for immediate execution.
A.2 EVALUATION DETAILS
The evaluation manually checks whether generated videos ever reach specified long-horizon goals and uses repeated video generation because each video is computationally expensive.
- Video evaluation manually checks whether any generated frame satisfies the specified long-horizon goal.
- Approximately 30 minutes are required to generate each long-horizon video, so evaluation uses 50 videos per goal for each task and method.
- Execution evaluation uses the ground-truth simulation state of each block.
A.3 DATASET DETAILS
The experiments draw on substantial trajectory datasets for several robot settings, including real and simulated data, teleoperated demonstrations, and diverse web-scale sources.
- Language Table: Language Table training uses approximately 10,000 long-horizon trajectories across simulation and real data, covering several hundred goals.The selected experiments use three goals, with roughly 20,000 combined trajectories and approximately 400,000 short-horizon text labels.
- Failure Cases: Failure examples include inaccurate dynamics for rarely seen octopus toys and incorrect web-knowledge transfer when the toy is unseen in training data.
- Failure Cases: Generated videos can violate object permanence by making objects disappear or reappear.Respecting object permanence in long videos remains a stated future direction.
- 14DoF Bi-Manual Manipulation: The 14DoF bi-manual manipulation setting uses approximately 1,200 teleoperated kitchen-stacking demonstrations.The demonstrations progress from stacking bowls, to cups on bowls, and utensils on cups.
A.4 TRAINING DETAILS
Training details cover multiresolution text-conditioned video generation, goal-conditioned robustness to synthesized-goal noise, and comparisons against multiple baselines.
- Video Models: The video models generate 16-frame clips at 24 × 40 resolution and upsample them through 48 × 80 to 192 × 320 resolution.The models use a base channel width of 256.
- Goal Policy Robustness: Goal-conditioned policies remain robust to noise and artifacts in synthesized goals when executing control actions in the real environment.
- Baselines: VLP is compared with LAVA, UniPi, and additional baselines using the reported training configurations.LAVA is trained on long-horizon goal text, while UniPi follows the text-conditioned video-model architecture and approach used for VLP.
A.5 PLANNING DETAILS
VLP planning details specify the search configuration, sampling choices, heuristic thresholds, and computation used to generate long-horizon video plans.
- Planning used a horizon of 16, beam width of 2, and language and video branching factors of 4.
- Video generation used DDIM sampling with 64 base-resolution timesteps, 4 higher-resolution timesteps, and classifier-free guidance scale 5.
- Sixteen videos were generated in parallel using a 4 TPU inference pod, while VLM policy actions used temperature 0.3 and heuristic decoding used temperature 0.0.
- The heuristic clipping threshold was 50, and videos were removed when improvement exceeded 50 after one rollout to preserve physical plausibility.
- A horizon-16 planning procedure took approximately 30 minutes, and additional plans were generated from only initial images and language goals.