Source-linked AI summary
Large Video Planner Enables Generalizable Robot Control
Boyuan Chen, Tianyuan Zhang, Haoran Geng, Caiyi Zhang, Peihao Li, Kiwhan Song, William T. Freeman, Jitendra Malik, Pieter Abbeel, Russ Tedrake, Vincent Sitzmann, Yilun Du
TL;DR
General-purpose robot planners still struggle to generalize across unseen tasks and environments, while robot action data remains scarce. The paper trains Large Video Planner on large-scale video data to generate motion-plan videos from observations and instructions, then extracts robot actions. Evaluations report task-level generalization in third-party tasks and successful physical execution, though generation latency and action extraction errors limit deployment.
Problem
Robot planning models must generalize across diverse unseen tasks and environments, but robot action data is much scarcer than web-scale text and image data.
Method
Large Video Planner is a video foundation model trained with an open embodied video dataset to generate motion plans conditioned on scene frames and text instructions, followed by action extraction.
Results
The model shows task-level generalization on third-party selected tasks and successfully executes extracted plans on real robots across diverse settings and embodiments.
Takeaways & Limitations
Video can serve as the backbone of a robot foundation model, providing an open basis for zero-shot visual planning and real-robot execution.
Takeaways & Limitations
A single video plan takes several minutes on one A100 GPU, and errors in 4D reconstruction, hand pose estimation, or retargeting can cause task failures.
Abstract
from arXiv · showhide
General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available at https://www.boyuan.space/large-video-planner/.
1 INTRODUCTION
The paper addresses the unresolved challenge of planning robot behaviors that generalize across unseen tasks and environments. It proposes video as the primary modality for learning visual action plans and evaluates this approach through independent task testing and real-robot execution.
- General-purpose robots need planners that comprehend unseen tasks, adapt to novel scenes, and produce physically coherent behaviors.
- Robot action data is scarcer than web-scale text and image data, limiting VLA generalization in new tasks and unseen situations.
- The proposed video foundation model generates predictive video plans from task instructions and initial observations, then extracts executable actions.
- The work combines an open embodied-decision-making video dataset with History Guidance and Diffusion Forcing to improve temporal coherence and causal consistency.
- Independent testers evaluated creative tasks across diverse conditions, while real-robot experiments deployed extracted actions on parallel grippers and dexterous hands.
- The contributions include Large Video Planner, an open internet-scale dataset, and systematic evaluation across unseen environments, tasks, and embodiments.
2 RELATED WORK
Prior robotics work includes video learning for visual policies and dynamics models, alongside VLAs, embodied language models, and trajectory generators. This paper instead treats video generation itself as a robot foundation model that produces interaction videos for subsequent action extraction and retargeting.
- Video diffusion research has developed temporal-generation methods beyond vanilla full-sequence diffusion, including sliding-window and related approaches.
- History Guidance conditions generation on preceding frames to strengthen temporal grounding in image-to-video generation and video extension.
- Vision–Language–Action policies map multimodal inputs directly to robot actions, while other systems generate trajectories or use language models for high-level planning.
- The paper formulates a robot foundation model as a video generator that produces realistic interaction videos, from which executable actions are extracted and retargeted.
- Video learning is attractive for robotics because demonstrations provide rich data, and recent methods use generated videos as visual policies or dynamics models.
- Continuous actions can be extracted using inverse dynamics, goal-conditioned policies, scene flow, or the paper’s reconstruction-and-retargeting pipeline.
3 METHOD
LVP plans robot behavior by generating instruction- and observation-conditioned videos, then extracting and retargeting their motions as executable actions. Its latent diffusion model uses temporally causal representations, flow matching, Diffusion Forcing, and history guidance, trained on diverse action-centric human and robot videos.
- 3 METHOD: LVP uses a two-stage design: a video foundation model generates visual task plans, and action extraction converts them into executable robot control signals.The extracted actions can target either dexterous five-fingered hands or parallel grippers.
- 3 METHOD: A temporally causal 3D VAE compresses video clips into compact latent representations, where a diffusion model is trained to generate videos.The VAE maps spatiotemporal patches to 16-channel embeddings, and its decoder reconstructs generated latent samples into videos.
- 3 METHOD: The video diffusion model uses flow matching to predict the denoising flow from noisy latents conditioned on input images, text instructions, and noise levels.Training perturbs encoded latents with zk = (1 − k)z0 + kϵ and minimizes a flow-matching loss.
- 3 METHOD: Diffusion Forcing jointly supports image-to-video and video-to-video conditioning by assigning independent noise levels to randomly sized history and future segments.History lengths are sampled from 0 through 6 latent frames, enabling flexible conditioning without an additional variable-length cross-attention module.
- 3 METHOD: History guidance combines history-conditioned and text-conditioned diffusion scores to improve adherence to context frames and instructions during video sampling.The model can condition on a single frame or a context video by setting the history noise level to zero, while fully masking context yields an unconditional score.
4 EVALUATING TASK-LEVEL GENERALIZATION
The evaluation targets zero-shot task-level generalization on third-party-selected manipulation tasks, using graded video-plan metrics and real-robot execution across robot morphologies.
- Motivation: Zero-shot task-level generalization tests whether models can perform drastically different tasks they have never encountered, beyond object- or configuration-level variation.
- In-the-wild evaluation: Third-party participants proposed short manipulation tasks in everyday scenes, including substantially out-of-distribution environments and challenging actions such as flushing a toilet or tearing tape.
- Video-plan evaluation: The video planner was evaluated on 100 in-the-wild prompts using four levels covering contact, end state, task completion, and physical realism.
- Video-plan results: 59.3% success at Level 3 (Task Complete) was achieved on the third-party test set, while performance decreased monotonically from Level 1 to Level 4.
- Robot execution evaluation: Real-robot evaluation covered a Franka arm with a parallel-jaw gripper and a G1 arm with an Inspire dexterous hand, including out-of-distribution objects and substantially different scenes.
- Robot execution results: LVP consistently outperformed existing baselines across both task suites, with especially strong performance on dexterous manipulation and challenging tasks such as scooping coffee beans and tearing tape.
5 LIMITATIONS
The approach remains constrained by video-generation latency, imperfect action extraction and retargeting, and open-loop execution limitations.
- A single video plan takes several minutes on one A100 GPU, making direct real-time robot deployment intractable.
- Action extraction can fail because 4D reconstruction and hand-pose models make mistakes, while retargeting may be insufficient for some dexterous hands.
- Parallel-jaw gripper retargeting is challenging because the gripper has far fewer degrees of freedom than a human hand.
- The overall robot execution framework is open-loop and is not sufficient for accomplishing dexterous tasks.
6 CONCLUSION
The paper presents LVP, a video-backed robot foundation model that generates motion plans and retargets them to robots, supported by a released dataset and code.
- LVP is a 14-billion-parameter video foundation model that generates motion plans from scene frames and task descriptions.
- The generated motion plans can be retargeted to dexterous robotic hands using open-source reconstruction and retargeting tools.
- The model, data, and training code are open-sourced to support reproducible video-based robot learning.
- The Pandas-70M preprocessing retains clips with rich, meaningful, normal-speed human hand actions and no scene changes, yielding 196K clips after filtering.
A.2 DATA PREPROCESSING PIPELINE
The preprocessing pipeline removes unsuitable motion and camera behavior through dataset visualization and optical-flow filtering.
- The curation pipeline and final LVP-1M composition are illustrated in Figures 10 and 11.
- The pipeline removes the top 30% of videos with the highest spatio-temporal average optical-flow magnitude to reduce rapid camera motions.
B DETAILS OF VIDEO GENERATIVE MODEL TRAINING
Training uses reweighted sampling across eight heterogeneous video sources, followed by continued pretraining and finetuning with specified video representations and optimization schedules.
- Training configuration: Each training sample is a 49-frame, 832 × 480 video encoded as a VAE latent with shape 104 × 60 × 13.
- Training configuration: 60,000 continued-pretraining iterations use batch size 128 and learning rate 1 × 10−5 after 1,000 warmup steps.
- Training configuration: Finetuning runs for 10,000 iterations with batch size 128 and a reduced learning rate of 2.5 × 10−6.
- Data reweighting: Eight sources receive sampling weights from 0.05 to 2.0 to balance their unequal contributions during training.Weights are assigned to AgiBot-World, DROID, Ego4D, Pandas, SomethingSomething, Bridge, Epic-Kitchens, and Language Table.
C METHOD: COMPARISON BETWEEN HAMER AND OUR 4D ALIGNMENT MODULE
The 4D Alignment Module is evaluated against HaMeR through an ablation focused on spatio-temporal consistency, alongside visualizations of dataset composition and source reweighting.
- Alignment comparison: The ablation evaluates whether the 4D Alignment Module enforces spatio-temporal consistency relative to HaMeR.The comparison is motivated by HaMeR's inaccurate metric-scale depth estimates and temporally inconsistent hand poses.
- Dataset composition: The LVP-1M curation pipeline combines quality and embodiment filtering, action-focused Gemini captioning, and temporal frequency alignment.Its final dataset contains 1.4 million clips with diverse, action-centric text captions.
- Dataset composition: Figure 11 visualizes filtered LVP-1M clips across eight sources and shows final sampling ratios after first-stage reweighting.Greenish tones denote teleoperated robot datasets, while reddish tones denote human-centric activity datasets.
D METHOD: DETAILS ABOUT OUR VIDEO PROCESSING MODULES
The processing modules align monocular hand motion into metrically consistent robot coordinates, smooth trajectories temporally, and support retargeting to dexterous hands or parallel-jaw grippers.
- 4D alignment: The method recovers metrically consistent, temporally smooth hand motion from monocular RGB sequences without known camera intrinsics or extrinsics.The representation includes per-frame joint angles and a wrist-pose trajectory in a fixed world frame.
- 4D alignment: MegaSaM estimates and bundle-adjusts depth and camera parameters, using the first frame as the world reference to improve temporal consistency and resolve scale ambiguity.HaMeR's per-frame weak-perspective predictions can otherwise produce noisy, drifting translations.
- Wrist recovery: Wrist localization combines HaMeR image coordinates with MegaSaM depth and optimized extrinsics to recover wrist positions in the reference world frame.The process samples depth at the predicted wrist pixel and back-projects it using the intrinsic matrix K.
- Robot-frame transformation: Camera-to-robot alignment applies an estimated rotation and translation between the reference camera world frame and the G1 base frame.This transformation maps wrist positions and orientations into the coordinate system required for robot execution.
- Gripper retargeting: GraspNet supplies candidate grasp poses for parallel-jaw grippers because direct five-finger-to-two-finger retargeting is under-constrained.Heuristics then trigger grasp execution based on the observed grasping motion.
F EXPERIMENTS: REAL-WORLD ROBOTS SET UP
Real-robot experiments use a Franka arm with a parallel-jaw gripper and a Unitree G1 humanoid with an Inspire dexterous hand, with hardware and control adaptations for execution.
- Experimental settings: Table 4 presents the real-world robot experimental settings.
- Robot platforms: Task Set 1 uses a Franka Emika Panda arm equipped with a parallel-jaw gripper for baseline grasping tasks.
- Robot platforms: Task Set 2 combines a Unitree G1 humanoid with an Inspire dexterous hand and synchronized arm-hand control at 5 Hz.Predicted joint angles are remapped into the hand's valid motor-command ranges for real-time execution.
- Hardware adjustment: Some humanoid trials use tape on the gripper to compensate for limited torque or insufficient friction.The authors characterize this as a common hardware adjustment that does not affect the main experimental conclusions.
G VIDEO RESULTS
The paper provides qualitative visualizations of generated video plans, showing task-conditioned frame sequences from input observations and comparisons with real-world robot executions.
- Each qualitative video is represented by eight uniformly sampled frames, beginning with the input observation image and followed by the task instruction.The instructions shown beneath the sequences precede rephrasing by the language model.
- Generated plans with a human hand and robot gripper visualize how the model’s task-conditioned videos evolve from the input scene.Each row contains eight uniformly sampled frames, with the first frame defining the scene condition.
- Two visualizations pair five sampled frames from generated plans with the robot executing the corresponding task in the real world.The generated video is conditioned on the input scene image, while the second row shows physical execution.