Source-linked AI summary
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation
Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, Sean Kirmani
TL;DR
Robot manipulation policies need to generalize to unseen object types and motions without expensive task-specific robot data collection. Gen2Act generates human videos from scene images and language goals, then translates those videos into robot actions with a closed-loop policy. It achieves ∼30% higher absolute success rate on average than the most competitive baseline for novel object and motion types, while supporting chained long-horizon activities.
Problem
Robot policies must generalize to novel tasks, object types, and motions without collecting robot interaction data for every task.
Method
Gen2Act uses zero-shot human video generation from web-trained models, then conditions a closed-loop human-to-robot translation policy on the generated video.
Results
∼30% higher absolute success rate is achieved on average over the most competitive baseline for novel object types and novel motions.
Takeaways & Limitations
The factorized system enables manipulation of unseen objects and novel motions with limited robot demonstrations and can be chained for long-horizon activities.
Abstract
from arXiv · showhide
How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data. Videos are at https://homangab.github.io/gen2act/
I. INTRODUCTION
Gen2Act addresses robot manipulation generalization by using zero-shot human video generation to provide motion information, then translating that information into robot actions. It targets unseen objects and motions while using limited robot data and reports substantially higher success on such tasks.
- Motivation: Robot policies should execute language-specified everyday tasks in unseen scenarios without collecting interaction data for every task.The motivating scope includes articulated-object manipulation, pouring, re-orienting objects, and wiping tables.
- Motivation: Prior web-video approaches predict task-specific visual attributes such as goal images, hand-object masks, or point tracks, but require intermediate models for supervision.This makes scaling those predictive models difficult.
- Approach: Gen2Act casts motion prediction from web data as zero-shot video prediction and conditions a robot policy on the generated human video.The video conveys both what to do and motion-centric information about how to perform the task.
- Approach: The system generates a human video from a scene image and language goal, then translates it into robot behavior with a closed-loop policy.Policy training uses offline robot demonstrations and combines behavior cloning with an auxiliary point-track prediction loss.
- Results: ∼30% higher absolute success rate is achieved on average over the most competitive baseline for novel object types and novel motions.Gen2Act also supports chaining tasks into long-horizon activities such as making coffee.
II. RELATED WORKS
Related work primarily scales robot imitation data or transfers representations and motion cues from non-robotic data. Gen2Act instead factorizes manipulation into web-scale human-video motion prediction and robot-action translation with limited robot interaction data.
- Visual imitation: Large-scale visual imitation approaches require expensive robot data collection and remain limited in generalizing to novel objects and motions.Some recent systems require years of data collection while still showing restricted generalization.
- Visual representations: Pre-trained visual representations from non-robot datasets can support manipulation policies but often remain task-specific because they require substantial in-domain robot data.Representation learning alone does not provide the full motion-to-action solution described here.
- Gen2Act: Gen2Act uses web-scale video models to generate human task videos and a learned translation model to infer robot actions from them.This factorization assigns motion synthesis to the video model and robot execution to the translation policy.
- Gen2Act: The translation policy is trained through behavior cloning on a small robot interaction dataset while conditioning on generated human video and recent robot observations.The policy is designed to convert motion cues from generated videos into executable robot actions.
B. Human Video Generation
Gen2Act uses off-the-shelf image-and-language-conditioned human video generation to produce motion cues, then conditions a closed-loop robot policy on those videos. Training adds point-track prediction to behavior cloning, while track prediction is omitted at deployment.
- Human video generation: The video-generation model receives a scene image and language task description and produces a human video for the requested manipulation.The generated video is intended to preserve the scene while depicting the desired action.
- Human video generation: Current video-generation models can produce human manipulation videos zero-shot without fine-tuning or adaptation.Gen2Act uses human rather than robot videos because zero-shot robot-video generation is not currently available without robot-specific fine-tuning.
- Training data: Paired generated-human-video and robot-trajectory datasets are created automatically from offline robot trajectories and language instructions.The procedure conditions generation on each trajectory’s first frame and does not require manually collected human videos.
- Deployment: At evaluation, Gen2Act generates a human video for the new scene and language goal, then feeds it to the translation policy for rollout.The approach is not tied to a specific video-generation model.
- Policy training: The closed-loop policy combines generated-video motion cues with robot observations to infer actions in the scene.Point-track prediction is used during training to make latent representations informative about motion, alongside behavior cloning.
- Policy training: Track prediction is an auxiliary training loss and is not used at test time, reducing deployment computation.Inference uses the video model and policy without the training-specific track-prediction transformer.
D. Deployment
Deployment first generates a human video from the task language and scene image, then rolls out the video-conditioned closed-loop policy. Long-horizon activities are handled by chaining policies for subtasks described by an LLM.
- Execution: Gen2Act deployment generates a human video conditioned on the task description and scene image before rolling out the closed-loop policy.
- Long-horizon tasks: For long-horizon activities, an off-the-shelf LLM produces language descriptions of subtasks that are executed sequentially with Gen2Act.
IV. EXPERIMENTS
The experiments evaluate Gen2Act across plausible human-video generation, multiple generalization levels, long-horizon task chaining, and improvement from additional tele-operated demonstrations.
- Evaluation questions: The experiments test whether Gen2Act generates plausible human manipulation videos in diverse everyday scenes.They consider kitchen, office, and lab scenes and ask whether generated videos plausibly depict manipulation.
- Evaluation questions: The evaluation tests generalization to new scenes, objects, and motions, including object types and motion types unseen in robot data.The evaluation terminology defines seen versus unseen with respect to the robot interaction dataset, while assuming broader coverage in web video data.
- Evaluation questions: The experiments examine whether chaining video generation and policy execution enables long-horizon manipulation.The stated long-horizon question concerns activities composed of several sequential tasks.
- Evaluation questions: The study tests whether co-training with a small amount of additional diverse tele-operated demonstrations improves performance on new tasks.This experiment evaluates whether extra human-operated robot trajectories can improve generalization.
- Evaluation setup: The evaluation measures task success across mild, standard, object-type, and motion-type generalization levels.Object-type and motion-type generalization involve completely unseen object or motion types in unseen scenes, relative to robot interaction data.
B. Dataset and hardware details
The experiments use a pretrained video model and a mobile manipulator, with robot demonstrations and generated or paired human-video data supporting policy training and evaluation.
- Video generation: Gen2Act uses VideoPoet with square-image and language conditioning, without fine-tuning the video model.The model directly generalizes to human-video generation in the robot experiment scenes.
- Robot hardware: The robot is a mobile manipulator with compliant two-finger grippers, deployed through end-effector control at 3Hz.Experiments span manually positioned offices, kitchens, and labs with different manipulated objects.
- Evaluation data: Table I compares Gen2Act and an ablated variant with baselines across the defined generalization levels.The table reports success-rate comparisons for different levels of generalization.
- Training data: Policy training combines an existing offline robot-demonstration dataset with paired human-video demonstrations and generated human-video/robot-demo pairs.Generated videos are conditioned on the first frame of corresponding robot demonstrations, while point tracks use off-the-shelf tracking.
C. Baselines and Comparisons
Gen2Act is compared with language-conditioned, real-video-conditioned, goal-image-conditioned, and track-ablation baselines, alongside qualitative and chained-task evaluations.
- Baselines: The comparisons include RT1, Vid2Robot, RT1-GC, and a Gen2Act variant without track prediction.RT1 uses the same robot data; Vid2Robot uses paired real human and robot videos; RT1-GC uses generated-video goal images.
- Human-video analysis: Qualitative generations plausibly manipulate instructed objects while preserving scene backgrounds and avoiding camera movements and artifacts.These zero-shot videos are presented as usable context for robot manipulation in novel scenes.
- Quantitative comparison: For higher generalization levels, Gen2Act achieves much higher success rates than the compared baselines and ablated variants.The authors attribute this result to combining human video generation with explicit motion extraction through track prediction.
- Long-horizon chaining: Long-horizon evaluation compares success rates for activities formed by sequentially executing multiple intermediate tasks.Table II covers activities such as making coffee, with each next video conditioned on the previous execution’s last scene image.
G. Co-Training with additional teleop demonstrations
Co-training adds a small amount of diverse tele-operated data while keeping video generation fixed, improving the translation model where the original robot dataset provides limited support.
- Setup: The co-training experiment adds diverse tele-operated trajectories to the existing offline dataset while keeping the video generation model fixed.The prior dataset had limited scene and task coverage, with less than 60% Gen2Act success at higher generalization levels.
- Setup: Table III analyzes co-training with approximately 400 additional tele-operated robot trajectories.The table focuses on the effect of this additional dataset on generalization.
- Results: A small amount of diverse demonstrations improves Gen2Act’s conditioning on generated videos at higher generalization levels.The reported improvement is most relevant where robot-data support is limited.
- Failure analysis: When generated videos are implausible in object-type or motion-type generalization, the policy fails more often because robot-data support is limited.The authors contrast this with mild and standard generalization, where video-generation inaccuracies are less correlated with policy failures.
- Limitations: The paper identifies current video-generation limitations, including unrealistic hands that restrict very dexterous manipulation.It suggests denser motion representations beyond point tracks as future work.
APPENDIX
The appendix details Gen2Act’s video-generation inputs, motion-aware policy training, and action prediction setup. It uses a pretrained video model, point-track auxiliary objectives, and discretized end-effector actions.
- Human Video Generation: VideoPoet generates human-manipulation videos from a task-language prompt and a square scene image without adaptation or fine-tuning.The model was pretrained on more than 270M videos and uses prompts such as “A person opening the microwave, static camera.”
- Video Encoding: Generated and robot videos are encoded with ViT features and Perceiver-Resampler Transformer modules that output N = 64 tokens.The encoders use gated cross-attention layers to process temporally uncorrelated video tokens.
- Motion Representation: Point-track prediction auxiliary losses make policy tokens informative of motion cues from generated videos and robot observations.Off-the-shelf tracking supplies tracks, while separate prediction transformers estimate them from video tokens during training.
- Action Prediction: The policy combines behavior cloning with track prediction and discretizes each action dimension into 256 uniformly distributed bins.It predicts end-effector actions, episode termination, and gripper open/close state.
C. Chaining Gen2Act for Long-Horizon Manipulation
Gen2Act can be deployed sequentially for long-horizon activities by generating task-specific human videos and chaining closed-loop policy rollouts. Task decomposition uses an off-the-shelf language model, while failures can arise from either video generation or execution.
- Chaining Gen2Act: Deployment first generates a human video from the task language and scene image, then rolls out the closed-loop policy conditioned on that video.For sequences, the last image of one task is used in chaining the next task.
- Task Decomposition: An off-the-shelf LLM such as Gemini decomposes long-horizon activities into language descriptions of intermediate tasks.The examples decompose table cleaning and coffee making into ordered subtasks.
- Example Sequence: The coffee-making sequence includes opening the Keurig lid and inserting a K-Cup before later subtasks such as closing the lid.These examples illustrate chaining multiple manipulation tasks rather than executing one monolithic instruction.
- Failure Examples: In object-type generalization examples, most robot-execution failures correlate with incorrect video generations, although plausible videos can still be followed incorrectly after grasping.The failure analysis distinguishes video-generation errors from trajectory-following errors during execution.
D. Analysis of Failures
Gen2Act’s failures depend on the generalization setting and reflect both video-generation quality and execution accuracy. Implausible videos are especially problematic for object- and motion-type generalization, while plausible videos do not guarantee successful execution.
- Failure Modes: For object-type and motion-type generalization, implausible generated videos are associated with failed task execution when robot-data support is limited.This pattern supports the policy’s use of generated human videos to infer motion cues.
- Failure Modes: Most illustrated failures correlate with video-generation errors, but plausible videos can still fail because of grasping or post-grasp trajectory following.The authors identify denser motion representations, such as object meshes, as a direction for future work.