Source-linked AI summary
VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation
Hanzhi Chen, Boyang Sun, Anran Zhang, Marc Pollefeys, Stefan Leutenegger
TL;DR
Robot manipulation needs scalable ways to bridge diverse embodiments without costly physical demonstrations. VidBot extracts metric-scale 3D affordances from RGB-only human videos and learns coarse-to-fine interaction trajectories, achieving strong zero-shot performance across 13 tasks while remaining embodiment-agnostic. Its data quality is constrained by the accuracy of the depth foundation model and Structure-from-Motion pipeline.
Problem
Deploying manipulation skills across diverse robot embodiments and environments remains difficult, while teleoperated demonstrations are costly and human-video transfer has lacked directly deployable 3D affordances.
Method
VidBot combines monocular depth and Structure-from-Motion to extract metric-scale 3D hand trajectories, then uses coarse-to-fine affordance learning with diffusion and test-time constraints.
Results
VidBot outperforms existing methods by 20% in success rate across 13 manipulation tasks in zero-shot simulator settings and also transfers to real-world robot platforms.
Takeaways & Limitations
The results support in-the-wild RGB-only human videos as a scalable source for embodiment-agnostic robot manipulation learning and downstream deployment.
Takeaways & Limitations
Data quality remains constrained by the accuracy of the depth foundation model and the Structure-from-Motion pipeline, despite filtering low-quality labels.
Abstract
from arXiv · showhide
Future robots are envisioned as versatile systems capable of performing a variety of household tasks. The big question remains, how can we bridge the embodiment gap while minimizing physical robot learning, which fundamentally does not scale well. We argue that learning from in-the-wild human videos offers a promising solution for robotic manipulation tasks, as vast amounts of relevant data already exist on the internet. In this work, we present VidBot, a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos. VidBot leverages a pipeline to extract explicit representations from them, namely 3D hand trajectories from videos, combining a depth foundation model with structure-from-motion techniques to reconstruct temporally consistent, metric-scale 3D affordance representations agnostic to embodiments. We introduce a coarse-to-fine affordance learning model that first identifies coarse actions from the pixel space and then generates fine-grained interaction trajectories with a diffusion model, conditioned on coarse actions and guided by test-time constraints for context-aware interaction planning, enabling substantial generalization to novel scenes and embodiments. Extensive experiments demonstrate the efficacy of VidBot, which significantly outperforms counterparts across 13 manipulation tasks in zero-shot settings and can be seamlessly deployed across robot systems in real-world environments. VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
1. Introduction
VidBot targets scalable zero-shot robot learning from unlabeled human videos by extracting 3D affordances and transferring them across scenes and embodiments. Its coarse-to-fine framework combines pixel-level action identification, diffusion-based trajectory generation, and test-time constraint guidance.
- Robot manipulation still depends heavily on costly, time-consuming, and labor-intensive teleoperated demonstrations.
- Human videos offer scalable task diversity, but prior transfer methods often require static scenes, depth sensors, motion capture, or embodiment-specific fine-tuning.
- VidBot extracts temporally consistent, metric-scale 3D hand trajectories from RGB-only videos by combining learned monocular depth, Structure-from-Motion, and geometric constraints.
- Its coarse stage identifies contact and goal points from RGB-D observations and instructions, while its fine stage generates interaction trajectories with a diffusion model.
- Test-time differentiable costs guide trajectory sampling toward constraints such as multi-goal reaching and collision avoidance.
- Experiments in simulation and real-world settings show that the 3D affordance model outperforms baselines and supports visual goal-reaching and exploration.
2. Related Work
Prior robot-learning approaches use annotated data, simulation, visual representations, rewards, or extracted human motion, but often face costly data collection, restricted setups, or limited action representations. VidBot instead uses everyday human videos to predict embodiment-agnostic affordances in 3D space.
- Manual affordance annotation is costly, while simulation-based collection avoids human intervention but requires diverse virtual assets.
- Human-video robot learning includes visual representation learning, reward inference, and motion extraction, but many methods require in-lab setups or additional teleoperated demonstrations.
- The closest prior work extracts embodiment-agnostic actions from everyday videos, but its 2D pixel-level motions are oversimplified and ambiguous for direct robot deployment.
- Diffusion models generate complex data through iterative denoising and have become strong frameworks for learning multimodal robot trajectories.
3. Method
VidBot learns embodiment-agnostic 3D affordances from human videos through metric-scale trajectory extraction and a coarse-to-fine affordance model. The model predicts contact and goal configurations, then generates context-guided interaction trajectories using diffusion and test-time costs.
- Problem definition: VidBot represents affordances as 3D contact points and interaction trajectories from RGB-D observations and language instructions.The representation is designed to remain agnostic to robot embodiment.
- 3D affordance acquisition: A video-processing pipeline combines SfM, metric-depth prediction, hand-object masks, inpainting, and pose refinement to extract temporally consistent 3D hand trajectories.Camera poses and scales are optimized before transforming hand centers into the first-frame coordinate system.
- Coarse-to-fine affordance learning: The affordance model factorizes into a coarse network predicting goal and contact points and a fine model planning low-level interaction trajectories.The coarse stage uses RGB-D inputs and language, while the fine stage conditions on the coarse outputs and task inputs.
- Coarse affordance prediction: The coarse stage predicts pixel-space goal and contact heatmaps with depth information, then lifts sampled points into 3D using camera intrinsics.Goal and contact predictions use separate networks and incorporate visual, spatial, and language features.
- Fine affordance prediction: The fine stage integrates voxelized TSDF features with trajectory features to generate spatially aware interaction trajectories.Waypoint features are obtained by trilinear interpolation from the TSDF representation.
- Cost-guided trajectory generation: Test-time cost guidance adjusts diffusion trajectories to handle multiple goals, collision avoidance, embodiment morphology, and unseen-object geometry.The final cost also provides a criterion for selecting among candidate interaction plans.
4. Experiments
VidBot achieves strong zero-shot manipulation performance across simulated and real-world settings, while ablations show that coarse affordance prediction, test-time guidance, and cost-based trajectory selection are important to performance. The model also transfers to downstream robot-learning applications and multiple robot embodiments.
- Results: 88.2% success rate across 13 simulator tasks gives VidBot the best overall performance, nearly 20% above the runner-up.Octo achieves the second-best success rate of 69.2%.
- Results: Using the same human-video source as VRB, VidBot boosts task success by ca. 30% through 3D priors and its affordance-learning strategy.Figure 5 visualizes more accurate 3D contact points and interaction trajectories than VRB’s ambiguous pixel-space predictions.
- Ablation Studies: Removing coarse goal prediction drops performance from 85.6% to 57.8%, indicating that coarse goal configurations simplify fine-grained trajectory generation.The ablation variant also removes goal conditioning and multi-goal guidance.
- Ablation Studies: Multi-goal guidance improves overall performance by 12.3%, while collision-avoidance guidance raises success on portable-object pickup by 26.7%.The guidance terms use test-time observations to produce more controllable trajectories under explicit constraints.
- Ablation Studies: Selecting trajectories by final guidance cost matters: random selection causes an 11.1% performance drop.The final cost value serves as a heuristic for choosing the optimal interaction plan.
- Robot Learning Applications: VidBot improves visual goal-reaching and exploration, and achieves 80.0% success over 55 real-world trials on Stretch 3 and Spot.The real-world tests cover household tasks across three human-suited environments.
5. Conclusion
VidBot learns manipulation skills from in-the-wild RGB-only human videos and generalizes across tasks, environments, and robot platforms. It outperforms existing methods in zero-shot simulation while achieving real-world household-task performance across multiple robots.
- The embodiment-agnostic design supports deployment across robot platforms for diverse household tasks in real-world environments.