Source-linked AI summary
Affordances from Human Videos as a Versatile Representation for Robotics
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Deepak Pathak
TL;DR
Robots need actionable knowledge of where and how to manipulate objects, which existing visual affordance models have not adequately connected to physical-robot deployment. VRB learns contact points and post-contact trajectories from human videos, then integrates them with four robot-learning paradigms. Across real-world experiments spanning 4 environments, over 10 tasks, and 2 robot platforms, VRB outperforms other state-of-the-art human hand-object affordance models and enables high-performance learning in the wild without simulation.
Problem
Existing affordance models are tested primarily on human-video datasets without physical-robot or in-the-wild experiments, leaving their robotic definition and representation unclear.
Method
VRB learns contact points and post-contact trajectories from egocentric human videos and integrates these affordance priors with four robot-learning paradigms.
Results
VRB outperforms other state-of-the-art human hand-object affordance models and enables high-performance robot learning in the wild without simulation.
Takeaways & Limitations
Human-interaction videos provide affordance priors that support robot learning across imitation, exploration, goal-conditioned learning, and action-space parameterization.
Abstract
from arXiv · showhide
Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild. Results, visualizations and videos at https://robo-affordances.github.io/
1. Introduction
VRB addresses the gap between visual affordance learning and direct robotic deployment by learning actionable interaction information from human videos. It integrates this representation across four robot-learning paradigms and evaluates it extensively on real robots in diverse settings.
- Humans use visual affordances to infer where to hold objects and how to move them across everyday tasks.
- Prior affordance methods primarily use human-video datasets, leaving their definition, representation, and robotic performance insufficiently evaluated.
- Robot learning often starts new tasks tabula rasa, while continuous-control complexity makes knowing how to manipulate objects difficult even with visual perception.
- VRB represents affordances with contact points and post-contact trajectories, learns them from egocentric videos, and integrates them across robot-learning paradigms.
- VRB is evaluated across 4 robotic paradigms, 10 real-world tasks, 4 environments, and 2 robot hardware platforms, including in-the-wild experiments.
- VRB outperforms other state-of-the-art human hand-object affordance models and enables high-performance robot learning in the wild without simulation.
2. Related Work
Related work spans human-interaction affordance learning, visual robot learning, and learning manipulation from human demonstrations. These approaches provide interaction or control priors but often require specialized data, can be sample inefficient, or make demonstration collection costly.
- Affordance-learning methods infer object, graph, environment, or heatmap information from geometry, physical attributes, segmentation, and large human datasets.
- Visual robot learning includes direct grasping and pushing, end-to-end control, structured policy spaces, offline robotics data, pretrained visual representations, and human demonstrations.
- End-to-end visual control is general but sample inefficient in the real world, while direct spatial policies can limit the types of tasks possible.
- Learning from demonstrations commonly uses teleoperation or kinesthetic teaching, both of which make collecting demonstrations tedious and slow.
3. Affordances from Human Videos (VRB)
VRB learns robot-transferable affordances from egocentric human videos using contact points and post-contact trajectories. It extracts multimodal supervision, compensates for camera and human-robot domain shifts, and predicts affordances for multiple robot-learning paradigms.
- 3. Affordances from Human Videos (VRB): VRB targets scalable extraction of affordance priors from egocentric human-interaction videos for robot learning in the wild.
- 3.1. Actionable Representation for Affordances: Affordance representations should be amenable to robot deployment because affordances describe interactions an actor can execute.
- 3.1. Actionable Representation for Affordances: VRB uses pixel-space contact points c and post-contact trajectories τ as robot-transferable affordance representations that are agnostic to robot morphology.The trajectory is modeled as τ = f(It, ht), using the image and human-hand location.
- 3.2.1 Extracting Affordances from Human Videos: The method finds contact timing, aggregates contact points with a GMM, and extracts the subsequent 2D hand trajectory from human videos.
- 3.2.1 Extracting Affordances from Human Videos: VRB compensates for camera motion with homography-based projection and maps human-frame affordances back to an initial frame without the human.
- 3.2.2 Training Affordance Model: Because interactions are multimodal, the model predicts multiple heatmaps as spatial probability distributions and uses a visual encoder to produce them.
- 3.2.2 Training Affordance Model: A trajectory network predicts relative post-contact movement from the latent representation, and VRB bootstraps four robot-learning paradigms rather than prescribing one use.
A. Imitation Learning from Offline Data Collection
VRB uses its affordance model to guide offline data collection, producing interaction-labeled datasets for subsequent imitation learning and goal-conditioned trajectory selection.
- Human demonstrations, teleoperation, and scripted policies are expensive, whereas affordance-guided collection is intended to yield interesting robot interactions.
- For each image, VRB stores the predicted contact point and trajectory with the image, then applies imitation learning after sufficient data are collected.
B. Reward-Free Exploration
VRB bootstraps reward-free exploration by biasing data collection toward trajectories that produce larger environment changes, avoiding exploration entirely from scratch.
- VRB guides reward-free exploration by bootstrapping trajectory sampling instead of starting from scratch.The approach targets more efficient discovery of diverse skills for downstream tasks.
C. Goal-Conditioned Learning
VRB uses affordances to expedite goal-conditioned learning by ranking trajectories according to their visual distance from a goal image and sampling from the resulting distribution.
- The goal-conditioned setting uses an image of the desired configuration to supervise policy search.
- Figure 4 compares predicted contact regions and post-grasp trajectories for VRB, HOI, Hotspots, and HAP.
- VRB ranks trajectories by distance to a goal image and fits a distribution to the top trajectories.The metric uses the final trajectory image and can also compare images in feature space.
- Subsequent data collection samples from the fitted trajectory distribution together with the affordance model.
D. Affordance as an Action Space
VRB parameterizes robot actions spatially using predicted affordances, pairing interaction locations with primitives to structure otherwise difficult continuous action spaces.
- VRB uses contact points and post-contact trajectories to define a spatially structured action space.This representation supplies locations and motion information for robot interaction.
- Spatial action parameterization offers a pragmatic alternative to optimizing directly over continuous robot actions.The approach assigns primitives such as grasping, pushing, or placing to locations, although this can limit task types.
4. Experimental Setup and Results
The paper evaluates VRB across four robot-learning paradigms using physical robots, simulation, multiple environments, and diverse tasks. Results report stronger exploration, action-space, representation, and offline-imitation performance than comparison methods.
- Experimental Setup: Experiments span two robot platforms, four environments, and more than ten real-world tasks, including in-the-wild manipulation.The platforms are the Franka Emika Panda and Hello Stretch mobile manipulator.
- Experimental Setup: The robot predicts pixel-space contact points and post-contact trajectories, projects them into 3D, and executes 6DOF end-effector actions.Each action samples a rotation, moves to contact, grasps, and moves to a post-contact position.
- Offline Imitation Learning: Table 1 reports success rates for k-NN and Behavior Cloning using offline data collected with different affordance models, with VRB vastly outperforming prior approaches.
- Reward-Free Exploration: VRB exploration raises coincidental-success performance 3× to 10× over HAP and random exploration for every task.
- Affordance as an Action Space: VRB produces more successes with DQN over a discretized affordance action space because more actions correspond to meaningful object interactions.
- Analyzing Visual Representations: VRB outperforms R3M on all three simulated Franka control tasks after finetuning for 2K steps.The result indicates that VRB representations contain information useful for control.
- Analyzing Visual Representations: VRB feature-space distance decreases almost monotonically during successful cabinet opening, correlating with actual task progress.
- Failure Modes: VRB has almost 2× as many successful cabinet-opening trajectories and more than 6× as many partial successes as baselines.
5. Conclusion
VRB learns robot-useful affordances from passive human videos and deploys them across multiple robot learning paradigms and real-world tasks.
- VRB learns affordances from passive human video data for deployment across imitation, exploration, goal-conditioned learning, and action-space parameterization.
- The approach represents affordances using contact points and post-contact trajectories.
- VRB is evaluated across four paradigms, 10 real-world robotics tasks, and experiments spanning over 200 hours.
- The experiments include many in-the-wild tasks and report that VRB drastically outperforms prior approaches.
- Future work includes more complex multi-stage tasks, force and tactile information, and visual-representation applications.
A. Result Videos
The paper provides qualitative rollout videos and describes the video-based training data and affordance-label extraction procedure.
- Qualitative rollout videos are available on the VRB website.
- Training uses EpicKitchens-100 sub-action videos, such as “open door” and “put cup on table,” as sequences.
- The extraction process identifies the first contact, selects the active hand, and smooths contact variables to remove spurious detections.
- Skin segmentation supplies contact points, which are clustered with a five-component GMM to estimate candidate contact means.
- The model input is the first pre-contact frame without a visible hand, and trajectories are discarded when camera motion removes relevant points from view.
- The trajectory predictor outputs length-5 trajectories using a two-layer MLP with hidden size 192.
C. Robotics Setup
The robotics setup deploys contact-point and trajectory affordances on two robot platforms for imitation, exploration, goal reaching, and discrete-action reinforcement learning.
- Robotics Setup: The Franka and Hello Robot platforms execute trajectories by moving to a contact point, closing the gripper, and following post-contact waypoints.
- Imitation: For imitation, trajectories are filtered by R3M distance to a goal image, then executed with k-NN or used to train behavior cloning.
- Exploration and Goal Reaching: For exploration, trajectories are ranked by visual change, while goal reaching ranks them by distance to the goal image.
- Action Space Parameterization: The action-space method clusters contact points and trajectories separately, then forms a discrete space with Nc*Nτ actions for DQN.
- Results: VRB produces more object-interacting discretized actions than HAP, and coincidental exploration success increases multiple folds over random exploration and HAP.
D. Baselines and Ablations
The evaluation compares VRB with established affordance baselines and includes simulation benchmarking alongside real-robot experiments.
- Baselines: Baselines include HOI, HAP, and Hotspots, using pretrained models with EpicKitchens-trained versions for HAP and Hotspots.
- Ablations: Behavior-cloning comparisons use frozen R3M features or an R3M encoder finetuned with affordance training.
- Simulation: The simulation benchmark follows Franka-Kitchen and evaluates turning on a light, opening a microwave, and lifting a kettle.
- Simulation: VRB significantly outperforms the baselines on the simulation benchmark’s success rates.
- Implementation: The implementation uses codebases for Franka control, hand-object detection, skin segmentation, affordance baselines, visual features, masking, and DQN.