Source-linked AI summary
Structured World Models from Human Videos
Russell Mendonca, Shikhar Bahl, Deepak Pathak
TL;DR
Robot learning needs broad, efficient ways to acquire manipulation skills beyond scarce robot interaction data and task-specific supervision. SWIM pre-trains a world model on human videos using structured visual affordances, fine-tunes it with reward-free robot data, and achieves strong performance across robots and tasks with under 30 minutes of interaction.
Problem
Robot world models need action information and robot-relevant actions, but robot-collected data is limited and expensive while abundant human videos use a different morphology.
Method
SWIM uses morphology-invariant visual affordances to pre-train a world model on human videos, then fine-tunes it with unsupervised robot interaction data for multi-task planning.
Results
SWIM reaches about 80% average success with joint models and about 75% with individual-task models, compared with around 40% for the next-best approaches in individual-task training.
Takeaways & Limitations
SWIM supports learning and continual improvement across diverse manipulation tasks by combining human-video pre-training with reward-free robot data.
Takeaways & Limitations
The reported system is evaluated on two robot systems: a Franka Arm and a Hello Stretch.
Abstract
from arXiv · showhide
We tackle the problem of learning complex, general behaviors directly in the real world. We propose an approach for robots to efficiently learn manipulation skills using only a handful of real-world interaction trajectories from many different settings. Inspired by the success of learning from large-scale datasets in the fields of computer vision and natural language, our belief is that in order to efficiently learn, a robot must be able to leverage internet-scale, human video data. Humans interact with the world in many interesting ways, which can allow a robot to not only build an understanding of useful actions and affordances but also how these actions affect the world for manipulation. Our approach builds a structured, human-centric action space grounded in visual affordances learned from human videos. Further, we train a world model on human videos and fine-tune on a small amount of robot interaction data without any task supervision. We show that this approach of affordance-space world models enables different robots to learn various manipulation skills in complex settings, in under 30 minutes of interaction. Videos can be found at https://human-world-model.github.io
I. INTRODUCTION
SWIM uses human videos to pre-train a world model in a morphology-invariant affordance space, then fine-tunes it with unsupervised robot data for general-purpose manipulation. Across robots and settings, it achieves roughly twice the success of prior approaches with less than 30 minutes of real-world interaction.
- Motivation: Robot learning is difficult to scale because reward structures and robot-collected interaction data do not transfer easily across new tasks and settings.Human videos offer abundant examples of diverse interactions, but their actions must be represented in a form usable by robots.
- Approach: SWIM represents manipulation actions through high-level, morphology-invariant affordances such as grasp locations and post-grasp waypoints.This abstracts away robot- and human-specific low-level control while preserving object interaction structure.
- Approach: SWIM pre-trains a world model on human videos, fine-tunes it on reward-free robot trajectories, and plans through the model to achieve goals.The same world model can use data from multiple tasks without task-specific reward supervision.
- Results: SWIM achieves roughly 2× higher success than prior approaches while requiring less than 30 minutes of real-world interaction data.The study evaluates the framework across multiple environments and robots.
II. RELATED WORK
The related work motivates SWIM through sample-efficient world models, visual and action pre-training, and human-video affordance learning. SWIM combines these directions by modeling temporal dynamics in a compact state space while using structured visual actions transferable between humans and robots.
- Efficient Real World Robot Learning: Robot learning methods reduce the burden of real-world control through demonstrations, autonomous interaction, simulations, hierarchical skills, physical priors, or visually grounded actions.SWIM additionally uses human-centric visual affordances to access out-of-domain internet videos.
- Visual and Action Pre-Training for Robotics: Prior visual pre-training methods learn generalizable representations from large-scale human videos, but primarily emphasize visual complexity rather than transferable action structure.SWIM uses human hand-object interactions to define affordance-based actions for robotics.
- Model-based learning: World models learn compact temporally structured states from high-dimensional observations and use them to predict dynamics and support planning.SWIM follows the Dreamer model structure and training procedure, including learned states, dynamics, reconstruction, and reward prediction.
IV. WORLD MODELS FROM HUMAN VIDEOS
SWIM defines transferable manipulation actions from human hand interactions using visual affordances, then augments them with Cartesian control for robot execution.
- Visual affordances as actions: Human-video actions are defined through grasp and post-grasp image locations to bridge the human–robot morphology gap.The affordance model learns these locations from video and uses them as transferable action targets.
- Visual affordances as actions: The affordance action is u_t = [p_t, d_t], combining an image-space pixel location with its corresponding depth.Depth enables projection into 3D robot-frame coordinates for grasping and manipulation.
- Hybrid action space: SWIM uses a hybrid action space that combines structured visual-affordance actions with arbitrary Cartesian end-effector actions.A mode index selects either affordance control or Cartesian adjustment at each timestep.
- Hybrid action space: The action representation includes a mode, gripper rotation, image-space action, and Cartesian end-effector action.Only one of the image-space or Cartesian actions is executed at a given timestep.
B. Structured Affordance-based World Models for Robotics
SWIM pre-trains world and affordance models on human videos, fine-tunes the world model with reward-free robot data, and plans toward goal images using model-based optimization.
- Human-video pre-training: Human video clips provide grasp and post-grasp pixels used to train the affordance model and world model.The procedure processes clips, creates structured actions, trains G, and trains W on trajectory sequences.
- Framework overview: SWIM pre-trains on passive human videos, fine-tunes with unsupervised robot data, and deploys the model to reach a goal image.These are the framework’s three stated phases.
- World-model predictions: Figure 5 presents human-video reconstruction pre-training and imagined high-reward rollouts for both human and robot data.The caption identifies the first highlighted image as the robot’s original observation.
- Robot fine-tuning and planning: Robot fine-tuning collects in-domain data with the affordance model before optimizing action sequences through the world model.CEM selects plans from affordance and fitted-distribution proposals.
- Human-video pre-training: During human-video training, the model uses image-space actions while randomly sampling unavailable depth and robot-rotation components.The resulting world model is trained on human trajectory sequences and visualized through passive-data predictions.
- Robot fine-tuning and planning: Planning ranks trajectories and evaluates predicted proposals by distance to the goal in the world-model feature space.The feature-space reward supports trajectory ranking and model-based planning.
A. Environments
SWIM is evaluated on two robot systems across six real-world manipulation tasks, with comparisons isolating human-video pre-training, affordance actions, world models, and action-space structure.
- Environments: The evaluation uses a Franka arm and Hello-Robot Stretch across six tasks in play-kitchen and real-kitchen environments.Tasks include opening cabinets, manipulating vegetables and a knife, opening a dishwasher, pulling a drawer, and opening a garbage can.
- Baselines and evaluation: Table I reports success rates for SWIM and baselines over 25 trials on six manipulation tasks.The experiments compare multiple world-model, behavior-cloning, affordance, pixel, and Cartesian-action variants.
- Baselines and evaluation: The MBRL-Affordance baseline removes human-video pre-training to test its contribution to generalizable world-model learning.MBRL-Pix additionally removes affordance-based action sampling, while MBRL uses only Cartesian actions.
- Baselines and evaluation: BC-Affordance tests whether filtered behavior cloning can replace a learned world model, using feature-space distance to select trajectories.BC-Pix retains behavior cloning but randomly samples action locations instead of using the affordance model.
C. Implementation details
Implementation uses large-scale human video clips to train affordance and world models, and evaluates success through repeated human-judged trials; removing human-video pre-training reduces success, especially for diverse tasks.
- Human-video implementation: Approximately 55K Epic-Kitchens clips are used for human-video world-model training, with grasp and post-grasp waypoints extracted from the clips.Depth and rotation components unavailable in the videos are randomly sampled.
- Ablation: Removing human-video pre-training causes a large success drop, especially for jointly trained models handling diverse robot tasks.The comparison is between SWIM and MBRL-Affordance.
- Evaluation: Success is evaluated over 25 trials using a predefined task-specific metric measured by a human.An example criterion is whether a cabinet is fully open.
VI. RESULTS
SWIM combines affordance-based world models, human-video pre-training, and robot fine-tuning to learn across tasks and continue improving with collected data. Experiments report strong multi-task performance, effective transfer from human videos, and over 90% success after continual training.
- SWIM achieves about 80% average success across environment settings and robots with joint models, versus about 40% for the next-best approaches on individual tasks.
- Sharing data across tasks improves SWIM performance, while human-video pre-training is especially important for joint models on diverse multi-task data.
- Human-video pre-training yields realistic world-model reconstructions early in training, unlike the pixelated reconstructions of MBRL-Affordance without pre-training.
- The affordance action space roughly doubles success relative to pixel-space actions in both filtered behavior cloning and world-model settings.
- SWIM continues improving through iterative fine-tuning and exceeds 90% success, outperforming BC-Affordance after continual training.
- The paper uses six real-world tasks and reports that the full system can collect its robot interaction samples in 30 minutes.
APPENDIX
The appendix describes the robots, sensing and control setup, and the six manipulation tasks used for evaluation. Experiments span Franka and Stretch robots in kitchen and everyday environments.
- Experiments use Franka Emika and Stretch RE1 robots controlled in end-effector space with task-specific rotational actions.
- The robots receive RGBD observations from Intel RealSense D415 cameras and use low-level impedance control for Franka end-effector actions.
- The evaluation contains six tasks, including vegetables, knife, cabinet, dishwasher, garbage-can-handle, and drawer manipulation.
3) Data Collection
Data collection begins with the affordance model and uses sampled grasp and post-grasp actions, while baselines vary pre-training, action-space structure, and planning strategy.
- Data Collection: The affordance model supplies 2D action locations and depth, which are converted into grasp and post-grasp trajectories for robot data collection.
- Data Collection: Human-video annotations provide hand-object contact points and post-contact motion, while rotation and depth values are randomly sampled during training.
- Data Collection: The grasp predictor uses a ResNet18 encoder-decoder with spatial softmax, and the post-grasp trajectory head is a six-layer, eight-head Transformer.
- Baselines: World-model baselines share SWIM's code but differ in pre-training or action space, including no pre-training for MBRL-Affordance and random pixel actions for MBRL-Pix.
- Baselines: BC-Affordance selects high-reward trajectories, fits a two-center Gaussian mixture model, and samples actions in the visual affordance space.
7) Training, Finetuning and Deployment
Training proceeds from large-scale human-video pre-training to robot-data fine-tuning, with separate sequence lengths and batch sizes for the different data sources.
- Training, Finetuning and Deployment: Human-video pre-training uses 100 batches per iteration, with sequence lengths of 2 for human clips, 3 for Hello robot data, and 10 for Franka data.
- Training, Finetuning and Deployment: Robot fine-tuning takes about 3–4 hours on an RTX 3090, using 300 epochs for joint models and 200 iterations for single-task models.
- Training, Finetuning and Deployment: Robot batches are smaller because fine-tuning uses longer sequences containing both affordance and Cartesian end-effector actions.
8) Evaluation
The evaluation executes world-model trajectories in the real world using open-loop control, with goal images specifying desired object manipulations. Each method or ablation is evaluated 25 times on average, alongside documented implementation codebases.
- World-model outputs are evaluated by executing their predicted trajectories in the real world with open-loop control.
- Goal images specify target manipulations including opening cabinets, lifting knives, pulling drawers, and opening garbage cans or dishwashers.
- 25 evaluations are performed for each method or ablation, and the average is reported.
- The implementation uses DreamerV2 for the world model, 100DoH for hand detection, Epic-Kitchens processing, R3M, and Polymetis for Franka end-effector control.