PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured World Models from Human Videos

Russell Mendonca, Shikhar Bahl, Deepak Pathak

arXiv:2308.10901v1cs.ROcs.AIcs.CVcs.LGcs.NE

TL;DR

Robot learning needs broad, efficient ways to acquire manipulation skills beyond scarce robot interaction data and task-specific supervision. SWIM pre-trains a world model on human videos using structured visual affordances, fine-tunes it with reward-free robot data, and achieves strong performance across robots and tasks with under 30 minutes of interaction.

  • Problem

    Robot world models need action information and robot-relevant actions, but robot-collected data is limited and expensive while abundant human videos use a different morphology.

    Source: §I · p. 1

  • Method

    SWIM uses morphology-invariant visual affordances to pre-train a world model on human videos, then fine-tunes it with unsupervised robot interaction data for multi-task planning.

    Source: §I · pp. 1–2, A. Visual Affordances as Actions · pp. 3–4

  • Results

    SWIM reaches about 80% average success with joint models and about 75% with individual-task models, compared with around 40% for the next-best approaches in individual-task training.

    Source: §VI · p. 7

  • Takeaways & Limitations

    SWIM supports learning and continual improvement across diverse manipulation tasks by combining human-video pre-training with reward-free robot data.

    Source: §I · p. 2, §VI · p. 8

    The reported system is evaluated on two robot systems: a Franka Arm and a Hello Stretch.

    Source: §VII · p. 8

Abstract

from arXiv · show

We tackle the problem of learning complex, general behaviors directly in the real world. We propose an approach for robots to efficiently learn manipulation skills using only a handful of real-world interaction trajectories from many different settings. Inspired by the success of learning from large-scale datasets in the fields of computer vision and natural language, our belief is that in order to efficiently learn, a robot must be able to leverage internet-scale, human video data. Humans interact with the world in many interesting ways, which can allow a robot to not only build an understanding of useful actions and affordances but also how these actions affect the world for manipulation. Our approach builds a structured, human-centric action space grounded in visual affordances learned from human videos. Further, we train a world model on human videos and fine-tune on a small amount of robot interaction data without any task supervision. We show that this approach of affordance-space world models enables different robots to learn various manipulation skills in complex settings, in under 30 minutes of interaction. Videos can be found at https://human-world-model.github.io

I. INTRODUCTION

SWIM uses human videos to pre-train a world model in a morphology-invariant affordance space, then fine-tunes it with unsupervised robot data for general-purpose manipulation. Across robots and settings, it achieves roughly twice the success of prior approaches with less than 30 minutes of real-world interaction.

Motivation

  • Robot learning is difficult to scale because reward structures and robot-collected interaction data do not transfer easily across new tasks and settings.Human videos offer abundant examples of diverse interactions, but their actions must be represented in a form usable by robots.Source: §I · p. 1

Approach

  • SWIM represents manipulation actions through high-level, morphology-invariant affordances such as grasp locations and post-grasp waypoints.This abstracts away robot- and human-specific low-level control while preserving object interaction structure.Source: §I · pp. 1–2
  • SWIM pre-trains a world model on human videos, fine-tunes it on reward-free robot trajectories, and plans through the model to achieve goals.The same world model can use data from multiple tasks without task-specific reward supervision.Source: §I · p. 2, Figure 2 · p. 2

Results

  • SWIM achieves roughly 2× higher success than prior approaches while requiring less than 30 minutes of real-world interaction data.The study evaluates the framework across multiple environments and robots.Source: §I · p. 2

II. RELATED WORK

The related work motivates SWIM through sample-efficient world models, visual and action pre-training, and human-video affordance learning. SWIM combines these directions by modeling temporal dynamics in a compact state space while using structured visual actions transferable between humans and robots.

Efficient Real World Robot Learning

  • Robot learning methods reduce the burden of real-world control through demonstrations, autonomous interaction, simulations, hierarchical skills, physical priors, or visually grounded actions.SWIM additionally uses human-centric visual affordances to access out-of-domain internet videos.Source: §II · p. 2

Visual and Action Pre-Training for Robotics

  • Prior visual pre-training methods learn generalizable representations from large-scale human videos, but primarily emphasize visual complexity rather than transferable action structure.SWIM uses human hand-object interactions to define affordance-based actions for robotics.Source: §II · p. 3, §III · p. 3

Model-based learning

  • World models learn compact temporally structured states from high-dimensional observations and use them to predict dynamics and support planning.SWIM follows the Dreamer model structure and training procedure, including learned states, dynamics, reconstruction, and reward prediction.Source: §III · p. 3, Figure 3 · p. 3

IV. WORLD MODELS FROM HUMAN VIDEOS

SWIM defines transferable manipulation actions from human hand interactions using visual affordances, then augments them with Cartesian control for robot execution.

Visual affordances as actions

  • Human-video actions are defined through grasp and post-grasp image locations to bridge the human–robot morphology gap.The affordance model learns these locations from video and uses them as transferable action targets.Source: A. Visual Affordances as Actions · p. 3
  • The affordance action is u_t = [p_t, d_t], combining an image-space pixel location with its corresponding depth.Depth enables projection into 3D robot-frame coordinates for grasping and manipulation.Source: A. Visual Affordances as Actions · p. 3

Hybrid action space

  • SWIM uses a hybrid action space that combines structured visual-affordance actions with arbitrary Cartesian end-effector actions.A mode index selects either affordance control or Cartesian adjustment at each timestep.Source: A. Visual Affordances as Actions · p. 4
  • The action representation includes a mode, gripper rotation, image-space action, and Cartesian end-effector action.Only one of the image-space or Cartesian actions is executed at a given timestep.Source: A. Visual Affordances as Actions · p. 4

B. Structured Affordance-based World Models for Robotics

SWIM pre-trains world and affordance models on human videos, fine-tunes the world model with reward-free robot data, and plans toward goal images using model-based optimization.

Human-video pre-training

  • Human video clips provide grasp and post-grasp pixels used to train the affordance model and world model.The procedure processes clips, creates structured actions, trains G, and trains W on trajectory sequences.Source: B. Structured Affordance-based World… · p. 4

Framework overview

World-model predictions

  • Figure 5 presents human-video reconstruction pre-training and imagined high-reward rollouts for both human and robot data.The caption identifies the first highlighted image as the robot’s original observation.Source: Figure 5 · p. 5

Robot fine-tuning and planning

  • Robot fine-tuning collects in-domain data with the affordance model before optimizing action sequences through the world model.CEM selects plans from affordance and fitted-distribution proposals.Source: B. Structured Affordance-based World… · p. 5

Human-video pre-training

  • During human-video training, the model uses image-space actions while randomly sampling unavailable depth and robot-rotation components.The resulting world model is trained on human trajectory sequences and visualized through passive-data predictions.Source: B. Structured Affordance-based World… · pp. 4–5

Robot fine-tuning and planning

  • Planning ranks trajectories and evaluates predicted proposals by distance to the goal in the world-model feature space.The feature-space reward supports trajectory ranking and model-based planning.Source: B. Structured Affordance-based World… · p. 5

A. Environments

SWIM is evaluated on two robot systems across six real-world manipulation tasks, with comparisons isolating human-video pre-training, affordance actions, world models, and action-space structure.

Environments

  • The evaluation uses a Franka arm and Hello-Robot Stretch across six tasks in play-kitchen and real-kitchen environments.Tasks include opening cabinets, manipulating vegetables and a knife, opening a dishwasher, pulling a drawer, and opening a garbage can.Source: A. Environments · p. 5

Baselines and evaluation

  • Table I reports success rates for SWIM and baselines over 25 trials on six manipulation tasks.The experiments compare multiple world-model, behavior-cloning, affordance, pixel, and Cartesian-action variants.Source: Table I · p. 6, B. Baselines and Ablations · p. 6
  • The MBRL-Affordance baseline removes human-video pre-training to test its contribution to generalizable world-model learning.MBRL-Pix additionally removes affordance-based action sampling, while MBRL uses only Cartesian actions.Source: B. Baselines and Ablations · p. 6
  • BC-Affordance tests whether filtered behavior cloning can replace a learned world model, using feature-space distance to select trajectories.BC-Pix retains behavior cloning but randomly samples action locations instead of using the affordance model.Source: B. Baselines and Ablations · p. 6

C. Implementation details

Implementation uses large-scale human video clips to train affordance and world models, and evaluates success through repeated human-judged trials; removing human-video pre-training reduces success, especially for diverse tasks.

Human-video implementation

  • Approximately 55K Epic-Kitchens clips are used for human-video world-model training, with grasp and post-grasp waypoints extracted from the clips.Depth and rotation components unavailable in the videos are randomly sampled.Source: C. Implementation details · p. 6

Ablation

  • Removing human-video pre-training causes a large success drop, especially for jointly trained models handling diverse robot tasks.The comparison is between SWIM and MBRL-Affordance.Source: Figure 6 · p. 7

Evaluation

  • Success is evaluated over 25 trials using a predefined task-specific metric measured by a human.An example criterion is whether a cabinet is fully open.Source: C. Implementation details · p. 7

VI. RESULTS

SWIM combines affordance-based world models, human-video pre-training, and robot fine-tuning to learn across tasks and continue improving with collected data. Experiments report strong multi-task performance, effective transfer from human videos, and over 90% success after continual training.

  • SWIM achieves about 80% average success across environment settings and robots with joint models, versus about 40% for the next-best approaches on individual tasks.Source: §VI · p. 7
  • Sharing data across tasks improves SWIM performance, while human-video pre-training is especially important for joint models on diverse multi-task data.Source: §VI · p. 7
  • Human-video pre-training yields realistic world-model reconstructions early in training, unlike the pixelated reconstructions of MBRL-Affordance without pre-training.Source: §VI · p. 7, Figure 8 · p. 8
  • The affordance action space roughly doubles success relative to pixel-space actions in both filtered behavior cloning and world-model settings.Source: §VI · p. 8
  • SWIM continues improving through iterative fine-tuning and exceeds 90% success, outperforming BC-Affordance after continual training.Source: §VI · p. 8, Figure 7 · p. 8
  • The paper uses six real-world tasks and reports that the full system can collect its robot interaction samples in 30 minutes.Source: §VII · p. 8

APPENDIX

The appendix describes the robots, sensing and control setup, and the six manipulation tasks used for evaluation. Experiments span Franka and Stretch robots in kitchen and everyday environments.

  • Experiments use Franka Emika and Stretch RE1 robots controlled in end-effector space with task-specific rotational actions.Source: 1) Robot Setup · p. 12
  • The robots receive RGBD observations from Intel RealSense D415 cameras and use low-level impedance control for Franka end-effector actions.Source: 1) Robot Setup · p. 12
  • The evaluation contains six tasks, including vegetables, knife, cabinet, dishwasher, garbage-can-handle, and drawer manipulation.Source: 2) Tasks and Environments · p. 12

3) Data Collection

Data collection begins with the affordance model and uses sampled grasp and post-grasp actions, while baselines vary pre-training, action-space structure, and planning strategy.

Data Collection

  • The affordance model supplies 2D action locations and depth, which are converted into grasp and post-grasp trajectories for robot data collection.Source: 3) Data Collection · p. 12
  • Human-video annotations provide hand-object contact points and post-contact motion, while rotation and depth values are randomly sampled during training.Source: 4) Human Videos · p. 12
  • The grasp predictor uses a ResNet18 encoder-decoder with spatial softmax, and the post-grasp trajectory head is a six-layer, eight-head Transformer.Source: 4) Human Videos · p. 12

Baselines

  • World-model baselines share SWIM's code but differ in pre-training or action space, including no pre-training for MBRL-Affordance and random pixel actions for MBRL-Pix.Source: 6) Baselines · p. 12
  • BC-Affordance selects high-reward trajectories, fits a two-center Gaussian mixture model, and samples actions in the visual affordance space.Source: 6) Baselines · p. 12

7) Training, Finetuning and Deployment

Training proceeds from large-scale human-video pre-training to robot-data fine-tuning, with separate sequence lengths and batch sizes for the different data sources.

Training, Finetuning and Deployment

8) Evaluation

The evaluation executes world-model trajectories in the real world using open-loop control, with goal images specifying desired object manipulations. Each method or ablation is evaluated 25 times on average, alongside documented implementation codebases.

  • World-model outputs are evaluated by executing their predicted trajectories in the real world with open-loop control.Source: 8) Evaluation · p. 13
  • Goal images specify target manipulations including opening cabinets, lifting knives, pulling drawers, and opening garbage cans or dishwashers.Source: 8) Evaluation · p. 13
  • 25 evaluations are performed for each method or ablation, and the average is reported.Source: 8) Evaluation · p. 13
  • The implementation uses DreamerV2 for the world model, 100DoH for hand detection, Epic-Kitchens processing, R3M, and Polymetis for Franka end-effector control.Source: 9) Codebases · p. 13

Related papers

Loading 2308.10901v1…