PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-to-Robot Imitation in the Wild

Shikhar Bahl, Abhinav Gupta, Deepak Pathak

arXiv:2207.09450v1cs.ROcs.AIcs.CVcs.LGeess.SY

TL;DR

Robot manipulation outside labs is difficult because existing reinforcement-learning and imitation-learning approaches are sample-inefficient, unsafe, or dependent on costly demonstrations. WHIRL watches human videos to extract priors, improves them through real-world interaction with video alignment and exploration, and reports one-shot generalization across 20 in-the-wild tasks.

  • Problem

    Existing robot manipulation research remains concentrated in lab or simulation settings, while real-world reinforcement learning and imitation learning face data, safety, and demonstration-cost limitations.

  • Method

    WHIRL extracts human trajectory and interaction priors, learns residual policies through real-world interactions, aligns human and robot videos, and explores around the prior.

  • Results

    WHIRL demonstrates one-shot generalization and success across 20 manipulation tasks in three real-world environments.

  • Takeaways & Limitations

    WHIRL provides an efficient framework for learning robot manipulation policies from human videos in in-the-wild settings.

Abstract

from arXiv · show

We approach the problem of learning by watching humans in the wild. While traditional approaches in Imitation and Reinforcement Learning are promising for learning in the real world, they are either sample inefficient or are constrained to lab settings. Meanwhile, there has been a lot of success in processing passive, unstructured human data. We propose tackling this problem via an efficient one-shot robot learning algorithm, centered around learning from a third-person perspective. We call our method WHIRL: In-the-Wild Human Imitating Robot Learning. WHIRL extracts a prior over the intent of the human demonstrator, using it to initialize our agent's policy. We introduce an efficient real-world policy learning scheme that improves using interactions. Our key contributions are a simple sampling-based policy optimization approach, a novel objective function for aligning human and robot videos as well as an exploration method to boost sample efficiency. We show one-shot generalization and success in real-world settings, including 20 different manipulation tasks in the wild. Videos and talk at https://human2robot.github.io

I. INTRODUCTION

WHIRL addresses the challenge of moving robot manipulation from lab and simulation settings into the wild, where reinforcement learning is data-hungry and unsafe and imitation learning requires costly demonstrations. It uses human videos to obtain priors, improves policies through real-world interaction, aligns human and robot videos, and demonstrates one-shot generalization across 20 tasks.

  • Real-world manipulation remains largely restricted to simulation or lab-like settings despite progress in robot manipulation.
  • Reinforcement learning can be data-hungry and unsafe in the real world, while imitation learning commonly requires many costly kinesthetic or teleoperated demonstrations.
  • Visual human imitation offers a third-person alternative, but must address embodiment mismatch, missing human actions, absent task rewards, and limited training samples.
  • WHIRL extracts human trajectories and interaction information as priors, learns an improvement policy through real-world interactions, and aligns human and robot videos with an agent-agnostic objective.
  • WHIRL demonstrates one-shot generalization and success across 20 manipulation tasks in three environments, including doors, fridges, shelves, shirts, whiteboards, and taps.

A. Detecting Humans

WHIRL builds on passive human video as scalable semantic supervision while addressing the challenge of transferring knowledge across human and robot embodiments. The approach extracts human priors and uses them to learn interaction policies in diverse real-world settings.

  • Human-video robotics research seeks scalable semantic supervision, but aligning human videos with robot videos remains challenging.
  • Kinesthetic teaching and teleoperation provide demonstrations but are expensive, whereas online human videos offer broad task and visual diversity.
  • First-person reacher-grabber setups reduce the action domain gap but limit achievable tasks and require additional data-collection effort.
  • Prior-based action policies can help control robots but may require task knowledge and careful design, limiting scalability.
  • WHIRL extracts priors from human actions, learns interaction policies in the real world, and explores around those priors across in-the-wild settings.

A. Human Priors

WHIRL converts human demonstrations into robot-executable priors by extracting hand motion, object contact, interaction waypoints, orientation, and gripper commands. These quantities are projected into the robot frame using depth-based 3D pose estimation and robot-specific heuristics.

  • Manipulation trajectories are decomposed into pre-interaction, interaction, and post-interaction primitives that can be estimated from human videos.
  • Hand position is extracted frame by frame, while contact detection identifies when object interaction starts and ends.
  • The human prior contains interaction waypoints, hand orientation, and time-indexed commands to open or close the hand.
  • Figure 4 traces the prior from hand and interaction detection to an interaction region, movement direction, robot action-space projection, and trajectory execution.
  • Depth-based 3D pose estimation projects human-derived waypoints into the robot frame, producing robot-frame waypoints, wrist rotation, and gripper parameters Ψk.

B. Policy Learning via Interaction

WHIRL learns from safe but exploratory real-world interactions by sampling residual actions around human-derived priors. It selects successful or informative trajectories and fits task and exploration policies to improve sample efficiency.

  • Directly executing human priors may fail because of morphology differences, detection inaccuracies, and calibration errors, motivating real-world policy learning.
  • WHIRL balances safe interactions with nonrestrictive exploration by combining a task policy with a task-agnostic exploration policy.
  • A zeroth-order optimization procedure similar to CEM samples real-world interactions and is used because traditional reinforcement learning is infeasible for many diverse manipulation tasks.
  • The task policy outputs residuals to the human prior, initializing behavior near the prior while allowing search around it.
  • WHIRL ranks collected trajectories by human-robot video alignment cost, fits the task policy to elite examples, and fits exploration to examples with high representation change.

1) Policy Structure

WHIRL learns task and exploration policies around a human-derived prior, using residual and multimodal policies to improve interaction while avoiding local minima.

  • Exploration policy: The exploration policy is needed because interaction-based task optimization can become trapped in a local minimum near the prior.WHIRL therefore trains both a task policy and an exploration policy.
  • Task policy: The task policy outputs residuals to a human-derived prior, initializing behavior near the demonstration while allowing search around it.Residual learning is intended to avoid unsafe behavior and permit improvement beyond the prior.
  • Task policy: A Conditional VAE models multiple robot ways to perform a task from residual samples conditioned on the human video embedding.The encoder represents residuals, while the decoder samples residuals conditioned on the video and a latent variable.
  • Exploration policy: The exploration policy maximizes environment change around the prior rather than relying on intrinsic motivation or state coverage objectives.Because actions remain close to the prior, the resulting changes are expected to be meaningful and non-destructive.
  • Exploration policy: The exploration objective uses frame-wise video embeddings, with the same inputs, setup, and CVAE architecture as the task policy.Different video frames are compared through the frame-by-frame embedding Φf.
  • Design motivation: Human and robot videos require a representation that compares task progress despite differences in morphology and execution.The objective must evaluate robot behavior against the target human video without assuming identical movements.

2) Representations for Human-to-Robot Video Alignment

WHIRL aligns human and robot videos by comparing their environment-level effects through agent-agnostic video representations, then optimizes policies with sampled real-world interactions.

  • Agent-agnostic alignment: Rather than learning tightly coupled human–robot correspondences, WHIRL compares videos at a high level because environmental effects matter more than agent movements.This design targets scaling to many in-the-wild manipulation tasks.
  • Agent-agnostic alignment: An action-recognition model trained on large-scale passive data embeds entire videos so differently timed or paced demonstrations can be compared semantically.The model addresses differences in video length, speed, and other minor variations.
  • Agent-agnostic alignment: Video inpainting removes the human or robot from frames, producing an agent-agnostic representation based on the remaining scene.Segmentation models provide human and robot masks used for inpainting.
  • Policy optimization: The task policy is trained with the agent-agnostic objective, while exploration maximizes change using frame-wise embeddings and augmented video-level costs.Costs are sampled over multiple embeddings generated with different video augmentations.
  • Policy optimization: WHIRL samples residuals around a human prior, executes them in the real world, ranks resulting trajectories, and fits policies to the highest-ranking samples.Later iterations sample both from the learned policy and from a Gaussian distribution, fitting to the 10 highest-ranking residuals.
  • Policy optimization: The sampling procedure produces trained exploration and task policies and is summarized in Algorithm 1.

IV. EXPERIMENTAL SETUP

The experiments evaluate WHIRL on mobile-robot manipulation tasks in everyday environments and compare its interaction-based approach with offline and imitation-learning baselines.

  • Hardware: Experiments use a Stretch mobile robot with a 6-DOF arm, gripper, Cartesian wrist control, orientation control, suction fingertips, and a depth camera.The platform is selected for manipulation in the wild.
  • Environment and data: The dataset covers everyday objects and settings including drawers, dishwashers, fridges, and cabinet doors, with each task trained over three demonstrations.The setup includes 20 tasks shown in the paper and appendix.
  • Baselines: The main baseline comparison uses offline Conservative Q-learning trained with the same samples as WHIRL, because real-world RL is considered infeasible for the task set.The baseline predicts residuals to extracted human priors and uses video-embedding rewards.
  • Evaluation design: Figure 6 reports training and testing success rates across kitchen tasks, shelf placement, multitask policies, and cross-task generalization over three training iterations.
  • Baselines: Additional baselines include CQL variants using WHIRL, TCN, or CycleGAN representations, plus Behavior Cloning without iterative refinement.Behavior Cloning uses an L2 action loss rather than WHIRL’s VAE-based policy.
  • Evaluation design: The evaluation tests task scale, generalization to new scenes and objects, component contributions, and comparison with state-of-the-art approaches.Experiments analyze iterative improvement, the agent-agnostic objective, and the exploration policy.

A. Robot Learning in the Wild

WHIRL scales to diverse real-world manipulation tasks and improves through iterative interaction, while precise shelf placement remains more difficult and the method outperforms offline baselines.

  • Robot learning in the wild: WHIRL trains 20 diverse tasks in a few hours across varied locations, including fixed, rigid, and soft-object manipulation.Examples include water-tap operation, ball-in-hoop, shirt folding, and whiteboard cleaning.
  • Training evaluation: Door-task training success increases from about 40% to 92% over three iterations, demonstrating iterative improvement with WHIRL.
  • Training evaluation: Drawer-task training success increases from about 43% to 83% after two iterations, while dishwasher and shelf-placement training curves also improve.Shelf placement shows a larger train–test gap because it requires greater precision.
  • Comparison to baselines: WHIRL strongly outperforms offline RL and Behavior Cloning baselines trained with the same data amount.CQL-ours generally exceeds other offline variants, while Behavior Cloning mostly exceeds offline RL approaches.
  • Comparison to baselines: The agent-agnostic objective differentiates good and bad trajectories across multiple algorithmic settings, including CQL variants and Behavior Cloning.

C. Generalization to New Instances

WHIRL generalizes beyond the training instances, improving performance on held-out drawers, objects, and shelves, though scene changes usually reduce success. Results are evaluated across drawer and door tasks with averages over 30 trials.

  • C. Generalization to New Instances: The evaluation compares third-person human imitation on drawer and door tasks, with results averaged over 30 trials.
  • C. Generalization to New Instances: WHIRL improves success on held-out drawers and object placements across multiple iterations.Held-out drawer performance is lower than training-drawer performance, while held-out object and shelf evaluations show improvement over iterations.
  • C. Generalization to New Instances: Door performance on a held-out instance exceeds training performance, which the authors attribute to a stronger prior for that door.
  • C. Generalization to New Instances: WHIRL generalizes to new scenes, but performance is usually worse than on new instances because visual, geometric, and calibration changes are larger.For a new door setting, success still improves from 20% to about 57%.
  • C. Generalization to New Instances: WHIRL also shows some task-level transfer between drawer and door policies, with performance improving after further training.

F. Multi-task Generalization

WHIRL supports joint multi-task policy learning with improvement across training and testing iterations, while individual-task policies achieve higher success rates. Its agent-agnostic objective and exploration policy are both important for effective and efficient learning.

  • Multi-task Generalization: Joint policies improve across iterations on both training and testing scenes, but achieve lower success rates than policies trained for individual tasks.The largest increase occurs from the first iteration, while the advantage from additional data does not persist with further training.
  • Multi-task Generalization: Iterative improvement boosts performance across tasks and outperforms offline reinforcement-learning methods.This pattern is reported across nearly all tasks and scenarios evaluated in Figure 6 and in the comparison summarized in Table I.
  • Sensitivity of WHIRL: Removing the agent-agnostic objective produces almost no success-rate gain beyond the initial prior samples on drawer and door tasks.The authors attribute this to video-alignment models focusing too heavily on the agents rather than selecting the best trajectories.
  • Sensitivity of WHIRL: Removing exploration lowers success rates compared with WHIRL, although the method still learns more slowly without exploration.The exploration policy biases sampling toward maximizing changes in the environment.
  • Sensitivity of WHIRL: Embedding cost decreases from failure to partial success to success trajectories, with the largest drop between partial success and success.Despite nonzero measurement noise, the embedding robustly differentiates successful and unsuccessful trials.
  • Multi-task Generalization: WHIRL combines sampling-based policy optimization, agent-agnostic representations, and exploration that maximizes environmental change.The method was evaluated on performance, state-of-the-art baseline comparisons, and generalization across real-kitchen tasks.

APPENDIX

The appendix describes WHIRL’s real-world setup, data collection, task coverage, and policy implementation. Experiments use everyday environments, one demonstration per task, and a policy that predicts residuals to a human-derived prior.

  • Environment and Data Collection: The experiments use a Stretch robot with a mobile base, suction-cup fingertips, six controllable degrees of freedom, and an Intel RealSense D415 camera.The camera captures both depth and RGB images for human demonstrations and robot episodes.
  • Environment and Data Collection: All tasks are performed outside laboratories using everyday objects and locations such as kitchens.Torque limits required non-standard objects in some settings, including the ball-in-hoop task.
  • Tasks: The task suite contains 20 tasks spanning large fixed objects, small rigid objects, and soft-object manipulation.Examples include fridges, ball-in-hoop, shirt folding, and whiteboard cleaning.
  • Environment and Data Collection: Each task is trained from one human demonstration lasting about 30 seconds; each iteration samples 30 robot episodes and fits the policy to the top 10.A robot episode takes less than one minute, or about 20 minutes per iteration.
  • Policy Implementation: The policy is a four-layer MLP that receives a demonstration-video embedding and prior, then outputs a residual to the prior.The policy is trained as a variational autoencoder with KL-divergence weight 0.0005 and latent dimension 4.
  • Policy Implementation: Human priors use hand-object interaction detection, while inpainting removes humans and robots before video alignment.Action-recognition representations include Multi-Moments and SlowFast 3D ResNets; VGG16 features measure environmental change for exploration.
  • Policy Implementation: The implementation uses 640 x 480 images and fits the policy to the top 10 of 30 samples.Approximately 200 labeled robot images train the instance-segmentation module.

4) Codebases

The appendix lists codebases used for perception, video processing, baselines, and representation computation. These components support WHIRL’s human-video processing, alignment, segmentation, and offline-RL comparisons.

  • Perception and Processing: Hand-object detection uses Shan et al.’s codebase, while wrist detection uses FrankMocap.These modules provide human interaction and wrist information for the method’s visual processing pipeline.
  • Perception and Processing: Robot instance segmentation uses LabelMe for annotation and Detectron2-based training code.The segmentation module is part of the robot-video processing pipeline.
  • Baselines: The TCN and CycleGAN baselines use publicly available implementations from their respective codebases.These provide comparison methods for video-based learning experiments.
  • Video Processing and Representations: The inpainting model uses Copy-and-Paste Networks, and representation costs use the model from Monfort et al.The representation model computes features used in the method’s cost functions.
  • Baselines: Offline reinforcement-learning baselines use the d3rlpy codebase.These baselines provide comparisons without environment interaction during training.
Loading 2207.09450v1…