PaperlayerView exact revision on arXiv

Source-linked AI summary

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning

Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, Sergey Levine

arXiv:1802.01557v1cs.LGcs.AIcs.CVcs.RO

TL;DR

The paper asks whether robots can learn manipulation from one raw human video despite shifts in viewpoint, environment, and embodiment. It meta-learns a prior and an action-free adaptation objective from human and robot demonstrations across tasks, then evaluates one-shot adaptation on new objects and settings. Experiments on PR2 and Sawyer show learning of placing, pushing, and pick-and-place skills from a single human video.

  • Problem

    Robots must address correspondence between human and robot embodiments and the large data demands of raw visual learning when imitating humans from video.

  • Method

    The method meta-learns an initialization and adaptation prior from human and robot demonstrations across tasks, adapting from one human video without human action labels.

  • Results

    Experiments on PR2 and Sawyer demonstrate one-shot learning of manipulation skills for new objects from RGB human videos across novel humans, scenes, and viewpoints.

  • Takeaways & Limitations

    After meta-learning, the robot can visually recognize and manipulate new objects from a single human demonstration in substantially different settings.

Abstract

from arXiv · show

Humans and animals are capable of learning a new behavior by observing others perform the skill just once. We consider the problem of allowing a robot to do the same -- learning from a raw video pixels of a human, even when there is substantial domain shift in the perspective, environment, and embodiment between the robot and the observed human. Prior approaches to this problem have hand-specified how human and robot actions correspond and often relied on explicit human pose detection systems. In this work, we present an approach for one-shot learning from a video of a human by using human and robot demonstration data from a variety of previous tasks to build up prior knowledge through meta-learning. Then, combining this prior knowledge and only a single video demonstration from a human, the robot can perform the task that the human demonstrated. We show experiments on both a PR2 arm and a Sawyer arm, demonstrating that after meta-learning, the robot can learn to place, push, and pick-and-place new objects using just one video of a human performing the manipulation.

I. INTRODUCTION

The paper targets one-shot robot imitation from raw human video despite visual and physical domain shifts. It uses meta-learning over human and robot demonstrations to infer robot skills without manually specifying human–robot action correspondence.

  • Motivation: Raw human-video imitation introduces correspondence problems from differences in appearance and morphology, while visual learning typically demands extensive data.The paper identifies domain shift and data requirements as the two central challenges.
  • Approach: The proposed system learns a task prior from human and teleoperated robot demonstrations collected across structurally similar tasks.This prior is built during meta-training before deployment on a new task.
  • Approach: After meta-training, the robot adapts to a new task using one human video, without manually specifying how human and robot actions correspond.The method combines learned prior knowledge with the single new demonstration.
  • Experiments: Experiments on two real robotic platforms evaluate learning from RGB human videos with novel objects, humans, scenes, and viewpoints.The experiments include PR2 and Sawyer platforms and test substantial visual variation.
  • Related Work: The paper positions watching-based imitation as an alternative to kinesthetic teaching, teleoperation, explicit human sensing, and reward-inference approaches.These prior approaches generally operate on trajectories, demonstrator sensors, or inferred goals and rewards.

III. PRELIMINARIES

The paper extends model-agnostic meta-learning to one-shot imitation from human video under domain shift. MAML learns an initialization that adapts quickly from few examples and is evaluated on held-out tasks.

  • Meta-learning: Meta-learning uses data from many tasks to discover shared structure, enabling rapid learning on new meta-test tasks.Meta-training and meta-testing draw tasks from a task distribution.
  • MAML: MAML optimizes initial parameters so one or a few gradient steps on K training examples produce good validation performance.The training data are partitioned into Dtr and Dval for adaptation and evaluation.
  • MAML: At meta-test time, gradient descent starts from the learned initialization using K examples from a held-out task.The resulting parameters are inferred specifically for the new task.
  • Motivation: The paper extends MAML because prior one-shot imitation used robot demonstrations but did not support raw human video or demonstration-to-robot domain shift.The extension addresses a different demonstration medium and embodiment.
  • Scope: The method is presented as one-shot imitation from humans with a learned temporal adaptation objective.The section introduces both the problem and the temporal adaptation component.

A. Problem Overview

The method treats human-video imitation as inferring robot policy parameters from one unlabeled human demonstration. It meta-trains a shared initialization and adaptation objective using paired human and robot demonstrations across tasks.

  • A. Problem Overview: The problem is to infer robot policy parameters that accomplish a task from one human video and a prior encoding visual, physical, and task knowledge.Manual priors are possible but task-specific and time-consuming, motivating learned priors from demonstrations.
  • A. Problem Overview: Human demonstrations contain image sequences, whereas robot demonstrations additionally contain robot states and actions.Robot state includes body configuration, while object information must be inferred from images.
  • A. Problem Overview: The method permits substantial differences between human and robot observations, including arm appearance, background clutter, and camera viewpoint.The formulation makes no assumptions about observation similarity.
  • Meta-training: Meta-training learns a policy prior from human and robot demonstrations across tasks, then combines that prior with one human demonstration for a new task.The task distribution supplies multiple training tasks and a held-out task for adaptation.
  • B. Domain-Adaptive Meta-Learning: Because human actions are unavailable and need not correspond directly to robot actions, adaptation uses a meta-learned objective operating only on policy activations.The learned loss produces parameter-update gradients that support effective robot actions after adaptation.
  • B. Domain-Adaptive Meta-Learning: The algorithm samples tasks and human videos, adapts policy parameters with the learned loss, and updates the meta-parameters using robot demonstrations.The displayed procedure separates inner adaptation from outer optimization.
  • B. Domain-Adaptive Meta-Learning: During meta-training, the system learns initialization θ and adaptation-loss parameters ψ, while robot actions supervise the behavioral-cloning meta-objective.At meta-test time, θ and ψ are retained and one human demonstration updates the policy parameters.

C. Learned Temporal Adaptation Objectives

The learned adaptation objective uses temporal information from human video and supplies gradients without human action labels, enabling policy adaptation from observations alone.

  • C. Learned Temporal Adaptation Objectives: The objective must provide useful gradient information without access to the human’s true actions.It operates on available policy inputs and activations instead of action labels.
  • C. Learned Temporal Adaptation Objectives: The adaptation objective processes video demonstrations with temporal convolutions rather than treating each timestep independently.The architecture uses multiple 1D convolutions over time to integrate sequential information.
  • C. Learned Temporal Adaptation Objectives: During meta-training, observations and actions are available to learn the prior and adaptation factor; during meta-testing, observations infer task-specific policy parameters.The graphical model distinguishes the information available in the two phases.
  • C. Learned Temporal Adaptation Objectives: The approach substantially outperforms a single-timestep linear loss.The supplied passage states this comparison without reporting a numerical value.

D. Probabilistic Interpretation

The probabilistic interpretation views the learned prior and adaptation objective as components of approximate MAP inference over task-specific policy parameters.

  • D. Probabilistic Interpretation: MAML-style gradient descent from θ is approximately equivalent to MAP inference under a Gaussian prior on policy parameters.The prior has mean θ, with covariance depending on the step size and number of gradient steps.
  • D. Probabilistic Interpretation: The proposed method adapts with gradient descent on the learned loss Lψ rather than the action likelihood.The learned loss replaces the likelihood as the adaptation objective.
  • D. Probabilistic Interpretation: Fixed-step adaptation still imposes the Gaussian prior centered at θ and yields approximate MAP inference on a joint distribution.The joint formulation combines the prior with a learned factor involving observations and policy parameters.
  • D. Probabilistic Interpretation: MAP inference provides a tractable alternative to Bayesian integration over φ and produces good results in practice.The probabilistic interpretation does not guarantee calibrated probabilities because training uses MAP inference.
  • D. Probabilistic Interpretation: The MAP interpretation is exact for linear functions but only a local approximation for nonlinear neural networks.
  • D. Probabilistic Interpretation: The learned adaptation objective Lψ induces a joint factor over observations and policy parameters φ.This factor is represented in a partially directed graphical model.

V. NETWORK ARCHITECTURES

The paper next introduces the policy π and the learned adaptation objective Lψ before describing their architectures.

  • V. NETWORK ARCHITECTURES: The section introduces the form of the policy π and the learned adaptation objective Lψ.

A. Policy Architecture

The policy maps visual observations and robot configuration into action distributions, while the adaptation objective uses feature and hidden representations with temporal convolutions.

  • A. Policy Architecture: The policy is a convolutional network that maps RGB images to a distribution over actions.
  • A. Policy Architecture: The policy concatenates visual feature points with the robot’s gripper pose before the fully connected control layers.The gripper pose is represented using the 3D positions of three non-axis-aligned points.
  • A. Policy Architecture: Continuous gripper velocities use a mixture density network, while the gripper open/close action is discrete.The supplied passage specifies the action-space structure and mixture-density representation for continuous actions.
  • A. Policy Architecture: The model also predicts the gripper pose at contact with the target object or container during meta-training.This auxiliary supervision comes from robot demonstrations and is not required at meta-test time with human video.
  • A. Policy Architecture: Five convolutional layers are followed by a spatial soft-argmax and fully connected layers.The spatial soft-argmax extracts 2D feature points from the final convolutional layer.
  • A. Policy Architecture: The adaptation objective decomposes into feature-level and hidden-layer terms: Lψ = Lψ1(f1:T) + Lψ2(h1:T).Both terms use temporal-convolution architectures whose output ℓ2 norm gives the scalar objective value.

VI. EXPERIMENTS

The experiments evaluate whether domain-adaptive meta-learning supports one-shot imitation from human videos, including viewpoint and embodiment variation, and compare alternative adaptation strategies.

  • The experiments test one-shot manipulation of new objects, generalization across viewpoints, backgrounds, and demonstrators, and transfer across robot platforms and demonstration types.
  • Experiments primarily use a 7-DoF PR2 with teleoperated robot demonstrations and RGB images, while Sawyer experiments use kinesthetic demonstrations.
  • The evaluation compares contextual, DA-LSTM, and DAML policies with linear or temporal adaptation objectives.

A. PR2 Placing, Pushing, and Pick & Place

PR2 experiments evaluate placing, pushing, and pick-and-place from single human videos, then test robustness to substantial visual domain shift.

  • PR2 Placing, Pushing, and Pick & Place: The initial tasks are placing, pushing, and pick-and-place amid distractor objects, using robot-perspective human demonstrations and RGB or RGB-D inputs.
  • PR2 Placing, Pushing, and Pick & Place: The evaluation uses 15, 12, and 15 novel target objects for placing, pushing, and pick-and-place, respectively, with one human demonstration and three policy trials per object.
  • PR2 Placing, Pushing, and Pick & Place: Across the tasks, the robot learns to interact with novel objects from one human video, while pick-and-place is the most difficult task.
  • PR2 Placing, Pushing, and Pick & Place: DA-LSTM and contextual policies struggle, and the linear adaptation loss performs poorly relative to the temporal adaptation objective.
  • Demonstrations with Large Domain Shift: The large-domain-shift evaluation uses human videos recorded in another room with a different camera and perspective, varying background and lighting.
  • Demonstrations with Large Domain Shift: With novel objects, a new demonstrator, and seen or novel backgrounds, performance degrades on novel backgrounds but remains about 70% successful.

C. Sawyer Experiments

The Sawyer experiments assess whether the approach transfers to another robot and to kinesthetic robot demonstrations, while the ablation isolates temporal adaptation.

  • C. Sawyer Experiments: The Sawyer study uses a 7-DoF robot with kinesthetic teaching, introducing the demonstrator into recorded images, and collects human demonstrations from the robot’s perspective.
  • D. Learned Adaptation Objective Ablation: The temporal adaptation ablation is conducted in simulated pushing without domain shift to isolate the adaptation loss.
  • D. Learned Adaptation Objective Ablation: 14% absolute improvement in success results from using the temporal adaptation objective in the simulated pushing experiment.
  • VII. Discussion: The paper concludes that robots can recognize and manipulate new objects from one human video after meta-learning with human and robot demonstrations on other objects.
  • VII. Discussion: The demonstrated meta-test behaviors remain structurally similar to those seen during meta-training, despite involving unseen objects and demonstrators.
  • Beyond Human Imitation: The method is presented as potentially applicable beyond human imitation, including animal imitation, simulated-robot imitation, and simulation-to-real transfer.

APPENDIX

The appendix documents data collection, validation, image inputs, model architectures, losses, and augmentation used in the experiments.

  • Human and robot demonstrations are collected at 10 Hz, last approximately 3–8 seconds, and use 40 randomly sampled frames during meta-training.
  • Separate models are trained for placing, pushing, and pick-and-place, using RGB inputs for the first two tasks and RGB-D inputs for pick-and-place.
  • Architecture and hyperparameters are selected using held-out validation objects for each task.
  • DAML and baseline methods share convolutional architectures, while the linear DAML variant replaces the learned temporal adaptation objective with one linear layer.
  • Pushing and pick-and-place use 20-mode mixture density networks with negative log likelihood behavioral cloning loss, and 100 sampled actions are evaluated at each test timestep.
  • Diverse human demonstrations: Large-domain-shift meta-training uses 80 objects, 10 backgrounds, fixed phone-camera viewpoints, and lighting noise augmentation.

C. Sawyer Robot Experiments

The Sawyer experiments use kinesthetic human demonstrations and an image-based policy adapted through a learned objective. Evaluation details span robot control, image augmentation, and pushing-task success measurement.

  • Demonstration collection: Sawyer demonstrations were collected by humans guiding the arm kinesthetically, at 10 Hz, rather than through teleoperation.Each timestep stored a monocular RGB observation.
  • Training: Training uses one human and one robot demonstration per sampled task, meta-batch size 8, step size α = 0.005, and 60k iterations.Inner gradients are clipped to [−30, 30].
  • Evaluation control: During evaluation, the first image predicts the final end-effector pose, followed by control using continuous end-effector velocity actions.Images are color-augmented during training but not during evaluation.
  • Pushing evaluation: Pushing success rates are reported over 74 tasks with 6 trials each, totaling 444 trials, with success requiring target placement for at least 10 timesteps.Each episode has a horizon of T = 100.
  • Pushing policy: The pushing policy combines 125 × 125 RGB images with robot joint angles, joint velocities, and end-effector information.Its architecture includes convolutional and fully connected layers plus a learned adaptation objective.
Loading 1802.01557v1…