PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos

Haoyu Xiong, Quanzhou Li, Yun-Chun Chen, Homanga Bharadhwaj, Samarth Sinha, Animesh Garg

arXiv:2101.07241v2cs.ROcs.CVcs.LG

TL;DR

Human-video imitation can reduce reliance on expert actions, but human–robot morphology mismatch and missing salient detail complicate learning from translated images. LbW translates human videos into robot videos and detects keypoints for reward and policy learning; it reports state-of-the-art results across five manipulation tasks, with limits tied to pose similarity and single-video generalization.

  • Problem

    Human-video imitation requires only video supervision, but human–robot morphology mismatch and image translation that misses salient detail complicate learning manipulation skills.

    Source: §I · p. 1

  • Method

    LbW translates human videos into robot videos, detects keypoints on the translated frames, and uses the resulting structured representations for reward and policy learning.

    Source: §I · p. 1, A. Algorithmic Overview · p. 4

  • Results

    LbW achieves the state of the art on five robot manipulation tasks; all three methods reach 100% success on reaching, while LbW is favorable on sliding, drawer closing, and coffee making.

    Source: §I · p. 2, D. Performance Evaluations · p. 7

  • Takeaways & Limitations

    Keypoint representations provide semantically meaningful information for reward computation and policy learning in physical imitation.

    Source: A. Algorithmic Overview · p. 4, §VI · p. 7

    The translation model requires similar human- and robot-arm poses, and learning from one human video limits generalization to new scenes.

    Source: E. Discussion of Limitations · p. 7

Abstract

from arXiv · show

Learning from visual data opens the potential to accrue a large range of manipulation behaviors by leveraging human demonstrations without specifying each of them mathematically, but rather through natural task specification. In this paper, we present Learning by Watching (LbW), an algorithmic framework for policy learning through imitation from a single video specifying the task. The key insights of our method are two-fold. First, since the human arms may not have the same morphology as robot arms, our framework learns unsupervised human to robot translation to overcome the morphology mismatch issue. Second, to capture the details in salient regions that are crucial for learning state representations, our model performs unsupervised keypoint detection on the translated robot videos. The detected keypoints form a structured representation that contains semantically meaningful information and can be used directly for computing reward and policy learning. We evaluate the effectiveness of our LbW framework on five robot manipulation tasks, including reaching, pushing, sliding, coffee making, and drawer closing. Extensive experimental evaluations demonstrate that our method performs favorably against the state-of-the-art approaches.

I. INTRODUCTION

LbW targets physical imitation from human videos, where expert actions are unavailable but human–robot morphology mismatch and lost salient detail complicate learning. It combines unsupervised human-to-robot translation with keypoint representations, and the paper reports state-of-the-art results across five tasks.

I. INTRODUCTION

  • Expert demonstrations can be expensive and challenging to collect, while imitation from human videos requires only video supervision but must address human–robot morphology mismatch and lost salient detail.Existing image-to-image translation methods may capture macro features while neglecting details important downstream.Source: §I · p. 1
  • LbW translates human videos into the robot domain, detects keypoints on translated videos, and uses the resulting semantically meaningful structured representations for downstream policy learning.The framework's perception module aims to reduce the domain gap and capture salient-region detail.Source: §I · p. 1
  • LbW achieves the state of the art on five robot manipulation tasks, with keypoint representations used directly for task rewards and policy learning.The paper reports favorable performance against state-of-the-art approaches.Source: §I · p. 2

II. RELATED WORK

  • Earlier video-imitation work includes image-to-image translation approaches, while AVID uses unpaired CycleGAN translation but may suffer when visual artifacts affect feature-based state representations.Other cited approaches use 3D detection or deep pose estimation; some require additional labeled data –,.Source: §II · p. 2
  • Unsupervised keypoint methods infer points using temporal transformations, differentiable bottlenecks, or known transformations and correspondences –; LbW adopts Transporter for downstream policy learning.The paper identifies unsupervised, semantically meaningful representations as its focus and leaves new keypoint methods for future work.Source: §II · p. 2

III. PRELIMINARIES

The paper decomposes physical imitation from human videos into human-to-robot translation, unsupervised keypoint representation learning, and reinforcement-learning-based imitation. For translation, it adopts MUNIT, which learns across unpaired domains by separating shared content from domain-specific style.

III. PRELIMINARIES

  • Physical imitation from human videos is decomposed into human-to-robot translation, unsupervised keypoint-based representation learning, and physical imitation with reinforcement learning.Source: §III · p. 2

A. Unsupervised Image-to-Image Translation

  • LbW uses MUNIT for unpaired human-to-robot image translation, representing images with domain-invariant content and domain-specific style codes.The content encoders share weights across domains, while the style encoders do not; translation combines content with a style code sampled from the other domain.Source: A. Unsupervised Image-to-Image… · p. 2
  • MUNIT training uses adversarial and image, content, and style reconstruction losses, with hyperparameters controlling the relative importance of the loss functions.The source-domain reconstruction losses are described, with corresponding target-domain losses derived similarly.Source: A. Unsupervised Image-to-Image… · p. 3

B. Unsupervised Keypoint Detection

The method detects keypoints in translated robot videos to represent arm pose and interacting-object location, rather than relying on translated-image features that can omit salient detail or contain artifacts. Transporter learns keypoints by reconstructing one video frame from another using features transported at detected keypoint locations.

B. Unsupervised Keypoint Detection

  • Keypoints represent robot-arm pose and interacting-object location, retaining semantically meaningful state information while avoiding negative effects from translation artifacts.The method motivates this representation because translated images may miss salient details needed for downstream tasks.Source: B. Unsupervised Keypoint Detection · p. 3
  • Transporter extracts feature maps and keypoint locations from two frames, then transports features around detected locations to synthesize a representation of the second frame.The refinement network reconstructs the second frame from the transported feature.Source: B. Unsupervised Keypoint Detection · pp. 3–4
  • The detected keypoints from translated video frames serve as structured state representations for defining the reward and as policy-network inputs for action prediction.Source: B. Unsupervised Keypoint Detection · p. 4

IV. PROPOSED METHOD

The proposed-method section outlines an approach overview, unsupervised domain transfer with keypoint-based representations, and physical imitation with reinforcement learning.

IV. PROPOSED METHOD

  • The section introduces the approach, then covers unsupervised domain transfer with keypoint-based representations and physical imitation with reinforcement learning.Source: §IV · p. 4

A. Algorithmic Overview

LbW learns manipulation from a single human demonstration video without human action labels. It translates the video into the robot domain, extracts keypoints, and uses those representations for reward computation and action selection.

A. Algorithmic Overview

  • LbW uses a single human demonstration video, with no human actions provided, as the basis for robot imitation.The framework combines image translation, keypoint detection, and a policy network.Source: A. Algorithmic Overview · p. 4
  • The translation network maps each human demonstration frame into a robot-domain frame before keypoint extraction.The framework applies T to each human frame and then applies Ψ to the translated frame.Source: A. Algorithmic Overview · p. 4
  • The keypoint detector Ψ represents translated demonstration frames and current robot observations as keypoint coordinates.These representations are used to compute the imitation reward.Source: A. Algorithmic Overview · p. 4
  • The policy network takes the current observation's keypoint representation and predicts an action for the robot.The resulting action guides interaction with the environment.Source: A. Algorithmic Overview · p. 4

B. Unsupervised Domain Transfer with Keypoints

LbW trains human-to-robot translation using human demonstration frames and robot videos generated with random actions, then learns keypoints without ground-truth annotations. The resulting keypoint representations cover both translated demonstrations and current robot observations.

B. Unsupervised Domain Transfer with Keypoints

  • The source training data includes the demonstration and extra human hand-motion footage, while target-domain robot videos come from randomly sampled actions.The robot-video collection does not require human expertise or effort.Source: B. Unsupervised Domain Transfer with… · p. 4
  • MUNIT translates demonstration frames by combining human-frame content codes with robot-domain style codes.The style code is randomly sampled from the robot domain.Source: B. Unsupervised Domain Transfer with… · p. 4
  • Transporter detects keypoints on translated robot frames without ground-truth keypoint annotations, using random-policy robot video frames during training.It reconstructs the translated demonstration frame from detected keypoint locations and features.Source: B. Unsupervised Domain Transfer with… · p. 5
  • The learned detector extracts keypoint representations for translated demonstration frames and current robot observations, which are used to compute reward and predict actions.The task overview includes pushing, sliding, drawer closing, and coffee making, with human videos shown against plain or noisy backgrounds.Source: B. Unsupervised Domain Transfer with… · p. 5, Figure 4 · p. 5

C. Physical Imitation with RL

LbW learns a policy with reinforcement learning from keypoint representations of the current observation. Its reward compares the agent's keypoint trajectory with the translated demonstration trajectory, and the experiments use SAC.

C. Physical Imitation with RL

  • The policy maps the current observation's keypoint representation to an action, while RL optimizes cumulative learned reward.The action is executed in the environment to obtain the next observation.Source: C. Physical Imitation with RL · p. 5
  • The policy-learning phase is decoupled from keypoint-representation learning, and the experiments use Soft-Actor Critic (SAC).The method adds state, action, next state, and reward tuples to a replay buffer.Source: C. Physical Imitation with RL · p. 5
  • The reward uses a distance to the closest demonstration keypoint representation and a first-order difference of that distance.Hyperparameters λr1 and λr2 balance the two reward terms.Source: C. Physical Imitation with RL · p. 5

V. EXPERIMENTS

The experiments assess perception and compare LbW with state-of-the-art baselines on five manipulation tasks. The supplied experimental passages show a visual comparison for pushing but do not report benchmark outcomes.

V. EXPERIMENTS

  • For pushing, the visual comparison contrasts CycleGAN translations with LbW translations and detected keypoints, which are reported to capture robot-arm pose and interacting-object location.The caption describes human-video input in the first row, CycleGAN images in the second, and LbW results in the third.Source: Figure 5 · p. 6
  • The experiments ask whether perception handles the human-robot domain gap and detects keypoints, and how LbW compares with state-of-the-art task baselines.The stated evaluation scope is five robot manipulation tasks.Source: §V · p. 6

A. Experimental Setting

Experiments use two simulation environments and five manipulation tasks, with policy learning from image observations under fixed initial configurations.

  • Experiments use Fetch-Robot manipulation in OpenAI Gym and Meta-World across reaching, pushing, sliding, coffee making, and drawer closing.Source: A. Experimental Setting · p. 6
  • Reaching targets an end-effector location; pushing moves a puck to a target, while sliding sends it beyond reach to stop through friction.Source: A. Experimental Setting · p. 6
  • Coffee making moves a cup beneath the machine outlet over a longer distance than pushing, and drawer closing moves the end-effector to close the drawer.Source: A. Experimental Setting · p. 6
  • The perception-training dataset statistics report video-frame counts for each task in the human source and robot target domains.Source: Table I · p. 6
  • Policy learning uses 84 × 84 × 3 RGB observations, end-effector-position control, and fixed initial object and target locations.Source: A. Experimental Setting · p. 6

B. Comparison to Baseline Methods

The comparison uses classifier-reward and AVID-m baselines, with a shared SAC+AE control model and success-rate evaluations.

C. Dataset Collection and Statistics

The section describes perception and policy-training data, evaluates task success, discusses limitations, and concludes that keypoint representations support physical imitation from human videos.

C. Dataset Collection and Statistics

  • Perception training uses human expert and random-action videos plus robot random-policy videos, not robot expert videos; policy learning uses one human expert video, versus 15 human expert videos for AVID-m and 35 robot expert videos for classifier-reward.Source: C. Dataset Collection and Statistics · p. 7

D. Performance Evaluations

  • All three methods achieve 100% success on reaching; LbW performs favorably on sliding, drawer closing, and coffee making.Success is measured within 50 steps for reaching and pushing, and 300 for sliding, coffee making, and drawer closing; each task has 10 test episodes.Source: D. Performance Evaluations · p. 7
  • AVID-m translations show visual artifacts, and comparisons indicate that translation alone can hurt downstream performance; LbW adds unsupervised keypoint representations with semantically meaningful information.Source: D. Performance Evaluations · p. 7

E. Discussion of Limitations

  • LbW shares image-translation methods’ dependence on similar human and robot arm poses, and single-video training limits generalization to new scenes.Source: E. Discussion of Limitations · p. 7

VI. CONCLUSIONS

  • The conclusion attributes LbW’s performance to keypoint representations that guide manipulation through physical imitation.Source: §VI · p. 7

Related papers

Loading 2101.07241v2…