PaperlayerView exact revision on arXiv

Source-linked AI summary

Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

YuXuan Liu, Abhishek Gupta, Pieter Abbeel, Sergey Levine

arXiv:1707.03374v2cs.LGcs.AIcs.CVcs.NEcs.RO

TL;DR

Standard imitation learning usually requires expert observation-action pairs, whereas imitation-from-observation seeks to learn from videos recorded in different contexts. The paper translates demonstrations into the learner’s context and uses deep reinforcement learning to track them, achieving varied simulated and real robotic skills, including human tool-use tasks. The method nevertheless requires substantial demonstrations and multiple training contexts, and assumes demonstration and test contexts share a distribution.

  • Problem

    Standard imitation learning assumes observation-action demonstrations, leaving the problem of learning from observation-only videos across differing contexts.

  • Method

    The method learns a context translation model that converts demonstrations into the learner’s context, then uses deep reinforcement learning to track the translated behavior.

  • Results

    The method successfully learned diverse simulated and real manipulation tasks from context-varied videos, including human tool-use tasks and ladling almonds at 66% success.

  • Takeaways & Limitations

    Imitation-from-observation can support real-world robotic control from videos of human demonstrations across varied contexts.

  • Takeaways & Limitations

    The method requires substantial demonstrations and observations from multiple contexts, while current experiments assume demonstration and test contexts share a distribution.

Abstract

from arXiv · show

Imitation learning is an effective approach for autonomous systems to acquire control policies when an explicit reward function is unavailable, using supervision provided as demonstrations from an expert, typically a human operator. However, standard imitation learning methods assume that the agent receives examples of observation-action tuples that could be provided, for instance, to a supervised learning algorithm. This stands in contrast to how humans and animals imitate: we observe another person performing some behavior and then figure out which actions will realize that behavior, compensating for changes in viewpoint, surroundings, object positions and types, and other factors. We term this kind of imitation learning "imitation-from-observation," and propose an imitation learning method based on video prediction with context translation and deep reinforcement learning. This lifts the assumption in imitation learning that the demonstration should consist of observations in the same environment configuration, and enables a variety of interesting applications, including learning robotic skills that involve tool use simply by observing videos of human tool use. Our experimental results show the effectiveness of our approach in learning a wide range of real-world robotic tasks modeled after common household chores from videos of a human demonstrator, including sweeping, ladling almonds, pushing objects as well as a number of tasks in simulation.

I. INTRODUCTION

The paper defines imitation-from-observation as learning behaviors from videos without expert actions despite context differences, and proposes context translation combined with reinforcement learning. It targets practical robotic imitation from natural human videos, including tool-use tasks.

  • Motivation: Standard imitation learning typically assumes expert observation-action trajectories for behavioral cloning or reward learning.These approaches learn policies or reward functions from state-action examples.
  • Problem: Imitation-from-observation learns policies from observation sequences alone, even when demonstrations and learner contexts differ.Context differences include viewpoint, environment, object positions or identities, and surroundings.
  • Approach: The proposed method translates demonstrations between contexts, then uses deep reinforcement learning to track the translated behavior.The translation model converts demonstrations from settings such as a human third-person view into the robot’s context.
  • Evaluation: The approach is evaluated on simulated and real manipulation tasks, including sweeping, pushing, ladling, striking, and human tool-use behaviors.The introduction presents these applications as a practical use of natural videos of people performing desired behaviors.
  • Related work: The method is designed for high-dimensional raw observations and context variation, unlike approaches relying only on invariant features or pretrained visual features.The related-work discussion identifies restrictive requirements or weak performance for several alternatives on complex manipulation tasks.

III. PROBLEM FORMULATION AND OVERVIEW

The problem formulation treats demonstrations as observation sequences generated under varying contexts, with actions and underlying states unobserved. Successful imitation therefore requires both context-appropriate tracking information and actions that realize the tracked behavior.

  • Problem setting: Each demonstration is an observation sequence generated under its own context, including varying viewpoints, backgrounds, objects, and agent properties.The context affects observation distributions, dynamics, and expert behavior.
  • Problem setting: The demonstrations reveal observations but not the expert’s actions or underlying Markovian states.The formulation represents states as unknown and actions as unobserved in demonstrations.
  • Challenges: The learner must determine which observation information remains meaningful in its own context and which actions can track the demonstrated behavior.These are identified as the two central challenges of imitation-from-observation.
  • Challenges: Raw pixel distances may provide poorly shaped objectives because matching image intensities does not necessarily represent semantically correct task execution.Reinforcement learning can use a demonstration distance as a reward, but the distance must be informative.

IV. LEARNING TO TRANSLATE BETWEEN CONTEXTS

The context translation model learns to convert a demonstration from one context into another using paired demonstrations and a target-context initial observation. Its learned features support both translated observations and behavior tracking.

  • Translation objective: Training pairs a source demonstration with a target demonstration and conditions prediction on the target demonstration’s first observation.The model predicts future target-context observations from a source-context sequence and one target-context frame.
  • Use at learning time: After training, the model can translate a demonstration into the learner’s context for tracking.The resulting feature extractor maps an observation and context image to translated features.
  • Translation objective: The model assumes demonstrations are temporally aligned while learning a translation function across demonstration contexts.The paper notes that iterative time alignment could relax this assumption in future work.
  • Architecture: The architecture encodes source observations and the target context, translates their features, and decodes the translated features into target-context observations.The model uses source and target encoders, a translator, and a target-context decoder, with tied weights optional and skip connections included.
  • Training: The training objective combines translation, reconstruction, and feature-alignment losses over randomly paired demonstrations.Feature alignment places translated expert features and encoded learning trajectories in a common space for tracking.

V. LEARNING POLICIES VIA CONTEXT TRANSLATION

The policy-learning stage uses deep reinforcement learning to acquire actions that track translated demonstrations in the learner’s context.

  • Policy learning: Deep reinforcement learning learns actions that track translated demonstration features using a reward function based on the tracking distance.The policy operates after the translation model has produced observations and features in the learner’s context.

A. Reward Functions for Feature Tracking

The reward combines feature tracking with weak image tracking to guide policies toward translated demonstrations while addressing observation-distribution mismatch.

  • A. Reward Functions for Feature Tracking: The feature-tracking reward penalizes squared Euclidean deviations between the learner’s encoded observation and translated demonstration features.The translation function maps demonstration observations into the learner’s context, approximately encouraging tracking of their average feature.
  • A. Reward Functions for Feature Tracking: Feature tracking alone may fail when poor initial policies generate observations outside the demonstrations’ training distribution.In that case, the feature encoder must handle out-of-distribution observations during policy learning.
  • A. Reward Functions for Feature Tracking: The final reward is a weighted combination of the feature-tracking and image-tracking components, with wrec tuned as a small hyperparameter.

B. Reinforcement Learning Algorithms for Feature Tracking

The method uses reinforcement learning with a reward based on translated demonstration features, selecting algorithms differently for simulated and real-world experiments.

  • B. Reinforcement Learning Algorithms for Feature Tracking: The reward supports any reinforcement-learning algorithm, with TRPO used for simulated experiments.
  • B. Reinforcement Learning Algorithms for Feature Tracking: Real-world experiments use trajectory-centric reinforcement learning from guided policy search, based on locally linear dynamics and LQR updates.Image features are included in the state, and GPS minimizes squared Euclidean distance in state space while omitting image tracking.
  • B. Reinforcement Learning Algorithms for Feature Tracking: The simulated benchmark includes reaching, pushing, sweeping, and striking tasks with distinct object-manipulation goals.

VI. EXPERIMENTS

The experiments test whether context translation supports imitation from raw images across viewpoint, appearance, object-position, and object-configuration changes, while comparing prior imitation methods.

  • VI. EXPERIMENTS: The experiments evaluate context translation on raw images and variation in viewpoint, object appearance, object positions, and contexts.
  • VI. EXPERIMENTS: Four simulated manipulation tasks use expert policies trained with hand-specified rewards, with video demonstrations collected across randomly generated contexts.
  • VI. EXPERIMENTS: The reaching task varies goal positions, color, and appearance while requiring the arm to reach a red disk.
  • VI. EXPERIMENTS: The pushing task varies object positions and distractors, the sweeping task varies viewpoint, and the striking task varies target position.

A. Network Architecture and Training

The models use convolutional encoders, a translation module, and a decoder, with separate architectural choices for simulation and real-world images and training across multiple video datasets.

  • A. Network Architecture and Training: Simulation encoders use four stride-2 convolutions followed by two 1024-unit fully connected layers with LeakyReLU activations.
  • A. Network Architecture and Training: The translation module is a 1024-unit hidden layer receiving the concatenated encoder features, while the decoder uses fractionally-strided convolutions.
  • A. Network Architecture and Training: Skip connections connect every context-encoder layer to its corresponding decoder layer in simulation.
  • A. Network Architecture and Training: Real-world models use smaller feature layers, dropout in fully connected layers, and tied encoder weights between Enc1 and Enc2.
  • A. Network Architecture and Training: Training uses ADAM with learning rate 10^-4 across video sets ranging from 60 to 4500 videos, with task-dependent image resolutions.

B. Comparative Evaluation of Context Translation

Across simulated reaching, pushing, sweeping, and striking, the proposed method learned tasks from demonstrations in random contexts more reliably than prior imitation methods. Ablations further show that removing translation, alignment, reconstruction, or tracking components substantially degrades imitation performance.

  • Evaluation metric: Performance was evaluated using normalized final distance between the task-specific target object and its goal.The target is the robot hand, cylinder, balls, or ball depending on the task.
  • Baseline comparison: Third person imitation learning and generative adversarial imitation learning both achieved 0% success rate in the comparison.
  • Comparative results: The proposed method successfully learned reaching, pushing, sweeping, and striking from demonstrations provided in random contexts, whereas prior methods failed on reaching, pushing, and striking.Prior methods also struggled with sweeping; only pretrained visual features made a reasonable improvement there.
  • Ablation study: The full method performed consistently best across simulated reaching and pushing ablations.For Push Simulation, some ablations were not performed: without Ltrans, without Lrec, and without Lrec, Lalign.
  • Ablation study: Removing translation, reconstruction, alignment, feature-tracking, or image-tracking losses significantly worsened subsequent imitation performance.

D. Natural Images and Real-World Robotic Manipulation

The method was evaluated on natural-image manipulation tasks where human demonstrations and robot execution differed in viewpoints, object positions, and surroundings. It successfully learned real-world pushing, sweeping, and ladling behaviors and outperformed the tested baselines, while domain shift remained outside the present scope.

  • Experimental setting: Human demonstrations and robot execution used differing viewpoints and object configurations, while both used the same tool because contexts were assumed to come from the same distribution.
  • Real-world tasks: The method successfully learned real-world pushing, sweeping, and ladling tasks from human video demonstrations, with robot executions shown alongside the demonstrations.
  • Scope boundary: Domain shift was left for future work because the experiments assumed demonstration and test contexts were sampled from the same distribution.The authors identify domain adaptation as one possible direction.
  • Comparative performance: The method consistently performed well across Sawyer robot tasks and outperformed the baseline methods, using task-specific success metrics.

2) Sweeping:

The sweeping experiments test imitation-from-observation on crumpled paper and granular almonds, showing that the method can handle manipulation scenes that are difficult to represent analytically. The broader approach translates demonstrations across contexts, tracks them with reinforcement learning, and has limitations in demonstrations and context diversity.

  • Sweeping: The experiments evaluate sweeping crumpled paper into a dustpan and sweeping a pile of almonds, using almonds instead of dirt or fluids to avoid damaging the robot.The tasks probe manipulation of granular media and summarize quantitative results in Fig 8.
  • Sweeping: On the easier crumpled paper task, both the proposed method and kinesthetic teaching work well, while pre-trained visual features are insufficient.The success metric is the average percentage of paper pieces or almonds ending inside the dustpan.
  • Sweeping: On almond sweeping, the proposed method achieves a higher success rate than the alternative approaches.This result is reported for the Fig 7 almond-sweeping task.
  • Method: The method translates demonstrations into a target context and uses reinforcement learning to track them, enabling reproduction of observed behavior.The translation model is trained across contexts and generalizes to the learner’s unseen context.
  • Limitations: The method requires many demonstrations and observations from multiple contexts, which can make task-specific training inefficient when available contexts are scarce.The paper also identifies large embodiment differences as a domain-shift challenge for future work.
Loading 1707.03374v2…