PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Perceptual Rewards for Imitation Learning

Pierre Sermanet, Kelvin Xu, Sergey Levine

arXiv:1612.06699v3cs.CVcs.RO

TL;DR

The paper addresses the difficulty of designing rewards and providing intermediate feedback for real-world RL tasks. It learns dense perceptual rewards and implicit sub-goals from a few human videos using pre-trained visual features, then uses those rewards for robotic learning. The method successfully supports real-world manipulation, including door opening from a human demonstration, while its reward classifiers outperform a random baseline by about a factor of 2.

  • Problem

    Real-world RL often requires hand-engineered rewards, extra sensors, and feedback for implicit intermediate steps that final-task measurements do not provide.

  • Method

    The method discovers intermediate steps from a few demonstration videos and learns incremental visual rewards using selected features from a pre-trained deep model.

  • Results

    The method learns a complex real-world door-opening skill from a human demonstration, while learned reward functions outperform a random baseline by about a factor of 2.

  • Takeaways & Limitations

    Pre-trained visual representations can be used without retraining to represent goals and sub-goals for real-world robotic learning.

Abstract

from arXiv · show

Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a reward function takes considerable hand engineering and often requires additional sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple implicit intermediate steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide feedback on these intermediate steps. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit specification of sub-goals. The resulting reward functions can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also show that our method can be used to learn a real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task. Supplementary material and data are available at https://sermanet.github.io/rewards

I. INTRODUCTION

The paper proposes learning dense visual rewards from a few human demonstrations, including implicit intermediate steps, so robots can learn complex manipulation without shared embodiment or extensive task-specific sensing.

  • Motivation and novelty: The method targets a limitation of prior reward-learning approaches that required kinesthetic demonstrations with robot state or low-dimensional state spaces and numerous demonstrations.The proposed setting instead uses videos of humans performing the task.
  • Contribution: The method learns perceptual reward functions from only a few demonstrations of real-world tasks, with automated discovery of intermediate steps.These rewards are intended to capture both high-level goals and the sub-goals comprising complex behaviors.
  • Empirical scope: The paper demonstrates vision-based reward learning for a complex robotic manipulation task from human demonstrations in real-world robotic experiments.The reward can be extracted from a demonstrator with a different embodiment, such as a human demonstrating for a robot.
  • Contribution: The approach uses pre-trained deep visual representations to identify goals and sub-goals without retraining or explicit sub-goal specification.The learned features are reported to generalize to new scenes and represent manipulation goals and sub-goals.

II. SIMPLE INVERSE REINFORCEMENT LEARNING WITH VISUAL FEATURES

The method approximates inverse reinforcement learning with a simple factorized model over pre-trained visual features, trading dynamical fidelity for efficient learning from very few demonstrations.

  • Approach: The approach exploits semantically meaningful features from a pre-trained deep network to infer task goals and sub-goals with a simple approximate inverse reinforcement learning method.The approximation is designed to reduce overfitting and support efficient learning in low-data settings.
  • Visual representation: The method uses ImageNet-pre-trained Inception features as visual reward representations and tests whether sparse feature subsets can encode manipulation goals without fine-tuning.The experiments suggest these representations can support reliable real-world robotic learning.
  • Inverse reinforcement learning: The model treats visual activations as states and approximates MaxEnt IRL by modeling demonstrated trajectories with a tractable reward distribution.Exact inference is infeasible for the high-dimensional visual state, while sampling would require many trials and introduce optimization challenges.
  • Model assumptions: The approximation assumes trajectories are dynamically feasible and uses independence between time steps and features to limit overfitting in the low-data regime.This is explicitly characterized as a simple, biased model.
  • Reward construction: The resulting reward decomposes across features, allowing each feature distribution to be fitted with its mean and variance and converted into a log-Gaussian reward.A quadratic feature reward provides an analytically normalizable Gaussian form.

B. Discovery of Intermediate Steps

The method discovers intermediate steps by segmenting demonstrations into visually consistent regions, then fitting separate reward functions for the resulting goals. Recursive similarity maximization minimizes within-segment feature variance, with a more efficient greedy variant producing qualitatively sensible results.

  • Algorithm 1 recursively evaluates candidate splits using average standard deviation, joining segment statistics to select the minimum-variance partition.
  • A single quadratic reward can be too coarse for complex multi-stage tasks, so the method fits one quadratic reward per automatically discovered intermediate step.
  • Learning identifies step boundaries and fits independent Gaussian feature distributions whose within-segment variance is minimized.
  • Recursive video segmentation splits demonstrations so frames within each segment are abstractly similar, treating the number of segments as a hyperparameter.
  • Algorithm 1 has complexity O(nm), while a greedy binary variant has complexity O(n^2 log m) and yields qualitatively sensible results.

C. Steps Classification

The steps classifier uses pretrained deep-model activations to find discriminative visual features for intermediate goals. Feature selection reduces the search space in low-data settings, while logistic regression uses all features and performs slightly better without severe overfitting.

  • The method learns a steps classifier from pretrained deep-model activations using either a regular linear classifier or a custom feature-selection classifier.
  • Feature selection targets discriminative features that remain invariant to unrelated variation such as lighting, color, and viewpoint.
  • Good feature-selection results occur with n = [32, 64], whereas accuracy collapses to 0% when n > 8192.
  • A small subset of features can reduce overfitting and make unsupervised step discovery tractable within the high-dimensional representation.
  • The logistic-regression layer uses 1,453,824 activations without fine-tuning and performs slightly better than feature selection without severe overfitting.

D. Using Perceptual Rewards for Robotic Learning

The learned perceptual rewards are integrated into a complete real-world skill-learning system using a reinforcement-learning algorithm and policy representation selected for efficient robotic evaluation. The empirical evaluation includes real-world robotic door opening.

  • The system uses PI2 reinforcement learning to evaluate learned perceptual rewards in real-world robotic skill learning.
  • The empirical evaluation covers learned reward structure, quantitative segmentation accuracy, and real-world validation on robotic door opening.

A. Perceptual Rewards Evaluation

The evaluation studies door opening and liquid pouring using roughly a dozen smartphone-recorded demonstrations per task, combining qualitative reward analysis with quantitative segmentation and reward-function tests. Learned rewards identify intermediate steps and outperform a random baseline, while some visual conditions produce failures.

  • Experimental setup: The experiments use door-opening and liquid-pouring tasks, with about a dozen smartphone-recorded training videos for each task.
  • Qualitative analysis: Qualitatively, intermediate steps are often correctly classified in partially completed pouring executions, while two-segment door splits are more sensible than three-segment splits.
  • Qualitative analysis: The method is robust to the presence or absence of a door handle and to opening direction, while four segments often give the most sensible pouring breakdown.
  • Failure cases: Failure cases include noisy door rewards for human-hand manipulation and a flat pouring reward when transparent liquid resembles the glass.
  • Quantitative analysis: Unsupervised segmentation is evaluated on training sets with human-provided ground-truth steps using Jaccard overlap, while reward functions are evaluated on test sets.
  • Quantitative analysis: Learned feature-selection and linear-classifier rewards outperform a random baseline by about a factor of 2 on individual test-set steps.
  • Qualitative analysis: Example pouring rewards can increase incrementally and saturate at completion, reject incomplete actions, or recognize that an action is already complete.
  • Quantitative analysis: The feature-selection approach works well for n in [32, 64] but collapses to 0% accuracy when n > 8192.

B. Real-World Robotic Door Opening

The door-opening experiments vary demonstration source, number of extracted subgoals, and door appearance while using cropped monocular RGB images as reward-learning input.

  • B. Real-World Robotic Door Opening: The experiments vary the source demonstration, number of extracted subgoals, and appearance of the door.Sources include human and robotic demonstrations.
  • B. Real-World Robotic Door Opening: The reward-learning setup compares human and robotic demonstrations across varied door configurations.The experiment adapts door opening to novel position or orientation configurations.
  • B. Real-World Robotic Door Opening: The system records monocular RGB images from an over-shoulder camera and crops and resizes the door before feature extraction.The shortest image side is resized to 299 pixels, followed by a 299x299 center crop.

2) Qualitative Analysis:

Qualitative and real-robot evaluations show that perceptual rewards remain useful across demonstrators and door appearances, with learning comparable to a sensor-based ground-truth baseline.

  • 2) Qualitative Analysis:: The learned rewards remain useful when trained from human demonstrations and applied to a robotic arm.The evaluation includes varied demonstrators and door types.
  • 2) Qualitative Analysis:: The method achieves similar convergence speeds to PI2 trained with a ground-truth reward from an instrumented door.PI2 runs for 11 iterations with 10 sampled trajectories per iteration.
  • 2) Qualitative Analysis:: The reward continues to work across some unseen door variations and can saturate when a robot opens a door despite no prior robot-arm images.The human-demonstration evaluation includes successful, failed, and varied-door videos.
  • 2) Qualitative Analysis:: The method consistently opens the door, and high-reward candidate trajectories support using perceptual rewards for global-policy training.The authors present this as evidence for extending the approach beyond local policy learning.
  • 2) Qualitative Analysis:: The method automatically identifies intermediate goals and learns incremental rewards from a few visual demonstrations using pre-trained deep-model features.The learned representations are reported to generalize without retraining.
  • 2) Qualitative Analysis:: A fixed viewpoint constrains the demonstrated imitation setting and is identified as a direction for future work.The paper connects viewpoint diversity with future lifelong robotic learning.

APPENDIX

The appendix specifies a recursive binary-splitting procedure for dividing a video into candidate segments and collecting their standard-deviation scores.

  • A. Binary Segmentation Algorithm: BINARYSPLIT recursively divides a video into the requested number of segments using SPLIT and a minimum split size.The algorithm returns segment splits and associated standard deviations.
  • A. Binary Segmentation Algorithm: The algorithm joins splits and standard-deviation values from the recursive branches into unified outputs.Branch outputs are combined with JOIN before returning all splits and standard deviations.

B. Combining Intermediate Rewards

The method combines rewards for discovered intermediate steps into one reward that partially credits earlier steps while emphasizing later ones.

  • B. Combining Intermediate Rewards: Each intermediate step produces a reward function that must be combined for use by the RL algorithm.The initial step is ignored because it represents the demonstrations’ starting state.
  • B. Combining Intermediate Rewards: Each later reward is assigned twice the maximum range of its preceding reward before the intermediate rewards are summed.This weighting makes later rewards more prominent in the combined function.
  • B. Combining Intermediate Rewards: The combination uses n detected intermediate rewards and an activations vector, with an example provided in Fig. 9.The resulting single reward is constructed from the discovered intermediate rewards.

C. Feature Selection Algorithm

The method selects a small set of discriminative features for each discovered sub-goal, then uses them to construct Gaussian perceptual rewards. At test time, these sub-goal rewards are combined into a single time-invariant reward that favors later steps.

  • Each feature is normalized using the mean and standard deviation across all training sequences before sub-goal-specific ranking.
  • Features are ranked for each sub-goal by their distance from the statistics of positive and negative frames.
  • Only the top-M features are retained, and the sub-goal reward is the log-probability under an independent Gaussian model.
  • The experiments use α = 5.0 and M = 32, while combining all goals into one time-invariant reward because goal-transition times are unknown at test time.

D. PI2 Reinforcement Learning Algorithm

The paper uses PI2 to improve a robot policy from image-based learned rewards, with demonstrations providing an initialization for local policy search. The reward pipeline combines intermediate step rewards, while qualitative evaluations examine discovered steps and reward signals across door and pouring tasks.

  • PI2 policy and reward: PI2 uses a linear-Gaussian policy whose learned controls are combined with fixed linear feedback over robot joint states.
  • PI2 policy and reward: Camera images are mapped through a visual feature network so that the learned reward can be evaluated from the robot’s observed state.
  • PI2 policy and reward: PI2 iteratively improves the policy by sampling trajectories and updating controls under a KL-divergence trust-region constraint.
  • Demonstration initialization: Demonstrations bootstrap PI2 through a trajectory-following controller, whose occasional successes provide a learning signal for subsequent improvement.
  • Reward composition: The combined pouring reward spans [0,2], while individual step rewards span [0,1] and represent the pouring action and the glass-full state after ignoring the initial resting step.
  • Real-world learning: The method learns reward functions from initial or human videos, allowing continued real-world policy improvement despite PI2’s need for demonstrations.
  • Qualitative evaluations: Qualitative analyses show unsupervised step splits for door and pouring videos, with testing reward signals that vary in reliability across sub-goals and ambiguous pouring cases.
Loading 1612.06699v3…