PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Videos: Combining Offline Observations with Interaction

Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, Chelsea Finn

arXiv:2011.06507v2cs.LGcs.AIcs.CVcs.RO

TL;DR

Robot RL requires substantial online data, while human videos provide broad experience but lack actions and differ visually from robots. RLV combines offline observations with robot interaction, and the paper reports that it learns tasks with half as many trials as standard RL and imitation-from-observation approaches.

  • Problem

    Robot RL requires substantial online data, while human videos offer broad experience but lack actions and exhibit visual domain shift relative to robots.

    Source: §1 · p. 1

  • Method

    RLV combines offline observations with online robot interaction, estimating robot-space actions and rewards and using domain-invariant features to incorporate observations into RL.

    Source: §3 · p. 3, §3.2 · p. 4

  • Results

    Across simulation experiments, RLV learned tasks with half as many trials as standard RL and prior imitation-from-observation methods, and extended to real human videos.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that human observations can make robotic policy learning more efficient than standard RL and can support improvement beyond observation data.

    Source: §5 · p. 8

    The formulation assumes observational and robot MDPs have isomorphic dynamics and rewards, and that demonstrations come from a successful, potentially suboptimal policy.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Reinforcement learning is a powerful framework for robots to acquire skills from experience, but often requires a substantial amount of online data collection. As a result, it is difficult to collect sufficiently diverse experiences that are needed for robots to generalize broadly. Videos of humans, on the other hand, are a readily available source of broad and interesting experiences. In this paper, we consider the question: can we perform reinforcement learning directly on experience collected by humans? This problem is particularly difficult, as such videos are not annotated with actions and exhibit substantial visual domain shift relative to the robot's embodiment. To address these challenges, we propose a framework for reinforcement learning with videos (RLV). RLV learns a policy and value function using experience collected by humans in combination with data collected by robots. In our experiments, we find that RLV is able to leverage such videos to learn challenging vision-based skills with less than half as many samples as RL methods that learn from scratch.

1 Introduction

The paper proposes using human videos to broaden robot RL experience despite missing actions and visual domain shift. RLV combines offline observations with robot interaction and reports learning with half as many trials as standard RL and imitation-from-observation approaches.

1 Introduction

  • Human videos offer broad training experience, while robot RL requires substantial online trial-and-error data to acquire behaviors.The paper frames human video as a readily available source of diverse experiences for robot training.Source: §1 · p. 1
  • Human videos lack corresponding actions and can differ from robot data in action space, morphology, viewpoint, and environment.These differences make direct use of human experience in robot RL challenging.Source: §1 · p. 1
  • Because demonstrations may be suboptimal or mix tasks, imitation can fail to learn successful policies; RL can use successful and unsuccessful trials and improve beyond observations.The paper argues that RL can better leverage information in observational data than perfect imitation alone.Source: §1 · p. 2
  • RLV addresses missing actions, rewards, and domain shift by inferring actions and rewards from domain-invariant image representations.It combines offline observation data with online robot interaction.Source: §1 · pp. 1–2
  • RLV learned tasks with half the trials of standard RL and prior imitation-from-observation methods across simulation experiments, and extended to real human videos.The authors report that it handled observational demonstrations of varying quality.Source: §1 · p. 2

2 Related Work

Prior work addresses action-free demonstrations, RL with demonstrations, and domain adaptation, but RLV combines offline observations with RL without requiring demonstration actions or a learned dynamics model.

Learning from demonstration

  • Unlike action-conditioned demonstration approaches, RLV does not require demonstrator actions, which may require instrumentation or be impossible across different morphologies.The cited early work sought to match an expert policy.Source: §2 · p. 2

Imitation learning without actions

  • Unlike action-free imitation methods, which are limited by demonstrator performance, RLV integrates observations into RL and can improve over them.The cited imitation approaches include matching state transitions [18] [3] [19] [20] [4] and learning rewards [22] [23] [24] [25] [26] [27] [28] [29].Source: §2 · p. 2

Reinforcement learning with demonstrations

  • Unlike demonstration-based RL methods [30] [31] [32] [33] [34] [35], RLV does not assume demonstrations include actions.Both approaches allow reinforcement learning to fine-tune beyond expert performance.Source: §2 · p. 2

Related action-free methods

  • Compared with action-free methods [36] [37], RLV does not require a good dynamics model and targets domain shift across morphologies, viewpoints, and backgrounds.The paper contrasts these methods with RLV's handling of more complex visual domain shifts.Source: §2 · p. 2
  • Navigation work using offline videos [38] does not address morphology and action-space differences required for manipulation from human videos.The paper positions RLV as a step toward visual manipulation with diverse open-world human data.Source: §2 · p. 2

Domain adaptation

  • Domain adaptation includes image translation [39] [40] [41] [42] and domain-confusion methods [46] [47] [48] [50]; RLV applies adversarial domain confusion to RL with offline observations.The passage also notes image translation for sim-to-real adaptation [43] [44] and demonstrations [24] [45] [26].Source: §2 · p. 3

3 Reinforcement Learning with Videos

RLV incorporates action-free observations into off-policy RL by estimating robot actions and rewards and learning features that reduce domain differences. Its formulation assumes the observational and robot environments have isomorphic dynamics and rewards, with observations from a successful, potentially suboptimal policy.

3 Reinforcement Learning with Videos

  • The framework addresses missing actions and rewards in observational data and domain shift relative to the robot.The section introduces the techniques for action estimation, reward estimation, and domain adaptation.Source: §3 · p. 3

3.1 Problem Formulation

  • The formulation treats robot interaction and observations as MDPs with distinct state and action spaces but isomorphic dynamics and rewards, and assumes observations come from a successful, possibly suboptimal policy.The isomorphism assumption simplifies the gap between human and robot states; the paper notes that human state is not actually isomorphic to robot state.Source: §3.1 · p. 3

3.2 Overview

  • RLV keeps separate replay pools for action-free observations and action-conditioned robot interaction, then combines their encoded samples for RL training.The observation pool contains the initial observations, while the interaction pool is updated during training.Source: §3.2 · p. 4

3.3 Action Prediction

  • An inverse model trained with robot transitions maps pairs of invariant features to robot-space actions, then annotates action-free observations for RL.The model is trained from interaction data and applied to observation-data feature pairs.Source: §3.3 · p. 4

3.4 Reward Generation

  • Observed trajectories receive a large constant reward at the final timestep and a small constant reward at every other timestep, encouraging the agent to reach their endpoints.This assumes the observed trajectories are good, but the authors report robustness to suboptimal trajectories because robot interaction can counter inaccuracies.Source: §3.4 · p. 5

3.5 Domain Adaptation

  • An adversarial discriminator encourages features from observations and robot interaction to be domain-invariant; using learned features for the inverse model while RL uses raw observations performed best.The encoder minimizes the discriminator's ability to classify the feature domain, while the discriminator tries to maximize it.Source: §3.5 · p. 5

3.6 Joint Optimization

  • The method jointly optimizes domain-adaptation loss, inverse-model loss, and the selected RL algorithm's objective, with hyperparameters weighting the terms.The passage identifies the relative-importance constants as hyperparameters.Source: §3.6 · p. 5

4 Experiments

Experiments test whether RLV can learn from suboptimal observations, robotic manipulation demonstrations, and real-world human videos. All experiments use SAC as the underlying reinforcement-learning algorithm and report means and standard errors over five seeds.

4.1 Learning from Observations of sub-optimal policies

  • RLV consistently achieved higher rewards than ILPO, BCO, and SAC on Acrobot, with the strongest improvements from medium-quality observations.ILPO was more data-efficient but could not improve on suboptimal trajectories; SAC required more samples than RLV.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Robotic Tasks

  • RLV required 30% as many samples as SAC to solve State Pusher, while BCO and ILPO did not reliably solve it.The task uses continuous two-dimensional actions and a sparse reward.Source: §4.2 · p. 7
  • RLV required 3x fewer samples than SAC on Visual Pusher and Visual Door Opening, where BCO and ILPO failed to achieve non-zero rewards.On Visual Door Opening, three of five SAC seeds converged to the optimal policy, compared with all RLV seeds.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Learning from Real-World Human Videos

  • RLV matched SAC’s performance in both human-video tasks with only 50% of SAC’s samples, while RND learned more slowly and finished with lower performance.The tasks were pushing and drawer opening.Source: Figure 6 · p. 7, §4.3 · p. 8, Figure 7 · p. 8

4.4 Ablations over action and reward prediction

  • In Visual Pusher, estimated and ground-truth actions performed well, while zero actions ended at a suboptimal score; estimated, ground-truth, and zero rewards all reached the optimal solution.The authors suggest zero rewards worked because ground-truth rewards were mostly zero, and conclude RLV is fairly robust to reward-label choice.Source: §4.4 · p. 8

5 Conclusion

RLV combines online robot interaction with offline human observations, addressing missing actions and rewards and domain shift. The paper reports greater efficiency than standard RL and advantages over prior observation-based imitation approaches.

5 Conclusion

  • RLV learns significantly more efficiently than standard RL methods and outperforms prior observation-based imitation approaches on complex vision-based tasks.The paper reports that it can learn tasks with half the number of trials compared with standard RL.Source: §5 · p. 8
  • RLV combines online robotic interaction with offline human observations, addressing missing actions and rewards and domain shift through simple mechanisms.Source: §5 · p. 8
  • Future work targets more visually complex scenes, more natural and diverse human demonstrations, and more complex learned behaviors.The paper states these challenges must be overcome to leverage broad human observations for robotic learning.Source: §5 · p. 8

A Revisions from CoRL Version

The revision corrects the account of paired data and clarifies the network architecture used in the implemented algorithm.

A Revisions from CoRL Version

B Data and Qualitative Results on Supplementary Website

The supplementary website hosts dataset downloads and videos of qualitative results.

B Data and Qualitative Results on Supplementary Website

C COVID-19 Statement

The authors could not evaluate RLV on a real robot because of COVID-19. They argue that simulation results provide evidence for likely real-robot success and practical deployment.

C COVID-19 Statement

  • COVID-19 prevented evaluation of RLV on a real robot, leaving its real-robot performance untested.Source: C COVID-19 Statement · p. 12
  • The authors argue that human-to-robot domain shift on a real robot would likely be no larger than the shift between real human videos and simulated robot observations in their experiments.Source: C COVID-19 Statement · p. 12
  • They cite raw-visual performance without shaped rewards and learning within 75,000 steps or 750 episodes as evidence that real-robot data collection is practical.They also note that RLV builds on SAC, previously demonstrated on a real robot with raw pixel observations.Source: C COVID-19 Statement · p. 12

D Architecture Details and Hyperparameters

RLV uses deep neural networks with SAC, alongside reward-generation, observation-architecture, and human-observation hyperparameters tailored to the experiments. Human-observation training uses random image crops, while ILPO and BCO use discretized actions for robotic pushing.

D Architecture Details and Hyperparameters

  • RLV is instantiated with deep neural networks and uses SAC as its underlying RL algorithm; hyperparameters for its different experiments are presented in Tables 1–5.Source: D Architecture Details and… · p. 12
  • The feature-representation losses are most sensitive to Table 5's balance parameters; clarge is robust across environments, while csmall can be set to the random-step reward at each starting state.Network architecture hyperparameters have a range of viable values.Source: D Architecture Details and… · p. 12
  • For human observational data, SAC and RLV use random crops from images padded with four black pixels on each side.Crops retain the original image size.Source: D Architecture Details and… · p. 12
  • For robotic pushing, ILPO and BCO use a 49-bin discretization of the two-dimensional State Pusher action space.The methods are designed for discrete actions; the resolution was selected by hyperparameter sweep.Source: D Architecture Details and… · p. 13

E Human Dataset

The human dataset contains videos of puck pushing and drawer opening across varied objects, backgrounds, and lighting. Videos are aligned to task landmarks and cropped to 48x48 pixels; the section also provides example images and identifies the image-observation and human-observation hyperparameter tables.

E Human Dataset

  • The pushing dataset has 198 videos totaling 806 seconds across four backgrounds, two hands, two pucks, and varied lighting; drawer opening has 108 videos totaling 376 seconds across two drawers, one hand, and varied lighting.Source: E Human Dataset · p. 13
  • Videos are transformed to align the pushing goal square or drawer handle, then cropped to 48x48 pixels.Figures 9–12 show example sequences and post-processed images for pushing and drawer opening.Source: E Human Dataset · p. 13, Figure 9 · p. 14, Figure 10 · p. 14
  • Table 4 specifies image-observation architecture hyperparameters for visual pushing, door opening, and drawer opening; Table 5 specifies RLV hyperparameters for human demonstrations.Source: Table 4 · p. 16, Table 5 · p. 16

Related papers

Loading 2011.06507v2…