PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement and Imitation Learning for Diverse Visuomotor Skills

Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, Nicolas Heess

arXiv:1802.09564v2cs.ROcs.AIcs.LG

TL;DR

Robotic visuomotor RL must handle difficult exploration, sparse rewards, complex dynamics, and limited real-world data. The paper combines model-free RL with demonstration-based imitation, simulation, and diversified training conditions to learn pixel-to-joint-velocity policies. The combined agent outperforms RL-only and imitation-only baselines across most tasks and achieves preliminary zero-shot transfer to a real robot.

  • Problem

    Robotic manipulation requires learning from partial visual observations in high-dimensional continuous settings, while real-world RL is constrained by sample complexity, safety, and difficult reward design.

  • Method

    The method combines model-free deep RL with demonstrations, using a hybrid imitation-and-task reward, demonstration-state curricula, simulation training, and diversified visual and dynamics conditions.

  • Results

    The full model achieves the highest returns on all six tasks except for parity on block lifting, while zero-shot real-robot success rates reach 64% for lifting and 35% for stacking.

  • Takeaways & Limitations

    Combining reinforcement and imitation learning improves training for challenging pixel-based manipulation, using only 30 demonstration trajectories per task in the experiments.

Abstract

from arXiv · show

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in https://youtu.be/EDl8SQUNjj0

I. INTRODUCTION

The paper addresses the difficulty of learning robust visuomotor manipulation policies under partial observations, complex dynamics, sparse rewards, and limited real-robot data. It combines demonstrations, simulation, reinforcement and imitation learning, and diverse training conditions to learn across six tasks and support sim2real transfer.

  • Robotic RL must map noisy, partial multimodal observations to coordinated control across many degrees of freedom and varying contact-rich environments.
  • High-dimensional continuous actions, difficult reward design, safety concerns, and costly real-world interaction make exploration and direct hardware training challenging.
  • The method combines imitation learning and reinforcement learning, using demonstrations for a hybrid task-and-imitation reward and a curriculum of training states.
  • Training uses simulation, privileged and task-specific information, and diversified visual, geometric, and dynamics conditions to improve generalization and sim2real transfer.
  • The same model and algorithm, with small task-specific training modifications, learn visuomotor controllers for six diverse robot-arm manipulation tasks.

II. RELATED WORK

Prior work developed robot-control, simulation-transfer, large-scale data, and demonstration-based approaches, but these efforts generally addressed subsets of the problem. This paper integrates complementary techniques into an end-to-end visuomotor method that does not require demonstrator actions.

  • Deep RL for robotics includes guided policy search, value-based methods, and policy-gradient methods, but sample inefficiency limits direct training on hardware.
  • Prior visuomotor studies used real-robot pretraining, large self-supervised grasp datasets, or simulation transfer with visual and dynamics randomization.
  • Demonstrations have been used to initialize policies, design costs, guide exploration, and augment training data through tracking or inverse-RL objectives.
  • Concurrent manipulation studies used demonstrations for exploration or behavioral cloning, often with low-level state or action assumptions.
  • This method integrates multiple techniques into one coherent approach and learns end-to-end visuomotor policies without relying on demonstrator actions.

III. MODEL

The model learns an end-to-end visuomotor policy from RGB observations and proprioceptive features, while GAIL supplies an imitation-learning framework for using demonstrations efficiently. The policy is trained with policy-gradient methods and stabilized trust-region techniques.

  • The policy receives RGB camera observations and proprioceptive features describing joint positions and angular velocities, then produces robot actions for transfer to real hardware.
  • Imitation learning trains a behavior policy by mimicking demonstrations, while behavior cloning maximizes the likelihood of demonstrated actions.
  • GAIL combines a policy network with a discriminator and uses agent-environment interaction to learn efficiently from demonstrations.
  • The GAIL objective encourages the learned policy to have an occupancy measure close to that of the expert policy.
  • The policy is trained with policy gradients using a clipped GAIL reward, while trust-region methods stabilize continuous-domain training.

B. Reinforcement and Imitation Learning Model

The model combines task and imitation rewards so demonstrations guide exploration while reinforcement learning preserves task optimization. Balanced contributions can solve tasks that either component alone cannot.

  • 1) Hybrid IL/RL Reward:: Sparse, piecewise-constant task rewards represent transitions between stages such as reaching, lifting, and stacking.
  • 1) Hybrid IL/RL Reward:: The hybrid reward is r(s_t, a_t) = λr_gail(s_t, a_t)+(1−λ)r_task(s_t, a_t), with λ ∈ [0, 1].
  • 1) Hybrid IL/RL Reward:: Imitation encourages trajectories closer to demonstrations, while the task reward encourages high task returns; λ = 0 or 1 recovers standard RL or GAIL.

2) Leveraging Physical States in Simulation:

The method uses privileged physical-state information available in simulation to improve training while keeping the learned visuomotor policy image-based. It combines demonstration-state curricula, state-based value learning, object-centric discrimination, and auxiliary state prediction.

  • Demonstration as a curriculum: Demonstration states form a curriculum that initializes episodes at clusters corresponding to different task stages.For pouring, clusters represent reaching the mug, grasping the mug, and pouring; episodes otherwise begin from random states with probability ϵ.
  • Learning value functions from states: The value function uses low-level physical states rather than pixels to estimate policy-gradient advantages.A smaller multilayer perceptron processes object and robot-arm positions and velocities, which the authors report stabilizes training and reduces performance oscillation.
  • Object-centric discriminator: An object-centric discriminator receives task-specific physical features, such as absolute and relative object positions, for adversarial imitation learning.Robot-arm states are excluded because they can focus the discriminator on irrelevant controller behavior.
  • State prediction auxiliary tasks: A state-prediction auxiliary layer regresses object locations from camera observations using an ℓ2 loss.The layer is added on top of the CNN module and uses ground-truth object locations during training.

3) Sim2Real Policy Transfer:

The policies are transferred from simulation to a real Kinova Jaco arm using approximate visual and dynamics matching plus domain randomization. They map raw pixels directly to joint velocities without object-position inputs or intermediate goals.

  • Transfer setup: The sim2real experiments use a real-world Kinova Jaco robot arm with simulation manually adjusted to roughly match laboratory appearance and dynamics.A Kinect camera was visually calibrated to match the simulated camera, and simulation dynamics parameters were manually adjusted to match the real arm.
  • Transfer setup: Figure 3 compares RGB pixel observations for six simulated tasks with real-robot views of block lifting and stacking.The caption identifies the left column as simulated task images and the right column as the two real-robot block tasks.
  • Transfer setup: Domain randomization of camera position and orientation supports policy transfer without professional calibration equipment.The policies do not rely on object-position information or intermediate goals.
  • Robot interface: The visuomotor policy controls the arm through 9-dimensional continuous joint-velocity commands at 20Hz.The robot has six arm joints and three actuated fingers, while visual observations come from a tabletop scene camera.
  • Training diversity: Training uses diverse procedurally generated objects with randomized physical properties, including dimensions, color, mass, and friction.Demonstrations comprise 30 episodes per task collected with a SpaceNavigator 3D motion controller.

B. Robot Arm Manipulation Tasks

The experiments cover six manipulation tasks spanning simple block interactions, multi-stage clearing and pouring, and abstract category-based order fulfillment. The tasks vary in dexterity, horizon, object diversity, and generalization demands.

  • Task suite: Figure 3 presents six manipulation tasks in simulation and real-world block lifting and stacking setups.The simulated and real views show visual discrepancies, while the first three tasks use simple colored blocks.
  • Block tasks: Block lifting evaluates robustness to randomized arm dynamics, lighting, camera pose, background color, and block properties.Each episode samples these factors uniformly from preset ranges and begins with a new configuration.
  • Block tasks: Block stacking requires placing one block on top of another and is included in the sim2real experiments.Together with block lifting, it provides a real-robot transfer setting.
  • Multi-stage tasks: Clearing the table with blocks requires lifting two blocks, potentially by stacking them first, creating a longer and more dexterous exploration problem.The task introduces greater exploration difficulty than shorter manipulation tasks.
  • Generalization tasks: The procedurally generated-shape tasks test generalization across object variations in long and complex tasks.These tasks are difficult to recreate in real environments because they use diverse procedurally generated 3D shapes.
  • Generalization tasks: Pouring liquid uses small spheres as liquid and requires transferring them between containers, with dexterity posing a particular challenge.Even humans struggled to demonstrate this task extensively with the 3D motion controller.
  • Generalization tasks: Order fulfillment requires categorizing variable numbers of planes and cars, grasping diverse shapes, and placing them into separate colored boxes.The task therefore requires abstract-level generalization and handling variable episode lengths.

C. Quantitative Evaluation

Across six manipulation tasks, the full reinforcement-and-imitation model performs best overall, while ablations show that its hybrid reward and selected state-based components are central to stacking performance.

  • The full model solves all six tasks with only occasional failures using one policy network, one algorithm, and fixed hyperparameters.
  • A balanced hybrid reward uses λ = 0.5, compared with λ = 0 for pure RL and λ = 1 for pure GAIL.The baselines otherwise use the same setup as the full model.
  • The full model achieves the highest returns in all six tasks, substantially outperforming reinforcement-only and imitation-only baselines in five tasks.Block lifting is the exception, where reinforcement learning and the full model perform similarly.
  • Demonstration-state curricula substantially improve reinforcement-learning efficiency by exposing the agent to states from different stages of each task.Starting from random states produces very slow learning progress.
  • The trained agents exceed the imperfect human demonstrations, including for the particularly difficult pouring task.
  • In stacking ablations, hybrid RL/IL reward, state-based value learning, and object-centred discriminator features are integral, whereas recurrent and auxiliary components are optional.The ablations separate into agents that stack, with average returns above 400, and agents that only lift, with returns between 200 and 300.

D. Sim2Real Policy Transfer Results

Simulation-trained policies achieve preliminary zero-shot transfer to a real Jaco arm, with stronger performance for lifting than stacking and reliable reaching behavior. The remaining reality gap limits transfer performance.

  • Zero-shot transfer succeeds at 64% for lifting over 25 trials and 35% for stacking over 20 trials on a real Jaco arm.The trials use repeatable object configurations and no additional training.
  • Stacking trajectories contain successful lifting behavior in 80% of trials and successful reaching behavior in 100% of trials.
  • The method demonstrates reaching, lifting, and stacking behaviors beyond the closest cited sim2real work, which demonstrated only reaching.The cited comparison reports 100% reaching success for this method.
  • A sizable reality gap constrains zero-shot performance because real foam blocks deform and bounce unpredictably, while camera-rendering discrepancies affect neural policies.The simulation uses rigid blocks, unlike the real-world foam blocks.

V. DISCUSSION

The paper presents a general model-free deep RL approach that combines demonstrations, simulation, and training-time privileged information to learn vision-based manipulation policies. It improves learning of difficult tasks and supports initial zero-shot sim2real deployment, while a remaining simulation-to-reality gap degrades real-robot performance.

  • Contribution: The method learns end-to-end manipulation policies from RGB images using joint velocity control, while requiring only a small number of demonstration trajectories.The experiments use 30 demonstrations per task, collected by teleoperation in less than thirty minutes per task.
  • Contribution: Training combines generative adversarial imitation learning with model-free reinforcement learning to support difficult-task learning and robust generalization.The approach also uses simulation and training-time privileged state information to facilitate learning.
  • Generalization: Policies are trained for significant variations in task conditions using parameterized procedural objects and randomized system dynamics.The stated goal is robustness across visual appearance, system dynamics, and task-related variation.
  • Simulation and deployment: Simulation avoids practical robotics constraints while allowing state information to support reward specification and stabilize visuomotor policy training.The resulting deployed policies rely only on vision and arm proprioception, despite privileged information used during training.
  • Simulation and deployment: A sizable simulation-to-real hardware domain gap remains, with visual, dynamic, and physical discrepancies causing performance degradation during transfer.Nevertheless, the real-world experiments demonstrate initial zero-shot sim2real success for pixel-to-joint-velocity policies.

APPENDIX A EXPERIMENT DETAILS

The experiments use RGB and proprioceptive observations, distributed simulated training, domain randomization, and action-drop fine-tuning to improve robustness. Tables report real-robot success rates across starting positions, while Fig. 6 illustrates randomized visual conditions.

  • Observations: The policy receives a 64 × 64 × 3 RGB image and a 24-dimensional proprioceptive feature describing arm and finger positions and velocities.Finger velocities are excluded because of noisy real-robot sensory readings.
  • Training: Policies are trained with distributed PPO using synchronous updates from 256 CPU workers, with task-dependent episode lengths up to 1000 time steps.Each worker completes an entire episode before parameter updates are computed.
  • Evaluation: Table I reports block-lifting success rates by starting position, while Table II reports block-stacking success rates by support-block initialization.Fig. 6 shows representative diversity from randomized colors, lighting, and backgrounds.
  • Robustness: Sim2real preparation lowers observation rates, adds sensor noise, and randomizes visual and physical conditions during training.Pixel observations are sampled at 5Hz while the controller runs at 20Hz; proprioceptive features are sampled at 10Hz.
  • Action Dropping: Action dropping addresses variable real-robot execution delays by fine-tuning the trained agents in simulation.The agents are fine-tuned for a further 2 million iterations on block lifting and block stacking.
  • Action Dropping: 64% of real-robot block-lifting trials succeed after action-drop fine-tuning, compared with 48% without action dropping.The comparison is made on the block lifting task.

APPENDIX C TASK DETAILS

The task suite represents progress through staged, sparse-reward objectives and uses state-defined stages to initialize training episodes as a curriculum. Tasks vary in duration, number of stages, objects, and auxiliary object-coordinate predictions.

  • Curriculum: Training episodes are initialized from clusters of demonstration states grouped by task-progress stages, forming a curriculum.For block stacking, the stages are reaching, lifting, and stacking.
  • Task details: Block lifting lasts 100 time steps and includes initial, reaching, and lifting stages, with an auxiliary task predicting the block’s 3D coordinates.Its object-centric feature is the relative gripper-to-block position.
  • Task details: Block stacking lasts 500 time steps and uses stages for reaching, lifting, and stacking an orange block onto a pink block.The auxiliary task predicts both blocks’ 3D coordinates.
  • Task details: Clearing-table tasks last 1000 or 500 time steps and use five stages ending in lifting both blocks or lifting the box, respectively.Their auxiliary tasks predict the 3D coordinates of the involved objects.
  • Task details: Pouring lasts 500 time steps and rewards the number of spheres transferred, while order fulfillment lasts 1000 time steps with one to four objects and rewards correctly placed toys.Object-centric features represent the relevant objects and their gripper-relative positions.
Loading 1802.09564v2…