PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymmetric Actor Critic for Image-Based Robot Learning

Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel

arXiv:1710.06542v1cs.ROcs.AIcs.LG

TL;DR

Robotic RL needs scalable, safe learning despite the mismatch between simulator full states and real-world partial observations. The paper trains image-based actors with full-state critics, then combines asymmetric training with domain randomization. It reports improved simulated performance and real-robot transfer without real-world training data.

  • Problem

    Robot RL is expensive and potentially dangerous on physical systems, while simulation provides full state information that is often unavailable in the real world.

  • Method

    The paper trains an actor-critic algorithm with full states for the critic and rendered RGBD observations for the actor, combined with domain randomization.

  • Results

    Asymmetric inputs improve performance across simulated tasks and support transfer of complex manipulation policies to real robots without real-world training data.

  • Takeaways & Limitations

    Simulator full-state access can be used during training while deploying visual policies, including for real-robot picking, pushing, and block-moving tasks.

Abstract

from arXiv · show

Deep reinforcement learning (RL) has proven a powerful technique in many sequential decision making domains. However, Robotics poses many challenges for RL, most notably training on a physical system can be expensive and dangerous, which has sparked significant interest in learning control policies using a physics simulator. While several recent works have shown promising results in transferring policies trained in simulation to the real world, they often do not fully utilize the advantage of working with a simulator. In this work, we exploit the full state observability in the simulator to train better policies which take as input only partial observations (RGBD images). We do this by employing an actor-critic training algorithm in which the critic is trained on full states while the actor (or policy) gets rendered images as input. We show experimentally on a range of simulated tasks that using these asymmetric inputs significantly improves performance. Finally, we combine this method with domain randomization and show real robot experiments for several tasks like picking, pushing, and moving a block. We achieve this simulation to real world transfer without training on any real world data.

I. INTRODUCTION

Robotic RL must balance expensive, unsafe physical exploration with limited observability when transferring policies from simulation. The paper trains image-based actors with full-state critics and combines this asymmetric design with domain randomization for real-world transfer.

  • I. INTRODUCTION: Physical robot learning is constrained by scalability and safety, motivating simulation-to-real training but introducing an observability challenge.Robots are slow and expensive, and exploration can damage the robot or its environment.
  • I. INTRODUCTION: Simulators expose full system state, whereas real robots often provide only partial visual observations.Full-state prediction from images can be infeasible and may suffer compounding errors.
  • I. INTRODUCTION: The proposed policy uses only RGBD images while training exploits simulator access to the full state.This addresses the tension between difficult full-state prediction and image policies hindered by dimensionality and partial observability.
  • I. INTRODUCTION: Asymmetric actor-critic training gives the critic full states and the actor visual partial observations, allowing the critic to learn state-action values faster.The resulting actor updates are better informed during training.
  • I. INTRODUCTION: Domain randomization is combined with asymmetric training to transfer simulator-learned visual policies without real-world training data.The paper discusses complex simulated and real-robot tasks, including picking, pushing, and moving a block.

B. Transfer from simulation to the real world

Simulation-to-real transfer remains difficult because simulators differ from physical environments. The paper emphasizes domain randomization, bottlenecks, and asymmetric actor-critic training for complex manipulation transfer.

  • B. Transfer from simulation to the real world: Reality-gap methods that match simulator physics to reality face a difficult system-identification problem and have had limited success.This approach attempts to make the simulator as close to the real world as possible.
  • B. Transfer from simulation to the real world: Domain adaptation and simulator fine-tuning can fail when simulation differs substantially from reality, limiting many methods to simple behaviours.Fine-tuning may then be no easier than training from scratch.
  • B. Transfer from simulation to the real world: Bottlenecks have supported domain adaptation for reaching, and this work extends their use to more complex fine manipulation tasks.The passage frames bottlenecks as a way to speed training and support adaptation.
  • B. Transfer from simulation to the real world: Domain randomization trains on randomized scene renderings to learn robust policies for transfer.The paper states that this approach is extended to complex behaviours when coupled with asymmetric actor critic.
  • B. Transfer from simulation to the real world: The real-robot experiments cover picking, pushing, and moving a block, with emphasis on fine manipulation behaviour.Picking includes moving to, grasping, and relocating an object.

III. BACKGROUND

The paper formulates continuous-space partially observable control and reviews actor-critic learning, including DDPG, replay-buffer training, Bellman-error critic updates, and policy-gradient actor updates.

  • III. BACKGROUND: The robotic problem is modeled as a continuous-space Markov Decision Process with states, partial observations, actions, transitions, rewards, discounting, and initial-state distribution.Partial observations correspond to states in the state space.
  • III. BACKGROUND: Episodes sample an initial state, select actions from a deterministic policy, receive rewards, transition through dynamics, and maximize expected discounted return.In partial observability, actions depend on the observation corresponding to the full state.
  • III. BACKGROUND: DDPG is an actor-critic algorithm for learning deterministic continuous action policies using separate actor and Q-function critic networks.The actor maps states to actions, while the critic approximates the Q-function over states and actions.
  • III. BACKGROUND: DDPG stores transitions in a replay buffer, minimizes Bellman error for the critic, and optimizes the actor through its Q-value objective.Training targets use reward plus discounted next-state value, while separate slowly changing target networks stabilize training.

C. Multigoal RL

For multigoal RL, the paper combines goal-conditioned policies, sparse rewards, hindsight experience replay, and asymmetric inputs so actors use rendered observations while critics use full states and goals.

  • C. Multigoal RL: Universal policies accept a goal as an additional input and can be trained with arbitrary reinforcement-learning algorithms.The policy is represented as a function of state and goal.
  • C. Multigoal RL: Sparse rewards provide positive feedback when the distance between the current state and goal is below a threshold.For picking and placing, reward is given only when the object is sufficiently near its desired location.
  • C. Multigoal RL: Hindsight Experience Replay adds achieved later states as alternative goals in the replay buffer, improving learning of universal policies with sparse rewards.This is valid because pursued goals do not influence environment dynamics under off-policy learning.
  • C. Multigoal RL: The method trains the critic on full states and the actor on partial observations, using images rendered from an external camera.In multigoal settings, the critic additionally receives full goal states while the actor receives partial goal observations.
  • C. Multigoal RL: The algorithm uses DDPG with an initialized replay buffer, rendered goals and observations, episodic transitions, minibatch updates, and hindsight experiences.The replay buffer stores states, observations, actions, rewards, successor states and observations, goals, and rendered goal observations.
  • C. Multigoal RL: Asymmetric inputs significantly improve performance and enable transfer of more complex manipulation behaviours to real robots.The actor is optimized from observation-based tuples while the critic uses state-based tuples.

B. Improvements with bottlenecks

Bottlenecks constrain an intermediate actor layer to predict the full state, using its smaller dimension to improve training efficiency.

  • B. Improvements with bottlenecks: The bottleneck constrains an intermediary actor-network layer to predict the full state.Because the full state is often lower-dimensional than other network layers, this state-predictive layer is called a bottleneck layer.

C. Randomization for transfer

Domain randomization transfers policies from rendered simulator images to real-world images by varying visual scene elements during training.

  • C. Randomization for transfer: Randomizing visual elements during rendering helps policies generalize to real-world observation errors.The stated goal is for policies to focus on important observation aspects despite differences between simulated and real images.
  • C. Randomization for transfer: The paper randomizes textures, lighting, camera location, and depth during training.Textures include random RGB values, gradients, and checker patterns; camera parameters and depth also receive randomized perturbations.
  • C. Randomization for transfer: Renderer randomization includes textures, lighting, and camera position to enable simulator-to-real transfer.

V. RESULTS

The evaluation section introduces experiments across simulated and real robot environments, comparing the method with baselines and examining training improvements.

  • V. RESULTS: Experiments span simulated and real robot environments, including baseline comparisons and analyses of training improvements.The section proceeds from environment descriptions to method comparisons, training analysis, and real robot experiments.

A. Environments

The paper evaluates multi-goal policies in three simulated environments and three real-robot manipulation tasks using visual observations and specified goal images.

  • A. Environments: Particle and Reacher are 2D MuJoCo environments, while Fetch Pick is a 3D simulated Fetch-robot task.Particle moves a point to a location; Reacher moves a two-link arm’s end-effector; Fetch Pick moves a block to an airborne target.
  • A. Environments: The simulated tasks use RGB or RGBD camera images, with state spaces containing system positions and velocities.Particle and Reacher use 100 × 100 RGB images; Fetch Pick uses 100 × 100 RGBD images and a 4D action space.
  • A. Environments: Multi-goal policies receive both the current observation and the desired episode goal.Goal observations represent the particle, arm target, or object at its desired location.
  • A. Environments: Real-robot evaluation uses a 7-DOF Fetch arm with an Intel RealSense R200 providing aligned RGBD images.Nearest-neighbour hole filling is used because real depth images can contain holes.
  • A. Environments: Real tasks are Pick, Forward Push, and Block Move, with goals specified by an image of the box at its target location.The actor receives a physical RGBD observation and a simulated goal observation for evaluation consistency.

C. Does asymmetric inputs to actor critic help?

Asymmetric inputs improve actor-critic performance over symmetric inputs, while asymmetric HER outperforms expert imitation after its initial learning advantage fades.

  • Asymmetric DDPG and HER perform much better than their symmetric-input counterparts on Particle.
  • On Fetch Pick, asymmetric HER significantly outperforms symmetric HER, highlighting HER’s importance for this harder sparse-reward task.
  • The BC baseline’s learning curves exclude the iterations used to train the expert policy.
  • DAgger initially learns faster from expert supervision but saturates below asymmetric HER across environments after many rollouts.The expert policy is trained on full states, whereas the cloned policy receives partial visual observations.

E. Can we speed up training?

Bottleneck-based auxiliary prediction improves training, while asymmetric HER with domain randomization transfers successfully to real-robot tasks and improves robustness to scene variation.

  • Bottlenecks improve the stability and speed of training by adding an L2 loss that predicts the full state from partial observations.
  • Asymmetric HER with domain randomization succeeds on Pick, Forward Push, and Block Move across five runs with different block initializations and goals.The learned policies also exhibit push-grasping and re-grasping behaviours.
  • Without randomization, policies perform perfectly in simulation but fail on the real robot; removing viewpoint randomization also severely harms performance except on Block Move.
  • Observation randomization makes a Pick policy trained on one red block work in the presence of distractor blocks.
  • Successive frames show asymmetric HER policies performing three complex real-robot tasks after simulation-to-real transfer.

H. Implementation Details

The implementation uses asymmetric actor-critic networks, with a full-state critic and image-based actor, trained through parallel DDPG rollouts and replay-buffer optimization. The setup also specifies exploration noise and target-network updates, while the reported evaluation finds improvements over actor-critic baselines and DAgger.

  • Network architecture: The critic is a three-layer fully connected network with 512-unit ReLU hidden layers, while the actor combines CNN processing of current and goal observations with fully connected layers.The critic receives state, goal state, and action; the actor processes both current and goal images before concatenation.
  • Optimization: Each DDPG iteration samples 16 parallel rollouts and performs 40 optimization steps using minibatches of 128 from a replay buffer containing 105 transitions.Target actor and critic networks are updated each iteration using Polyak averaging of 0.98.
  • Exploration: Exploration selects a uniformly random valid action with probability 20%, while the actor output receives coordinate-independent Normal noise during the remaining 80%.The Normal noise standard deviation equals 5% of the action range.
  • Evaluation: The method reports significant improvements over standard actor-critic baselines and performance superior to DAgger, despite using no expert demonstrations.Combined with domain randomization, it learns visual policies in simulation that work on a real robot without real-world training data.
Loading 1710.06542v1…