Source-linked AI summary
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Andrei A. Rusu, Mel Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, Raia Hadsell
TL;DR
Pixel-based deep reinforcement learning is too interaction-intensive for practical real-robot training, despite promising simulated performance. The paper uses progressive networks to transfer policies and representations from simulation to real robots, reporting reliable, fast transfer for pixel-to-action policies across robot manipulation tasks. The approach remains an initial demonstration whose real-domain exploration depends on preserving the prior policy when rewards are sparse.
Problem
Pixel-based deep reinforcement learning requires too much real-robot interaction for practical training, although its control capabilities have mainly been demonstrated in simulation.
Method
Progressive networks transfer simulation-trained robot policies through lateral connections, heterogeneous new columns, and output initialization that preserves the previous policy initially.
Results
The experiments demonstrate reliable, fast transfer for pixel-to-action reinforcement-learning policies from simulation to a real robot across manipulation tasks.
Takeaways & Limitations
Progressive networks provide a proof-of-concept route for learning real-robot manipulation from raw visual input using deep reinforcement learning and sparse rewards.
Takeaways & Limitations
The approach assumes real-domain rewards are sufficiently available, so the new column is initialized to preserve the prior policy and increase the likelihood of reward during exploration.
Abstract
from arXiv · showhide
Applying end-to-end learning to solve complex, interactive, pixel-driven control tasks on a robot is an unsolved problem. Deep Reinforcement Learning algorithms are too slow to achieve performance on a real robot, but their potential has been demonstrated in simulated environments. We propose using progressive networks to bridge the reality gap and transfer learned policies from simulation to the real world. The progressive net approach is a general framework that enables reuse of everything from low-level visual features to high-level policies for transfer to new tasks, enabling a compositional, yet simple, approach to building complex skills. We present an early demonstration of this approach with a number of experiments in the domain of robot manipulation that focus on bridging the reality gap. Unlike other proposed approaches, our real-world experiments demonstrate successful task learning from raw visual input on a fully actuated robot manipulator. Moreover, rather than relying on model-based trajectory optimisation, the task learning is accomplished using only deep reinforcement learning and sparse rewards.
1 Introduction
Pixel-based deep reinforcement learning has shown sophisticated control mainly in simulation, while real-robot learning remains difficult because training is data-hungry and robot interaction is constrained. The paper proposes progressive networks as a transfer-learning approach from simulation to real robots.
- Deep reinforcement learning has achieved sophisticated pixel-to-action control primarily in simulation rather than on physical robots.
- Real-robot learning is difficult because current pixel-based deep RL requires extensive interaction and research robots are relatively fragile.
- Progressive networks are used to bridge the simulation-to-reality gap through transfer learning.
- The approach transfers a simulation-trained actor-critic network to a real robot using lateral nonlinear connections into a new network column.The simulation policy uses a convolutional encoder, LSTM, and linear action outputs; the real-robot column is trained on a similar task.
- The initial findings report a dramatic learning speed-up on the real robot from the features and encoded policy learned in simulation.
2 Transfer Learning from Simulation to Real
Progressive networks transfer knowledge by freezing earlier columns and adding new columns connected through lateral pathways, allowing heterogeneous capacities and task-specific inputs. For sim-to-real robot learning, the architecture uses a large simulation column, narrow robot columns, and output initialization designed to preserve the prior policy initially.
- Progressive networks freeze a trained column and add a randomly initialized column for each new task, while connecting corresponding layers through lateral connections.
- Progressive networks allow arbitrary width per column or layer, accommodating different task difficulties and ensembles of independent networks.
- Each layer combines the current column’s preceding activation with lateral inputs from earlier columns through learned weight matrices and an element-wise nonlinearity.The intermediate nonlinearity used in the paper is f(x) = max(0, x).
- Unlike pretrain-and-finetune methods, progressive networks do not assume task overlap and can allocate different structures or inputs to new tasks.
- 2.2 Approach: For sim-to-real transfer, the simulation column has sufficient capacity and depth, whereas robot-trained columns use minimal capacity to encourage fast learning and limit parameter growth.The modified architecture permits columns with different capacities or structures and omits adapters at output layers for complementary task sequences.
- 2.2 Approach: The new robot column is initialized to match the previous policy by copying prior output weights and setting incoming weights from its last hidden layer to zero.Other parameters in the new column are randomly initialized, so training can still change the policy after initialization.
3 Related Literature
Prior work has explored domain adaptation and sim-to-real transfer, but evidence remains limited for end-to-end pixel-to-action control on real robots. Existing approaches include aligned-data methods, feature-distribution constraints, adversarial representations, and restricted task transfers.
- Few methods address simulation-to-reality transfer for interactive, rich-sensor robot domains using end-to-end pixel-to-action learning.
- Some domain-transfer methods augment target data with source data when aligned data exists, while others constrain feature distributions across domains.
- Domain-confusion losses and adversarial representations aim to reduce sensitivity to inter-domain variations, but these ideas had not yet been tested in control in the cited discussion.
- A simulation-to-reality method for robotic-arm pose estimation used aligned data and a triple loss but did not demonstrate efficiency for learning novel complex policies.
- Reported transfer successes primarily move from simpler task versions to more difficult full tasks, while direct neural-policy learning on real robots has used both low-dimensional state and visual input.
4 Experiments
The experiments evaluate progressive-network transfer from MuJoCo simulation to a real Jaco robot on sparse-reward visual manipulation tasks, then test transfer across task and environment changes. Simulation learning is substantially faster than real-robot learning, while progressive columns outperform scratch training and finetuning in the reported settings.
- Experimental setup: A3C trains policies for nine Jaco degrees of freedom from RGB observations, using velocity commands and a factored discrete-action policy with a shared value function.The networks use convolutional inputs followed by either a fully connected layer or an LSTM; simulation columns are wide and robot columns reduced-capacity.
- Experimental setup: The task randomizes arm starts and target locations, provides only a +1 reward within 10cm of the target, and terminates episodes after 50 steps or safety violations.Targets vary within a 40cm by 30cm area, requiring visual state inference and control over a high-dimensional state space.
- Transfer to the robot: The progressive second column reaches 34 points on the real robot, whereas finetuning does not reach the same score and randomly initialized scratch baselines receive no reward.Real-robot experiments train recurrent RGB-input models for approximately 60,000 steps, or four hours.
- Training in simulation: 50 million simulation steps—more than one million episodes—are needed for stable performance, and recurrent models achieve higher final mean scores than feedforward models.The LSTM model outperforms the feedforward model by an average of 3 points per episode, while narrow networks learn more slowly and perform worse.
- Training in simulation: 53 days of continuous real-robot training would be required to match the 24-hour simulation convergence time, motivating simulation-to-real transfer.Simulation benefits from fast rendering, multithreaded learning, continuous unattended training, and parallel hyperparameter experiments.
- Transfer across conditions: Across 300 simulated runs with color and perspective changes, progressive networks achieve higher final performance and greater stability than finetuning.The experiments vary random seeds, learning rates, and entropy costs to assess sensitivity to hyperparameter selection.
- Transfer to a dynamic task: Adding a third progressive column enables immediate transfer from the static reacher task to the dynamic conveyor task, unlike directly training the conveyor policy.Direct second-column training reaches full performance after 50,000 steps, while the third-column curriculum reaches full performance almost immediately.
5 Discussion
The experiments show that progressive networks support transfer learning across simulation and real-robot settings, including faster learning when a dynamic task is assigned to a later column.
- Progressive networks enabled successful transfer of a visually trained control skill from simulation to an agent training on a real robot.The approach used simulation to compare hyperparameters and architectures before transferring the skill.
- The experiments demonstrate reliable, fast transfer for pixel-to-action reinforcement-learning policies using progressive networks.The authors frame this as an initial set of experiments toward more efficient real-world deep reinforcement learning.
- A conveyor task learned as the third progressive-network column trained significantly faster than when learned as the second column.The third column used only proprioception, while the second combined vision and proprioception.