Source-linked AI summary
Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, Sergey Levine
TL;DR
Robotic reinforcement learning faces substantial data demands, motivating transfer across skills and robots. The paper decomposes neural policies into task-specific and robot-specific modules, then recombines them for unseen robot-task pairs. In simulation, this enables zero-shot generalization and faster learning across visual and non-visual tasks.
Problem
Deep reinforcement learning for robotic skills requires onerous data collection, while unstructured policies make transfer across tasks and robots difficult.
Method
The paper trains neural policies as task-specific modules shared across robots and robot-specific modules shared across tasks, then mixes them for new combinations.
Results
The composed modules enable zero-shot performance or significantly sped-up learning for unseen robot-task combinations in simulated visual and non-visual tasks.
Takeaways & Limitations
Modular policy networks transfer task information between robots and robot information between tasks, including perception, dynamics, and kinematics.
Takeaways & Limitations
The method requires different task-robot combinations to be trained simultaneously, potentially requiring multiple robots to learn at once.
Abstract
from arXiv · showhide
Reinforcement learning (RL) can automate a wide variety of robotic skills, but learning each new skill requires considerable real-world data collection and manual representation engineering to design policy classes or features. Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations. Transfer learning can mitigate this problem by enabling us to transfer information from one skill to another and even from one robot to another. We show that neural network policies can be decomposed into "task-specific" and "robot-specific" modules, where the task-specific modules are shared across robots, and the robot-specific modules are shared across all tasks on that robot. This allows for sharing task information, such as perception, between robots and sharing robot information, such as dynamics and kinematics, between tasks. We exploit this decomposition to train mix-and-match modules that can solve new robot-task combinations that were not seen during training. Using a novel neural network architecture, we demonstrate the effectiveness of our transfer method for enabling zero-shot generalization with a variety of robots and tasks in simulation for both visual and non-visual tasks.
I. INTRODUCTION
The paper targets inefficient transfer in robotic reinforcement learning by decomposing policies into task-specific and robot-specific modules. These modules can be recombined for unseen robot-task combinations, supporting zero-shot performance or faster learning.
- Deep reinforcement learning reduces manual representation engineering but requires onerous data collection, while learned policies lack clear structure for reuse.
- Robotic-skill information such as dynamics, perception, and task steps is partly decoupled between tasks and robots, creating opportunities for transfer.
- The proposed policy decomposition shares task-specific components across robots and robot-specific components across tasks.
- The modules can be trained on observed robot-task combinations and composed to enable zero-shot performance or significantly faster learning on unseen combinations.
- The work analyzes regularization for generalizable module interfaces and evaluates the approach on simulated visual and non-visual tasks.
II. RELATED WORK
Prior work established transfer and modular representations in several domains, but this paper focuses on recombining modules for robotic policies across task and robot variation. Its setting combines discrete robot and task choices into possible worlds.
- Transfer learning has been studied to reduce expensive on-policy data collection for large repertoires of robotic skills.
- Pretraining, domain adaptation, and modular networks address transfer in passive perception, but do not directly solve robotic policy learning across task-relevant differences and invariances.
- The experiments use guided policy search because it supports high-dimensional neural-network policies that can be decomposed into interconnected modules.
- The paper’s universe varies along robot morphology and task identity, with Fig. 1 illustrating 3DoF and 4DoF robots and drawer-opening and block-pushing tasks.
- Unlike prior module-recombination approaches described here, the proposed method targets direct zero-shot generalization for new robotic domains.
III. MODULAR POLICY NETWORKS
The paper formalizes transfer across discrete degrees of variation by treating each robot-task combination as a world. Modular policies exploit shared structure while preserving different controls and strategies required by robots and tasks.
- Problem setting: A universe consists of worlds formed by discrete degrees of variation, such as robot structure and task, producing four combinations for two robots and two tasks.
- Problem setting: A single policy for all worlds is non-optimal because different tasks require different strategies and robots with different joint counts require different controls.
- Transfer setting: Standard reinforcement learning treats each world separately, although worlds sharing a robot or another degree of variation can share commonalities in their optimal policies.
- Preliminaries: A policy maps observations to a distribution over motor controls, with observations potentially including images, encoder readings, motion-capture markers, or full state.
- Modularity: Observations can be divided into robot-specific intrinsic information and task-specific extrinsic information, including joint and sensor readings versus images and object locations.
- Modularity: The decomposition assumes costs separate into robot-dependent and task-dependent terms, with actions affecting the robot-dependent term.
B. Modularity
The policy is decomposed into robot-specific and task-specific modules, which are composed to transfer information across changing robot-task combinations. Module ordering reflects task planning followed by robot-dependent control, while more general decompositions can form a DAG.
- Module reuse: Policies for worlds sharing a degree of variation reuse the corresponding module while other degrees of variation change.For example, two tasks performed by the same 3 DoF arm share the arm-specific policy component.
- Module decomposition: Policies share a robot module across tasks and a task module across robots.The robot-specific function f_r and task-specific function g_k operate on corresponding observation components.
- Composition order: The composition order places task information before robot-dependent control because task identity affects planning and robot configuration affects control output.The task module output is a learned latent representation rather than a supervised semantic variable.
- Generalization: For multiple degrees of variation, modules may be arranged as a directed acyclic graph whose root outputs the action.Each module receives child-module outputs and observations for its own degree of variation.
- Zero-shot composition: An unseen robot-task combination can be formed by composing previously trained modules, provided each robot and task appeared separately during training.The robot and task need not have been trained together before.
- Generalization: Training more varied worlds encourages modules to become invariant to changes in other degrees of variation.Task modules must work with varied robot modules, while robot modules must work with varied task modules.
C. Architecture and Training
The architecture represents policy means as compositions of neural-network robot and task modules and trains tied module combinations jointly across selected worlds. Reinforcement-learning loss is aggregated across worlds, with gradients computed by standard backpropagation.
- Architecture: Each policy uses a Gaussian action distribution whose mean is a composition of robot-specific and task-specific neural-network modules.The covariance is learned but independent of observations.
- Training setup: Training worlds are selected so every module participates in at least one robot-task combination.The resulting policy grid ties weights between modules shared across worlds.
- Training setup: The tied policy networks are trained synchronously by collecting samples from each world and forwarding them through corresponding modules.The text notes that asynchronous training methods could also be explored.
- Optimization: The total reinforcement-learning loss is the sum of the individual losses from all training worlds.The specific loss and optimization procedure depends on the reinforcement-learning algorithm.
- Optimization: The experiments use guided policy search, where local policy solvers supervise neural-network regression onto generated training actions.A policy-gradient loss is described as an alternative.
- Optimization: Gradients through the composed modules are computed using standard neural-network backpropagation.The decomposition rewrites parameter gradients for the robot and task modules.
D. Regularization
Regularization is used to make module interfaces compact and less specialized to observed robot-task combinations. The figure illustrates weight-shared task and robot modules being recombined for an unseen world.
- Motivation: Overfitting can cause modules to specialize to particular training-world combinations instead of learning standardized interfaces.This is problematic when only a few robots and tasks are available.
- Bottleneck regularization: Limiting the first module’s hidden-unit output forces information to pass through a compact representation.The bottleneck reduces the capacity to partition and specialize separately for each training world.
- Figure 3: Fig. 3 shows 2 task modules and 2 robot modules forming policies for a 2-task, 2-robot universe.Modules with the same color share weights, and training-world compositions are trained end-to-end.
- Figure 3: For an unseen world, the appropriate previously trained modules are composed to support zero-shot performance.The figure depicts recombination rather than training a separate module for the new combination.
- Dropout: Dropout randomly zeros activations during each minibatch, encouraging redundancy rather than dependence on individual hidden units.This limits information flow between task and robot modules and reduces overspecialization.
IV. EXPERIMENTS
The experiments evaluate modular policy networks in simulated environments spanning multiple robots and tasks, including visual and non-visual policies. The method is compared with training a separate policy for each world.
- Evaluation design: The evaluation uses multiple robots and tasks across simulated environments with visual and non-visual policies.Experiments train on a subset of possible worlds and assess unseen-world learning.
- Evaluation design: Modular policy networks are evaluated for faster or zero-shot learning on unseen worlds.The baseline trains a separate policy network for each world instantiation.
A. Reinforcement Learning Algorithm
The method uses guided policy search to train decomposable neural policies, with convolutional and spatial-softmax layers forming the task module and later fully connected layers forming the robot module.
- Policy learning: Guided policy search combines trajectory optimization with supervised learning to train global neural-network policies.The experiments use its BADMM-based variant to stabilize learning.
- Module architecture: For visual tasks, two convolutional layers and spatial softmax form the task module, while later fully connected layers form the robot module.For non-visual tasks, convolutional layers are replaced with fully connected layers.
- Module architecture: The task module also receives the robot end-effector position, which is present across robots and made available in the earliest policy layers.
C. Reaching Colored Blocks in Simulation
The colored-block experiment tests whether modular policies can disentangle visual task information from robot-specific control and transfer to an unseen robot-task combination.
- Setup: The experiment uses three robots and colored-block reaching tasks, with each robot assigned its own robot module.The blocks are red, green, yellow, and black, and each task targets one color.
- Setup: Training uses 11 of 12 possible worlds, holding out the 4-link robot reaching for the black block.The held-out combination cannot be trivially transferred from the 3-link robots or other color-specific task modules.
- Results: The modular policy performs the held-out task well without additional training, while random and mismatched-task baselines have significant error.Table I measures average end-effector distance to the black block over the final five timesteps; zero is perfect.
- Results: The result demonstrates transfer of visual recognition capabilities across robots.
D. Object Manipulation
Object-manipulation experiments test transfer across robots and physically different tasks, showing that modular policies can accelerate learning and sometimes generalize zero-shot.
- Object manipulation: The experiment combines three robots with three target-coordinate tasks: horizontal drawer pulling, vertical drawer pushing, and block pushing.The robots have 3, 4, or 5 links and therefore different-dimensional state spaces.
- Object manipulation: Successful transfer requires sharing both task kinematics and robot dynamics across the held-out combinations.
- Drawer transfer: For the held-out 3-link horizontal-drawer task, modular initialization learns faster without shaping than learning from scratch with shaping.Learning from scratch without shaping makes no progress, whereas the incorrectly transferred vertical-drawer module is too different for direct transfer.
- Visually distinct tasks: In visually distinct manipulation tasks, the held-out 4-link block-pushing combination succeeds with zero-shot transfer and significantly outperforms random and mismatched combinations.The result is attributed to separating perception and kinematic goals from robot-specific feedback control.
V. DISCUSSION AND FUTURE WORK
Modular policy networks recombine robot- and task-specific modules to address unseen combinations, either directly or as learning initializations, but current training requires simultaneous task-robot learning.
- Discussion: Task-specific modules are robot-invariant and robot-specific modules are task-invariant, enabling recombination for unseen robot-task combinations.Modules are trained together end-to-end with standard reinforcement learning algorithms.
- Discussion: Previously untrained module combinations may generalize immediately or provide a strong initialization that speeds subsequent learning.
- Future work: The current method requires different task-robot combinations to be trained simultaneously.The authors identify sequential transfer and lifelong multi-robot learning as future directions.