PaperlayerView exact revision on arXiv

Source-linked AI summary

One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control

Wenlong Huang, Igor Mordatch, Deepak Pathak

arXiv:2007.04976v1cs.LGcs.CVstat.ML

TL;DR

The paper investigates whether a single controller can generalize across robot morphologies that differ in structure and state/action dimensionality. It proposes Shared Modular Policies, reusing locally sensing modules at each actuator with message passing, and reports coordinated locomotion across diverse agents, including unseen variants.

  • Problem

    Robot-control policies are typically tailored and tuned to individual morphologies, leaving the possibility of general pre-trained controllers unresolved.

  • Method

    Shared Modular Policies reinstantiate one reusable module per actuator, using local sensor inputs and learned messages between neighboring limbs.

  • Results

    A single SMP controls locomotion for several planar morphologies, generalizes to unseen variants, and matches corresponding individually trained oracle performance.

  • Takeaways & Limitations

    Locally acting and sensing modules can produce globally coordinated locomotion across a wide variety of agents simultaneously.

  • Takeaways & Limitations

    The communication design can struggle when a common message is sent to all child limbs, creating body-level coordination issues in some cases.

Abstract

from arXiv · show

Reinforcement learning is typically concerned with learning control policies tailored to a particular agent. We investigate whether there exists a single global policy that can generalize to control a wide variety of agent morphologies -- ones in which even dimensionality of state and action spaces changes. We propose to express this global policy as a collection of identical modular neural networks, dubbed as Shared Modular Policies (SMP), that correspond to each of the agent's actuators. Every module is only responsible for controlling its corresponding actuator and receives information from only its local sensors. In addition, messages are passed between modules, propagating information between distant modules. We show that a single modular policy can successfully generate locomotion behaviors for several planar agents with different skeletal structures such as monopod hoppers, quadrupeds, bipeds, and generalize to variants not seen during training -- a process that would normally require training and manual hyperparameter tuning for each morphology. We observe that a wide variety of drastically diverse locomotion styles across morphologies as well as centralized coordination emerges via message passing between decentralized modules purely from the reinforcement learning objective. Videos and code at https://huangwl18.github.io/modular-rl/

1. Introduction

The paper asks whether one controller can generalize across diverse robot morphologies, rather than requiring a separately tuned policy for each agent. It introduces Shared Modular Policies, which reuse locally sensing, communicating modules across actuators and achieve coordinated locomotion across morphologies.

  • Motivation: Policies are typically trained from scratch for one robot morphology, limiting their use as general pre-trained control priors.The paper frames differing morphology and changing state/action dimensionality as central obstacles to reusable robot-control policies.
  • Motivation: Centralized policies jointly control all actuators, missing opportunities for modularity and reuse during training and execution.The paper asks whether artificial policies can generalize across agents while exploiting these properties.
  • Approach: Shared Modular Policies re-instantiate one reusable module at every actuator, with local sensing and learned messages exchanged between neighboring limbs.This decentralized-but-communicating arrangement is intended to support globally coherent behavior.
  • Results: SMP uses standard policy-gradient reinforcement learning and generalizes to variants not seen during training.The unified policy must perform across all morphologies rather than being specialized to one agent.
  • Results: The resulting policies control several planar agents simultaneously while matching the performance of corresponding individually trained oracle methods.The paper also analyzes how centralized coordination emerges from decentralized components.

2. Learning General-Purpose Controllers

The method represents each agent as a limb graph and applies one shared modular policy independently to every actuator. Local observations produce local torques, while joint reinforcement learning and communication support control across diverse morphologies.

  • Learning General-Purpose Controllers: The shared controller is trained across multiple agents and can generalize zero-shot to held-out morphologies without further fine-tuning.The objective is a single controller for agents with different physical structures and limb counts.
  • Representing Agent Morphologies as Graphs: Each agent morphology is represented as a graph whose nodes are limbs and whose edges connect limbs through joints.The method description assumes a connected acyclic tree with one limb designated as the root, while noting cycles can be incorporated.
  • Sensorimotor Modules: The same module parameters are reused across every limb of every agent, with each instance mapping local sensory state to its actuator’s torque.Local limb state includes quantities such as position, velocity, and rotation.
  • Modular Policy Optimization: After combined limb actions execute, the environment returns per-limb next states and one overall reward for the morphology.The shared parameters are optimized against this joint reward rather than separate limb rewards.
  • Modular Policy Optimization: The architecture can be trained with standard continuous-control reinforcement learning, including an actor-critic deterministic policy-gradient implementation using TD3.The authors state that most cross-morphology ability stems from controller modularity rather than extensive algorithmic changes.
  • Modular Communication: Independent modular policies condition actions only on local limb states, whereas communication is needed to coordinate diverse locomotion gaits across morphologies.The paper identifies the absence of a common gait as a major challenge for joint training.

3. Modular Communication

SMP uses identical local modules connected by learned messages to coordinate locomotion across diverse morphologies. Both-way communication supports emergent centralization by aggregating information upward and distributing actions downward.

  • Communication via Messages: Independent limb modules receive local states and learned messages from neighboring limbs to produce coordinated actions.Messages travel along the edges of the morphology graph, which represents limbs as nodes and joints as connections.
  • Message Passing Schemes: Bottom-up and top-down schemes pass messages in opposite directions, while both-way passing combines both directions and can produce centralized coordination.The upward pass aggregates information toward the root; the downward pass generates actions and messages for children.
  • Bottom-Up Message Passing: The aggregator f(.) collects child messages and combines them into a fixed-dimensional output for bottom-up message passing.Examples include element-wise sum, average, and maximum operators.
  • Top-down Message Passing: Passing identical messages to children can prevent symmetry breaking when branches, such as left and right legs, differ.The implementation can instead allow a parent to send different messages to each child.
  • Both-way Message Passing: Emergence of Centralization: The upward pass outputs messages only, whereas the downward pass produces final actions and messages to children.Both passes proceed sequentially through the tree, from leaves to root and then root to leaves.
  • Generalization to Cycles: For cyclic morphologies, repeated both-way message passing can continue until messages converge.This extends the tree-based procedure using an approach analogous to loopy belief propagation.

4. Experiment Setup

The experiments evaluate shared controllers on modified Gym MuJoCo locomotion agents and systematically vary their morphologies. Training uses multiple variants, with held-out variants for testing, while local limb states describe each actuator’s physical context.

  • Evaluation Protocol: Each experiment runs environments in parallel with the shared controller, uses four seeds, and reports mean and standard error over 1,000-step episode rewards.An environment reward is the sum of instantaneous rewards across the episode.
  • Environment and Agents: Experiments use Walker2D-v2, Humanoid-v2, Hopper-v2, and HalfCheetah-v2, with the humanoid constrained to a 2D plane.The environments are selected from Gym MuJoCo to study general-purpose locomotion across agents.
  • Environment and Agents: The study constructs collections containing 12 walker, 8 humanoid, and 15 cheetah variants, plus cross-category walker-hopper collections.Variants are combined into several multi-task environment sets.
  • Environment and Agents: Variants are generated by retaining connected torso-containing subsets of each agent’s tree structure and excluding structurally infeasible locomotion configurations.The same procedure can be implemented by successively removing leaf nodes and enumerating combinations.
  • Implementation: Shared modules across more than 20 agents use two four-layer fully connected networks with 32-dimensional message vectors.One network handles bottom-up messaging and the other handles top-down messaging, using TD3 for reinforcement learning.

5. Results and Ablations

The experiments compare SMP with multi-task and message-passing baselines, test zero-shot morphology generalization, and examine robustness to root-node choice. Both-way message passing performs best across diverse morphologies and generalizes to held-out variants without fine-tuning.

  • Evaluation setup: The evaluation compares SMP against a monolithic multi-task baseline, multiple message-passing schemes, zero-shot held-out morphologies, and alternative root-node choices.The experiments proceed through baseline comparison, message-passing analysis, unseen-morphology testing, and root-node robustness.
  • Multi-Task RL Baseline: The monolithic multi-task baseline fails to perform well across environments, whereas SMP with both-way message passing models diverse gaits across drastically different agents.The baseline is trained jointly across environments using padded state and action spaces.
  • Role of Message Passing: Both-way message passing outperforms the multi-task baseline and decentralized top-down-only and bottom-up-only schemes across the evaluated morphologies.Message-passing direction has a significant impact on morphology performance.
  • Role of Message Passing: Both-way message passing learns multiple gaits simultaneously, while decentralized schemes fail to model different motions across morphologies with drastically different gait requirements.Examples include alternating walking for bipeds and hopping for two-limb walkers.
  • Zero-Shot Generalization: Both-way message passing achieves high rewards on held-out morphologies in zero-shot evaluation, demonstrating generalization without fine-tuning.Policies are trained on 80% of variants and tested on the remaining 20% unseen variants.
  • Training with a Non-Torso Limb as Root: SMP remains robust to the choice of root node, with a left-foot-rooted walker performing slightly better than the default torso-rooted setup.The comparison uses four seeds and reports mean and standard deviation of training rewards at 1M timesteps.

6. Analysis of Message Passing

The analysis tests whether message passing carries information relevant to globally coordinated locomotion. Learned torso messages exhibit temporal patterns aligned with alternating gait and agent pose.

  • Analysis of Message Passing: The analysis examines whether message passing conveys contextual information needed for general-purpose control rather than merely increasing task-modeling capacity.The study focuses on the role of learned messages in coordination and generalization.
  • Consistency over Time: A clear torso-message pattern emerges over episode time, and the message remains consistent with the agent’s pose.The torso message aggregates global information after bottom-up message passing and is visualized with one-dimensional t-SNE.
  • Consistency over Time: The observed message structure supports centralized coordination emerging from decentralized controllers through message passing.The pattern corresponds to alternating walker legs and hopper contraction and relaxation.

7. Related Works

Related work connects SMP to modular robotics, virtual evolution, graph-structured neural networks, communicating multi-agent systems, and biological sensorimotor organization. These lines of work provide context for morphology-aware control, modular reuse, and decentralized coordination.

  • Modular Robotics and Virtual Evolution: Prior robotics work conditions control policies on robot morphology, while trajectory optimization offers a nonparametric alternative for robot control.These approaches address customizable or reconfigurable platforms and fixed morphologies with developmental variation.
  • Modular Robotics and Virtual Evolution: Virtual evolution studies jointly adapt morphology and control mechanisms, with modular control advantages reported for evolving or reconfigurable agents.The cited work includes modular approaches to customizable robot platforms and virtual evolution.
  • Graph-Structured Neural Networks: Graph-structured neural networks provide another framework for modular reuse, using global aggregation or decentralized message passing for coordination.The related work situates message passing within graph-based neural architectures.
  • Communication and Biological Modularity: Communicating multi-agent systems and biological sensorimotor research likewise report global coordination emerging from decentralized components.These connections relate SMP’s message-passing design to both learned communication and biological modularity.

8. Conclusion

The paper presents a reusable local module that produces globally coordinated locomotion across diverse agents, including morphologies unseen during training. It positions SMP as a foundation for general-purpose pre-trained sensorimotor priors.

  • Conclusion: A single reusable module can sense and act locally while producing globally coordinated complex movement behaviors.The architecture is built from one module instantiated across actuators.
  • Conclusion: The architecture produces locomotion for a wide variety of agents simultaneously, including agents not seen during training.The conclusion frames this capability as groundwork for general-purpose pre-trained sensorimotor control.

A.1. Result Videos

A single 4-layer network policy represents different gait behaviors across agent morphologies, with videos provided for all variants.

  • A.1. Result Videos: A single 4-layer network policy can represent different gait behaviors across agent morphologies.The project website provides videos for all variants trained with one policy.
  • A.1. Result Videos: Both-way message passing can represent multiple gates through centralized coordination emerging from decentralized modules.One-way message passing sometimes learns more than one morphology but cannot represent multiple gates.
  • A.1. Result Videos: Videos for all variants are available on the project website.The authors recommend using them to observe the differing gait behaviors.

A.2. Implementation and Training

The implementation trains shared modular policies with TD3 across agent environments, using graph-structured message passing and environment-specific replay buffers. Analyses show that message content is more correlated with nearby leaves, while both-way SMP achieves performance comparable to a monolithic baseline.

  • Implementation and Training: TD3 trains the shared modular policy, with randomized initial states and velocities and specified exploration and optimization settings.Training uses Adam, a learning rate of 4e−4, tau 0.046, exploration noise 0.13, and 4-layer internal modules.
  • Implementation and Training: Multi-morphology training assigns each morphology an independent environment and replay buffer, with total replay capacity capped at 1e7.Environments run in parallel using vectorized environments and dynamic batching.
  • Implementation and Training: Both-way message passing sends information from leaves to the root and then back to the leaves.The procedure is used to investigate whether messages convey global information.
  • Implementation and Training: Leaves’ states and actions are more correlated with closer messages, indicating that messages convey meaningful contextual information for locomotion.The analysis reduces dimensions with PCA and averages results over an episode.
  • Implementation and Training: Both-way Shared Modular Policies achieve performance comparable to a monolithic baseline in all single-agent sanity-check experiments.The comparison evaluates average rewards under different message-passing schemes and the monolithic baseline.
  • Implementation and Training: Training and pseudocode jointly share policy modules across the motors of all agents and optimize SMP with each environment’s replay buffer.The algorithms dynamically adapt the policy graph to the agent’s limb structure before bottom-up and top-down message passing.
Loading 2007.04976v1…