Source-linked AI summary
Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, Alexei A. Efros
TL;DR
The paper addresses limited evidence on learning control when an agent’s morphology is not fixed. It trains primitive limbs to self-assemble and coordinate through a morphology-matched modular graph policy, then evaluates standing and locomotion under environmental and structural changes. The dynamic self-assemblies generalize better than static and monolithic baselines, including across unseen limb counts, although experiments use a fixed six-limb chain as the static morphology.
Problem
The paper investigates modular self-assembling agents because standard sensorimotor-control benchmarks do not support jointly co-evolving control and morphology.
Method
Primitive limbs learn torque, linking, and unlinking actions through a shared-parameter Dynamic Graph Network whose topology matches the assembled morphology.
Results
Dynamic self-assembling agents generalize better than static and monolithic baselines across novel environments and agent structures, including unseen limb counts.
Takeaways & Limitations
A single modular policy can control multiple morphologies, including 3- and 12-limb agents after training to build a 6-limb tower.
Takeaways & Limitations
The fixed-morphology baseline uses a six-limb straight chain, which may be optimal for standing tall but not necessarily for learning to stand or for locomotion.
Abstract
from arXiv · showhide
Contemporary sensorimotor learning approaches typically start with an existing complex agent (e.g., a robotic arm), which they learn to control. In contrast, this paper investigates a modular co-evolution strategy: a collection of primitive agents learns to dynamically self-assemble into composite bodies while also learning to coordinate their behavior to control these bodies. Each primitive agent consists of a limb with a motor attached at one end. Limbs may choose to link up to form collectives. When a limb initiates a link-up action, and there is another limb nearby, the latter is magnetically connected to the 'parent' limb's motor. This forms a new single agent, which may further link with other agents. In this way, complex morphologies can emerge, controlled by a policy whose architecture is in explicit correspondence with the morphology. We evaluate the performance of these dynamic and modular agents in simulated environments. We demonstrate better generalization to test-time changes both in the environment, as well as in the structure of the agent, compared to static and monolithic baselines. Project video and code are available at https://pathak22.github.io/modular-assemblies/
1 Introduction
The paper studies learned modular self-assemblies in which primitive limbs jointly evolve morphology and control. A shared dynamic graph policy supports coordination across changing morphologies, improving generalization to novel environments and limb counts.
- Motivation and contribution: Primitive limbs link into self-assembled collectives while jointly learning control and morphology.Linking and unlinking are dynamic actions, and connected limbs pass messages and share rewards.
- Modularity: The policy uses a Dynamic Graph Network with shared parameters across limbs and topology matching the physical morphology.Each limb receives neighboring messages and outputs torque plus linking or unlinking actions.
- Evaluation: The study evaluates standing up and locomotion under test-time changes to terrain, environmental conditions, and the number of available limbs.These dynamic assemblies are compared with static and monolithic baselines.
- Generalization: A single modular policy trained to build a 6-limb tower can control 3- or 12-limb morphologies at test time.The policy performs on these unseen limb counts without the passage specifying additional fine-tuning.
- Results: The paper reports that self-assembling agents train and generalize better than fixed-morphology baselines.This contribution is framed as using modularity to improve adaptability and robustness to novel environmental conditions.
2 Environment and Agents
The paper creates minimalist simulated arenas and primitive limb agents that co-evolve self-assembly and control. Limbs use local sensing, torque actions, and complementary attachment decisions to form joint-reward collectives evaluated across procedurally varied terrains.
- Environment Structure: The arena uses gravity and friction, while procedural tile-height changes create novel terrains for evaluating generalization.The paper also illustrates standing-up and locomotion tasks across newly generated environments.
- Environment Structure: Training begins with six primitive limbs that can self-assemble into collectives for complex control tasks.The minimalist environment and reward structure are intended to let complex morphologies emerge naturally rather than forcing them.
- Agent Structure: Each limb has a cylindrical body, a motorized parent-end, and a free child-end that can connect to another limb’s motor as a three-degree-of-freedom joint.Multiple limbs can attach their child-ends to other limbs’ parent-ends, producing modular morphologies.
- Agent Structure: Assembled limbs become one agent whose members maximize a joint reward, unlike typical multi-agent scenarios.Primitive limbs jointly solve the task while applying continuous torque controls.
- Agent Structure: Binary linking and unlinking decisions assign linking to parent-ends and unlinking to child-ends, enabling decentralized self-assembly control.This complements the limbs’ continuous torque actions.
- Sensory Inputs: Each limb receives local dynamics, nearby-agent position, and trinary touch-sensor information rather than global morphology state.The sensory inputs include 3-D position, velocity, angular rotation, and angular velocity.
- Simulation: Unity ML provides an efficient simulator for simultaneously simulating many primitive limbs while retaining realistic physics details.The implementation keeps contact forces and control frequency similar to those in other physics-based environments.
3 Learning to Control Self-Assemblies
The paper jointly learns control and morphology by treating assembly decisions as actions and representing each changing limb morphology with a shared, modular graph policy.
- 3.1 Co-evolution: Linking/Unlinking as an Action: Linking and unlinking are added to each limb’s torque actions, turning morphological evolution into a reinforcement-learning problem.The action space combines three continuous torques with binary link and unlink decisions.
- 3.3 Dynamic Graph Networks (DGN): A self-assembled agent is modeled as a graph whose nodes are limbs and whose edges represent physical joints.The policy graph mirrors the morphology’s connectivity and parent-child relationships.
- 3.3 Dynamic Graph Networks (DGN): The Dynamic Graph Network uses shared parameters across limb controllers and passes messages between physically connected neighbors.Messages provide context about each limb’s role within its connected component.
- 3.3 Dynamic Graph Networks (DGN): Because linking actions change physical connectivity, the DGN topology also changes at every timestep according to previous policy outputs.This dynamic topology distinguishes the model from a fixed graph policy.
- 3.3 Dynamic Graph Networks (DGN): The shared policy parameters are optimized to jointly maximize limb rewards using policy gradients, specifically PPO.Training rollouts contain torque sequences for each limb at each timestep.
- 3.3 Dynamic Graph Networks (DGN): Message passing aggregates information recursively through the morphology, typically from leaf limbs toward the root.When no messages are used, limbs can still coordinate through responses to other agents’ previous actions.
4 Experiments
The experiments evaluate dynamic modular policies on standing and locomotion, comparing them with monolithic and fixed-morphology baselines. DGN policies generally train better and retain performance under zero-shot changes to limb count and environmental conditions.
- Experimental setup: The experiments compare dynamic graph policies with monolithic policies across standing-up and locomotion tasks, including dynamic and fixed morphologies.Training uses six limbs, while evaluation includes zero-shot changes in limb count and novel environments.
- Scalability: Monolithic policies become harder to train as limb count increases because their observation and action spaces grow, while the shared DGN limb policy has fixed-size spaces.The monolithic policy performs well up to three limbs but does not reach the optimum for four to six limbs in the appendix experiment.
- Training performance: DGN variants outperform monolithic baselines during locomotion training and show superior performance in standing with wind.Message passing is particularly helpful for standing, where long-term coordination across limbs is needed.
- Ablation: Modular DGN policies with static morphology perform better than monolithic static-graph policies but worse than self-assembling DGN, supporting contributions from both software and hardware modularity.The comparison separates modular policy structure from dynamic physical morphology.
- Zero-shot generalization to limb count: DGN retains similar standing-up performance when trained with 6 limbs and tested with 12 or 4 limbs without fine-tuning, whereas monolithic policies require retraining for different limb counts.For 6-to-4-limb standing, DGN scores 17518 at training and 10839 at test, while the retrained baseline scores 5351 and 7356, respectively.
- Zero-shot generalization to environments: DGN retains most of its performance when standing policies face stronger or underwater wind and when locomotion policies encounter water, hurdles, gaps, bumps, stairs, or valleys.These evaluations use novel environments without further fine-tuning.
5 Related Work
The paper situates its approach within modular robotics, morphology optimization, and graph-based controllers. Unlike prior methods that optimize fixed structures, it learns morphology through dynamic linking during an agent’s lifetime.
- Prior modular robotics research developed hardware modules that dock to form complex robotic morphologies.
- Earlier morphology-optimization methods used genetic algorithms, energy-based minimization, or fixed-morphology limb design.
- This paper uses deep reinforcement learning to co-evolve control and morphology through linking actions taken during agents’ lifetimes.
- Related graph controllers support multi-limb generalization, but this work studies agents whose morphology can change during both training and operation.
6 Discussion
The discussion frames modular self-assembling agents as a way to study whether modularity improves generalization. Simulated experiments provide promising evidence, while the authors describe the direction as an initial step.
- The paper revisits self-assembling agents with deep networks and reinforcement learning, treating modularity as a route to adaptability and robustness.
- The experiments report promising results suggesting that modularity improves generalization in simulated agents.
- The authors characterize the work as initial steps and release code, models, and environments to support follow-up research.
A Appendix
The appendix introduces additional experimental setup details and extra experiments.
- This appendix section provides additional experimental setup details and extra experiments.
A.1 Implementation and Training details
The implementation trains the joint Dynamic Graph Network objective with PPO using specified network, optimization, communication, and training settings.
- PPO optimizes the joint DGN objective for the modular control policy.
- Each limb policy is a 4-layer fully connected ReLU network trained with learning rate 3e −4, discount factor 0.995, entropy coefficient 0.01, and advantage parameter 0.95.
- The system uses 32-length float messages, batch size 2048, and RMS-Prop as the PPO optimizer.
A.2 Project Video
The project video provides a visual resource for understanding the simulated environment terrains and the reported generalization results.
- The project video is available at the linked YouTube address.
- The video is intended to clarify the environment terrains used in the experiments.
- The video is also intended to clarify the reported generalization results.
A.3 Performance of Monolithic Policy w/ Static Morphology Baseline vs. Number of Limbs
The monolithic static-morphology baseline performs well with fewer limbs but fails to reach the optimum as morphology size increases, including the six-limb case.
- The six-limb failure is attributed to the fixed morphology graph rather than an implementation problem with the baseline.
- The baseline performs well with fewer limbs, particularly one to three limbs.
- The monolithic baseline fails with six limbs during training.
A.4 Performance of Modular DGN Policy on Static Morphology
A modular DGN policy is evaluated on static morphology as an ablation separating policy modularity from physical-morphology modularity. Its performance exceeds the monolithic static-graph baseline but remains below the fully self-assembling DGN.
- The static-morphology modular DGN performs significantly better than the monolithic policy with a static graph.
- The static-morphology modular DGN performs worse than the final self-assembling DGN, suggesting that both software and hardware modularity are necessary for successful training and generalization.
- The modular DGN policy uses message passing over a graph whose structure corresponds to the agent morphology.