Source-linked AI summary
Reinforcement Learning for Improving Agent Design
David Ha
TL;DR
The paper addresses reinforcement learning with fixed agent bodies by jointly learning body design and policy. It parameterizes parts of the environment and optimizes these parameters with policy learning, showing better task-suited bodies, faster policy learning, and potentially useful design principles. The approach is limited to optimizing existing design properties rather than inventing truly novel morphologies.
Problem
Reinforcement learning commonly optimizes policies for fixed agent bodies, although those bodies are rarely optimized for the task.
Method
The method parameterizes the agent’s environment and body alongside policy parameters, then learns them jointly with population-based REINFORCE.
Results
Morphology learning consistently produces higher scores than the normal task across multiple experimental runs, while the learnable BipedalWalkerHardcore-v2 version scores 335 ± 37 versus 313 ± 53 for the baseline.
Takeaways & Limitations
Joint body and policy learning can produce task-adapted designs, faster policy learning, and design principles relevant to machine-learning-assisted design.
Takeaways & Limitations
The approach optimizes existing body-design properties rather than learning truly novel morphology, and unrestricted morphology search may produce unrealistic or unusable designs.
Abstract
from arXiv · showhide
In many reinforcement learning tasks, the goal is to learn a policy to manipulate an agent, whose design is fixed, to maximize some notion of cumulative reward. The design of the agent's physical structure is rarely optimized for the task at hand. In this work, we explore the possibility of learning a version of the agent's design that is better suited for its task, jointly with the policy. We propose an alteration to the popular OpenAI Gym framework, where we parameterize parts of an environment, and allow an agent to jointly learn to modify these environment parameters along with its policy. We demonstrate that an agent can learn a better structure of its body that is not only better suited for the task, but also facilitates policy learning. Joint learning of policy and structure may even uncover design principles that are useful for assisted-design applications. Videos of results at https://designrl.github.io/
1 Introduction
The paper brings embodied cognition into reinforcement learning by jointly learning an agent’s body design and policy for task-specific performance. It situates this approach within morphology learning, standardized Gym environments, and possible assisted-design applications.
- Embodied cognition holds that an organism’s cognitive abilities are shaped by its body and sensorimotor experiences.
- Organisms can adapt their bodies during life while improving the mental skills needed for particular activities.
- RL tasks usually learn policies for agents with fixed bodies, even though those designs may be poorly suited to the task.
- The proposed direction jointly learns an agent’s body structure and policy so the body becomes better suited to its task.
- The experiments apply morphology-learning ideas to standardized OpenAI Gym tasks to connect artificial-life and reinforcement-learning research.
2 Related Work
Related work spans evolutionary computation, artificial life, robotics, passive dynamics, and methods that jointly optimize robot morphology and control. The paper positions its reinforcement-learning approach within this broader morphology-learning landscape.
- Evolutionary computation and artificial-life research has produced novel morphologies, modular robots, and soft robots through methods including indirect encoding.
- Passive-dynamics studies design robots whose body components exploit natural motion rather than motors at every joint.
- Prior robotics work jointly optimizes body design and control using constraint-based modeling, CMA-ES, or reinforcement learning.
3 Method
The method parameterizes both policy weights and aspects of the agent’s environment, including body structure, then optimizes their shared parameter vector with population-based REINFORCE. Rewards can additionally favor desired design properties.
- The learnable vector w includes policy-network weights and environment parameters governing body-part properties such as width, length, radius, mass, and orientation.
- Each rollout deploys an agent and environment parameterized by the same vector w, coupling policy and body design.
- 3.1 Overview of Population-based Policy Gradient Method (REINFORCE): The algorithm searches for w by maximizing expected cumulative reward with a population-based policy-gradient method based on REINFORCE.
- Reward augmentation can encourage more difficult or resource-efficient designs, such as carrying a payload with smaller legs.
- 3.1 Overview of Population-based Policy Gradient Method (REINFORCE): Population-based REINFORCE samples candidate parameter vectors from a parameterized distribution, estimates gradients from rollout rewards, and updates the distribution by gradient ascent.
- Population-based REINFORCE is connected to non-elitist Evolution Strategies, including NES and OpenAI-ES, and to natural-gradient methods related to CMA-ES.
4 Experiments
Experiments show that jointly learning morphology and policy produces task-specific body designs and can improve performance and learning speed across Ant and BipedalWalker environments.
- 4 Experiments: 5789 ± 479 versus 3447 ± 251 average cumulative score shows the learned Ant body outperforming the original design over 100 trials.The learned morphology developed longer, thinner legs and broke the original symmetry by biasing toward larger rear legs.
- 4 Experiments: On BipedalWalker-v2, the learned body hopped using its legs as springs and achieved a score of 359, compared with 347 for the original body.Lengths and widths of the four leg parts were learnable within ± 75% of the original design.
- 4.1 Joint learning of body design facilitates policy learning: BipedalWalkerHardcore-v2: Morphology learning solved BipedalWalkerHardcore-v2 in under 1400 generations (12 hours), versus under 4600 generations (40 hours) for the original version.The comparison is reported for population-based training over 100 random trials.
- 4.1 Joint learning of body design facilitates policy learning: BipedalWalkerHardcore-v2: 335 ± 37 versus 313 ± 53 average score shows the learnable morphology version outperforming the static version on BipedalWalkerHardcore-v2.The learned design generally used longer, thinner legs, with a thicker rear lower limb for stability.
- 4.2 Optimize for both the task and desired design properties: Reward augmentation reduced BipedalWalker leg area to 8% of the original design while retaining an average unaugmented score of 323 ± 68.For BipedalWalkerHardcore-v2, a similar small body could not solve the task; the learned design used 27% of the original leg area and scored 312 ± 69.
- 4.3 Results over Multiple Experimental Runs: Across 10 independent runs, morphology learning consistently produced higher scores than the normal task, although morphology and policy varied with random seed.The authors also note that reproducing these variations can require several days on a powerful 96-core CPU machine.
5 Discussion and Future Work
Jointly learning policy and body parameters can produce task-adapted agents that learn better policies more quickly, while revealing design principles and potential assisted-design applications. The approach remains limited to optimizing existing body properties and depends on the chosen learning method and design constraints.
- Joint policy and body learning produces policies that are better suited to the task and learned more quickly within design constraints.
- The learned morphologies reveal task-specific principles, including larger rear limbs for navigation and reduced or task-appropriate limb sizes under material constraints.These patterns were observed in RoboschoolAnt, BipedalWalker, and BipedalWalkerHardcore experiments.
- The approach may support machine learning-assisted design by optimizing agent or game-character specifications while retaining constraints on the original design.
- The study evaluates only a simple population-based policy-gradient method, leaving comparisons with other RL algorithms for future work.The authors relate this choice to sparse rewards created by setting body design only once at rollout start.
- The method optimizes existing body properties rather than discovering truly novel morphologies, although constrained existing designs may be practical for applications.
- RL-based morphology learning could be extended toward novel designs and may encourage further exchange between reinforcement learning and evolutionary-computation communities.