PaperlayerView exact revision on arXiv

Source-linked AI summary

Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning

Charles Schaff, David Yunis, Ayan Chakrabarti, Matthew R. Walter

arXiv:1801.01432v3cs.ROcs.LG

TL;DR

Evaluating robot designs with learned controllers requires costly separate training, making broad design search impractical. This paper jointly optimizes a design distribution and a neural control policy using reinforcement learning, with the policy conditioned on design parameters. In legged locomotion experiments, the method outperforms baselines on level terrain across three morphologies and on inclined terrain for the Hopper and Walker.

  • Problem

    Evaluating each physical design requires training a controller for it, making it computationally infeasible to explore a diverse design space.

    Source: B. Simultaneous Optimization of… · p. 3

  • Method

    The method jointly optimizes a design distribution and a neural policy with reinforcement learning, conditioning the policy on design parameters and shifting the distribution toward higher-performing designs.

    Source: §I · p. 1

  • Results

    The method exceeds all baselines for all three morphologies on level terrain and outperforms baselines for the Hopper and Walker on inclined terrain.

    Source: Level Terrain · p. 6, Inclined Terrain · p. 6

  • Takeaways & Limitations

    The framework learns design-policy pairs from random initialization without a dynamics model, expert supervision, or good initial solutions.

    Source: Level Terrain · p. 6

    The current approach assumes fixed morphology; handling different action spaces and transferring learned policies to physical robots remain future-work directions.

    Source: §V · p. 7

Abstract

from arXiv · show

The physical design of a robot and the policy that controls its motion are inherently coupled, and should be determined according to the task and environment. In an increasing number of applications, data-driven and learning-based approaches, such as deep reinforcement learning, have proven effective at designing control policies. For most tasks, the only way to evaluate a physical design with respect to such control policies is empirical--i.e., by picking a design and training a control policy for it. Since training these policies is time-consuming, it is computationally infeasible to train separate policies for all possible designs as a means to identify the best one. In this work, we address this limitation by introducing a method that performs simultaneous joint optimization of the physical design and control network. Our approach maintains a distribution over designs and uses reinforcement learning to optimize a control policy to maximize expected reward over the design distribution. We give the controller access to design parameters to allow it to tailor its policy to each design in the distribution. Throughout training, we shift the distribution towards higher-performing designs, eventually converging to a design and control policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel designs and walking gaits, outperforming baselines in both performance and efficiency.

I. INTRODUCTION

A robot’s physical design and control are coupled: different designs can call for different strategies and suit a task and environment differently. Because evaluating designs with learned controllers requires costly training, this work jointly optimizes design and control, finding novel designs and gaits that outperform baselines across morphologies and environments.

  • Different physical designs can require different control strategies, and some may prevent fast locomotion while others enable naturally efficient walking.The passage gives short legs and passive walking as examples.Source: §I · p. 1
  • Prior joint design-and-control methods often rely on robot-dynamics models or near-ideal motion trajectories chosen using expert intuition.Source: §I · p. 1
  • Training a controller for each design makes it computationally infeasible to evaluate a diverse design set for most complex tasks.The challenge remains even when methods such as Bayesian optimization help select designs to explore.Source: §I · p. 1
  • The method jointly optimizes design and control by training a neural policy over a design distribution, giving the policy design parameters and shifting the distribution toward higher-performing designs.The distribution and policy are optimized together rather than evaluating each design with a separately trained controller.Source: §I · p. 1
  • Starting from random initializations, the method finds novel designs and gaits that outperform manually designed agents and other optimization baselines across morphologies and environments.Source: §I · p. 1

II. RELATED WORK

Prior co-design work includes model-based methods, task-specific synthesis, and learned controllers, but each approach has constraints. This paper’s context includes computationally costly design search for neural controllers and gradient-based co-optimization methods that rely on differentiable functions.

  • Model-based co-design methods and trajectory-optimization approaches rely on dynamics models or impose initialization and user-input requirements.Spielberg et al. require initialization near a feasible solution and an estimated robot configuration at each time step.Source: §II · p. 2
  • Demonstration-driven and compositional design methods target specified behaviors or structures, while Censi selects parts by functional constraints without reasoning over geometry or motion.Source: §II · p. 3
  • Deep-RL neural controllers have succeeded across several domains [27] [28] [31] [33], but their iterative, data-driven training makes joint optimization with physical structure non-trivial.Source: §II · p. 3
  • Bayesian optimization can treat design specifications as hyperparameters, but becomes computationally infeasible for high-dimensional spaces.Earlier evolutionary co-design methods were restricted in scope.Source: §II · p. 3
  • Jointly learning sensor parameters and neural estimators uses gradients through differentiable losses, unlike this paper’s reinforcement-learning approach based on environment rewards.Source: §II · p. 3

III. APPROACH

The approach section first reviews standard reinforcement-learning policy training, then presents simultaneous optimization of robot design and control.

  • The approach comprises a review of standard policy learning for a fixed design followed by simultaneous optimization of design and policy.Source: §III · p. 3

A. Control Policy Training by Reinforcement Learning

The section frames control as continuous Markov decision process optimization: a policy maps states to actions to maximize discounted future reward. It then describes neural stochastic policies and gradient-based training, including PPO’s clipped objective.

  • The control problem is a continuous MDP with state and action spaces, unknown transitions, a reward function, and an initial-state distribution; the objective is to maximize expected discounted return.Source: A. Control Policy Training by… · p. 3
  • For continuous action spaces, policy-gradient methods update neural stochastic policies using sample-based gradients of expected return.The described vanilla estimator is unbiased, and later methods aim to improve training stability.Source: A. Control Policy Training by… · p. 3
  • PPO alternates between environment sampling and objective optimization; its clipped objective limits policy-space steps while maximizing expected return.Source: A. Control Policy Training by… · p. 3

B. Simultaneous Optimization of Physical Design and Control

Design quality depends on the control policy trained for it, making separate evaluation of candidate designs costly. This section proposes optimizing design and control jointly through a distribution over designs.

B. Simultaneous Optimization of Physical Design and Control

  • Evaluating a candidate design requires training its corresponding policy, making exhaustive search computationally expensive in high-dimensional design spaces.The design acts like a hyper-parameter whose quality cannot be assessed without a full round of policy training.Source: B. Simultaneous Optimization of… · p. 3
  • The method searches over physical designs and control policies simultaneously instead of training separate policies for each candidate design.The design changes the transition dynamics, while designs share the state space, action space, reward, and initial-state distribution; the common action space restricts the search to a shared morphology.Source: B. Simultaneous Optimization of… · p. 4
  • The controller receives each sampled design’s parameters, allowing its policy to tailor actions to that design while training across the design distribution.The distribution pφ(ω) represents candidate designs, and the policy πθ(at|st, ω) is conditioned on the current design.Source: B. Simultaneous Optimization of… · p. 4, Figure 2 · p. 4
  • PPO trains the policy on sampled designs while episode returns update the design distribution toward higher-performing designs.A Gaussian mixture maintains distinct design hypotheses; at intervals, the method removes half of the components with the lowest average reward.Source: B. Simultaneous Optimization of… · p. 4
  • Training starts with high-entropy designs and a policy-only warm-up, then narrows the distribution before fixing its mode for final policy fine-tuning.As poorly performing designs are eliminated, distribution variance decreases and expected reward increases.Source: B. Simultaneous Optimization of… · p. 4

IV. EXPERIMENTS

The experiments evaluate joint design and control for legged locomotion and report that the method finds designs and gaits outperforming the listed alternatives.

IV. EXPERIMENTS

  • The method discovers novel robot designs and gaits that outperform standard Roboschool designs and pairs found by random sampling or Bayesian optimization.The evaluation includes Hopper, Walker, and Ant morphologies.Source: §IV · p. 5
  • The evaluation studies legged robots tasked with locomotion in different environments.The tested morphologies are Hopper, Walker (Walker2D in Roboschool), and Ant.Source: §IV · p. 5
  • The experiments compare the learned design-controller pairs with policies trained for standard hand-crafted designs and with random-sampling and Bayesian-optimization searches.The authors describe their objective as maximizing a standard reward function.Source: §IV · p. 5

A. Experiment Setup

The setup tests three legged morphologies on level and inclined terrain, with design parameters, policy inputs, and training budgets specified for each experiment.

A. Experiment Setup

  • Locomotion is tested for Hopper, Walker, and Ant on level ground and a five-degree incline, using Bullet Physics and the Roboschool reward.The reward combines forward progress and upright posture with penalties for torque and joint-limit violations; episodes end at a fall or the timestep limit.Source: A. Experiment Setup · p. 5
  • Design search retains each default Roboschool morphology and varies link lengths and radii, with eight parameters for Hopper and Walker and 25 for Ant.Walker legs are symmetric; Ant legs have independent parameters, and parameter ranges serve as a proxy for fabrication constraints.Source: A. Experiment Setup · p. 5
  • The controller is a three-layer, 128-unit-per-layer tanh network that receives joint and body state variables plus the robot’s design parameters.Its output is a vector of joint torques.Source: A. Experiment Setup · p. 5
  • The design distribution is an eight-component Gaussian mixture with diagonal covariances, random initial means, broad initial variances, and uniform component weights.Weights remain uniform among surviving components after low-reward components are removed.Source: A. Experiment Setup · p. 5
  • Training uses eight parallel robot samples; design updates begin after 100 million timesteps, or 200 million for Ant, and the final 100 million use a fixed design.Runs total 1 billion timesteps, except Ant runs total 1.5 billion; components are evaluated and pruned every 100 million timesteps, or 200 million for Ant.Source: A. Experiment Setup · p. 5

B. Baselines

The baselines compare the method with fixed default designs and two searches that train policies for sampled designs.

B. Baselines

  • The default-design baseline trains the same policy architecture on the standard hand-crafted Roboschool robots until convergence.It provides a comparison against the unchanged default physical designs.Source: Default Designs · p. 5
  • Random sampling draws designs uniformly within the parameter ranges and trains a separate policy for each until convergence.It receives three times the method’s environment timesteps, and the best design-policy pair across samples is compared with a single method run.Source: Random Sampling · p. 5
  • Bayesian optimization trains a policy to convergence for each sampled design and uses its average episode reward to select the next design.It receives the same environment-timestep budget as the method, allowing roughly ten designs; four random-seed runs are reported.Source: Bayesian Optimization · p. 5

C. Results

The method outperforms baselines across all three morphologies on level terrain and for Hopper and Walker on inclines; modified-reward training also yields strong Ant results on the incline. The learned designs pair morphology-specific structures with distinct gaits, while standard inclined-Ant training exposes a local-optimum failure.

C. Results

  • In Fig. 3, the method beats all baselines for all three morphologies on level terrain and for Hopper and Walker on inclines; modified-reward Ant also beats baselines.The modified-reward Ant comparison uses the original reward for evaluation.Source: Figure 3 · p. 6, Level Terrain · p. 6, Inclined Terrain · p. 6
  • Hopper designs use long bodies: heavier versions take short, quick strides, while narrow, light versions use bounding gaits; the longer torso improves stability at faster pace.Source: Level Terrain · p. 6
  • Seven Walker seeds beat every BayesOpt design, using long thin legs and small feet for long-stride running; the remaining seed adopts stable knee-walking.Source: Level Terrain · p. 6
  • Across Ant seeds, similar-sized legs emerge despite unshared leg parameters, and the gait mainly walks on three legs while keeping the front leg airborne except for balance.Source: Level Terrain · p. 6
  • On inclines, Hopper designs use long feet to gain height on push-off; the higher-reward Walker gait pushes off with both feet simultaneously, avoiding leg-crossing torque penalties.Source: Inclined Terrain · p. 6
  • Standard inclined-Ant training leaves seven seeds balancing in a low-reward local optimum; halving upright reward frees all eight to beat baselines on original reward.A similar reward modification may also have helped the baselines.Source: Inclined Terrain · p. 6

D. Evolution of Design and Control During Training

The authors track sampled-design rewards alongside design-parameter distributions during training, observing early design diversity followed by convergence and linked shifts in rewards and distribution.

D. Evolution of Design and Control During Training

V. CONCLUSION

The paper presents a model-free method for jointly optimizing robot designs and control policies, reporting novel designs and gaits that outperform baselines. It identifies broader tasks, variable morphologies, and sim-to-real transfer as directions for future work.

V. CONCLUSION

  • Across varied legged morphologies and locomotion tasks, the method produces novel robot designs and walking gaits that outperform Bayesian-optimization and random-design baselines.The method converges to a design and policy that are locally optimal.Source: §V · p. 7
  • The model-free method jointly updates a distribution over physical design parameters and a neural control policy that maximizes expected reward over that distribution.The policy accesses design parameters to generalize to novel parts of the design space; the distribution shifts toward higher-performing designs.Source: §V · p. 7
  • Future work includes more tasks and domains, while relaxing the fixed-morphology assumption would require a policy that handles different action spaces across morphologies.Examples include uneven terrain, obstacles, friction variations, and manipulation.Source: §V · p. 7
  • The authors also propose fabricating learned designs and transferring the control policy from simulation to a real robot.They point to prior work by Tan et al. as showing this is possible for legged locomotion.Source: §V · p. 7

Related papers

Loading 1801.01432v3…