PaperlayerView exact revision on arXiv

Source-linked AI summary

Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimization

Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss, Andrew Spielberg, Rika Antonova

arXiv:2610.02366v1cs.LGcs.RO

TL;DR

Benchmarks commonly hold embodiment fixed, although embodiment and policy choices are coupled and optimizing them separately can be suboptimal. Co-Design Gym provides a broad benchmark suite and evaluates representative co-design methods; across tested presets, no method dominates and many tasks remain far from solved.

  • Problem

    Most benchmarks focus on policy learning with a fixed embodiment, despite embodiment and policy being coupled choices whose separate optimization can be suboptimal.

    Source: §1 · p. 2

  • Method

    The paper formalizes co-design and builds Co-Design Gym by extending the Gymnasium API, adapting existing environments, and adding new ones across 20 environment families and over 85 presets.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Across 18 presets where all four baselines were run, no method dominates, rankings depend on task and budget, and many presets remain far from solved.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The benchmark provides a unified setting for comparable co-design evaluation and leaves substantial room for new methods across its tasks.

    Source: §1 · p. 2, §6 · p. 11

    The design spaces do not cover every possible degree of freedom, baselines cover only a subset of presets, and fixed wall-clock budgets may make some performance gaps reflect engineering rather than algorithmic differences.

    Source: §6 · p. 10

Abstract

from arXiv · show

Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructure, communication networks, and multi-agent systems. Numerous benchmarks have been developed to support such research, but the vast majority assume that the agent's embodiment (design) is fixed, focusing instead on policy learning alone. Lifting this assumption gives rise to a broader class of problems in which optimizing embodiment and policy separately is highly suboptimal. An agent's embodiment strongly shapes which control policies can be discovered, while the optimal embodiment is in turn defined by the policies it admits. To help the research community study this class of problems explicitly and systematically, we introduce Co-Design Gym - a suite of benchmark environments for jointly optimizing embodiment and policy. Our environments span domains such as robotic manipulation and locomotion, multi-robot cooperation, deformable and soft dynamics, video games, electricity grids, wireless networks, F1 racing, multi-agent warehouses, and optimal control, offering 20 environment families (domains), with over 85 distinct co-design presets in total. We further contribute a systematic evaluation of representative co-design algorithms, characterizing the current state of the art. Together, these contributions lay the groundwork for cumulative, comparable progress in co-design.

1 INTRODUCTION

Co-design jointly optimizes embodiment and policy because each shapes the performance available from the other. Co-Design Gym provides a unified benchmark for this problem and evaluates representative methods across diverse tasks.

1 INTRODUCTION

  • Co-design jointly improves embodiment and policy toward performance goals, extending beyond policy learning with a fixed agent design.Source: §1 · p. 2
  • Embodiment and policy are coupled: an embodiment’s quality depends on the best policy it permits, so fully assessing it requires policy optimization across downstream tasks.Completing expensive policy optimization before assessing an embodiment’s potential can be wasteful, motivating tighter co-optimization.Source: §1 · p. 2
  • The field lacks a unified benchmark across domains that can track co-design methods and distinguish co-design progress from gains in embodiment search or policy optimization alone.Source: §1 · p. 2
  • Co-Design Gym extends the Gymnasium API with an explicit design space, adapts existing policy-optimization environments into co-design tasks, and adds new environments where embodiment optimization shapes policy capability.The suite contains 20 environment families and over 85 presets, with GPU-based resets for evaluating thousands of embodiments in parallel.Source: §1 · p. 2
  • Across the benchmark, no single evaluated method dominates, and many tasks remain far from solved, leaving substantial headroom for new methods.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Co-design spans established hardware–software and physical-design applications as well as newer robot embodiment-policy work. Existing benchmarks cover narrower settings, motivating a more extensible shared framework.

2 BACKGROUND AND RELATED WORK

  • Co-design has long appeared in computer systems, optics, aeronautical design, and chemical plants, and more recently in virtual and physical robotics.Robotics combines dynamical tasks, planning, discrete and continuous choices, and real-time operation; transferring simulation success to reality is another challenge.Source: §2 · pp. 2–3
  • Evolutionary and gradient-based methods search morphology, materials, controllers, and observers, while newer generative systems combine sampling- and gradient-based approaches.Sampling can be inefficient when simulation is the computational bottleneck; gradient methods can be more efficient given continuous, smoothly differentiable parameterizations.Source: §2 · p. 3
  • Co-designed robots have been physically demonstrated across quadrupeds, grippers, humanoids, and robots at larger and smaller scales.Source: §2 · p. 3
  • SoftZoo benchmarks soft-robot co-design across terrains, materials, and morphology but is limited to 3D elastic dynamics; EvoGym offers many complex-control tasks but uses 2D, low-fidelity pixel representations.Source: §2 · p. 3
  • Other evaluation setups focus on narrow task sets or lack realistic modeling, while Gymnasium lacks the extensibility to search over morphology and other embodiment choices.Source: §2 · p. 3

3 AN OPTIMIZATION PERSPECTIVE ON CO-DESIGN

The paper frames co-design as joint optimization over feasible embodiment-policy pairs, with each embodiment determining which policies and performance are available. It distinguishes the theoretical opportunity from fixed-design limits from what an algorithm can achieve under practical budgets.

3 AN OPTIMIZATION PERSPECTIVE ON CO-DESIGN

  • Co-design treats embodiment and policy as coupled choices: each affects the capabilities and performance available through the other.A controller developed for one robot leg geometry may misrepresent a changed geometry’s potential.Source: §3 · p. 3
  • An embodiment need not be physical, and design choices may be selected once before an episode while policy actions occur repeatedly during it.The paper focuses on embodiment and policy under a shared objective, although environment structure, tasks, and objectives can also be design variables.Source: §3 · p. 3
  • The formulation represents each admissible embodiment e with a control problem M_e and a nonempty set Π_e of policies, which may differ in dynamics, actions, initial conditions, or available information.The policies also incorporate what an embodiment can execute and any controller restrictions.Source: §3 · pp. 3–4
  • Performance J(e, π) is the expected performance of policy π with embodiment e; the setup assumes finite expectations and comparable rewards under a shared objective.The argument also applies to other well-defined scalar performance criteria.Source: §3 · p. 4
  • The feasible set need not be a Cartesian product because a policy executable with one embodiment may be unavailable with another.Changing the embodiment can change its policy slice, its best available return, and which policy attains that return.Source: §3 · p. 4
  • A fixed embodiment’s performance ceiling is the supremum over its available policies, while the design gap compares that ceiling with the joint optimum.The gap is nonnegative; zero means the fixed embodiment still permits a global optimum, while a positive gap means alternatives can exceed its ceiling.Source: §3 · pp. 4–5
  • A positive design gap is distinct from a policy’s failure to reach its fixed-embodiment ceiling: better policy optimization can address the latter but not the former with the embodiment fixed.The gap is a comparison of available performance, not a guarantee about what an algorithm will find; under limited budgets, fixed-embodiment optimization may outperform joint search.Source: §3 · p. 5
  • A plateau at one embodiment cannot establish that its policy has reached its ceiling or that other embodiments offer no improvement; measured gains depend on both embodiments considered and policy-optimization effectiveness.Determining whether a design gap exists requires comparisons across embodiments or additional knowledge linking design choices to behavior.Source: §3 · p. 5

4 CO-DESIGN GYM

Co-Design Gym brings together co-design environments from multiple domains, including from-scratch tasks, MuJoCo Control extensions, and lifted RL environments. Its minimal Gymnasium-based API supports paired embodiment and policy optimization, with evaluation controls for within-episode design changes.

4 CO-DESIGN GYM

  • The suite spans robotics, video games, and grid management, combining from-scratch tasks, MuJoCo Control extensions, and lifted RL environments.The environments pair an inner policy-optimization loop with an outer embodiment problem and offer presets of varying complexity.Source: §4 · p. 6
  • Ball Catching varies robot limbs and torso; Truck Unloading modifies robot arms; Nero-LEAP Grasping replaces thumb and index fingertip meshes.These tasks cover dynamic catching, coordinated unloading, and fine contact-rich manipulation, respectively.Source: §4 · p. 6
  • Soft Walkers pairs three terrains with 15 solver presets, while Car Racing co-designs vehicle setup parameters to minimize lap time.Soft-robot designs vary shape, mass, material, and actuators; car parameters include tyre grip, cornering stiffness, and drive power.Source: §4 · p. 7
  • Linear-Quadratic Control spans continuous, noisy placement, and budgeted saturated-input design; Microgrid presets budget solar, battery, and diesel capacity against 24–72-hour demand.Microgrid policies dispatch energy, and design costs are charged in the reward.Source: §4 · p. 7
  • Wireless Network varies base-station count, placement, and power; warehouse layouts scale from 2 to 8 robots; Pokémon Showdown searches six-Pokémon teams.The warehouse trades route length against congestion, while Pokémon opponents range from random players to League teams across nine generations.Source: §4 · p. 7

4.1 API

  • The API adds an embodiment space and reset argument while preserving the original dynamics, reward, termination logic, and base-embodiment task.The embodiment space uses Gymnasium Space semantics, including sampling and membership checks.Source: §4.1 · p. 8
  • An outer optimizer supplies embodiments at reset for an inner policy; evaluation mode restricts design changes to reset, though training may change them mid-episode.Passing no embodiment retains the current design, which defaults to the base embodiment.Source: Figure 7 · p. 8, §4.1 · p. 8

5 EXPERIMENTS WITH REPRESENTATIVE OPTIMIZATION METHODS

The paper benchmarks four representative co-design approaches under a shared evaluation setup. Its contained LLM study finds that PPO+LLM remains near zero return throughout training.

5 EXPERIMENTS WITH REPRESENTATIVE OPTIMIZATION METHODS

  • Four baselines span evolutionary, reinforcement-learning, alternating, and generative approaches to co-design.CMA-ES jointly searches controller and embodiment parameters; FastTD3 treats embodiment parameters as first-step actions; PPO+NGOpt alternates controller updates and embodiment proposals; LOKI uses a pretrained Transformer VAE and PPO-trained Transformer controllers.Source: §5 · pp. 8–9
  • The baselines are evaluated every 5% of training on held-out seeds, with final evaluations reported across five seeds.Most methods receive matched four-hour wall-clock budgets; LOKI receives three hours online after about one hour of preparation.Source: §5 · pp. 8–9
  • Table 1 reports final evaluation return as mean ± standard deviation over five seeds under matched wall-clock budgets.Every FastTD3 design on NeroGraspAll failed structural validation and received the −250 rejection reward.Source: Table 1 · p. 9
  • PPO+LLM stays near zero return throughout training, despite receiving additional task-relevant information in its prompt.The study uses Gemma 4 31B as the outer-loop designer and PPO to learn the controller.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

Across the evaluated presets, no co-design method dominates: rankings vary by task and training budget, and many tasks remain far from solved. The paper presents Co-Design Gym as a unified benchmark for studying this progress, while noting limits in preset coverage and experimental comparability.

6 DISCUSSION AND CONCLUSION

  • No method leads across the 18 presets run with all four baselines, and the leading method changes with task and budget.The leader changes at least once on 12 of 16 presets shown in Figure 8.Source: §6 · p. 9, Figure 8 · p. 10
  • Final winners cluster by domain, but none of the evaluated methods shows general capability across the benchmark’s varied domains.CMA-ES leads on most extended locomotion tasks, FastTD3 on ball-catching, racing, and most microgrid presets, and PPO+NGOpt on most warehouse, wireless, and Pokémon presets.Source: §6 · p. 10
  • Many presets remain far from solved: BallCatcher-Pitch’s best method completes only 16.6% of catches, while several harder tasks score near zero.SolarCleaner, WarehouseMedium, and WarehouseCongested are among the near-zero cases.Source: §6 · p. 10
  • The design spaces do not cover every possible degree of freedom, and baselines cover only a subset of available presets.Fixed wall-clock budgets favor faster implementations, so observed performance gaps may partly reflect engineering rather than algorithmic differences.Source: §6 · p. 10
  • Co-Design Gym formalizes joint embodiment-policy optimization, extends Gymnasium with a design space, and combines existing and purpose-built co-design environments.The paper presents the benchmark as a basis for cumulative, comparable progress in co-design.Source: §6 · pp. 10–11

APPENDIX A: ENVIRONMENTS

The appendix describes co-design environments across robotics, soft-body dynamics, games, infrastructure, racing, warehouses, and control. Each pairs a policy-learning task with design variables that shape what the agent can do.

APPENDIX A: ENVIRONMENTS

  • Ball Catching pairs an arm-equipped quadruped’s catching policy with arm design, while Solar Panel Cleaning jointly designs robot legs and arms for careful locomotion and precise cleaning.Ball-CatcherPitch, Arc, and Fielding vary throw conditions; SolarCleaner presets vary panel layout and terrain.Source: Appendix A · p. 11
  • Nero-LEAP Grasping optimizes thumb and index fingertip meshes for stable object lifts, while MorphHand tasks optimize hand surfaces or height-field displacements for object manipulation.The MorphHand tasks cover palm reorientation and pick-up; the supplied description specifies the Rotation preset.Source: Appendix A · p. 12
  • Soft Walkers vary robot shape, mass, actuation, and material across three terrains and five solver configurations, yielding 15 presets.The configurations span 2D and 3D Spring, MPM, Beam, and FEM-based simulation.Source: Appendix A · p. 13
  • Pokémon Showdown pairs battle policies with team design across random, power-based, heuristic, and sequential-league opponents.The four main presets use different generations and opponent behaviors; PokenvLeague pits teams against five successive opponents.Source: Appendix A · p. 13
  • Microgrid tasks optimize energy-flow policies alongside solar, battery, and generator capacities, while wireless tasks pair user-station assignment with base-station deployment and configuration.The microgrid design costs are charged at the first step; wireless designs specify station count, placement, and transmission capacity.Source: Appendix A · p. 14
  • Car Racing pairs lap-time policies with vehicle setup, Robotic Warehouse pairs fleet control with shelf layout, and Linear-Quadratic Control pairs regulation with system and actuator or sensor design.The environments include two racing tracks, three warehouse scales, and continuous, combinatorial, and budget-constrained control presets.Source: Appendix A · pp. 14–15

A.1 ADDITIONAL ENVIRONMENT PRESETS

The appendix exposes additional Nero-LEAP Grasping and Pokémon Showdown configurations beyond the main presets. The Pokémon League variants span nine game generations and differ in whether the team is restored between battles.

A.1 ADDITIONAL ENVIRONMENT PRESETS

  • Additional configurations are exposed for Nero-LEAP Grasping and Pokémon Showdown beyond the environments’ main presets.The appendix lists nine additional single-object NeroGrasp presets and further PokenvLeague variants.Source: §A.1 · p. 16

A.1.1 NERO-LEAP GRASPING

  • Each of the ten NeroGrasp objects can be run as its own preset, adding nine configurations beyond the main presets.The objects include scissors, tweezers, spatula, sticks, screws, nuts, and a washer.Source: §A.1.1 · p. 16

A.1.2 POKEMON SHOWDOWN

  • PokenvLeague adds nine generation-specific leagues with healed and unhealed variants, plus two presets that randomly select a league.Each league pits the designed team against five opponents in sequence; the unhealed variant requires surviving successive battles without restoration.Source: §A.1.2 · p. 16

B BASELINE OPTIMIZATION METHODS

The baselines search embodiment and policy jointly or in alternating stages, using different mechanisms to propose designs and train controllers.

B BASELINE OPTIMIZATION METHODS

  • CMA-ES samples embodiment and controller parameters from one joint search space, then evaluates the best pair on held-out seeds at 5% training intervals.The sampled controller parameters configure a neural-network controller, while embodiment values pass directly to the environment.Source: B BASELINE OPTIMIZATION METHODS · p. 17
  • FastTD3 expands its action space to include embodiment parameters and control actions, using the former to set up each episode and the latter to control it.Corrected replay-buffer samples reflect which actions were actually taken; evaluations occur every 5% of training on held-out seeds.Source: B BASELINE OPTIMIZATION METHODS · p. 17
  • PPO+NGOpt alternates PPO controller training with NGOpt embodiment sampling, scoring designs by mean rollout return and evaluating the best embodiment every 5% of training.Evaluation uses the current PPO policy on held-out seeds.Source: B BASELINE OPTIMIZATION METHODS · p. 17
  • LOKI pre-trains a Transformer VAE on 500,000 valid embodiments, then runs online training with six clusters and PPO-trained Transformer controllers.At evaluation boundaries, it selects the best-cluster controller and evaluates top-performing embodiments on held-out seeds.Source: B BASELINE OPTIMIZATION METHODS · p. 17

C DETAILED RESULTS

The detailed results show sharply different learning trajectories across presets: catching is rare, method rankings shift over time, and strong early designs may not be retained.

C DETAILED RESULTS

  • Table 2 reports final evaluation return as mean ± standard deviation over five seeds under a matched four-hour budget for CMA-ES, FastTD3, and PPO+NGOpt.Bold marks the best mean; some runs were still in progress, and some presets used fewer than five seeds.Source: Table 2 · p. 18
  • 16.6% full catches for FastTD3 versus ≤1.1% for the others, with final return 3.42 versus ≤0.44; its returns vary widely across seeds (1.35–5.98).None of the outer-loop methods reaches catching behaviour within the budget.Source: Figure 23 · p. 18
  • On Hopper, the leading method changes six times in four hours, and the four families follow distinct learning curves.FastTD3 peaks at 2,800 at 0.6 h and falls to 897; PPO+NGOpt rises from 761 to 1,588 in the final hour, while LOKI improves steadily and CMA-ES plateaus near 1,260.Source: Figure 24 · p. 18
  • On RacingSpielberg, PPO+NGOpt peaks at eval 157 after one hour but ends at 64 after about 245 design rounds, while FastTD3 stays near 77 and CMA-ES near 7.PPO+NGOpt’s train return stays near 13, far below eval because most rounds train on exploratory setups.Source: Figure 25 · p. 19
  • On RacingLevine, PPO+NGOpt peaks at 63 within the first hour and declines to 40, while FastTD3 ends with more completed laps (37% vs. 20%).The early design-search gains that are not retained recur on a second track.Source: Figure 26 · p. 19
  • BallCatcher-Pitch methods converge to visibly different final bodies; FastTD3 and LOKI also vary across seeds, while CMA-ES designs look alike.Similar returns can therefore arise from different morphologies.Source: Figure 27 · p. 19

D LLM CO-DESIGN

The PPO+LLM co-design loop uses a language model to propose embodiments, but it performs poorly against PPO+NGOpt on BallCatcher-Pitch despite receiving more design information. The LLM’s design proposals are often unusable or impractical, and the methods update their best designs on different schedules.

D LLM CO-DESIGN

  • PPO+LLM uses Gemma 4 31B as the outer-loop design optimizer in a PPO loop, with the controller conditioned on embodiment parameters.The setup mirrors PPO+NGOpt, replacing NGOpt with the LLM.Source: D LLM CO-DESIGN · p. 20
  • Each design round uses 16 LLM-proposed templates, each perturbed into 1,024 variants, matching PPO+NGOpt’s 16,384 designs per round.Both methods evaluate the single best-scoring design found so far; the prompt informs the LLM of the perturbation scheme.Source: D LLM CO-DESIGN · p. 20
  • Despite receiving environment details and embodiment-parameter context unavailable to other baselines, PPO+LLM performs poorly and often proposes impractical, over-engineered designs.The best design was updated three and four times in the two PPO+LLM runs, versus one to 15 times across PPO+NGOpt seeds; for seed 0, updates occurred at 0.21, 0.64, and 2.84 h for PPO+LLM and 1.91 h for PPO+NGOpt.Source: D LLM CO-DESIGN · pp. 20, 22, Figure 31 · p. 22
  • PPO+NGOpt rises from ≈0.08 to 0.43 eval return, while both PPO+LLM runs stay at or near zero across comparable PPO iterations.Between 27% and 36% of generated LLM designs were unusable.Source: Figure 30 · p. 22, D LLM CO-DESIGN · p. 22

E API ILLUSTRATION

The API illustration presents an interface between co-design algorithms and environments that preserves the established Gymnasium structure.

E API ILLUSTRATION

  • The API functions as an interface between co-design algorithms and environments while minimizing deviations from Gymnasium.The stated aim is broad compatibility with established frameworks.Source: E API ILLUSTRATION · p. 23
  • Figure 32 is a flowchart of the Co-design Gym API.Source: Figure 32 · p. 23
Loading 2610.02366v1…