PaperlayerView exact revision on arXiv

Source-linked AI summary

MOCHA: Multi-Objective Co-Design using Hypernetwork Architectures

Varun Madabushi, Neil Janwani, Maegan Tucker

arXiv:2609.30570v1cs.RO

TL;DR

Robot co-design can require a separate control-policy training run for each design, while universal policies tied to a fixed reward do not cover uncertain objective priorities. MOCHA uses design- and trade-off-conditioned hypernetworks to learn multi-objective policies and search for Pareto design-policy pairs; against MDMLP, it achieves greater hypervolume on both reported environments.

  • Problem

    Traditional co-design requires a time-consuming training run for each candidate design, while design-conditioned universal policies remain tied to a predefined reward function.

    Source: §I · p. 1

  • Method

    MOCHA combines multi-objective learning and design-conditioned policies in hypernetworks, then uses evolutionary search over design and trade-off inputs to find Pareto design-policy pairs.

    Source: §I · p. 1, E. Computing the Design Pareto Frontier · p. 5

  • Results

    Against MDMLP, MDH achieved 28% greater hypervolume for Cheetah and 6% greater hypervolume for the walker, with training time one-third that of MDMLP in both environments.

    Source: C. Comparison to Multi-Objective… · p. 6

  • Takeaways & Limitations

    MOCHA represents diverse design-dependent strategies and supports searching for Pareto-optimal design-policy combinations across competing objectives.

    Source: B. Analysis of the Design Pareto… · p. 6

    The design space must be isomorphic to an nd-dimensional hypercube, so the method cannot explicitly represent complex constraints such as self-collision avoidance.

    Source: §V · p. 7

Abstract

from arXiv · show

In this work, we present MOCHA, the first, to our knowledge, reinforcement learning based approach to computing a family of Pareto-optimal policies across the design space of a robot using a single network. Specifically, MOCHA leverages the hypernetwork architecture to learn a network that produces specialized network parameters optimized for a given objective and parameterized robot design; we term this a multi-objective design hypernetwork (MDH). We demonstrate the capabilities of MDHs to represent a complex family of design-dependent strategies on two distinct robot morphologies, each with six design dimensions and across 2-3 objectives. Moreover, we propose an approach for efficiently producing a Design Pareto set using evolutionary search of the learned policy network, generating the optimal design-policy combination for each objective prioritization. Lastly, we provide an efficient method for computing generalist robot designs which achieve the best cumulative performance across the entire set of objectives.

I. INTRODUCTION

Robot co-design jointly optimizes mechanics and control, but traditional approaches require a separate training run per candidate design, while reward-conditioned universal policies depend on a predefined objective. MOCHA combines multi-objective learning with design-conditioned hypernetworks to represent policies across designs and objective trade-offs.

I. INTRODUCTION

  • Jointly optimizing a robot’s mechanical and control parameters can outperform designing them separately, but traditional bilevel methods require a time-consuming training run for each candidate design.The outer loop selects a design and the inner loop selects a controller.Source: §I · p. 1
  • A design-conditioned universal policy flattens co-design into one search loop, but its behavior remains tied to a predefined reward and task prioritization.Once trained, it can express only behavior consistent with that reward.Source: §I · p. 1
  • MORL learns policies that trade off multiple objectives, but had not been extended to design-conditioned policies for robots with varying morphologies.A Pareto set can represent alternative objective trade-offs without retraining for multiple tasks.Source: §I · p. 1
  • MOCHA combines MORL and design conditioning in a Multi-Objective Universal Policy, using value and policy hypernetworks to generate specialized parameters for designs and objective trade-offs.The paper calls this hypernetwork class a Multi-Objective Design Hypernetwork and uses it to search for a Pareto frontier of design-policy pairs.Source: §I · p. 1
  • The work contributes algorithms for representing multi-objective policies across robot designs, computing Pareto design-policy sets, and finding generalist designs with the best cumulative performance across objectives.It also provides an open-source, GPU-compatible implementation and benchmarks.Source: §I · p. 2

II. RELATED WORK

Prior work addresses continuous Pareto policy families and reusable policies across domains, while robot co-design methods have typically remained single-objective. MOCHA builds on hypernetwork-based MORL and multi-domain ideas to apply multi-objective policies across robot designs.

II. RELATED WORK

  • Robot co-design methods include bilevel optimization, evolutionary search, designer policies, graph heuristics, and value-function-guided design search, but these methods use design-input networks for single-objective tasks.Multi-objective co-design examples include graph-heuristic search with MPC control and MO-BBO, which has not been demonstrated for high-dimensional control parameterizations.Source: §II · p. 2

A. Multi-Objective Reinforcement Learning

  • MORL learns a continuous Pareto set of policies for vector-valued rewards, and hypernetworks have represented such policy families parameter-efficiently; MOCHA builds on MORLAX to extend this approach across robot designs.AMOR and PD-MORL condition a single policy on trade-offs, while HYPER-MORL, PSL-MORL, and MORLAX use hypernetworks.Source: A. Multi-Objective Reinforcement… · p. 2

B. Multi-Domain Reinforcement Learning

  • Multi-domain RL trains reusable policies across domains, including robot morphologies, while prior hardware-conditioned policies generally keep the robot’s kinematic structure and observation and action spaces fixed.Hypernetworks have also been used in meta-learning to represent diverse task solutions while sharing parameters.Source: B. Multi-Domain Reinforcement Learning · p. 2

C. Co-Design

Robot co-design methods jointly select mechanical and control parameters, but existing approaches are largely single-objective. MOCHA frames design-aware control over a normalized morphology space and trains a multi-objective policy across designs and trade-offs.

C. Co-Design

  • Existing co-design approaches optimize mechanical and control parameters using methods such as evolutionary search, designer policies, graph heuristics, or value functions, but are limited to single-objective learning.Multi-objective alternatives include graph-heuristic search and Bayesian optimization, with the latter not demonstrated for high-dimensional control parameterizations.Source: C. Co-Design · p. 2
  • The MO-UP training loop samples M designs and K trade-offs, rolls out those combinations, and uses their rewards to update value and policy networks with Multi-Objective PPO.This trains the policy across designs and trade-offs rather than selecting a separate controller for each combination.Source: Figure 2 · p. 3

A. Problem Setup

  • The problem setup defines a design-aware MOMDP over a normalized design space D = [0, 1]^nd, with design-dependent transitions and a vector-valued reward.The trade-off vector w specifies relative priorities among reward components and lies on the (nr − 1)-dimensional simplex.Source: A. Problem Setup · p. 3

B. Hypernetwork Architecture

MOCHA uses actor and critic hypernetworks to generate policy and value-network parameters from design and trade-off context. Its multi-objective learning objective optimizes weighted expected returns and evaluates the resulting Pareto frontiers by hypervolume and spacing.

B. Hypernetwork Architecture

  • Actor and critic hypernetworks map design and trade-off context to parameters of policy and value networks, using an encoder and affine transformation to represent specialized controllers.The task encoder maps context into a higher-dimensional embedding; the trained architecture represents relevant policies on a lower-dimensional manifold.Source: B. Hypernetwork Architecture · p. 3

C. Multi-Objective Reinforcement Learning

  • MORL maximizes the scalarized expected return wT Jπ, with vector-valued rewards and values; a multi-objective extension of PPO trains the policy to represent Pareto-optimal trade-offs.A policy is Pareto optimal when no other policy dominates it across all reward components.Source: C. Multi-Objective Reinforcement… · p. 3
  • Pareto-frontier quality is measured by hypervolume relative to the origin and by spacing, the standard deviation of each frontier point’s nearest-neighbor distance in reward space.With the origin as reference, rewards must be positive; high-quality frontiers have high hypervolume and low spacing.Source: C. Multi-Objective Reinforcement… · p. 3

D. Multi-Objective Design-Conditioned Hypernetwork

The proposed MDH represents a multi-objective universal policy across robot designs and is trained to maximize scalarized reward across the design space.

D. Multi-Objective Design-Conditioned Hypernetwork

  • The MDH architecture represents a multi-objective universal policy across robot designs.It is trained to maximize wᵀR over every design in D.Source: D. Multi-Objective Design-Conditioned… · pp. 3–4
  • The procedure initializes sampled designs, trade-offs, and trials, with zero-initialized MDH weight matrices and Kaiming-initialized biases and task encoders.The authors report that this initialization empirically promotes policy diversity.Source: 1) Initialization · p. 4

2) Sampling and Rollout:

Training samples objective trade-offs and robot designs for parallel rollouts, then updates the hypernetworks using design-grouped advantage normalization to avoid mode collapse.

2) Sampling and Rollout:

  • Each epoch samples trade-offs, including simplex corners, and Sobol-sequence designs to instantiate parallel environments for rollouts.Including the simplex corners guarantees exploration of extreme trade-offs.Source: 2) Sampling and Rollout · p. 4
  • Rollouts use a policy conditioned on each sampled trade-off and design, producing a dataset of design-conditioned state-action transitions.The dataset entries are indexed by design, trade-off, and trial.Source: 2) Sampling and Rollout · p. 4
  • The method estimates a vector-valued advantage for each trajectory by applying generalized advantage estimation independently to each objective.These advantages are used in the subsequent hypernetwork update.Source: 3) Gradient Update · p. 4

3) Gradient Update:

  • Normalizing advantages without regard to design causes mode collapse; the method instead normalizes advantages within each design group before updating the hypernetworks.The update uses stochastic gradient descent to minimize the stated losses.Source: 3) Gradient Update · p. 4

E. Computing the Design Pareto Frontier

The Design Pareto Set pairs each target objective trade-off with a design and conditioning trade-off that maximizes its scalarized return; evolutionary search approximates this set.

E. Computing the Design Pareto Frontier

  • For each target trade-off, the Design Pareto Set selects a design and conditioning trade-off whose induced policy maximizes scalarized return.The conditioning trade-off is searched independently of the evaluation trade-off and is often nearly identical to it in practice.Source: E. Computing the Design Pareto Frontier · p. 4
  • Evolutionary search is used because random sampling becomes computationally infeasible in moderate-to-high-dimensional design and trade-off spaces.The search seeks a dense collection of target trade-off, conditioning trade-off, and design tuples.Source: E. Computing the Design Pareto Frontier · pp. 4–5
  • Figure 4 shows Cheetah Design Pareto frontier slices along Run–Energy, Run–Height, and Energy–Height axes.Source: Figure 4 · p. 5
  • NSGA-III searches the design space and simplex-constrained trade-off space, repairing sampled points back onto the simplex after genetic mixing.Batched search fits the approach's massively parallelized simulation.Source: E. Computing the Design Pareto Frontier · p. 5
  • With parallelized simulation, NSGA-III produces denser, higher-quality Design Pareto sets than random sampling; frontier hypervolume also indicates performance and convergence.Convergence is associated with the hypervolume plateauing.Source: E. Computing the Design Pareto Frontier · p. 5

F. Finding a Generalist Design

A generalist design is intended for settings where one robot must handle unknown missions; the method seeks the design whose multi-objective policy has the highest hypervolume.

F. Finding a Generalist Design

  • A generalist design is motivated by missions in which only one robot can be fabricated before the mission is known.Such a design should handle a large volume of potential missions.Source: F. Finding a Generalist Design · p. 5
  • The method defines a generalist design as one whose multi-objective policy has high hypervolume, reflecting high reward across many possible trade-offs.Source: F. Finding a Generalist Design · p. 5
  • Figure 5 presents the Walker's two-objective Design Pareto frontier and rollouts of Pareto-optimal designs.Source: Figure 5 · p. 5
  • Trust Region Bayesian Optimization is used to find the design with the highest hypervolume because its computation is expensive and non-differentiable.Computing it requires forward-simulating policies across many sampled trade-offs and designs.Source: F. Finding a Generalist Design · p. 5

IV. RESULTS

The results section evaluates MOCHA on Cheetah and Walker robots and frames comparisons around Pareto-frontier quality, design-conditioned MLPs, and how practitioners can interpret design trade-offs.

IV. RESULTS

  • The evaluation asks whether the MDH represents optimal design–trade-off policies, improves on a design-conditioned MLP, and helps practitioners interpret the resulting Pareto set.The study evaluates robots based on Cheetah and Walker environments.Source: §IV · p. 5, A. Implementation Details · p. 5

A. Implementation Details

  • Cheetah has six link-length design dimensions and Run, Height, and Energy rewards; Walker also has six segment-length dimensions and Run and Energy rewards.Both designs use a scaling procedure based on segment lengths.Source: A. Implementation Details · pp. 5–6
  • For both Cheetah and Walker, the Hypernetwork achieves higher hypervolume and lower spacing than the MLP on Design Pareto Frontier metrics.Figure 6 compares Cheetah (top) and Walker (bottom).Source: Figure 6 · p. 6
  • The environments use MO-Playground with MuJoCo JAX for parallel computation, while pymoo supplies frontier metrics and NSGA-III and botorch supplies TuRBO.Source: A. Implementation Details · p. 6
  • The MDH uses a task encoder to map design and trade-off context to a 128-dimensional embedding, with separate policy and value hypernetworks.The task encoder has two 1024-unit hidden layers; each hypernetwork has three hidden layers sized [256, 256, 128].Source: A. Implementation Details · p. 6

B. Analysis of the Design Pareto Frontier

MOCHA produces design-dependent Pareto frontiers and generalist or specialist designs. Its hypernetwork policy outperforms a parameter-matched multi-objective MLP on reported hypervolume and training time.

B. Analysis of the Design Pareto Frontier

  • The Cheetah frontier includes policies that jump for Height, sit for Energy, and run for speed; long legs favor Height, while short legs run fast through lower inertia.Almost every design performs well on Energy because doing nothing can maximize that reward.Source: B. Analysis of the Design Pareto… · p. 6
  • For Walker, symmetric designs dominate most reward combinations, except that a much shorter leg is favored for a purely Energy-maximizing strategy.The authors report that the algorithm identifies relevant subspaces of the design space.Source: B. Analysis of the Design Pareto… · p. 6
  • The run-optimized Cheetah outperforms the other selected designs on running and energy savings, while the generalist performs approximately like the height-optimized design.Figure 7 shows fixed-design Pareto frontiers for the generalist and selected specialists.Source: Figure 7 · p. 6

C. Comparison to Multi-Objective Design-Conditioned MLP Policy

  • 28% greater hypervolume for Cheetah and 6% greater hypervolume for Walker were achieved by the MDH over the MDMLP after 3 billion timesteps; MDH training took one-third as long.The MDMLP performed similarly on Height and Energy but failed at Run; the authors attribute the training-time difference to using the hypernetwork once per rollout and the smaller policy at each timestep.Source: C. Comparison to Multi-Objective… · p. 6

B. Analysis of the Design Pareto Frontier

  • The generalist Cheetah maximizes the hypervolume of its associated frontier, while each task-optimized design maximizes scalar reward for its objective prioritization.Figure 8 presents the procedurally generated generalist and specialist robots.Source: Figure 8 · p. 7

D. Comparison of Generalist and Specialists

The paper uses frontier analysis to relate Cheetah morphology to performance and identifies a generalist design similar to the height specialist. It also identifies limits involving policy scoring, design constraints, and sim-to-real transfer.

D. Comparison of Generalist and Specialists

  • Short legs help the run-optimized Cheetah run faster and save energy, while the generalist is morphologically and behaviorally similar to the height-optimized design.The authors describe the generalist as a nonobvious design found through MOCHA's multi-objective formulation.Source: D. Comparison of Generalist and… · p. 7

V. LIMITATIONS

  • If the universal policy misses optimal strategies, it may overrate a suboptimal design; more studies are needed to understand hypernetwork scaling.Training a larger hypernetwork is suggested as a possible remedy.Source: §V · p. 7
  • The design-space representation must be isomorphic to an n-dimensional hypercube, so the algorithm cannot explicitly represent constraints such as self-collision avoidance.The authors suggest adding a cost for design-constraint violations as a possible solution.Source: §V · p. 7
  • The work does not consider sim-to-real transfer of learned policies.Domain randomization and a design-space trust region are proposed as future directions.Source: §V · p. 7

VI. CONCLUSION AND FUTURE WORK

  • Across two environments with six-dimensional design spaces and multiple objectives, the hypernetwork policy achieves higher hypervolume in fewer training iterations and less wall-clock time.MOCHA also computes a generalist design with maximal performance across all objectives.Source: §VI · p. 8
Loading 2609.30570v1…