PaperlayerView exact revision on arXiv

Source-linked AI summary

PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots

Amr Mousa, Rifny Rachman, Neil Karavis, Michele Caprio, Richard Allmendinger

arXiv:2610.01260v1cs.ROcs.AIcs.HCcs.LGcs.SYeess.SY

TL;DR

Quadrupedal locomotion must balance tracking, stability, and efficiency, but conventional RL fixes those priorities in a training-time scalar reward. PROMO conditions one policy on runtime preferences while keeping locomotion priors fixed. Simulation and zero-shot Unitree Go2 results show preference-responsive behavior, including measured reductions relative to a balanced setting.

  • Problem

    Conventional RL fixes the trade-off among command tracking, stability, and energy efficiency in a scalar reward, although priorities may need to change during deployment.

    Source: §I · p. 1

  • Method

    PROMO conditions a single locomotion policy on deployment-time preferences over tracking, stability, and efficiency while keeping embodiment-specific locomotion priors fixed.

    Source: §I · p. 2

  • Results

    On a Unitree Go2, preference changes reduce specific energy by up to 30.4%, position error by up to 38.7%, and peak body-attitude deviation by up to 59.0% relative to the balanced setting.

    Source: §VII · p. 11

  • Takeaways & Limitations

    The results support semantic MORL as a deployment mechanism that exposes the objective trade-off as a runtime control input rather than only constructing an offline Pareto set.

    Source: §VII · p. 11

    The hardware study uses five trials per condition and four anchor preferences, demonstrating preference modulation rather than dense hardware controllability or statistical significance.

    Source: C. Limitations · p. 11

Abstract

from arXiv · show

Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/.

I. Introduction

PROMO makes locomotion priorities selectable at deployment, separating operator-facing objectives from fixed locomotion priors. The paper motivates this interface against fixed-reward control and evaluates preference responsiveness in simulation and on hardware.

I. Introduction

  • PROMO conditions a single policy on runtime preferences over tracking, stability, and efficiency while keeping embodiment-specific locomotion priors fixed.The preference vector represents behavioral intent rather than implementation-level reward weights.Source: §I · p. 2
  • In simulation, one policy specializes across tracking, stability, and efficiency, with lower failure rates than evaluated preference-conditioned and specialist baselines.A 100-preference study places the anchor behaviors within a broader controllable interface; the hardware study reports directionally consistent changes across five trials per condition.Source: §I · p. 2
  • Preference changes alone reduce specific energy by up to 30.4%, position error by up to 38.7%, and peak body-attitude deviation by up to 59.0% on a zero-shot Unitree Go2 policy.The comparison is with a balanced configuration.Source: §I · p. 2

A. Learned Quadruped Locomotion

  • Learned quadruped controllers commonly fix reward-level priorities even when commands and observations can change at runtime.This motivates making the objective trade-off selectable after training rather than requiring reward redesign, retraining, or controller switching.Source: §I · p. 1, A. Learned Quadruped Locomotion · p. 2

B. Reward Design and Runtime Behavioral Interfaces

  • PROMO keeps posture, contact, and smoothness requirements as fixed priors instead of exposing reward-shaping terms as operator controls.Its preference dimensions represent task-level trade-offs.Source: B. Reward Design and Runtime… · pp. 2–3

C. MORL Preference Conditioning

D. From Pareto Coverage to Behavioral Control

The paper argues that Pareto coverage alone is insufficient to establish a reliable physical control interface. PROMO evaluates semantic preferences while keeping embodiment-level requirements fixed.

D. From Pareto Coverage to Behavioral Control

  • Strong Pareto performance does not by itself establish that preferences reliably control behavior on a physical robot.Standard MORL benchmarks omit failures, dense rewards, embodiment-specific priors, sim-to-real transfer, and hardware deployment.Source: D. From Pareto Coverage to Behavioral… · p. 3
  • Hypervolume summarizes convergence and diversity, but does not show whether changing preferences predictably changes behavior.The paper therefore motivates evaluation of preference sensitivity, monotonicity, objective alignment, and behavioral consistency.Source: D. From Pareto Coverage to Behavioral… · p. 3
  • Physical deployment evaluation should consider preference controllability, robustness, and semantic interpretability alongside Pareto quality.PROMO combines these concerns with semantic objectives, locomotion-specific priors, and physical deployment.Source: D. From Pareto Coverage to Behavioral… · p. 3

A. Problem Formulation and Semantic Interface

  • PROMO models locomotion as a partially observed multiobjective decision process, with semantic objectives for tracking, stability, and efficiency.A sampled preference is held fixed within each rollout and supplied as a runtime input; embodiment-level regularizers remain fixed priors outside the semantic objective space.Source: A. Problem Formulation and Semantic… · pp. 3–4

B. Architecture Overview

PROMO uses distinct privileged and history representations in training, then deploys an actor conditioned on preference and deployable inputs. Its semantic value learning keeps locomotion priors separate from preference objectives.

B. Architecture Overview

  • The critic receives privileged simulator inputs and predicts one value per semantic objective, while the actor uses deployable observations, history representation, and preference to output joint-position targets.The preference enters the critic representation through the privileged encoder and is applied explicitly when objective-wise values are scalarized.Source: B. Architecture Overview · p. 4
  • The history representation is aligned to a detached privileged target, and the velocity estimator is trained from the history latent and latest five observations.The history encoder uses alignment and velocity losses; the velocity estimator uses the velocity loss.Source: B. Architecture Overview · p. 4
  • PROMO adapts TAR with a multilayer-perceptron history encoder, removes the forward-dynamics auxiliary model, and adds preference conditioning throughout the actor–critic architecture.The resulting architecture is described as lightweight for preference-conditioned flat-terrain locomotion while retaining privileged representation alignment.Source: B. Architecture Overview · p. 4

C. Semantic Objective Factorization and Value Learning

  • Semantic values are scalarized by the preference for the PPO update, while the fixed locomotion prior remains independent of preference and is not exposed as a preference axis.The privileged encoder and critic are discarded at deployment; the history encoder, velocity estimator, and actor are retained.Source: C. Semantic Objective Factorization… · pp. 4–5

D. Preference-Conditioned Policy Optimization

PROMO uses the preference to scalarize semantic values and rewards, while retaining a fixed locomotion prior. It combines these terms in a GAE actor advantage and optimizes the actor with a clipped PPO objective.

D. Preference-Conditioned Policy Optimization

  • PROMO scalarizes its semantic value heads and semantic reward using the preference.The scalarized critic and reward are used in actor optimization.Source: D. Preference-Conditioned Policy… · p. 5
  • The actor advantage combines the preference-scalarized semantic critic with the fixed prior through GAE.With λ_GAE = 0.95, the prior contribution is a biased, lower-variance estimate rather than an unbiased estimate of its full return.Source: D. Preference-Conditioned Policy… · p. 5
  • The actor uses the resulting advantage in a clipped PPO surrogate objective.The behavior-policy parameters and PPO clipping width define the surrogate.Source: D. Preference-Conditioned Policy… · pp. 5–6

E. Preference Responsiveness and Deployment

PROMO adds action-diversity regularization because preference conditioning alone does not ensure that the policy visibly uses the preference. At deployment, the preference is the sole behavior-modulation input to the retained modules.

E. Preference Responsiveness and Deployment

  • An action-diversity regularizer encourages different preferences to produce different actions at the same observation.The comparison holds the history latent and velocity estimate detached.Source: E. Preference Responsiveness and… · p. 6
  • The regularizer uses symmetric KL divergence between diagonal-Gaussian action distributions, summed across action dimensions.Its coefficient is λ_div = 10^-5.Source: E. Preference Responsiveness and… · p. 6
  • At deployment, the retained history encoder, velocity estimator, and actor require observation history and preference; preference alone modulates behavior.The privileged encoder, privileged state, ground-truth base velocity, and critic are discarded.Source: E. Preference Responsiveness and… · p. 6

IV. Experimental Evaluation

The experiments compare PROMO with single-objective controllers, MORL methods, and scalar specialists, using separate training and shared semantic evaluations. PROMO reaches higher reported training reward than the named baselines while retaining locomotion performance comparable to specialists on episode length.

A. Setup and Baselines

  • Training uses a 12-DoF Go2 in 4096 parallel flat-terrain environments with domain randomization; comparisons include RMA, TAR, MOPPO, DPMORL, and PROMO-SORL specialists.The evaluation also describes shared controls to keep comparisons methodologically consistent.Source: A. Setup and Baselines · p. 6

B. Evaluation Metrics

  • Dense preference evaluation measures objective-wise Spearman controllability and range-normalized hypervolume, alongside response trends and locomotion failures across 100 preferences.The study distinguishes four-anchor controllability for baseline comparisons from dense controllability for PROMO.Source: B. Evaluation Metrics · p. 7
  • Training reward and episode length are reported over three seeds, separately from common semantic metrics including rewards, fall rate, and lower-tail body-attitude risk.The balanced evaluation also reports requested-objective scores, aggregate fall rate, and scalarized return; Table I identifies the balanced evaluation metrics and their better directions.Source: B. Evaluation Metrics · p. 6, C. Training and Baseline Comparison · p. 7, Table I · p. 7

C. Training and Baseline Comparison

  • 27.5 reward for PROMO exceeds 25.3 for PROMO-SORL, 24.8 for RMA, 24.2 for TAR, and 23.5 for MOPPO.PROMO continues improving after locomotion stabilizes; against MOPPO, it also has lower variance.Source: C. Training and Baseline Comparison · p. 7
  • PROMO reaches the episode horizon after 550–650 steps, near the balanced specialist (600–700) and tracking specialist (400–500), while the efficiency specialist takes 1700–1900.The passage attributes the efficiency specialist's later convergence to effort minimization being a weak gait-discovery signal; PROMO learns four semantic behaviors at roughly the optimization cost of one balanced controller.Source: C. Training and Baseline Comparison · p. 7

D. Common Semantic Evaluation Against Baselines

Across common semantic evaluation and dense preference sampling, PROMO combines competitive balanced locomotion with a single policy that expresses multiple requested behaviors. Hardware evaluation uses the same deployed checkpoint across preference and command conditions, with gait differences visible on the Unitree Go2.

D. Common Semantic Evaluation Against Baselines

  • PROMO records the highest mean tracking reward (18.703 vs. 16.225 for PROMO-SORL), lower mean fall rate than TAR and RMA (7.4% vs. 14.5% and 18.2%), and the lowest mean CVaR tilt (0.105).Reward rankings vary by objective: TAR has the highest mean aggregate stability reward, while RMA has the highest mean efficiency reward.Source: D. Common Semantic Evaluation Against… · p. 7
  • PROMO exceeds specialists on requested tracking (20.099 vs. 17.833) and efficiency (−2.189 vs. −2.942), while the stability specialist scores higher (−1.916 vs. −2.245).PROMO's aggregate fall mean is 8.4%, 4.2 percentage points below the specialist-family mean of 12.6%.Source: D. Common Semantic Evaluation Against… · p. 7, Table II · p. 8
  • Against MOPPO, PROMO records higher mean scalarized return (3.964 vs. 2.212), an 11.0-percentage-point lower mean fall rate, and higher four-anchor controllability (0.662 vs. 0.492).DPMORL and PROMO have comparable mean requested efficiency (−2.133 vs. −2.189), while DPMORL's mean fall rate is 26.6%.Source: D. Common Semantic Evaluation Against… · p. 7

E. Pareto Coverage and Preference Controllability

  • PROMO converges to higher reward than shared-scale locomotion and MORL references while matching the balanced specialist's episode-length convergence.Specialist rewards use differently scaled scalar objectives, making episode length the more informative cross-specialist comparison.Source: Figure 3 · p. 8
  • 67 of 100 sampled preference-induced policies are non-dominated; ε-dominance pruning retains 37 solutions at ε = 1%, with tracking and efficiency in the clearest conflict.Stability is partly complementary to tracking and efficiency.Source: E. Pareto Coverage and Preference… · p. 8
  • Preference-to-behavior mappings rise near-monotonically, and dense evaluation records 0.8434 controllability with a 7.50% mean fall rate.Failures concentrate near aggressive preference boundaries; efficiency has the largest per-objective response, while tracking and stability remain strongly aligned despite tighter gait coupling.Source: E. Pareto Coverage and Preference… · p. 8, Figure 4 · p. 9

A. Protocol

  • The hardware study deploys one simulation-trained PROMO checkpoint on a Unitree Go2 without retraining or controller switching, testing four preference anchors across three command regimes.The protocol comprises 12 conditions with five trials each; replay uses the same recorded trajectory for every preference.Source: §V · p. 8, A. Protocol · p. 8

V. Real-Robot Evaluation

  • On hardware, tracking-heavy preferences produce larger foot excursions, efficiency-heavy preferences compact low-clearance trajectories, and stability-heavy preferences an intermediate gait with increased clearance.The figure shows chronophotographic overlays and 2-s tracked foot trajectories for the same policy.Source: Figure 5 · p. 9

B. Preference-Conditioned Behavior

PROMO uses one fixed locomotion policy whose semantic preference input shifts hardware behavior toward tracking, stability, or efficiency, while exposing trade-offs among those objectives. The discussion frames this as a deployment interface, not only an offline Pareto set, while noting limits to the evaluation scope.

B. Preference-Conditioned Behavior

  • Efficiency preference reduced mean specific energy in replay, fast, and slow regimes; tracking preference lowered position MAE, while stability preference reduced peak attitude deviation.Efficiency reductions were 17.9%, 30.4%, and 11.3%; tracking reductions were 24.2%, 25.5%, and 38.7%; stability reductions were 19.4%, 59.0%, and 8.7%, respectively, relative to balanced.Source: B. Preference-Conditioned Behavior · p. 9, Figure 6 · p. 10, Table III · p. 10
  • On low-traction terrain, the balanced preference recovered viable locomotion after whole-body, rotational, and direct-limb disturbances, including an approximately 230° yaw rotation.Because disturbance forces were not instrumented, these sequences are qualitative stress tests, not force-normalized robustness measurements.Source: B. Preference-Conditioned Behavior · p. 10, Figure 7 · p. 11

A. Semantic Preferences as a Deployment Interface

  • The semantic preference is a runtime input to one policy, separating operator-facing objectives from fixed embodiment-level locomotion priors.The policy's operator-facing objective weights are meaningful for deployment decisions, but do not causally explain every joint command.Source: A. Semantic Preferences as a… · pp. 10–11

B. From Pareto Optimization to Deployable Control

  • Deployment-interface quality differs from individual objective extrema: PROMO combines objective specialization, lower failures, preference response, and one-policy deployment.The discussion recommends evaluating utility, lower-tail outcomes, controllability, and fall rate together; aggregate metrics can improve while the operator-facing interface degrades.Source: B. From Pareto Optimization to… · p. 11
  • Broad preference coverage may diversify locomotion behaviors, but its causal role remains untested because an ablation of coverage width is future work.The discussion presents this as a plausible mechanism for balanced-preference performance, not an established explanation.Source: B. From Pareto Optimization to… · p. 11

C. Limitations

  • PROMO addresses operator-induced changes in objective priorities, not changing transition dynamics, hidden regime shifts, or online adaptation.The hardware study used five trials per condition and four anchor preferences, demonstrating modulation rather than dense hardware controllability or statistical significance.Source: C. Limitations · p. 11

VII. Conclusion

  • 30.4%, 38.7%, and 59.0%: preference changes reduced specific energy, position error, and peak body-attitude deviation, respectively, on the same Unitree Go2 policy relative to balanced.The policy transferred zero-shot; preference changes alone produced these reductions.Source: §VII · p. 11

Appendix A Reward and Preference Specification

PROMO groups reward terms into tracking, stability, and efficiency objectives, while keeping fixed locomotion priors outside the deployment preference vector. The appendix specifies the measures and shaping terms used for these groups and priors.

A. Semantic Reward Decomposition

B. Per-Step Objective Definitions

  • The per-step objective definitions use exponential kernels for tracking and specify stability and efficiency penalties; sums are over configured joints or action dimensions, not dimension-normalized means.The appendix also states a 0.5 kernel scale for planar velocity and yaw rate.Source: B. Per-Step Objective Definitions · p. 13
  • Fixed locomotion priors are optimized during training but remain independent of the semantic preference space.The appendix describes feet air-time, contact-balance, and joint-position priors, including stronger posture regularization near stationary operation.Source: B. Per-Step Objective Definitions · pp. 13–14

C. Preference Space and Sampling

Training preferences are sampled within a truncated semantic simplex, while PROMO retains nominal reward scales so the balanced preference corresponds to the locomotion environment's default trade-off. The appendix also provides the implementation and training configuration.

C. Preference Space and Sampling

  • Training samples preferences from a Dirichlet distribution mapped to the simplex interior with a minimum component of 0.1, so no semantic objective disappears.Each sampled preference is held fixed for the rollout used to estimate critic targets and actor advantages.Source: C. Preference Space and Sampling · p. 14

D. Objective Scaling and Semantic Interpretation

  • PROMO retains nominal reward scales: the balanced anchor recovers the environment's tuned trade-off, and other preferences act as relative semantic modifiers.The appendix cautions that scalar utility values are tied to nominal scales and are not scale-normalized objective importance.Source: D. Objective Scaling and Semantic… · p. 14
  • Tracking supplies the main locomotion-driving signal, while stability and efficiency mainly regularize; scaling the latter two by 2–5× relative to tracking degraded locomotion quality and preference-conditioned behavior.Source: D. Objective Scaling and Semantic… · p. 14

Appendix B Architecture and Training Configuration

B. Rollout and Optimization Configuration

The appendices describe PROMO's rollout and evaluation protocols, then analyze objective/prior factorization and the order of scalarization and PPO clipping. The ablations show that aggregate reward or hypervolume alone does not establish a controllable, reliable deployment interface.

B. Rollout and Optimization Configuration

  • The rollout and optimization appendix describes objective-wise TD targets, a scalarized semantic critic baseline, fixed-prior reward in the GAE residual, and batch-normalized advantages before PPO.A configuration table lists rollout and optimizer constants.Source: B. Rollout and Optimization… · p. 14, Table B.3 · p. 15

A. Simulation Evaluation Protocol

  • Simulation protocols use scripted conditions for representative preferences and 100 sampled preferences; 67 behaviors are non-dominated under exact Pareto dominance, with 37 retained at ε = 1%.Representative evaluations reuse a disturbance script; dense evaluations produce preference-performance maps under standardized conditions.Source: A. Simulation Evaluation Protocol · p. 15, Table C.2 · p. 15

B. Hardware Evaluation Protocol

  • Hardware evaluation distinguishes command-matched replay, dynamic fast locomotion, and low-speed slow operation; replay uses the same recorded commands across preferences.The regime descriptions distinguish command matching, joystick-driven dynamics, and near-standstill transitions.Source: B. Hardware Evaluation Protocol · p. 15, Table C.3 · p. 16

A. Objective/Prior Factorization

  • Keeping locomotion priors fixed outside semantic objectives gives the selected balance of controllability and reliability; alternative formulations can improve aggregate metrics while degrading the interface.Moving priors inside the objectives slows learning and raises fall rate from 0.060 to 0.165; removing priors or exposing them as a fourth objective also degrades controllability or reliability.Source: A. Objective/Prior Factorization · p. 16, Table D.1 · p. 16, Figure D.1 · p. 16

B. Scalarization Before PPO Clipping

  • Semantic scalarization and PPO clipping do not generally commute, so PROMO scalarizes semantic critic values and includes fixed-prior reward in the actor residual before applying the clipped surrogate.Fixed locomotion priors do not receive critic heads.Source: B. Scalarization Before PPO Clipping · p. 16

C. Diversity Regularization

Diversity regularization can improve preference-conditioned behavior, but its effects depend on the coefficient and metric. The sweep favors action-level regularization as a more reliable interface-level mechanism, with coefficient selection guided by controllability, robustness, and Pareto coverage.

C. Diversity Regularization

  • Action diversity separates policy output distributions for different preferences at the same observation and detached history latent; latent diversity regularizes the preference-conditioned representation.The coefficient α sets the target scale between representation distance and preference distance.Source: C. Diversity Regularization · p. 16
  • In the action/latent sweep, action diversity at 10−6 gives the largest controllability gain, increasing the metric from 0.803 to 0.879; no coefficient dominates every metric.Sensitivity results are averaged over three seeds and shown as relative improvement over no diversity, exposing metric-specific optima and coefficient sensitivity.Source: C. Diversity Regularization · p. 16, Figure D.2 · p. 17
  • At 10−5, action diversity broadens Pareto spread (0.657 vs. 0.561 hypervolume), reduces falls (0.098 →0.083), and slightly improves final reward and episode length.Table D.2 reports final-window training means and evaluation metrics averaged over three seeds.Source: Table D.2 · p. 17, C. Diversity Regularization · p. 17

Appendix E Baseline Implementations

  • All simulation baselines share flat-terrain evaluation with a 12-DoF quadruped, 50 Hz joint-position actions, 20 s episodes, matched termination, and PPO-style training.Commands span [−1, 1] m/s planar velocity and [−1, 1] rad/s yaw rate; evaluation uses the semantic decomposition in Appendix A.Source: Appendix E · p. 17

A. Single-Objective Locomotion Baselines

  • TAR and RMA are fixed-trade-off controllers without deployment preference inputs; TAR uses history-actor auxiliary losses, while RMA distills privileged environment information into a history-based adaptation module..Source: A. Single-Objective Locomotion Baselines · p. 17

B. MORL Baselines

  • MOPPO uses one preference-conditioned PPO policy, whereas DPMORL archives four fixed-utility policies and divides the interaction budget equally among them.MOPPO receives a three-dimensional preference held fixed per rollout; DPMORL uses scalar PPO on incremental utility rewards.Source: B. MORL Baselines · p. 17

C. PROMO-SORL Specialists

  • PROMO-SORL compares PROMO with independently trained balanced, tracking-, stability-, and efficiency-focused scalar policies without preference conditioning or parameter sharing.These specialists retain fixed locomotion priors and omit objective-wise semantic critics.Source: C. PROMO-SORL Specialists · p. 17
Loading 2610.01260v1…