Source-linked AI summary
One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?
Ziqian Huang, Yuzhuo Chen, Shimin Zhang, Junyang He, Xinyue Annie Yang, Animesh Garg
TL;DR
Preference-conditioned policies enable runtime style changes, but conditioning alone does not ensure strong, selective changes that preserve task performance. We evaluate steerability and compare aggregation strategies across two humanoid-control frameworks; no fixed strategy dominates, with earlier aggregation widening behavioral range but increasing leakage and later aggregation improving selectivity.
Problem
Preference conditioning does not guarantee intended behavioral responsiveness, selectivity, and task preservation, properties not captured by aggregate return or coverage alone.
Source: §1 · p. 1Method
The study evaluates steerability with responsiveness, leakage, selectivity, and reliability measures and compares PPO-based aggregation strategies across two humanoid-control systems.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The findings motivate multidimensional steerability evaluation and investigation of adaptive methods that balance responsiveness and selectivity.
Source: §6 · p. 6, §7 · p. 7The study covers two simulated humanoid environments, a narrow task range, and a small set of hand-designed preference dimensions.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.
1 Introduction
The paper evaluates preference-conditioned steerability as more than task return: it measures intended responsiveness, unintended behavioral changes, and task preservation. Comparing aggregation strategies across humanoid-control frameworks, it finds that no strategy dominates across settings and criteria.
1 Introduction
- MORL conditions one policy on a preference vector to expose runtime objective trade-offs [Basaklar et al., 2023a, Liu et al., 2025], but conditioning alone may yield weak intended changes, unintended behaviors, or compromised task performance.Aggregate return and Pareto coverage alone do not capture these distinctions [de las Heras Molins et al., 2026].Source: §1 · p. 1
- The evaluation suite combines the PRR matrix, Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy with utility measures to capture responsiveness, cross-objective leakage, monotonicity, task preservation, and failures.It addresses whether a preference change strongly affects its intended behavior, unintentionally affects others, and preserves reliable task performance.Source: §1 · pp. 1–2
- Reward-level aggregation achieves the widest controllable range in ProtoMotions, while later aggregation achieves higher hypervolume in fixed-direction MimicKit and higher selectivity with lower cross-objective leakage across both frameworks.The experiments do not isolate why coverage rankings differ between frameworks; a preliminary MetaDrive check shows a broadly similar pattern.Source: §1 · p. 2
- The study compares reward-, advantage-, and loss-level scalarization within PPO across ProtoMotions and MimicKit, and tests PCGrad gradient surgery in ProtoMotions.The two humanoid-control systems are independently implemented; PCGrad resolves objective conflicts in gradient space.Source: §1 · p. 2
2 Background and Related work
Preference-conditioned MORL supports runtime trade-offs across objectives [Basaklar et al., 2023b, Ambadkar et al., 2026], but how training-time aggregation affects controllability and cross-objective leakage remains underexplored. This work compares aggregation levels in embodied control using measures of responsiveness, selectivity, task preservation, and failure behavior.
2 Background and Related work
- Recent humanoid-control work uses a mixture-of-experts architecture for preference-conditioned MORL [Li et al., 2026a], but fixes one aggregation point and does not measure cross-objective leakage.The present study examines how the aggregation point affects whether changing the preference input predictably changes behavior.Source: §2 · p. 2
- LLM work motivates explicit behavioral control through runtime attribute conditioning and multi-objective alignment, extending toward fine-grained value priorities and Pareto-frontier adaptation [Dong et al., 2023, Wang et al., 2024, Shi et al., 2024, Guo et al., 2025, Li et al., 2026b].Constitutional AI uses written principles during training rather than as continuous runtime control [Bai et al., 2022].Source: §2 · p. 2
- The study compares reward-, advantage-, and loss-level combination, alongside PCGrad, across two robot-control settings using responsiveness, monotonicity, selectivity, task preservation, and failure behavior.Pareto coverage and utility alone do not directly distinguish intended behavioral response from cross-objective leakage.Source: §2 · p. 2
3 Preliminary
Preference-conditioned MORL trains one shared policy that can be steered at test time by changing a preference vector over multiple objectives. The paper evaluates recovered trade-offs using hypervolume and expected utility [Hayes et al., 2022, Felten et al., 2023].
3.1 Preference-Conditioned Multi-Objective Reinforcement Learning
- A preference vector w specifies desired trade-offs among objectives in a multi-objective Markov decision process [Hayes et al., 2022, Felten et al., 2023].The setting uses vector rewards, with each dimension corresponding to one objective.Source: §3.1 · pp. 2–3
- One policy shares parameters across preferences and uses input w to specify the desired behavior at test time.Changing w steers the policy without changing its shared parameters.Source: §3.1 · p. 3
- The experiments report hypervolume (HV) for Pareto-front quality and expected utility (EU) for average utility under sampled preferences [Hayes et al., 2022, Felten et al., 2023].MORL evaluates trade-offs using non-dominated solutions rather than a single reward value.Source: §3.1 · p. 3
4 Method
The method compares four stages for aggregating preference objectives and evaluates steerability with metrics that distinguish intended responses, spillover, selectivity, and reliability.
4.1 Aggregation Strategies
- The four methods vary when K+1 objectives collapse into one actor-update signal, from reward scalarization through advantage and loss aggregation to gradient-level PCGrad conflict resolution.Reward-level aggregation leaves no per-objective structure after reward scalarization; loss-level aggregation clips objectives separately, while PCGrad resolves gradient conflicts.Source: §4.1 · pp. 3–4
4.2 Steerability Evaluation Suite
- The suite extends rank-based controllability to embodied continuous control, measuring intended response (Self-PRR), spillover (Cross-PRR), relative selectivity, and directional reliability (Rank Accuracy).Self-PRR measures response of the preference-controlled objective, while Cross-PRR measures disturbance to other objectives; Rank Accuracy tests whether preference ordering predicts achieved-reward ordering.Source: §4.2 · p. 4
5 Experiments
Aggregation level shifts the balance between responsiveness or coverage and precision, with no method leading across all environments and criteria. The preferred level varies by framework and metric.
5.1 ProtoMotions: G1 Humanoid Preference-Conditioned Locomotion
- In ProtoMotions, reward-level achieved roughly 3× the hypervolume, 2× self-responsiveness, and 2× Jtask of the other methods, but had the weakest precision.The G1 experiment used five style preferences alongside a fixed-weight heading-tracking reward. The authors hypothesize that early scalarization prevents downstream mechanisms from limiting objective-gradient dominance or leakage, whereas later aggregation preserves per-objective structure and favors cleaner separation over raw responsiveness.Source: Table 1 · p. 5, §5.1 · p. 5
- Reward-level curves were higher and saturated earlier on four of five style dimensions, while all methods converged on energy by α≈0.4.The curves show the per-dimension sweeps underlying Self-PRR; the authors note that the responsiveness-versus-precision trade-off is driven more strongly by some style dimensions than others.Source: §5.1 · p. 5, Figure 1 · p. 6
5.2 MimicKit: Cross-Framework Validation
- In the MetaDrive ablation, advantage-level achieved the highest paired expected utility and Self-PRR, while loss-level had the lowest Cross-PRR.This further supports a precision benefit from later aggregation, while the metric-optimal level remains environment-dependent.Source: §5.2 · p. 6
- In MimicKit’s fixed-direction preference-conditioned control (mean ± std over 3 training seeds at 1.2B environment steps), reward-level achieved highest mean Self-PRR (0.510) and task return; advantage-level led hypervolume and expected utility, and loss-level selectivity.Advantage-level also minimized Cross-PRR and falls; Rank Accuracy was nearly tied between the two later aggregation methods.Source: Table 2 · p. 6, §5.2 · p. 6
6 Limitations and Future Work
The study is limited to two simulated humanoid environments, a narrow range of tasks, and a small set of hand-designed preference dimensions; no strategy resolves the responsiveness–precision trade-off.
6 Limitations and Future Work
- The study covers only two simulated humanoid environments, a narrow task range, and few hand-designed preference dimensions, and no strategy resolves the responsiveness–precision trade-off.Future work should investigate adaptive objective-combination methods, gradient-conflict methods beyond PCGrad, robustness to out-of-distribution and dynamically changing preferences, and transfer to real robotic systems.Source: §6 · p. 6
7 Conclusion
Across two humanoid-control frameworks, objective-signal aggregation shapes the trade-off between steerability and cross-objective interference, and no fixed strategy dominates. The evaluation suite measures responsiveness, selectivity, utility, and failures beyond standard MORL metrics.
7 Conclusion
- The evaluation suite measures Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy alongside utility and failure metrics because standard MORL metrics miss steerability properties.It evaluates whether preference changes selectively alter intended behavior without compromising the underlying task.Source: §7 · p. 6
- Across two humanoid-control frameworks, aggregation level shapes the trade-off between responsiveness and cross-objective interference, and no fixed strategy dominates.The findings motivate multidimensional steerability evaluation and adaptive aggregation methods.Source: §7 · pp. 6–7
- In ProtoMotions, reward-level aggregation yields the widest controllable range and highest utility but greater leakage, while later aggregation and PCGrad improve selectivity at the cost of range.In MimicKit, all aggregation levels steer reliably, and later aggregation improves coverage or precision without materially reducing task performance.Source: §7 · p. 7
A Evaluation Metrics · B PCGrad Aggregation · C ProtoMotions Experimental Details
The evaluation uses hypervolume and expected utility to assess achieved objectives under sampled preferences. PCGrad replaces loss-level weighted aggregation with conflict-aware gradient projection, using independently randomized objective comparisons.
A Evaluation Metrics
- Hypervolume measures the region spanned from a reference point to achieved objective vectors using the Lebesgue measure.The reference point is dominated by every point in the finite set.Source: A Evaluation Metrics · p. 10
- Expected Utility evaluates policies conditioned on preference vectors drawn from a distribution over preferences.The formulation is identified as one used in prior preference-conditioned MORL evaluation [Ambadkar et al., 2026, de las Heras Molins et al., 2026].Source: A Evaluation Metrics · p. 10
- Evaluation uses 50 achieved objective vectors from preferences sampled from Dirichlet(0.5), with the origin as reference in normalized [0, 1]K objective space.The preference distribution matches the training-time distribution.Source: A Evaluation Metrics · p. 10
- Hypervolume is estimated with 2 × 10^5 uniform samples, while Expected Utility is estimated as the sample mean over the same 50 points.Source: A Evaluation Metrics · p. 10
B PCGrad Aggregation
- PCGrad projects each objective’s gradient away from conflicting gradients, sums the projected gradients, and leaves non-conflicting pairs untouched.It starts from the same per-objective clipped losses as loss-level aggregation and adds entropy and bound-loss gradients.Source: B PCGrad Aggregation · p. 10
- For each objective, PCGrad compares its gradient with other objectives’ original gradients in an independently sampled random order, excluding itself.A correction earlier in an objective’s traversal does not change the original gradient used for later projections.Source: B PCGrad Aggregation · p. 10
C.1 Task and Reward Setup … D.2 Optimization and Evaluation
The appendices define preference-conditioned task setups and evaluation protocols for ProtoMotions and MimicKit, and test whether reported tradeoffs hold across style dimensions and scoring conventions. Reward-level’s responsiveness advantage varies by dimension, while the method rankings reported in the main text persist under the alive convention.
C.1 Task and Reward Setup
- ProtoMotions trains a G1 humanoid with PrefAMP on a fixed-weight task reward and five preference-weighted style rewards, resampling the style preference from Dirichlet(0.5) each episode.Each method trains for 1.2 × 10^9 environment steps with 4096 parallel environments, 16-step PPO rollouts, and three random seeds; the style reward dimensions are normalized to [0, 1].Source: §C.1 · pp. 10–11, Table 3 · p. 11
C.2 Evaluation Metrics and Protocol
- Evaluation measures preference responsiveness with Self-PRR, Cross-PRR, Selectivity, and Rank Accuracy, alongside Hypervolume and Expected Utility across preference vectors.Self-PRR measures response of a style dimension to its own preference weight, Cross-PRR measures spillover to other dimensions, and Selectivity compares their means. The protocol uses fixed-horizon returns that penalize falls, 15-point sweeps per each of five dimensions, and 50 sampled preferences.Source: §C.2 · pp. 11–12
C.3 Per-Dimension Breakdown
- Reward-level’s Self-PRR advantage is driven mainly by arm_swing and foot_clearance, but it is weakest on energy, where all four methods’ curves nearly converge.Table 5 decomposes Self-PRR, Cross-PRR, and Rank Accuracy by style dimension, showing that the overall tradeoff is not uniform across dimensions.Source: §C.3 · p. 12, Table 5 · p. 13
C.4 Measurement Validity: Fixed vs. Alive Convention
- Under the alive convention, reward-level remains best on Hypervolume, Expected Utility, Self-PRR, and Rank Accuracy, while advantage-level leads Selectivity and has the lowest Cross-PRR.The authors conclude that the reported tradeoff is not an artifact of the fixed-horizon failure penalty; alive scores are uniformly higher because they do not penalize falls in the same way.Source: §C.4 · p. 12, Table 6 · p. 13
- Every method’s Fixed Hypervolume exceeds the preference-insensitive baseline of about 0.5^K ≈ 0.03 by 3× to 10×, indicating genuinely preference-responsive behavior.The baseline assumes every sampled preference yields the same near-uniform objective vector; the passage reports the comparison for all four methods.Source: Table 6 · p. 13, §C.4 · p. 13
D MimicKit Experimental Details
- MimicKit uses its native 28-actuated-DoF humanoid and framework-specific motion data, while matching ProtoMotions on style dimensions and evaluation protocol.Its locomotion data contain 56 forward, left, and right running and walking clips; the target direction is fixed along world +x, with target speed resampled every 4–7 seconds.Source: §D.1 · pp. 13–14
D.1 Task and Reward Setup
- MimicKit’s reward combines fixed task and AMP terms with five preference-weighted style terms, and resamples the style preference from Dirichlet(0.5) each episode.The normalized weights are wtask = 0.1875, wAMP = 0.1875, and wpref = 0.625; the preference is supplied to both actor and critic. Three aggregation methods train for 1.2 × 10^9 environment steps with 4096 environments and 32-step PPO rollouts.Source: §D.1 · pp. 13–14
D.2 Optimization and Evaluation
- MimicKit evaluates checkpoints with 256 parallel environments, H=300, M=15, N=50, and three deterministic evaluation seeds, reporting sample standard deviation across training seeds.Reported fall rates use the native ground-contact failure flag, accumulate returns through the first failure step, and zero later rewards without resetting failed environments; absolute fall rates should not be compared directly with ProtoMotions.Source: §D.2 · p. 14
D.3 Reactive-Steering Task Variant
In reactive steering, later aggregation methods retain an advantage in hypervolume and selectivity and reduce Cross-PRR relative to reward-level, while the highest mean Self-PRR shifts to advantage-level. The comparison changes both task reward and facing-direction observation, so direction changes alone are not isolated.
D.3 Reactive-Steering Task Variant
- Target directions are sampled uniformly and target speeds from [0.5, 2.0] m/s, with both resampled every 4–7 seconds.The reactive-steering task enables the facing-direction observation.Source: §D.3 · p. 14
- Because task reward and facing-direction observation also change, this comparison does not isolate direction changes alone; Jtask should be compared across methods within each task setting.Source: §D.3 · p. 15
- Advantage-level has the highest mean expected utility, Self-PRR, and selectivity; loss-level leads in hypervolume and Rank Accuracy and has the lowest Cross-PRR and fall rate, while reward-level has the highest task return.Methods have similar mean Self-PRR (0.471–0.480), with Rank Accuracy of at least 0.966. Both later methods outperform reward-level in mean hypervolume and selectivity and have lower Cross-PRR; the highest mean Self-PRR shifts from reward-level under fixed direction to advantage-level here.Source: Table 8 · p. 15, §D.3 · p. 15
E MetaDrive
The MetaDrive environment ablation compares reward-, loss-, and advantage-level aggregation using preference-conditioned driving policies while keeping the native task-reward weight fixed.
E MetaDrive
- Each policy uses a K=9 preference vector over driving objectives, and all variants share the same architecture.The compared objectives include progress, speed, lane keeping, road clearance, heading alignment, and route completion; MetaDrive’s native task reward retains a fixed unit weight.Source: E MetaDrive · p. 16
- Table 9 reports the MetaDrive ablation as mean ± std over three evaluation seeds.Bold marks the best value, with arrows indicating the preferred direction.Source: Table 9 · p. 16