Source-linked AI summary
MPBRS: Multi-Potential-Based Reward Shaping for Unified Reward Configuration in Humanoid Locomotion
Van Chung Nguyen, Hung Manh La
TL;DR
Humanoid locomotion RL depends on reward designs that can be sensitive to scaling and difficult to transfer across morphologies. This paper applies one unified multi-PBRS configuration across nine humanoid robots; it reports modest convergence gains but lower tuning sensitivity and more reliable reuse.
Problem
Humanoid locomotion reward weights can produce markedly different behaviors, and reward designs often require extensive tuning and do not transfer cleanly across morphologies.
Source: §I · p. 1, A. Reward shaping for locomotion · p. 2Method
The study combines direct shaping for tracking and posture with three PBRS potentials and reuses the unified reward configuration across nine humanoid robots.
Source: §VI · p. 8Results
Across diverse humanoid morphologies, MPBRS provided modest convergence-speed improvements but was substantially less sensitive to reward scaling and weight choices.
Source: §VI · p. 8Takeaways & Limitations
The same reward design enabled more reliable transfer across humanoids and simplified tuning, with robot-specific modification limited to metadata such as joint mappings and nominal height targets.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Designing and tuning reward functions remains one of the main challenges in building effective reinforcement learning (RL) pipelines for humanoid locomotion. While shaping rewards can accelerate learning, naively formulated terms often conflict with the desired behavior, overfit to a specific robot, or become unstable under small weight changes. Potential-Based Reward Shaping (PBRS) offers an alternative solution: shaping the reward via differences of a potential function to guide exploration without changing the optimal policy. In this paper, we study Multi-PBRS for humanoid locomotion and propose a unified reward configuration. We then evaluate the same reward structure and weights across nine humanoid robots, requiring only robot-specific metadata (joint/link mappings and default poses) rather than per-robot reward redesign. Our experiments show that, compared to standard shaping, MPBRS provides more consistent scaling behavior and reduced tuning sensitivity across robots, making it easier to tune without altering the optimal policy.
I. INTRODUCTION
Humanoid locomotion RL depends heavily on reward design, yet reward tuning is time-consuming and often fails to transfer across robots. The paper evaluates a unified MPBRS configuration across nine humanoids, finding improved stability and portability rather than large convergence gains.
I. INTRODUCTION
- Reward design strongly affects learned behavior, and naive or poorly scaled terms can produce unintended, brittle, or unstable policies.Developers often spend substantial time tuning terms and weights to balance training stability and behavior quality.Source: §I · p. 1
- Humanoid reward tuning is difficult to reproduce and transfer because morphology, actuation, and physical design differ across robots.A reward effective on one robot may require significant modification on another, even among similar humanoids.Source: §I · p. 1
- PBRS shapes rewards using potential differences while preserving optimal policies in discounted MDPs.Prior PBRS evidence largely comes from grid-worlds and other low-dimensional domains, while high-performing legged locomotion commonly uses manually tuned shaping without policy-invariance guarantees –.Source: §I · p. 1
- MPBRS reuses one reward design across nine humanoid robots and primarily improves training stability and portability, with lower sensitivity to weight selection and reward magnitude.The configuration requires robot-specific metadata rather than per-robot reward redesign.Source: §I · pp. 1–2
II. LITERATURE REVIEW
Locomotion reward shaping can provide dense guidance, but direct shaping is sensitive to weight choices and imitation objectives may not transfer across morphologies. PBRS preserves optimal policies, and this work tests whether multi-PBRS improves scaling robustness and portability across humanoids.
A. Reward shaping for locomotion
- Direct shaping weights can produce behaviors from overly conservative gaits to unstable oscillations, while imitation references can bias solutions and transfer poorly across morphologies.DRS terms encode objectives such as velocity tracking, uprightness, base-height regulation, and energy or action-change penalties –; imitation methods use reference tracking –.Source: A. Reward shaping for locomotion · p. 2
B. Potential-based reward shaping (PBRS)
- PBRS constructs shaping rewards from potential differences and preserves the set of optimal policies in discounted MDPs.Its separation of task reward from auxiliary guidance offers dense feedback without changing an optimal solution.Source: B. Potential-based reward shaping (PBRS) · p. 2
- PBRS can redistribute long-horizon value into denser step-wise feedback, while reported faster convergence has largely been observed in low-dimensional environments.The cited examples include gridworlds and classic control,.Source: B. Potential-based reward shaping (PBRS) · p. 2
- For humanoid locomotion, PBRS speed gains may be modest, while its robustness to reward scaling motivates testing a unified configuration across nine robots.The evaluation measures scaling robustness and the degree of per-robot reward retuning required.Source: B. Potential-based reward shaping (PBRS) · p. 2
III. METHODOLOGY
The paper formalizes humanoid locomotion as an MDP and frames reinforcement learning as maximizing expected discounted return.
A. Markov Decision Processes and Reinforcement Learning
- Humanoid locomotion is formalized as an MDP with state space, action space, transition dynamics, reward, and discount factor.The notation is M = (S, A, T, r, γ).Source: A. Markov Decision Processes and… · p. 2
- At each timestep, the policy selects an action from the current state, the system transitions, and the agent receives a scalar reward.The policy is written as a_t = π(s_t).Source: A. Markov Decision Processes and… · p. 2
- The stated reinforcement-learning objective is to find a policy maximizing expected discounted return.Source: A. Markov Decision Processes and… · p. 2
B. Reward shaping: DRS vs. Multi-PBRS
Direct reward shaping adds instantaneous signals but is sensitive to weights and morphology, whereas PBRS rewards changes in potentials across transitions. The paper combines multiple potentials in one configuration intended for use across diverse humanoids with minimal robot-specific modification.
a) Direct reward shaping (DRS)
- DRS adds instantaneous signals for desirable locomotion properties, but small weight changes can produce sharply different behaviors and impede transfer across humanoids.The cited examples include overly cautious gaits, oscillations, standing-still solutions, and proxy exploitation.Source: B. Reward shaping: DRS vs. Multi-PBRS · p. 3
b) Potential-based reward shaping (PBRS)
- PBRS shifts shaping from rewarding a good state to rewarding progress between states, making signals less dominated by absolute potential offsets across morphologies.The shaping contribution is near zero when a potential is stationary and becomes positive when the next state increases it.Source: B. Reward shaping: DRS vs. Multi-PBRS · p. 3
c) Multi-PBRS
- Multi-PBRS applies three shared potentials for base height, uprightness, and joint regularization, with robot-specific joint-name mappings and nominal height targets.The joint-regularization potential uses robot-specific mappings to encourage left–right symmetry and nominal hip yaw.Source: B. Reward shaping: DRS vs. Multi-PBRS · pp. 3–4
- Shaping is set to zero on reset steps, while paired potential values are tracked consistently across transitions.The stated purpose of zeroing reset-step shaping is to avoid spurious impulses at episode boundaries.Source: B. Reward shaping: DRS vs. Multi-PBRS · p. 4
IV. SYSTEM OVERVIEW
The evaluation spans nine simulated humanoids in one unified training pipeline, holding the learning setup fixed to compare DRS and MPBRS shaping under the same task objectives.
IV. SYSTEM OVERVIEW
- Nine simulated humanoids with substantially different morphologies are evaluated in a unified training pipeline.The setup is held fixed and compares DRS with MPBRS under the same task objectives, isolating the shaping formulation from robot-specific reward engineering.Source: §IV · p. 4
A. Case study
For running locomotion, the case study evaluates nine humanoid robots while fixing the arm joints at nominal angles and limiting system control to the legs and waist yaw.
A. Case study
- The running-locomotion case study evaluates nine humanoid robots.The arms are fixed at nominal angles; control is limited to the legs and waist yaw, with waist roll and pitch fixed at defaults when those joints are present.Source: A. Case study · p. 4
- When a waist has roll, pitch, and yaw joints, only yaw is controlled; roll and pitch remain at their default values.Source: A. Case study · p. 4
B. Low-level control (PD tracking)
The low-level controller maps policy actions to desired joint positions and tracks them with a joint-space PD controller. The training setup specifies observations and rewards, then applies PBRS to three locomotion objectives.
B. Low-level control (PD tracking)
- Policy actions are mapped to desired joint-position targets tracked by a joint-space PD controller, with per-joint gains tuned to actuator characteristics.The gain tuning is intended to produce smooth motion while keeping control inputs within actuator limits.Source: B. Low-level control (PD tracking) · p. 4
C. Training setup
- The baseline reward directly targets linear-velocity and yaw-rate tracking; its remaining terms regularize motion smoothness and energy efficiency.The observation dimension is 18 + 3Na, with Na depending on the robot type; observation noise is added to model sensor imperfections.Source: C. Training setup · p. 4
- PBRS uses three locomotion potentials: height, uprightness, and joint regularization, with γΦ = 1.00 for every term.The height and joint-regularization settings use σh = 0.08 and σq = 0.5, respectively; uprightness uses the base-frame gravity component.Source: C. Training setup · p. 5
D. Implementation Details
The setup uses IsaacLab with PPO-Clip, while the results benchmark one reward configuration across nine humanoids against two baselines.
D. Implementation Details
- Training used IsaacLab with PPO-Clip and RSL-RL, 2048 agents, 1000 interaction steps, and a 200 Hz simulation with 100 Hz control.Each training run took approximately 50 minutes.Source: D. Implementation Details · p. 5
V. RESULTS
- The benchmark reused the same reward configuration across nine robots, keeping the MPBRS weight at 1.0 without robot-specific reward tuning.Baseline and DRS weights were tuned for stable locomotion; all agents started with randomized joint positions.Source: §V · p. 5
A. Training Benchmark
Across nine humanoids, MPBRS broadly preserves baseline-like tracking and fall-rate trends, while DRS changes reward scales and can show larger deviations. The Fourier N1 evaluation also shows accurate tracking through command changes and recovery after respawns.
A. Training Benchmark
- All three settings converge on all nine platforms, but DRS's apparent faster convergence is not directly comparable because its added terms change the reward scale.DRS also requires more careful reward-weight tuning.Source: A. Training Benchmark · p. 5
- MPBRS remains behaviorally consistent with Baseline, whereas persistent DRS auxiliary rewards can influence the learned policy when weights are not carefully tuned.The MPBRS shaping contribution approaches zero near the optimal state.Source: A. Training Benchmark · p. 5
- Across nine robots, fall-rate trends generally align between MPBRS and Baseline, and all three methods achieve broadly comparable planar- and yaw-tracking rewards.DRS shows larger total-reward deviations and, on some robots, fall-rate deviations; Fig. 3 lays out the benchmark comparisons.Source: A. Training Benchmark · p. 5, Figure 3 · p. 6
- Convergence speed varies by robot, with morphology, actuator limits, and nominal dynamics remaining important factors despite using a shared reward design.The results report operation across platforms without reward retuning and lower sensitivity to reward tuning.Source: A. Training Benchmark · p. 5
- DRS may appear to learn faster, but MPBRS offers a more conservative trade-off with baseline-like behavior and comparable tracking across the humanoid set.DRS may introduce behavioral bias when shaping weights are not carefully tuned.Source: A. Training Benchmark · p. 5
- On Fourier N1, MPBRS tracks x-axis commands up to 3.5 m/s, follows changing commands stably, and quickly resumes tracking after 10-second respawns.Respawn-related tracking-error spikes are expected transients; the policy recovers quickly.Source: A. Training Benchmark · pp. 6–7, Figure 4 · p. 6
B. Sensitivity analysis
The sensitivity analysis varies reward weights from 0.1 to 500 and contrasts MPBRS's tracking at high weight with DRS's standstill behavior.
B. Sensitivity analysis
- At a shaping weight of 500, MPBRS still converges to the optimal policy and tracks velocity, while DRS leaves the robot standing in place.The analysis varies reward weights from 0.1 to 500.Source: B. Sensitivity analysis · pp. 7–8
- With DRS, excessive weight lets height, uprightness, and joint-regulation rewards dominate the other terms.Source: B. Sensitivity analysis · p. 8
- In the analysis, convergence of the height, uprightness, and joint-regularization rewards to zero indicates the stated optimal state.Source: B. Sensitivity analysis · p. 8
VI. CONCLUSION
The paper concludes that a unified MPBRS reward configuration can be reused across nine humanoid robots. Its main reported advantage is lower sensitivity to reward scaling and weight choices, rather than a large convergence-speed gain.
VI. CONCLUSION
- MPBRS combines command-tracking and posture-stabilization shaping terms with ray-based base-height, gravity-alignment uprightness, and morphology-aware joint-regularization potentials.Source: §VI · p. 8
- Across nine humanoids, MPBRS provides modest convergence-speed improvements but is less sensitive to reward scaling and weight choices, simplifying tuning and reuse across robots.Source: §VI · p. 8
- Future work will investigate learned or adaptive potentials, improved sensor and noise modeling for sim-to-real robustness, and tighter integration with morphology-conditioned policies and domain randomization.Source: §VI · p. 8