PaperlayerView exact revision on arXiv

Source-linked AI summary

ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walking

Weidong Huang, Jingwen Zhang, Jiongye Li, Shibowen Zhang, Jiayang Wu, Jiayi Wang, Hangxin Liu, Yaodong Yang, Yao Su

arXiv:2602.06445v1cs.RO

TL;DR

Humanoid robots consume more energy than biological counterparts, and reward-shaped locomotion methods require difficult tuning of competing objectives. ECO separates energy and reference motion from rewards as explicit constraints, and experiments report lower energy consumption with robust walking in simulation and on BRUCE.

  • Problem

    Humanoid robots’ high energy consumption limits operational range, endurance, and payload, while reward-based optimization requires extensive tuning of competing objectives.

  • Method

    ECO uses constrained RL to separate energy consumption and reference motion from task rewards, enforcing them through constraints and Lagrangian optimization.

  • Results

    ECO achieves accurate velocity tracking with energy consumption approximately 3 times lower than MPC and 1.4 times lower than PPO.

  • Takeaways & Limitations

    ECO consistently reduces humanoid walking energy consumption while maintaining stable, robust behavior and avoiding extensive reward-weight tuning.

Abstract

from arXiv · show

Achieving stable and energy-efficient locomotion is essential for humanoid robots to operate continuously in real-world applications. Existing MPC and RL approaches often rely on energy-related metrics embedded within a multi-objective optimization framework, which require extensive hyperparameter tuning and often result in suboptimal policies. To address these challenges, we propose ECO (Energy-Constrained Optimization), a constrained RL framework that separates energy-related metrics from rewards, reformulating them as explicit inequality constraints. This method provides a clear and interpretable physical representation of energy costs, enabling more efficient and intuitive hyperparameter tuning for improved energy efficiency. ECO introduces dedicated constraints for energy consumption and reference motion, enforced by the Lagrangian method, to achieve stable, symmetric, and energy-efficient walking for humanoid robots. We evaluated ECO against MPC, standard RL with reward shaping, and four state-of-the-art constrained RL methods. Experiments, including sim-to-sim and sim-to-real transfers on the kid-sized humanoid robot BRUCE, demonstrate that ECO significantly reduces energy consumption compared to baselines while maintaining robust walking performance. These results highlight a substantial advancement in energy-efficient humanoid locomotion. All experimental demonstrations can be found on the project website: https://sites.google.com/view/eco-humanoid.

I. INTRODUCTION

Humanoid locomotion remains energy-intensive, while reward-based RL methods make energy optimization difficult to tune. ECO addresses this by treating energy and reference motion as explicit constraints and evaluating constrained RL across simulation and hardware.

  • Humanoid robots consume substantially more energy than biological counterparts, limiting operational range, endurance, and payload.
  • Reward-shaped RL methods require time-consuming, non-intuitive tuning of energy, task, and stability weights.Conflicting objectives can produce suboptimal solutions or convergence failures.
  • ECO separates energy consumption and reference motion from rewards, enforcing them as constraints while optimizing task-related rewards.Energy thresholds are tuned through incremental linear search using direct energy criteria.
  • The study compares four constrained RL algorithms and multiple constraint settings across simulation and physical BRUCE experiments.
  • ECO is designed to improve humanoid walking energy efficiency while preserving stable and robust locomotion.

B. Constrained Markov Decision Processes

A constrained Markov decision process extends an MDP with cost functions and thresholds. Its policy maximizes reward while satisfying all specified cost constraints, which may govern long-term energy or per-step motion properties.

  • A CMDP adds cost functions with corresponding thresholds to the standard MDP framework.
  • The CMDP objective maximizes policy reward subject to every cost remaining at or below its threshold.
  • Discounted-sum constraints limit cumulative quantities over an episode or long horizon, making them suitable for energy consumption requirements.
  • Average-sum constraints require a variable’s mean value to remain below a threshold at each time step.They can enforce motion styles such as walking symmetry.

C. Constraints RL Methods

The paper reviews constrained RL methods that optimize rewards while controlling costs through dual updates, alternating objectives, barriers, or penalties. It selects PPO-Lagrangian for ECO after comparing these alternatives.

  • PPO-Lag: PPO-Lagrangian reformulates constrained optimization using Lagrange multipliers and alternates primal-policy with dual-variable updates.
  • CRPO: CRPO alternates reward maximization with cost minimization and randomly selects one violated constraint when multiple constraints fail.
  • IPO: IPO converts constrained RL into unconstrained optimization with a logarithmic barrier controlled by a fixed steepness hyperparameter.
  • P3O: P3O extends PPO by adding penalties for constraint violations to its objective function.
  • The study selects PPO-Lagrangian for ECO and compares constrained methods to validate that choice.

IV. PROPOSED ECO FRAMEWORK

ECO uses neural-network control and critic policies trained with PPO-Lagrangian in IsaacGym, then deploys the learned policy directly on BRUCE. The policy converts proprioception and velocity commands into leg-joint targets for PD torque control.

  • The policy and reward critic are fully connected neural networks trained with PPO-Lagrangian in IsaacGym.The trained policy is deployed directly on the physical robot.
  • The policy receives proprioceptive observations and velocity commands, including a history of consecutive frames.
  • The reward critic additionally receives privileged information such as velocities, external forces, friction, mass, phases, and contacts.
  • The policy outputs deviations from nominal leg-joint positions, while arm joints remain fixed at nominal angles.
  • A PD controller converts the policy action and current joint state into desired torque commands.The controller uses current joint position and velocity in the torque computation.

C. Constraints

ECO treats energy consumption and reference motion as explicit constraints rather than mixing them with task rewards. These constraints are implemented through motor-energy and mirror-loss costs to promote efficient, stable, and symmetric locomotion.

  • Constraint selection: Energy consumption is selected as a constraint because reward-weight tuning alone did not converge within the feasible domain.Reference motion is also constrained using Mirror Loss to encourage natural motions.
  • Energy constraint: Motor energy is used as the evaluation metric because total robot energy cannot be directly measured from link inertia, friction, and heat losses.The metric integrates absolute motor power over time.
  • Energy constraint: The energy constraint threshold b1 is estimated from the robot’s speed range and walking duration.
  • Reference motion constraint: Mirror Loss constrains mirrored observations to produce mirrored actions, enforcing reference-motion stability and left-right symmetry.For example, lifting the left leg at the appropriate moment also constrains the policy to lift the right leg correspondingly.
  • Reward design: Reference-motion and energy costs are integrated into constraints, allowing the reward to omit reference joint-position tracking, joint-velocity penalty, and joint-torque penalty terms.The resulting reward configuration remains similar to Humanoid-Gym while separating these costs from task rewards.

E. Constrained Policy Optimization

ECO solves its two-constraint locomotion problem with Lagrangian optimization integrated into PPO. Policy updates combine reward and cost gradients while nonnegative multipliers are adjusted to enforce the constraints.

  • Lagrangian formulation: The Lagrangian method transforms ECO’s constrained objective into an unconstrained optimization problem with two designed constraints.
  • Optimization updates: Policy parameters and Lagrange multipliers are updated iteratively using reward and cost gradients to enforce both constraints.The updates alternate between policy improvement and multiplier adjustment.
  • PPO integration: PPO-Lagrangian integrates the Lagrangian approach into PPO’s Actor-Critic framework, optimizing a combined reward-and-constraint objective.Lagrange multipliers remain nonnegative through projection.
  • PPO objective: PPO’s clipped surrogate objective uses the behavior policy, GAE advantage estimates, and clipping parameters c1 and c2.
  • Algorithm selection: PPO-Lag required minimal parameter tuning, achieved low constraint violations, and showed better convergence speed and stability than P3O.P3O demonstrated comparable energy performance, but PPO-Lag was selected for ECO.
  • Cost estimation: Unlike cost-critic approaches, ECO estimates energy cost returns with Monte Carlo methods, requiring fewer network parameters and minimal design choices.The paper reports comparable performance to prior cost-critic methods.

V. SIMULATION

ECO is trained and evaluated on the kid-sized BRUCE humanoid using massively parallel simulation, domain randomization, and PPO-Lagrangian. Simulation setups specify robot hardware, reward functions, randomized parameters, and training hyperparameters.

  • Robot platform: BRUCE is a 70cm-tall, 4.8kg humanoid with 16 degrees of freedom, including five leg degrees of freedom per leg.Its leg actuators are liquid-cooled proprioceptive motors with 10.5Nm peak torque.
  • Training setup: ECO training uses IsaacGym with 8192 parallel environments, PPO-Lagrangian, domain randomization, and external perturbations for robustness.The perturbations include forces applied every 2 seconds for 0.001 seconds.
  • Reward configuration: Table I summarizes additional reward functions and defines tracking errors, velocity errors, stance indicators, foot-air duration, and joint-position terms.
  • Training targets: The base-height target is 0.45m, the maximum contact force is 50N, and the target swing-foot height is 0.03m.
  • Domain randomization: Table II lists domain-randomization terms and parameter ranges, distinguishing additive increments from multiplicative scaling.
  • Training duration: Training runs use 2000 policy iterations, typically converge in approximately 4 hours, and use a policy trained for 10 hours in sim-to-real experiments.

C. Baselines

ECO is evaluated against model-based, reward-shaped, and constrained RL baselines using reward, episode length, energy, and mirror-cost metrics. Across simulation and robustness studies, ECO satisfies energy and motion constraints while maintaining stable walking and substantially reducing energy use.

  • ECO is compared with MPC, PPO, P3O, IPO, and CRPO using normalized reward, episode length, energy cost, and mirror cost.The metrics respectively capture task performance, stability and robustness, energy efficiency, and motion symmetry.
  • 60J: ECO consistently converges to the target energy threshold while maintaining a normalized episode length close to 1.0.The threshold is selected as the best trade-off between energy efficiency and locomotion stability after searching 20J–100J.
  • P3O satisfies both constraints but requires more iterations to converge, while IPO is sensitive to its penalty coefficient and CRPO converges unstably.IPO either fails the energy constraint or becomes overly conservative when its penalty coefficient is increased.
  • Three orders of magnitude of PPO’s energy coefficient fail to produce a satisfactory balance: low values preserve energy use, whereas high values cause instability and falls.ECO instead directly enforces energy and motion constraints without manually tuning reward weights.
  • ECO’s constrained policy maintains the 0.05 mirror-motion threshold with less conservative behavior than IPO, P3O, and CRPO, whose gaits rapidly converge to that threshold.PPO with varied energy coefficients reaches differing mirror costs and often produces asymmetric, unstable gaits.
  • ECO transfers across IsaacGym, MuJoCo, and Gazebo with stable walking, accurate velocity tracking, and energy consumption approximately 3 times lower than MPC and 1.4 times lower than PPO.MPC has the highest energy cost and fails to track higher target speeds such as 0.2m{s.
  • ECO shows greater perturbation robustness than MPC, especially sagittally, and comparable robustness to PPO, including under longer pushes than those seen during training.The evaluation also includes random dragging of different robot links under varied disturbance directions and magnitudes.
  • Adding constraints reduces episode lengths relative to ECO’s original two-constraint setup; all evaluated algorithms fail to converge on self-collision cost.Removing foot-contact velocity and self-collision constraints improves episode length or energy satisfaction, but remains below the two-constraint configuration.

A. Real World Results

On real BRUCE hardware, ECO maintained comparable walking speeds while using substantially less motor energy than MPC and PPO, and it remained robust to disturbances and outdoor terrain.

  • Approximately 6 times lower than MPC and 2.3 times lower than PPO, ECO’s cumulative motor energy was measured at similar walking speeds.ECO maintained motor power near or below the 60J energy-cost threshold.
  • ECO’s learned policy enabled BRUCE to reject disturbances and traverse different outdoor terrains.

B. Emergent Behaviors

ECO produces energy-efficient walking behaviors, including less knee flexion and lower knee torque, while exposing limitations in transferring additional constraints and heel-to-toe walking to humanoids.

  • Emergent Behaviors: Minimizing energy consumption produces straight-knee-like behavior without explicitly prescribing it.The paper connects this emergent behavior to energy-efficient locomotion observed in humans and humanoid robots.
  • Emergent Behaviors: ECO generates less flexed knees and lower knee torque than MPC and PPO during real-world walking.Lower knee torque supports lighter steps and reduced energy expenditure while maintaining stability and velocity tracking.
  • Emergent Behaviors: Heel-to-toe walking is absent because the current BRUCE hardware requires additional control effort and higher swing-leg lift.The additional hip and knee motor effort would increase energy consumption.
  • Constraint Limitations: Adding foot clearance, foot-contact velocity, and self-collision constraints is challenging because humanoid morphology and balance dynamics differ from quadrupeds.The five-constraint setting uses thresholds for energy, mirror reference motion, foot clearance, foot-contact velocity, and self-collision.
  • Constraint Limitations: Smaller humanoid support polygons make feasible trajectories more restricted, so many feasibility constraints can complicate constrained-RL search.The paper leaves handling larger numbers of constraints in humanoid locomotion for future research.

B. Optimization Method Selection

The experiments identify PPO-Lag as the most effective constrained-RL algorithm for this setting, while highlighting optimization and transfer challenges as constraints and environments become more complex.

  • Optimization Method Selection: PPO-Lag is the most effective of the four constrained-RL algorithms evaluated.CRPO is simpler but lacks smooth reward-constraint transitions, while P3O and IPO require coefficient tuning.
  • Optimization Method Selection: ECO uses PPO-Lag to adapt Lagrange multipliers and learning rates to each constraint’s magnitude.This adaptive update supports smooth transitions between reward and constraint optimization and stable humanoid convergence.
  • Optimization Method Selection: ECO faces challenges when optimizing five constraints, motivating improved constrained-RL optimization techniques.The paper suggests leveraging humanoid dynamics models to improve convergence within restricted feasible regions.
  • Sim-to-Real Transfer: Real-world transfer requires careful low-level PD-gain tuning and a compliant ankle strategy to address hardware dynamics and sensor noise.Observation noise and biases are randomized to match hardware measurements.
  • More Complex Settings: Stable walking does not directly extend to transitions from flat terrain to stairs, where higher foot lifts and mode-aware constraints may be needed.The paper proposes terrain-aware thresholds and hierarchical model-based planning as future directions.
  • Conclusion: ECO improves energy efficiency without compromising robustness across simulation and real-world hardware experiments.
Loading 2602.06445v1…