PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion

Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Ji, Moonkyu Jung, Donghoon Youm, Jemin Hwangbo

arXiv:2308.12517v4cs.ROcs.AIcs.LG

TL;DR

Reward engineering for reinforcement-learning controllers can require extensive design and tuning of reward terms. This paper proposes a framework combining rewards and constraints, with constraint types and an efficient policy-optimization algorithm. Experiments on diverse legged robots report robust, high-performance control with less reward engineering.

  • Problem

    Designing and tuning numerous reward terms to obtain high-performing controllers is laborious and time-consuming.

    Source: B. Legged robot locomotion · p. 3

  • Method

    The framework combines rewards and constraints, with two constraint types and an efficient policy-optimization algorithm for finding reward-maximizing policies that satisfy multiple constraints.

    Source: §I · p. 2, C. Constrained reinforcement learning · p. 4, A. Constraint Types · p. 5

  • Results

    Across diverse legged robots, experiments demonstrate robust, high-performance controllers with less reward engineering, using only three reward coefficients and one coefficient modification.

    Source: §I · p. 2

  • Takeaways & Limitations

    Constraints offer a more generalizable and intuitive way to reflect engineering intent, including through physically meaningful limits such as joint-angle bounds.

    Source: §I · p. 2

Abstract

from arXiv · show

Several earlier studies have shown impressive control performance in complex robotic systems by designing the controller using a neural network and training it with model-free reinforcement learning. However, these outstanding controllers with natural motion style and high task performance are developed through extensive reward engineering, which is a highly laborious and time-consuming process of designing numerous reward terms and determining suitable reward coefficients. In this work, we propose a novel reinforcement learning framework for training neural network controllers for complex robotic systems consisting of both rewards and constraints. To let the engineers appropriately reflect their intent to constraints and handle them with minimal computation overhead, two constraint types and an efficient policy optimization algorithm are suggested. The learning framework is applied to train locomotion controllers for several legged robots with different morphology and physical attributes to traverse challenging terrains. Extensive simulation and real-world experiments demonstrate that performant controllers can be trained with significantly less reward engineering, by tuning only a single reward coefficient. Furthermore, a more straightforward and intuitive engineering process can be utilized, thanks to the interpretability and generalizability of constraints. The summary video is available at https://youtu.be/KAlm3yskhvM.

I. INTRODUCTION

Reward engineering is a time-consuming part of building reinforcement-learning controllers, while constraints offer a way to encode desirable or feasible behavior. The paper combines rewards and constraints in a learning framework and evaluates it on diverse legged robots.

I. INTRODUCTION

  • Reward-term design and coefficient tuning are the most time-consuming controller-engineering steps, often involving more than ten terms.Relative weights are difficult to choose from the resulting robot motion.Source: §I · p. 1
  • Constraints can generalize across similar robot platforms and use physically meaningful limits that are more intuitive to set than reward coefficients.The paper gives joint-angle limits and values derived from robot description files as examples.Source: §I · p. 2

A. Contributions

  • The framework combines rewards and constraints with an efficient policy optimization algorithm designed to scale to multiple constraints.The authors describe the framework as enabling high-performance policy learning.Source: A. Contributions · p. 2, §I · p. 2

I. INTRODUCTION

  • Simulation and real-world experiments across six quadrupeds and one biped demonstrate robust control while improving generalizability and engineering intuitiveness.The robots vary in morphology and physical properties.Source: Figure 1 · p. 1, A. Contributions · p. 2

A. Contributions

  • The authors present constraints as a tool to use alongside or instead of rewards, potentially reducing tedious reward engineering.They frame this as a direction for designing neural-network controllers.Source: A. Contributions · p. 2

A. Control of complex robotic systems

Learning-based control, especially model-free reinforcement learning, offers an alternative for complex robotic systems where demonstrations are limited and environmental uncertainty challenges model-based methods. This paper addresses the months-long, robot-specific reward-engineering burden in legged locomotion by combining rewards and constraints, with an optimization method intended to scale to many constraints.

A. Control of complex robotic systems

  • Model-based control becomes challenging as systems grow more complex and environmental uncertainty increases; learning-based methods instead train neural controllers from demonstrations or interaction data.Demonstrations are often unavailable for complex articulated systems, while reinforcement learning can use interaction data from simulation and the real world.Source: A. Control of complex robotic systems · p. 3

B. Legged robot locomotion

  • Reward engineering for robust, natural legged locomotion often takes months, as engineers design ten or more terms and tune their weights using experience and intuition.Prior approaches using expert motion are limited by available demonstrations, while morphology randomization can yield conservative controllers and was tested only indoors on simple behaviors.Source: B. Legged robot locomotion · p. 3

C. Constrained reinforcement learning

III. BACKGROUND

The background formulates constrained reinforcement learning as maximizing expected reward while satisfying cost limits. It distinguishes probabilistic constraints on undesirable events from average constraints on physical quantities.

A. Constrained Markov Decision Process

  • A CMDP represents constrained reinforcement learning with states, actions, transitions, rewards, cost functions, an initial-state distribution, and a discount factor.The policy objective is to maximize expected reward while satisfying each cost threshold.Source: A. Constrained Markov Decision Process · pp. 4–5

A. Constraint Types

  • The paper uses probabilistic and average constraints, allowing each constraint to match the engineer’s intent for stochastic policies.Probabilistic constraints limit the chance of an undesirable event; average constraints limit the average of a physical variable.Source: A. Constraint Types · p. 5

1) Probabilistic Constraints:

  • A probabilistic constraint bounds the probability of leaving a desirable event space by threshold Dk; zero forbids the undesirable event, but the paper uses a small positive threshold because zero is challenging in stochastic MDPs.The example constrains a robot’s center of mass to remain within a specified distance of the ground when passing beneath overhangs.Source: 1) Probabilistic Constraints · p. 5

2) Average Constraints:

Average constraints limit mean physical quantities rather than enforcing a per-step bound. The policy optimization section describes converting constraints into discounted cumulative limits and adapting IPO for multiple constraints.

2) Average Constraints:

  • An average constraint bounds the expected physical variable by a threshold, making it suitable for quantities such as foot-contact velocity that need not remain below the limit at every step.The paper contrasts this with probabilistic constraints, which are undesirable for nonzero foot velocity on slippery surfaces.Source: 2) Average Constraints · p. 5

B. Policy optimization

  • The method converts probabilistic and average constraints into discounted cumulative constraints using modified limit dk = Dk/(1 − γ), then applies Interior-point Policy Optimization.IPO uses logarithmic barrier functions; the authors optimize it with TRPO rather than PPO, citing TRPO’s explicit line search for more stable improvement and feasibility checking.Source: B. Policy optimization · pp. 5–6
  • Scaling constrained optimization is a challenge because randomly initialized policies may be infeasible for IPO, and addressing violated constraints one at a time becomes inefficient as their number grows.The paper identifies scalability to many constraints as a limitation in earlier CPO and IPO applications.Source: B. Policy optimization · p. 6

1) Adaptive constraint thresholding:

The policy optimization modifications address infeasible initialization and the cost of handling many constraints. Adaptive thresholds enlarge the feasible region during training, while a shared multi-head value network predicts all constraint costs.

1) Adaptive constraint thresholding:

  • Adaptive thresholding sets each constraint limit based on the current policy’s cost, widening the feasible region so randomly initialized policies can enter it without sequentially repairing violated constraints.The enlargement is controlled by α and parameterized by dk; barrier gradients then guide the policy toward the desired region.Source: 1) Adaptive constraint thresholding · p. 6

2) Multi-head cost value function:

  • A multi-head cost value function shares one neural-network backbone across cost predictions, increasing only the output dimension as constraints are added instead of training independent networks.The algorithm samples trajectories, computes reward and cost advantages with GAE, updates thresholds and policy parameters, then updates value functions.Source: 2) Multi-head cost value function · pp. 6–7

V. APPLICATIONS ON LEGGED ROBOT LOCOMOTION

The locomotion application trains a blind controller to track commanded velocities using proprioceptive sensing, while representing hardware and motion requirements as constraints rather than reward terms. The teacher policy combines three rewards with 11 constraints, including probabilistic and average constraints.

A. Overview

  • The blind controller uses proprioceptive data to track commanded forward velocity, lateral velocity, and turning rate despite unexpected terrain variations.Blind locomotion here means relying on proprioceptive sensors, such as joint positions and body orientation, to adjust base motion and foot placement.Source: A. Overview · p. 7
  • Only three reward terms—command tracking, joint torque, and action smoothness—were sufficient for a natural-gait policy performing on par with the state-of-the-art policy.The smoothness reward regulates policy output to help prevent motor vibration in real-world deployment.Source: A. Overview · p. 7, B. Teacher policy · p. 8

B. Teacher policy

  • The teacher state combines robot-accessible observations with simulation-only privileged information, which an encoder represents before the actor produces actions.Observations include commands, body and joint measurements, histories, and leg phases; privileged information includes terrain and contact properties.Source: B. Teacher policy · pp. 7, 9
  • The policy outputs joint PD targets, with desired positions formed from nominal joint positions plus scaled policy actions and desired velocity targets set to zero.A PD controller converts these targets to torques at a higher control frequency.Source: B. Teacher policy · p. 7
  • Constraint ranges for joint angles, velocities, and torques can come from a robot description file or be set manually, while most constraint limits are kept small to restrict undesirable states.Average-constraint thresholds correspond to physical quantities with known units, such as velocity and height.Source: B. Teacher policy · pp. 8–9
  • The teacher-training formulation defines 11 constraints: six probabilistic and five average constraints, with costs chosen according to whether requirements apply at each time step or are naturally expressed as physical quantities.The constraints cover joint limits, body contact and posture, gait pattern, motion and contact velocities, and foot clearance or height.Source: B. Teacher policy · pp. 8–9

C. Student policy

The student policy is trained for zero-shot real-world deployment using only robot sensor information. It imitates the teacher’s actions while predicting its encoded privileged representation from observation history.

C. Student policy

  • The student imitates the teacher’s actions and predicts its encoded privileged representation from observation histories, without using the teacher’s simulation-only privileged input.The student is intended for zero-shot real-world deployment using information directly available from robot sensors.Source: C. Student policy · p. 9
  • A GRU encoder predicts the encoded privileged representation from observations and its hidden state, then supplies that representation with the current observation to an MLP actor.Source: C. Student policy · p. 9
  • The student is trained end-to-end with a loss combining squared error in teacher-action imitation and encoded privileged-representation prediction.Source: C. Student policy · p. 9

D. Terrain curriculum

Teacher training uses procedurally generated terrains and an adaptive curriculum, while domain randomization targets sim-to-real transfer. The experimental section frames evaluation around robust performance, cross-platform generalizability, and engineering intuitiveness.

D. Terrain curriculum

  • Training terrains include hills, discrete hills, steps, inclining steps, and stairs, sampled in equal proportion across two parameterizations per terrain type.Source: D. Terrain curriculum · p. 9
  • The teacher’s adaptive curriculum expands each terrain type’s available sampling stages when the policy’s traversability score at its current maximum stage exceeds a threshold.The authors report that this strategy prevented catastrophic forgetting in their experience, with less related hyperparameter engineering while enabling terrain exploration.Source: D. Terrain curriculum · p. 9

VI. EXPERIMENTAL RESULTS

  • The experimental questions ask whether the framework can match reward-only robust control, generalize across robot platforms, and make motion engineering more straightforward and intuitive.Source: §VI · pp. 9–10
  • Table II reports constraint-satisfaction results after training for 1.6 billion time steps, with thresholds shown in the top row or brackets.Bold text indicates violated or barely satisfied constraints.Source: Table II · p. 10, §VI · p. 10

A. Implementation detail

The framework trains legged-robot controllers with limited reward tuning and transfers them to real robots that traverse harsh terrain. It also uses constraint changes to generate distinct motions and evaluates reward, cost, and constraint behavior.

A. Implementation detail

  • Training used RaiSim, taking about 24 hours for the teacher policy and 8 hours for the student; the deployed student policy ran at 100 Hz on Raibo and Mini-cheetah.The student network was reimplemented in C++ and forward-passed on the CPU.Source: A. Implementation detail · p. 10

B. Framework Evaluation

  • Across quadrupedal and bipedal robots with varied morphologies and physical properties, robust controllers trained with minimal parameter changes, adjusting only the torque-reward scaling factor among reward coefficients.The robots ranged from 11 kg to 135 kg; constraint limits and desirable regions were also adapted to morphology and size.Source: B. Framework Evaluation · pp. 10–11, Table III · p. 10
  • Figure 3 evaluates total reward and costs, reward-component sensitivity, and constraint satisfaction for foot clearance, gait pattern, and joint angle.The reward components are standardized to range from -1 to 0.Source: Figure 3 · p. 11
  • Real-world sensor data showed joint-angle, joint-velocity, and joint-torque costs of 0.007, 0.001, and 0.002, respectively, below the 0.025 thresholds.The verification used about 25 minutes of Raibo data from outdoor routes.Source: B. Framework Evaluation · pp. 11–12
  • Changing gait-pattern or COM-height and joint-angle constraints generated bounding and crawling motions from a trotting controller.The real-world examples include bounding and crawling beneath an overhanging obstacle.Source: Figure 5 · p. 13
  • Real-world controllers traversed harsh terrain, while the framework required only one reward coefficient adjustment among three reward terms for robot transfer.Raibo also sprinted over grass at 2 m/s and climbed slippery slopes of about 32° at 1 m/s.Source: B. Framework Evaluation · pp. 11, 13

C. Framework Analyzation

The framework separates rewards, which express values to maximize, from constraints, which specify desirable regions for converged policies. Experiments show objective sensitivity and constraint satisfaction, supporting constraint-led motion design with fewer reward terms.

C. Framework Analyzation

  • Rewards express values to maximize, while constraints specify desirable regions that the converged policy should satisfy.The framework targets objective sensitivity to reward changes and constraint satisfaction when a feasible policy exists.Source: C. Framework Analyzation · p. 13
  • Doubling torque- and smoothness-reward coefficients increased their corresponding reward values while constraints remained satisfied.This tests whether policies respond to objective changes without abandoning existing constraints.Source: C. Framework Analyzation · p. 13
  • Changing constraint regions or limits guided Mini-cheetah to different foot clearances, gait patterns, and joint-angle ranges in simulation.In real-world tests, changing gait phase produced bounding, while changing COM-height and hip-joint ranges produced crawling; both used the trotting controller's parameter set.Source: C. Framework Analyzation · pp. 13–14
  • Constraints restrict the policy's convergence region through parameters tied to physical variables, allowing engineers to tune a minimal set of reward terms more directly than reward coefficients.The passage points to Table IV for the number of reward terms and an appendix comparison with reward-only engineering.Source: C. Framework Analyzation · p. 14, Table IV · p. 14

D. Comparison with the Reward-only Framework

The constraint-based framework generalizes better than a reward-only controller when transferred from Raibo to Anymal C. Ablations report that IPO and a multi-head cost value function support efficient optimization across multiple constraints.

D. Comparison with the Reward-only Framework

  • On Anymal C, the constraint-based controller retained comparable command tracking, whereas the reward-only controller degraded and transferred locomotion style poorly.The reward-only controller's nominal-configuration reward was too small for Anymal C without coefficient adjustment.Source: Figure 6 · p. 14, D. Comparison with the Reward-only… · p. 14
  • Reward distributions differed across Raibo and Anymal C, shifting control behavior and making reward-coefficient retuning difficult because reward terms are interdependent.The figure marks reward types with highly different distributions in red.Source: Figure 7 · p. 14, D. Comparison with the Reward-only… · pp. 14–15

1) Algorithm:

  • IPO achieved higher reward, lower torque use, smoother actions, and higher traversability than P3O, while showing clearer constraint satisfaction.IPO's logarithmic barrier retains a smooth cost gradient inside the feasible region; IPO's computation time changes little as constraint count grows, unlike CPO's linear increase.Source: Table V · p. 15, 1) Algorithm · p. 15

2) Multi-head cost value function:

  • The multi-head cost value function used 0.1 million trainable parameters versus 1 million for single-head, with similar policy reward.Its shared backbone leaves only the output layer to grow with the number of constraints.Source: Table VI · p. 15, 2) Multi-head cost value function · pp. 15–16
  • Multi-head training had the least computation time and does not scale linearly with constraint count, while reducing single-head network size can worsen value-prediction accuracy and variance.The resulting prediction errors can affect policy-gradient derivation as constraints increase.Source: 2) Multi-head cost value function · p. 16, Figure 8 · p. 16

3) Algorithm hyperparameters:

Hyperparameter choices trade off constraint satisfaction, reward acquisition, and training speed, but one setting worked reliably across the tested robots and constraint designs. The conclusion highlights reduced reward engineering and identifies future directions in constraint formulations, tasks, platforms, and exploration.

3) Algorithm hyperparameters:

  • Large t or α can leave constraints violated, while small t can reduce task reward and small α can slow training.The effects depend on barrier-gradient shape and feasible-region enlargement; small α particularly limits early policy updates during TRPO line search.Source: 3) Algorithm hyperparameters · p. 16
  • Using one set (t = 100 and alpha = 0.02) across all robots and various constraint designs resulted in reliable constraint satisfaction and superior reward acquisition, demonstrating hyperparameter robustness across RL environments.Source: 3) Algorithm hyperparameters · p. 17

VII. CONCLUSION AND FUTURE WORK

  • Simulation and real-world experiments across diverse legged robots showed that constraints support performant locomotion with significantly less reward engineering.The authors characterize this as a more straightforward and efficient engineering process than using rewards alone.Source: §VII · p. 17
  • Future work includes new constraint formulations for specific use cases, applying the framework to diverse tasks and platforms, and using advanced exploration methods to improve data efficiency.Source: §VII · p. 17

APPENDIX

With an exponential command-tracking reward, the framework achieved similar tracking performance and torque usage to the reward-only comparison, while constraints enabled a toy controller to meet specified gait and contact goals. The examples contrast reward-coefficient tuning with directly specifying constraints.

A. Comparison of velocity command tracking performance with the reward-only framework

  • The framework achieved the same command-tracking performance with similar torque usage using an exponential reward, and similar maximum speeds under the tested velocity commands.The comparison reports forward-velocity commands from -1 m/s to 2 m/s and turning-rate commands from -1 rad/s to 1 rad/s.Source: A. Comparison of velocity command… · p. 17, Table VII · p. 17, Figure 9 · p. 17
  • An exponential reward kernel restored tracking performance relative to the quadratic-kernel result, and similar results on Anymal C supported generalization across reward designs.The authors attribute robot generalizability to incorporating constraints rather than to a specific reward design.Source: A. Comparison of velocity command… · p. 17

B. Extra demonstration of constraint’s engineering advantages

  • Adding body-contact, contact-velocity, and gait-pattern constraints solved the toy controller’s excessive contact, foot slippage, and failure to trot.The constraints used probabilistic thresholds of 0.025 for body contact and gait pattern, and an average threshold of 0.1 m/s for contact velocity.Source: B. Extra demonstration of constraint’s… · p. 18
  • Reward-only engineering required trial and error because coefficient sets made outcomes unpredictable and could interfere with velocity performance; the authors say constraints made intent easier to represent.The toy task began with two rewards for forward speed above 1 m/s and minimal torque usage.Source: B. Extra demonstration of constraint’s… · p. 18

Related papers

Loading 2308.12517v4…