Source-linked AI summary
Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach
Dohyeong Kim, Hyeokjin Kwon, Junseok Kim, Gunmin Lee, Songhwai Oh
TL;DR
Complex RL tasks make reward shaping difficult, especially when acrobatics require different reward emphasis across stages. The paper uses stage-wise rewards and costs in a CMORL framework with a PPO-based algorithm. It reports successful acrobatic demonstrations in simulation and the real world, and says its method alone completed tasks in comparisons with existing RL and constrained RL algorithms.
Problem
Defining and weighting many reward terms is laborious, while acrobatic tasks can require different emphasis on movements such as jumping and landing.
Source: §I · p. 1Method
The method defines independent reward and cost functions for each task stage within CMORL and adapts PPO to handle multiple objectives and constraints.
Source: §I · pp. 1–2Results
The method demonstrated acrobatic tasks in simulation and the real world and was the only compared RL or constrained RL method to complete the tasks.
Source: §I · p. 2Takeaways & Limitations
The comparisons support the authors’ conclusion that the CMORL framework is necessary for the demonstrated acrobatic tasks.
Source: §VI · p. 6Tasks are manually segmented into stages; the authors identify more efficient automatic stage division as future work.
Source: §VI · p. 6
Abstract
from arXiv · showhide
As the complexity of tasks addressed through reinforcement learning (RL) increases, the definition of reward functions also has become highly complicated. We introduce an RL method aimed at simplifying the reward-shaping process through intuitive strategies. Initially, instead of a single reward function composed of various terms, we define multiple reward and cost functions within a constrained multi-objective RL (CMORL) framework. For tasks involving sequential complex movements, we segment the task into distinct stages and define multiple rewards and costs for each stage. Finally, we introduce a practical CMORL algorithm that maximizes objectives based on these rewards while satisfying constraints defined by the costs. The proposed method has been successfully demonstrated across a variety of acrobatic tasks in both simulation and real-world environments. Additionally, it has been shown to successfully perform tasks compared to existing RL and constrained RL algorithms. Our code is available at https://github.com/rllab-snu/Stage-Wise-CMORL.
I. INTRODUCTION
Reward shaping becomes difficult as tasks grow more complex, particularly for acrobatics that require shifting emphasis across movement stages. The paper introduces stage-wise rewards and costs in CMORL, with a PPO-based algorithm, and reports successful demonstrations and comparisons with existing methods.
I. INTRODUCTION
- Reward shaping is laborious because many performance, safety, and regularization terms must be defined and weighted, with acrobatics adding stage-dependent reward emphasis.For a back-flip, jumping matters early and landing later; imitation approaches require extensive task-specific data.Source: §I · p. 1
- The method treats each task term as an independent reward or safety-related cost, then defines reward and cost functions for each task stage.CoMOPPO adapts PPO to handle multiple objectives and constraints; sim-to-real techniques include domain randomization and teacher-student distillation.Source: §I · pp. 1–2
- The proposed method was the only compared RL or constrained RL algorithm to complete the tasks, and it demonstrated two-hand walks, back-flips, side-flips, and side-rolls, including real-world demonstrations.The real-world quadrupedal demonstrations were side-roll, back-flip, and two-hand walk tasks.Source: §I · p. 2
II. RELATED WORK
Prior CMORL work includes single-policy and multi-policy approaches, while constrained RL for legged robots has mainly addressed locomotion. This paper places its approach within those lines of work and extends legged-robot methods to acrobatic movements.
A. Constrained Multi-Objective Reinforcement Learning
- CMORL algorithms extend multi-objective methods to constraints; CoMOPPO is described as a single-policy method that simplifies CoMOGA’s implementation.The related work distinguishes single-policy methods, which seek one Pareto-optimal policy, from multi-policy methods, which seek a set.Source: A. Constrained Multi-Objective… · p. 2
B. Reinforcement Learning for Legged Robots
- Constrained RL for legged robots has used explicit safety constraints and shown robustness to reward weights, but its tasks have primarily been limited to locomotion.The paper extends these approaches to CMORL for more complex acrobatic movements.Source: B. Reinforcement Learning for Legged… · p. 2
III. BACKGROUND
The background defines a constrained multi-objective Markov decision process and the policy, trajectory, and value-function concepts used to formulate its objectives and constraints.
A. Constrained Multi-Objective Markov Decision Processes
- A CMOMDP is specified by state and action spaces, transition model, initial-state distribution, discount factor, reward functions, and cost functions.The formulation includes N reward functions and M cost functions.Source: A. Constrained Multi-Objective Markov… · p. 2
- A policy maps states to action distributions, while a trajectory is a sequence of states and actions generated under the policy and transition model.The initial state is drawn from the initial distribution.Source: A. Constrained Multi-Objective Markov… · p. 2
- Reward and cost value and advantage functions provide the quantities used to construct CMORL objectives and constraints, respectively.The cost functions’ value and advantage functions are defined by substituting costs for rewards.Source: A. Constrained Multi-Objective Markov… · p. 2
B. CMORL Problem Setup
The CMORL setup seeks policies that optimize multiple rewards while respecting cost thresholds, with constrained-Pareto optimality as its target. The back-flip example also specifies stage transitions and reward/cost definitions.
B. CMORL Problem Setup
- A CMORL problem seeks a constrained-Pareto optimal policy whose costs meet their thresholds while optimizing reward objectives.The back-flip example’s table separates reward definitions from cost definitions.Source: B. CMORL Problem Setup · p. 2, Table I · p. 3
- For the back-flip, Table I lists five reward functions followed by five cost functions, with foot-contact cost set to its threshold outside the jump stage.The foot-contact cost is undefined in those other phases.Source: Table I · p. 3
- The back-flip transitions from stand to jump below 0.25 m base height, to air when all feet detach, and to landing at first foot contact.The figure describes sitting after a back-flip command before the jump stage.Source: Figure 3 · p. 3
- A feasible policy is CP optimal when no other feasible policy dominates it; a preference vector can select a specific policy from the non-unique set.The objectives may be combined by a weighted sum or another scalarization method.Source: B. CMORL Problem Setup · p. 3
IV. PROPOSED METHOD
The proposed method combines stage-wise reward shaping with a CMORL policy update and sim-to-real techniques. For sequential acrobatics, it assigns rewards and costs by stage, illustrated through back-flip definitions and trained-task snapshots.
IV. PROPOSED METHOD
- The framework combines stage-wise reward shaping, a multi-objective constrained policy update, and sim-to-real techniques for real-world deployment.The method section presents these as its three main parts.Source: §IV · p. 3
A. Stage-Wise Reward Shaping
- Stage-wise CMORL keeps reward terms as separate objectives and safety-related terms as constraints, while dividing sequential tasks into stages to clarify required motions.The authors state that this makes reward shaping more straightforward than dynamically adjusting individual reward-term weights.Source: A. Stage-Wise Reward Shaping · p. 3
2) Reward and Cost Functions:
- For the back-flip, the five stages are Stand-Sit-Jump-Air-Land, with transitions keyed to the robot’s command, height, foot detachment, and ground contact.The stage example is shown in Fig. 3.Source: 2) Reward and Cost Functions · p. 3
- Back-flip rewards target stillness in stand, sit, and land; backward rotation and height in jump and air; balance, natural motion, and bounded joint values are also addressed.Body-contact cost discourages falling, while rear-foot detachment before front-foot detachment incurs a jump-specific cost.Source: 2) Reward and Cost Functions · p. 3
- Trained-policy snapshots show Go1 side-roll and two-hand walk in the real world, and Go1 side-flip and H1 back-flip in simulation.Fig. 4 presents these motion sequences.Source: Figure 4 · p. 4
B. CMORL Policy Update
CoMOPPO updates a policy to maximize multiple objectives while satisfying constraints, using weighted reward and cost advantages. Reward normalization and advantage standardization help keep objectives and constraints on consistent scales during updates.
B. CMORL Policy Update
- CoMOPPO aggregates reward and cost advantages with weighted summation, then uses the aggregate for policy updates.The weights must satisfy conditions described in Kim et al. [14].Source: B. CMORL Policy Update · p. 4
- Reward normalization brings each reward function and stage to a consistent scale and is also used to train the value functions.This adjusts the relative scale of objectives.Source: B. CMORL Policy Update · p. 4
- Standard-deviation normalization of reward and cost advantages maintains a consistent ratio between objectives and constraints.The paper notes that an inconsistent ratio can lead to over-maximizing objectives rather than satisfying constraints, potentially destabilizing training.Source: B. CMORL Policy Update · p. 4
- The aggregated advantages are used in a PPO update to simplify implementation.The update is formulated as PPO [15].Source: B. CMORL Policy Update · p. 4
C. Sim-to-Real Techniques
For real-world deployment, the paper uses domain randomization and teacher-student learning to transfer policies trained in simulation. Randomized simulator properties and sensor-only student inputs are part of this process.
C. Sim-to-Real Techniques
- The two techniques are used to deploy policies trained in simulation to real-world environments.The paper identifies domain randomization and teacher-student learning as its sim-to-real techniques.Source: C. Sim-to-Real Techniques · pp. 4–5
- Domain randomization varies robot and environment properties, including motor strength and offset, gravity, friction, restitution, sensor noise, and base orientation.The randomization ranges are given in the appendix of.Source: C. Sim-to-Real Techniques · p. 5
- Teacher-student learning distills a teacher using privileged state information into a student that relies on sensor data such as joint position and base orientation.Teacher and student actions are alternated at fixed intervals to collect a dataset aligned with the student policy.Source: C. Sim-to-Real Techniques · p. 5
V. EXPERIMENTS
The experiments cover simulation and real-world settings, using Go1 and H1 robots in simulation and deploying Go1 in the real world. The state representation includes shared robot state and additional privileged information for the teacher policy.
A. Environmental Setup
- Experiments use Isaac Gym with Go1 quadruped and H1 humanoid robots in simulation, and deploy Go1 in the real world.Go1 has 23 body links and 12 motors; H1 has 20 body links and 19 motors.Source: A. Environmental Setup · p. 5
- The state representation includes body orientation, joint positions and velocities, commands, and the previous action.Source: A. Environmental Setup · p. 5
- The teacher policy additionally receives simulated privileged information, including linear and angular velocities, height, and foot contact.These quantities are difficult to access in the real world but available in simulation.Source: A. Environmental Setup · p. 5
B. Task Details
The paper evaluates four acrobatic tasks: back-flip, side-flip, side-roll, and two-hand walk. The descriptions specify the motions and, for the side-roll, its five-stage sequence.
B. Task Details
- The four tasks are back-flip, side-flip, side-roll, and two-hand walk; back-flip and two-hand walk are used on both robots, while the other two are quadruped tasks.Source: B. Task Details · p. 5
- The back-flip requires a backward jump, a 360-degree airborne rotation without ground contact, and landing on the feet in the initial pose.Source: B. Task Details · p. 5
2) Side-Flip:
- The side-flip resembles the back-flip but jumps to the right rather than backward, with identical stage transitions.Source: 2) Side-Flip · p. 5
3) Side-Roll:
- The side-roll proceeds through Stand, Sit, Half-roll, Full-roll, and Recover stages.The robot rolls along its right side and returns to its original pose.Source: 3) Side-Roll · p. 5
1) Back-Flip:
The two-hand walk is divided into Stand, Tilt, and Walk; the results describe successful acrobatic performances in simulation and real-world settings, with closely matched sim-to-real behavior.
4) Two-Hand Walk:
- The two-hand walk has three stages: Stand maintains the default pose, Tilt prepares the robot, and Walk uses only its two hands.Preparation involves lowering the front legs or placing the hands on the ground.Source: 4) Two-Hand Walk · p. 5
C. Results
- The robots successfully performed acrobatic tasks in simulation and real-world settings, and real-world behavior closely matched simulation patterns.The comparison tracks joint positions and body orientation across tasks.Source: C. Results · pp. 5–6
- The Go1 side-flip showed lowering and rightward rotation, while the H1 back-flip showed backward rotation and the H1 two-hand walk maintained balance by alternating hand support.In the two-hand walk, forward tilt and joint oscillations indicated repeated foot lifting and placement to stabilize balance.Source: C. Results · p. 6
D. Ablation Study
CoMOPPO alone completed the Go1 back-flip among the compared algorithms. The paper concludes that its stages were manually segmented and identifies automatic segmentation as future work.
D. Ablation Study
- PPO weight-summed reward and cost functions, while P3O summed rewards and retained the same constraints as CoMOPPO.The comparison used weights derived from CoMOPPO's average objective and constraint ratios during training.Source: D. Ablation Study · p. 6
- CoMOPPO alone successfully completed the Go1 back-flip; PPO and P3O remained primarily in Sit and did not properly execute the jump.CoMOPPO moved quickly through brief Sit and Jump stages into Air.Source: D. Ablation Study · p. 6
VI. CONCLUSIONS
- The method defines stage-wise rewards and costs in CMORL and extends PPO to handle multiple objectives and constraints; stages were manually segmented.The authors identify automatic division of complex tasks into stages as future research.Source: §VI · p. 6