Source-linked AI summary
Learning Getting-Up Policies for Real-World Humanoid Robots
Xialin He, Runpei Dong, Zixuan Chen, Saurabh Gupta
TL;DR
Humanoid fall recovery must handle varied configurations, complex contacts, and sparse rewards across challenging terrains. HUMANUP uses two-stage reinforcement learning to discover recovery trajectories with minimal constraints, then refine them into slowed, regularized policies for deployment. The framework enables a real-world G1 humanoid to recover from supine and prone situations across varied terrains.
Problem
Humanoid fall recovery is difficult because getting up involves non-periodic, rich contacts and sparse rewards across varied configurations and terrains.
Method
HUMANUP first discovers getting-up motions with weak constraints, then trains a policy to track 8× slowed trajectories under strong regularization and varied conditions.
Results
The framework enables a real-world G1 robot to get up from supine and prone poses across six terrains and achieves a 78.3% getting-up success rate.
Takeaways & Limitations
The two-stage strategy is effective in simulation and the real world for recovery from different starting poses and terrains.
Takeaways & Limitations
HUMANUP depends on high-performance, high-frequency physics simulation, and its under-specified reinforcement-learning formulation may permit reward hacking.
Abstract
from arXiv · showhide
Automatic fall recovery is a crucial prerequisite before humanoid robots can be reliably deployed. Hand-designing controllers for getting up is difficult because of the varied configurations a humanoid can end up in after a fall and the challenging terrains humanoid robots are expected to operate on. This paper develops a learning framework to produce controllers that enable humanoid robots to get up from varying configurations on varying terrains. Unlike previous successful applications of learning to humanoid locomotion, the getting-up task involves complex contact patterns (which necessitates accurately modeling of the collision geometry) and sparser rewards. We address these challenges through a two-phase approach that induces a curriculum. The first stage focuses on discovering a good getting-up trajectory under minimal constraints on smoothness or speed / torque limits. The second stage then refines the discovered motions into deployable (i.e. smooth and slow) motions that are robust to variations in initial configuration and terrains. We find these innovations enable a real-world G1 humanoid robot to get up from two main situations that we considered: a) lying face up and b) lying face down, both tested on flat, deformable, slippery surfaces and slopes (e.g., sloppy grass and snowfield). This is one of the first successful demonstrations of learned getting-up policies for human-sized humanoid robots in the real world.
I. INTRODUCTION
HUMANUP addresses the challenge of recovering humanoid robots from unpredictable falls across varied terrains. The task is harder than typical locomotion because contacts are non-periodic and rich, rewards are sparse, and deployable control must be learned.
- Motivation: Humanoid fall recovery is important because unpredictable configurations and terrains can require human intervention after falls.At the DARPA Robotics Challenge, 26 of 46 trials involved falls, and 25 of those required human recovery intervention.
- Challenges: Getting up differs from locomotion because the required contact sequence is non-periodic and must be discovered.Structured cyclic foot contacts and phase coupling used in locomotion do not directly apply.
- Challenges: Getting-up policies must model rich contacts involving body parts beyond the feet and detailed collision geometry.Freezing or coarsely modeling the upper body, common in locomotion, is unsuitable for this task.
- Challenges: Sparse rewards make learning difficult because parts of the body may move downward for seconds before the robot rises.Unlike velocity tracking, meaningful progress is not immediately visible during many phases of getting up.
- Approach: HUMANUP uses two-stage reinforcement learning: Stage I discovers motion with weak constraints, while Stage II makes it deployable under stronger regularization and variability.The curriculum progresses from simplified settings toward varied terrains, initial poses, and stronger control requirements.
- Results: Real-world experiments with the G1 cover supine and prone recovery across six terrains, while simulation handles varied terrains and starting poses.The manufacturer controller is reported to succeed only from supine poses on a flat surface without bumps.
1) Learning for humanoid control:
Prior work established learning-based control for locomotion and recovery, but learned getting-up policies for real humanoid robots remained difficult to transfer and generalize. HUMANUP builds on simulation-based reinforcement learning while targeting complex contacts and real-world deployment.
- Learning for humanoid control:: Simulation-based reinforcement learning has enabled quadruped and humanoid locomotion on challenging terrain and agile motions.Related work includes in-the-wild locomotion, jumping, visual locomotion, dancing, and naturalistic walking.
- Fall recovery: Humanoid robots are vulnerable to falls because of under-actuated dynamics, high-dimensional states, and unstructured environments.Fall recovery is therefore treated as a significant control problem.
- Prior recovery methods: Earlier getting-up methods used configuration graphs and discrete subgoals, while later work also addressed toy-sized humanoids.These approaches span hierarchical reinforcement learning and stored transitions between lying and standing states.
- Prior recovery methods: Commercial recovery controllers commonly replay manually designed trajectories, whose main drawback is limited adaptability to varied fallen states.The related work describes such predefined trajectories for Unitree and Booster robots.
- Quadruped recovery: Reinforcement learning and sim-to-real have produced quadruped recovery across complex configurations, snowy terrain, and dynamic scenarios.These results motivate applying learned recovery methods beyond quadrupeds.
- Character animation: Character-animation systems have demonstrated simulated getting-up through motion imitation, curricula, sampling, and online model-predictive control.Their humanoid characters often use larger degrees of freedom and simplified dynamics.
- Open problem: Policies learned for simulated characters can use infeasible high-velocity body motions, leaving generalizable real-robot recovery an open problem.The passage explicitly contrasts simplified character dynamics with transfer requirements for humanoid robots.
III. HUMANUP: SIM-TO-REAL HUMANOID GETTING UP
HUMANUP decomposes getting up into motion discovery and deployable imitation. It trains both policies with reinforcement learning, using task rewards for discovery and trajectory tracking for deployment.
- Task decomposition: HUMANUP handles supine and prone lying postures, which may require different behaviors.The prone task is decomposed by first rolling over before getting up.
- Stage I: Stage I trains a discovery policy f without deployment constraints to find standing-up or rolling-over motions.Only task and symmetry rewards are used in this stage.
- Policy architecture: Both policy models use proprioceptive observations, a 10-step state history, and learned 54-dimensional environment latents.The proprioceptive state includes roll, pitch, angular velocity, joint velocities, and joint positions.
- Stage I: The getting-up reward combines height, height change, uprightness, standing on the feet, foot contact forces, and symmetry.These terms provide task-specific incentives for discovering a recovery motion.
- Stage I: The rolling-over reward encourages body orientation toward the projected-gravity configuration associated with lying face up.This reward is used for the rolling-over component of the task.
- Stage II: Stage II trains deployable policy π to imitate an 8× slowed version of Stage I trajectories under strong control regularization.Tracking rewards encourage matching reference joint positions and body states.
2) Stage II: Deployable Policy:
Stage II converts discovered motions into deployable policies by tracking slowed trajectories under strong regularization, full collision geometry, varied initial poses, and terrain variation.
- Tracking: Tracking rewards make the policy follow reference joint positions and body states from the discovered motion.Body tracking includes upright head height for getting up and head orientation for rolling over.
- Curriculum: The two-stage design forms a curriculum from motion discovery in simplified settings to deployable, generalizable motion.Stage I uses weak regularization and simpler collision geometry; Stage II increases task and environment complexity.
- Collision modeling: Stage I uses a simplified collision mesh, whereas Stage II uses the full mesh for improved sim-to-real performance.The mesh transition addresses the richer collision modeling needed during deployment-oriented training.
- Initial poses: Stage II randomizes initial poses using 20K supine and 20K prone poses generated from randomized drops and collision-resolution simulation.The passage states that 10K poses from each set are used for training.
- Sim-to-real transfer: Stage II strengthens control regularization and slows motion to make discovered behaviors more suitable for sim-to-real transfer.Without strong regularization, Stage I can discover a fast getting-up motion taking less than 1s, which is infeasible for real-world deployment.
- Platform configuration: The G1 platform has 29 actuated degrees of freedom, with wrist joints disabled to leave 23 controlled degrees of freedom.Position control uses a PD controller operating at 50 Hz.
B. Simulation Configurations
The simulation evaluates three getting-up subtasks and compares HUMANUP with learning and ablated baselines that vary deployment constraints, collision modeling, posture randomization, symmetry, and curriculum design.
- Simulation setup: Simulation training uses simplified collision geometry for Stage I and the official whole-body URDF for Stage II.
- Tasks: The evaluation covers supine getting up, prone-to-supine rolling over, and prone getting up by consecutively solving the first two tasks.
- Baselines: Baselines include simple task-reward RL, HUMANUP without Stage II, simplified collision geometry, posture randomization, hard symmetry, or two-stage learning.
- Ablations: The single-stage ablation applies the full collision mesh, posture randomization, and all rewards and regularization terms simultaneously.
C. Metrics •
The paper defines task success, task-specific performance, smoothness, and safety metrics for evaluating getting-up policies on held-out postures.
- Task success: Getting-up success requires the robot’s head height to reach at least 1.1m at termination and remain standing.
- Task success: Rolling-over success requires the base, knee, and torso orientations to align with the target at cosine similarity at least 0.9.
- Smoothness: Smoothness is measured with action jitter, DoF position jitter, and mean energy, with jitter computed from third derivatives of motion.
- Safety: Safety scores measure commanded torque and joint displacement relative to their respective limits, using a threshold δ for excessive values.
- Safety: Evaluation uses δ = 0.8, α = 0.5, and β = 0.5 as default safety parameters.
- Evaluation protocol: Results are reported on held-out validation sets containing 10K supine and 10K prone posture samples.
1) Ignoring Torque / Control Limits Leads to Undeployable Policies:
The ablations show that unconstrained or insufficiently randomized policies are difficult to deploy, while two-stage learning and full collision modeling support real-world recovery across varied terrains.
- Deployment constraints: The simple task-reward baseline achieves similar success but has nearly 18× higher average action jitter and cannot be safely deployed.
- Deployment constraints: Removing Stage II produces policies with unsatisfying smoothness and safety, making deployment inappropriate.
- Real-world comparison: HUMANUP succeeds on 78.3% of real-world trials versus 41.7% for the manufacturer controller across terrains ranging from concrete to snow and slopes near 10°.
- Curriculum: Single-stage training with all Sim2Real regularization fails to solve the task, whereas the two-stage curriculum combines task completion with safe operation.
- Collision modeling: Without the full URDF, the policy generalizes in simulation but fails all 5 real-world trials on flat terrain because expected and actual contacts differ.
- Robustness: Posture randomization improves generalizable control, as removing it makes HUMANUP perform much worse.
- Execution comparison: HUMANUP reaches standing within 6 seconds, compared with 11 seconds for the manufacturer controller, while relying more on stronger leg motors.
A. Results
HUMANUP outperforms the manufacturer-provided controller across challenging terrains and achieves strong success rates for getting-up and rolling-over tasks.
- HUMANUP policies perform better than the manufacturer-provided controller and the variant without posture randomization.
- Terrain and domain randomization improve robustness to slipperiness, bumps, and slopes, while posture randomization supports diverse initial configurations.
- HUMANUP achieves a 78.3% getting-up success rate across varied real-world conditions.
- HUMANUP achieves a 98.3% success rate for rolling over, with the full model more robust than the version without posture randomization.
1) Motor temperature:
HUMANUP produces smoother, faster, and more load-efficient getting-up motions, while challenging terrains still expose failure modes involving foot stability and slippage.
- Motor temperature:: HUMANUP uses stronger leg motors more extensively, while the manufacturer controller causes significantly greater arm-motor heating.The leg motors have an 83N torque limit compared with 25N for the arm motors.
- Failure modes:: On slopes and snow, HUMANUP may partially recover but can fall because of unstable foot placement or ground slippage.
- Efficiency:: HUMANUP gets the robot standing in about 6 seconds, over 2× more efficiently than the manufacturer controller’s nearly 11 seconds.
- Limitations:: The authors identify complex terrain extension and stronger-arm behaviors as areas requiring further study.
- Method:: The two-stage framework discovers motions with minimal constraints, then refines them for smooth, deployable behavior under varied conditions.
APPENDIX
The appendix organizes reward definitions, additional visualizations, and training details for the two-stage HUMANUP framework.
- Appendix contents: Additional appendix sections cover reward definitions, baseline visualization, recovery from sitting or leaning, and external-turbulence robustness.
- Training details: Table II summarizes Stage I reward components and their weights for preventing undesired behavior, refining motion, and completing the task.
- Training details: The appendix separately documents Stage I and Stage II reward components.
- Training details: Table III summarizes Stage II reward components and weights for penalty, regularization, and successful whole-body tracking in real time.
B.1 Additional Baseline Result Visualization
Additional experiments show that HUMANUP generalizes beyond lying poses and tolerates certain disturbances, while the comparison baseline produces unstable motions unsuitable for deployment.
- Baseline visualization: The baseline method generates highly unstable, unsafe prone-recovery motions with jittering joints, stumbling feet, and repeated jumping.
- Additional initial poses: HUMANUP generalizes to sitting on stairs and leaning against a wall, with convergence approximately 4× faster than recovery from lying.The authors attribute the easier recovery to additional support from the ground, chairs, or walls.
- External turbulence: HUMANUP is practically robust to certain real-world disturbances, including hard-stick thumps and hurled objects.
- Training details: The getting-up trajectory is slowed to 8 seconds for deployment after Stage I discovery.The authors also tested 4× and 10× slowdowns; 4× produced large torques and velocities, while 10× did not converge.