PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Humanoid Standing-up Control across Diverse Postures

Tao Huang, Junli Ren, Huayi Wang, Zirui Wang, Qingwei Ben, Muning Wen, Xiao Chen, Jianan Li, Jiangmiao Pang

arXiv:2502.08378v2cs.ROcs.AIcs.LG

TL;DR

Humanoid standing-up control must work across postures and real-world conditions, but prior approaches rely on ground-specific trajectories or overlook hardware constraints. HoST learns control from scratch with multi-critic reinforcement learning, curriculum training, and motion constraints, achieving smooth, stable, robust deployment on Unitree G1 robots across laboratory and outdoor environments.

  • Problem

    Existing standing-up methods are limited by simulation-only hardware omissions or predefined ground-specific trajectories, leaving diverse real-world postures insufficiently addressed.

  • Method

    HoST trains from scratch with multi-critic reinforcement learning, diverse-terrain curriculum training, domain randomization, smoothness regularization, and implicit motion-speed constraints.

  • Results

    100% success rate and high motion smoothness are achieved across all evaluated scenes, including unseen outdoor terrains and complex postures.

  • Takeaways & Limitations

    HoST enables direct sim-to-real deployment of posture-adaptive standing-up motions that remain smooth, stable, and robust under external disturbances.

  • Takeaways & Limitations

    Outdoor failures occurred for some seated-standing attempts and collisions, indicating a need for environmental perception; training supine and prone postures also caused interference.

Abstract

from arXiv · show

Standing-up control is crucial for humanoid robots, with the potential for integration into current locomotion and loco-manipulation systems, such as fall recovery. Existing approaches are either limited to simulations that overlook hardware constraints or rely on predefined ground-specific motion trajectories, failing to enable standing up across postures in real-world scenes. To bridge this gap, we present HoST (Humanoid Standing-up Control), a reinforcement learning framework that learns standing-up control from scratch, enabling robust sim-to-real transfer across diverse postures. HoST effectively learns posture-adaptive motions by leveraging a multi-critic architecture and curriculum-based training on diverse simulated terrains. To ensure successful real-world deployment, we constrain the motion with smoothness regularization and implicit motion speed bound to alleviate oscillatory and violent motions on physical hardware, respectively. After simulation-based training, the learned control policies are directly deployed on the Unitree G1 humanoid robot. Our experimental results demonstrate that the controllers achieve smooth, stable, and robust standing-up motions across a wide range of laboratory and outdoor environments. Videos and code are available at https://taohuang13.github.io/humanoid-standingup.github.io/.

I. INTRODUCTION

Humanoid standing-up control remains underexplored despite its importance for transitions from seated postures and balance recovery. HoST learns posture-adaptive motions from scratch in simulation and directly deploys them on a Unitree G1 robot with smooth, stable, robust performance.

  • Motivation: Standing-up control is a foundational capability needed for seated transitions and recovery after loss of balance.
  • Limitations of existing methods: Existing trajectory-optimization methods require extensive model tuning and are computationally intensive, disturbance-sensitive, and dependent on accurate actuator models.
  • Challenges: RL-based standing-up control remains challenging because it combines dynamic whole-body coordination, time-varying contacts, multi-stage skills, and angular-momentum control.
  • HoST framework: HoST learns posture-adaptive standing-up policies from scratch using diverse terrains, force curriculum, multi-critic RL, smoothness regularization, and motion-speed constraints.
  • Contributions: The learned policies achieve real-world posture-adaptive motions without predefined trajectories or sim-to-real adaptation techniques.
  • Results: Smoothness, stability, and robustness are demonstrated consistently under challenging external disturbances.

C. Learning Quadrupedal Robot Standing-up Control

The control formulation combines reinforcement learning with proprioceptive observations, PPO optimization, and PD-based torque actuation. Action rescaling implicitly constrains motion speed, while the state includes temporal information for contact-rich control.

  • Problem formulation: The standing-up task is formulated as a finite-horizon Markov decision process and optimized with PPO for stable, efficient large-scale parallel training.
  • Observation space: The policy uses proprioceptive measurements from the IMU and joint encoders, including base motion, orientation, joint states, the previous action, and an action-scale factor.
  • Observation space: The previous five states are provided to the policy to enhance contact detection in the contact-rich standing-up task.
  • Action space: A PD controller converts position targets into torques using stiffness and damping coefficients.
  • Action space: The action rescaler constrains each action dimension to [−1, 1] and regulates motion speed implicitly through the target-position update.

IV. METHOD

HoST learns multi-stage humanoid standing-up control across diverse postures using staged rewards, multi-critic reinforcement learning, exploration assistance, diverse terrains, and sim-to-real randomization.

  • Framework overview: HoST formulates standing up as a dynamic, multi-stage control problem and learns it with reinforcement learning.The framework targets posture-adaptive standing up rather than a single predefined starting posture.
  • Reward design: The task is divided into righting, rising, and standing stages, with corresponding rewards activated according to the robot base height.The reward groups comprise task, style, regularization, and post-task objectives.
  • Exploration: A curriculum-based upward force on the robot base facilitates exploration during difficult early transitions.The strategy is motivated by external support during human infant motor-skill development.
  • Sim-to-real preparation: Four simulated terrains diversify initial postures, while domain randomization addresses physical discrepancies between simulation and the real robot.The terrain set includes ground, platform, wall, and slope; randomized factors include mass, center-of-mass offset, gains, torque offset, and initial pose.
  • Multiple critics: Multiple critics estimate returns independently for each reward group and integrate their advantages within PPO optimization.This separates reward-group learning to improve reward balancing and reduce tuning difficulty.

C. Force Curriculum as Exploration Strategy

HoST uses an upward force curriculum on the robot base to make exploration more effective during difficult standing-up transitions.

  • C. Force Curriculum as Exploration Strategy: An upward base force assists exploration during the transition from falling to stable kneeling.The force is largely set at training start and provides environmental assistance inspired by human infant skill development.

D. Motion Constraints

HoST constrains learned motion to improve physical deployability, using action rescaling and smoothness regularization while evaluating motion quality across diverse terrains.

  • D. Motion Constraints: Action rescaling gradually tightens output bounds, implicitly limiting joint torques and motion speed while preserving exploration.The constraint addresses violent ground impacts and rapid bouncing caused by high-power actuators and wide joint limits.
  • D. Motion Constraints: Smoothness regularization is applied to both the actor and critics through bounded-distance state sampling.The regularization objective is combined with ordinary PPO objectives.
  • Motion analysis: UMAP and 3D trajectory visualizations show terrain-specific but consistent motion patterns with coordinated hand-foot trajectories.Point colors encode motion progression from earlier lighter shades to later darker shades.
  • Terrain diversity: The four terrain types diversify starting postures by supporting different parts of the robot, including the trunk or whole body.The terrains are ground, platform, wall, and slope.

2) Domain Randomization:

HoST evaluates standing-up control using stability, smoothness, energy, and robustness-oriented measures while analyzing disturbance and speed–smoothness trade-offs.

  • 2) Domain Randomization:: Domain randomization varies mass, center-of-mass offset, PD gains, torque offset, and initial pose to bridge simulation and reality.Center-of-mass randomization improves robustness to real-world model and torque discrepancies.
  • 2) Domain Randomization:: The implementation uses 200 Hz simulation PD control, 500 Hz real-robot PD control, and 50 Hz policy execution.These settings are intended to support accurate tracking of PD targets.
  • Evaluation metrics: Success requires exceeding a target base height and maintaining it for the remainder of the episode, indicating stable standing.Feet movement after reaching the target height separately measures standing-pose stability.
  • Robustness analysis: Four environmental disturbances are used to evaluate robustness, with historical information and action-bound curriculum affecting robustness and energy consumption.The HOST-History1 and HOST-Bound0.25 variants expose these respective design effects.
  • Trade-off analysis: Motion speed and smoothness exhibit an inverse relationship across terrains, motivating constrained motion speed for deployment.The analysis also considers energy as a third motion-quality dimension.
  • Evaluation metrics: Motion smoothness aggregates consecutive-step joint-angle movement, while energy measures consumption before reaching the target height.Together, these metrics assess smooth motion and avoidance of violent standing-up behavior.

2) Baselines:

HoST is evaluated through ablations and trajectory analyses, showing that multiple critics, force curriculum, action bounds, and historical states shape standing-up performance, smoothness, and robustness across terrains.

  • Main results: HoST demonstrates effective standing-up control across all evaluated terrains.The study uses ablations to examine critics, exploration, motion constraints, and historical states.
  • Ablations: A single critic achieved zero success rates across all terrains despite using the same reward functions.This ablation highlights the role of multiple critics in learning and integrating motor skills.
  • Ablations: Force curriculum improves exploration when terrains require recovery from a fully fallen state to stable kneeling.Random-noise and curiosity-based exploration only partially alleviate this challenge.
  • Ablations: Action bounds prevent excessively violent motions and improve smoothness and success rates, while overly restrictive bounds produce less natural movements.The comparison includes HOST-w/o-Bound and HOST-Bound0.25.
  • Trajectory analysis: UMAP and 3D trajectory analyses show terrain-specific motion patterns with coordinated hand-foot adjustments that reflect posture adaptability.Flat-ground motion is smooth and controlled, whereas Wall trajectories are more complex but consistent.
  • Trade-off analysis: Motion speed and smoothness are inversely related, while energy consumption generally increases with fall-to-standing time and varies by terrain.Slope balancing requires more energy, and Wall becomes especially energy-intensive at longer durations.

VI. REAL ROBOT EXPERIMENTS

Real-robot experiments test direct sim-to-real transfer in laboratory and outdoor settings, finding smooth, stable, and successful standing-up alongside measurable sim-to-real discrepancies.

  • Laboratory experiments: Smoothness regularization produces smoother and more stable motions than the baseline without L2C2, especially on the 10.5° slope.The experiments compare HoST with HOST-w/o-L2C2.
  • Laboratory experiments: 100% success rate and high motion smoothness were achieved across all tested laboratory scenes.Without smoothness regularization, oscillations appeared in every scene and often caused failures.
  • Outdoor experiments: Controllers generalize to unseen outdoor terrains including grassland, wooden platforms, and stone roads while maintaining stable, smooth motion.They also handle stone platforms and tree-leaning postures.
  • Sim-to-real analysis: Domain randomization reduces the sim-to-real gap, with Center of Mass position having particularly strong influence.The analysis evaluates the contribution of individual randomization terms.
  • Posture diversity: HoST learns across prone ground postures and handles side-lying postures without tuning.The passage on posture diversity is included in the real-robot evaluation scope.
  • Sim-to-real analysis: Simulated and real knee and hip joint velocities differ notably, suggesting a gap in joint torques despite effective controller performance.These joints are identified as important for standing-up motions.

C. Emergent Properties

Emergent experiments show that HoST extends beyond standard standing-up tests to disturbances, dynamic balancing, diverse postures, and larger robots, while exposing deployment limitations.

  • Emergent properties: Controllers withstand 3 kg ball impacts, ground obstructions, and payloads up to 12 kg, twice the humanoid trunk mass.They also recover from large forces without fully falling and balance on a 15° slippery slope.
  • Prone and side-lying postures: HoST handles prone and side-lying postures, but jointly training prone and supine postures remains unclear because stronger hip constraints may be needed.Differences in motion patterns between prone and supine postures limit current training.
  • Larger robots: HoST extends to Unitree H1 and H1-2 robots with successful motions but requires high-stiffness joints to compensate for insufficient actuation.The larger platforms also rely more on upper-body ground contact and high hip actuation.
  • Conclusion: The framework achieves posture-adaptive, smooth, stable, and robust standing-up motions through reinforcement learning from scratch and deployment-oriented constraints.The conclusion emphasizes diverse terrains and direct real-world deployment on Unitree G1.
  • Limitations: Outdoor failures included standing from a seated position and collisions with surroundings, motivating integration of environmental perception.The paper also reports interference when training both supine and prone postures.

A. More Experimental Details

The experiments use a Unitree G1 platform and report motion metrics from repeated evaluations, including acceleration-based smoothness and torque-velocity-based energy.

  • Hardware and protocol: Experiments use a 35 kg, 1.32 m Unitree G1 with 23 actuated degrees of freedom, onboard Jetson computation, IMU, and joint encoders.Each policy is evaluated over 1,250 episodes per terrain using five repetitions of 250 episodes.
  • Metrics: Smoothness is computed from the squared second difference of joint positions over time.The expression uses p_t as joint positions.
  • Metrics: Energy is computed approximately from joint torque magnitude multiplied by joint velocity magnitude and integrated over time.τ_t denotes joint torques, ṗ_t joint velocities, and each policy step is 0.02 s.

B. More Implementation Details

Implementation details specify the curriculum, reward organization, PPO networks, terrain settings, and deployment adjustments used to train and transfer HoST across robot configurations.

  • Curriculum and training: The curriculum reduces vertical assistance from 200N by 20N and the action bound β from 1 by 0.02 whenever head height reaches its target, down to 0N and 0.25.
  • PPO implementation: The PPO actor is a 3-layer MLP with hidden dimensions [512, 256, 128], while each critic is a 2-layer MLP with dimensions [512, 256].
  • Terrains: Training terrains include platforms from 20cm to 92cm, slopes of approximately 1° to 14°, and walls inclined approximately 14° to 84°.
  • Reward design: Reward functions are grouped and independently normalized, with distinct critics estimating the corresponding advantage functions.
  • Reward design: The reward design penalizes excessive joint acceleration, action changes, oscillation, torque, power, velocity, tracking error, and joint-limit violations.
  • Hardware deployment: Deployment adapts PD stiffness to the simulation-to-reality torque gap, including hip and knee stiffness adjustments to 200 and 275, respectively.
  • Robot and posture extensions: Extensions to prone postures and Unitree H1/H1-2 modify reward constraints, curriculum scales, stage heights, target postures, controllers, and observation or action spaces.
Loading 2502.08378v2…