PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Humanoid Standing-up Control across Diverse Postures

Tao Huang, Junli Ren, Huayi Wang, Zirui Wang, Qingwei Ben, Muning Wen, Xiao Chen, Jianan Li, Jiangmiao Pang

arXiv:2502.08378v2cs.ROcs.AIcs.LG

TL;DR

Humanoid standing-up control remains limited by approaches that overlook hardware constraints or rely on ground-specific trajectories. HoST learns posture-adaptive motions from scratch, and its Unitree G1 policies achieved a 100% success rate with high motion smoothness across tested scenes.

  • Problem

    Existing RL-based standing-up control faces a highly dynamic, multi-stage, contact-rich task, while trajectory-guided approaches are typically ground-posture-specific, leaving scalability to other postures unclear.

    Source: §I · p. 2

  • Method

    HoST trains standing-up policies from scratch across diverse terrains using multi-critic reinforcement learning and force curriculum, with domain randomization and motion constraints for real-world deployment.

    Source: §I · p. 2, 4) Sim-to-Real Gap (Section IV-E) · p. 4

  • Results

    100% success rate and high motion smoothness across all scenes; controllers also handled outdoor terrains absent from training.

    Source: A. Main Results · p. 8

  • Takeaways & Limitations

    Experiments on the Unitree G1 demonstrate that standing-up policies can operate across varied real-world laboratory and outdoor scenarios.

    Source: §VII · p. 9

    Outdoor seated-start tests sometimes failed when the robot collided with surroundings, motivating integration of environmental perception.

    Source: §VIII · p. 10

Abstract

from arXiv · show

Standing-up control is crucial for humanoid robots, with the potential for integration into current locomotion and loco-manipulation systems, such as fall recovery. Existing approaches are either limited to simulations that overlook hardware constraints or rely on predefined ground-specific motion trajectories, failing to enable standing up across postures in real-world scenes. To bridge this gap, we present HoST (Humanoid Standing-up Control), a reinforcement learning framework that learns standing-up control from scratch, enabling robust sim-to-real transfer across diverse postures. HoST effectively learns posture-adaptive motions by leveraging a multi-critic architecture and curriculum-based training on diverse simulated terrains. To ensure successful real-world deployment, we constrain the motion with smoothness regularization and implicit motion speed bound to alleviate oscillatory and violent motions on physical hardware, respectively. After simulation-based training, the learned control policies are directly deployed on the Unitree G1 humanoid robot. Our experimental results demonstrate that the controllers achieve smooth, stable, and robust standing-up motions across a wide range of laboratory and outdoor environments. Videos and code are available at https://taohuang13.github.io/humanoid-standingup.github.io/.

I. INTRODUCTION · II. RELATED WORK · A. Learning Humanoid Standing-up Control

HoST learns humanoid standing-up control from scratch across diverse postures and deploys directly on the Unitree G1. Its simulated training combines multi-critic RL, terrain diversity, and motion constraints to support smooth, stable, and robust real-world motions.

I. INTRODUCTION

  • Prior standing-up systems often assume a pre-standing posture, leaving seated transitions and recovery after loss of balance less addressed.The paper frames standing up across diverse postures in real environments as an underexplored capability.Source: §I · pp. 1–2
  • Handcrafted trajectory tracking can generate motions but requires analytical-model tuning and is sensitive to disturbances and inaccurate actuator models.Real-time optimization is computationally intensive, prompting reduced precision or offboard computation, each with practical limitations [34] [8].Source: §I · p. 2
  • RL standing-up is challenging because it combines time-varying contacts, multi-stage motor skills, and angular-momentum control; trajectory-guided approaches remain ground-specific [35] [12].Wide exploration from scratch can produce violent, abrupt motions that hinder deployment, particularly for robots with many actuators and wide joint limits.Source: §I · p. 2
  • HoST trains from scratch across diverse postures using multiple terrains, an initial vertical pull force, multi-critic RL, smoothness regularization, and motion speed constraints.The multi-critic architecture optimizes distinct reward groups independently; the motion constraints target oscillatory and violent behavior.Source: §I · p. 2
  • HoST achieves real-world posture-adaptive standing-up without predefined trajectories or sim-to-real adaptation, with smooth, stable, and disturbance-robust motions.The policies were trained in simulation with domain randomization and tested in laboratory and outdoor environments, including under forces, stumbling blocks, and heavy payloads.Source: §I · p. 2
  • The paper presents evaluation protocols designed to comprehensively analyze standing-up control and guide future research and development.The reported real-world evaluations span laboratory and outdoor environments.Source: §I · p. 2

A. Learning Humanoid Standing-up Control

  • Earlier methods had not demonstrated real-world standing-up across diverse postures; HoST targets posture adaptivity and deployability without predefined motions.Classical optimization methods are computationally intensive, disturbance-sensitive, and require precise actuator models [15] [23].Source: A. Learning Humanoid Standing-up Control · p. 2

B. Reinforcement Learning for Humanoid Control … 2) Action Space:

The paper frames humanoid standing-up as a distinct, contact-rich control problem and develops an RL approach for posture-adaptive motions. It specifies the MDP, observations, and PD-controller actions used to train and deploy the policy.

B. Reinforcement Learning for Humanoid Control

C. Learning Quadrupedal Robot Standing-up Control

  • Quadruped standing-up methods include model-based optimization and predefined primitives and RL recovery or pose transitions [23] [30]; HoST extends these ideas to bipedal control.The framework targets posture adaptivity and robust motions across diverse environments.Source: C. Learning Quadrupedal Robot… · p. 3

III. PROBLEM FORMULATION

  • The task is a finite-horizon MDP, and PPO [42] is used for stable, efficient large-scale parallel training.The policy seeks to maximize discounted cumulative reward, estimated with a value-function critic.Source: §III · p. 3

1) Observation Space:

  • The policy observes IMU and encoder proprioception, including base angular velocity, roll and pitch, joint positions and velocities, the previous action, and action scale β.It also receives the previous five states to implicitly enhance contact detection.Source: 1) Observation Space · p. 3

2) Action Space:

  • Each action dimension is bounded to [−1, 1] and offsets current joint positions to form PD targets; the scalar β rescales actions to implicitly regulate motion speed.The action-space dimension equals the number of actuators; the PD controller uses stiffness Kp and damping Kd.Source: 2) Action Space · p. 3

IV. METHOD … 3) Motion Constraints (Section IV-D):

HoST is a reinforcement-learning framework for learning humanoid robots to stand up across diverse postures. Its method addresses the task’s dynamic, multi-stage, contact-rich challenges through staged rewards, multi-critic learning, curriculum-based exploration, and motion constraints.

IV. METHOD

  • HoST frames humanoid standing up across diverse postures as a challenging, dynamic, multi-stage, contact-rich reinforcement-learning task.The method overview introduces the framework and its task scope.Source: §IV · p. 3

1) Reward Design & Optimization (Section IV-B):

2) Exploration Challenges (Section IV-C):

  • A curriculum-based vertical pulling force on the robot’s base link facilitates exploration despite its high degrees of freedom and wide joint limits, drawing inspiration from infant skill development.Source: 2) Exploration Challenges (Section IV-C) · p. 3

3) Motion Constraints (Section IV-D):

4) Sim-to-Real Gap (Section IV-E): · B. Reward Functions & Multiple Critics

HoST addresses sim-to-real transfer with diverse terrains and domain randomization, and handles standing-up’s staged objectives using reward groups with multiple critics. The critics estimate reward-group returns independently and are integrated into PPO optimization.

4) Sim-to-Real Gap (Section IV-E):

  • Diverse terrains model real-world starting postures, while domain randomization reduces the effect of physical discrepancies between simulation and reality.Source: 4) Sim-to-Real Gap (Section IV-E) · p. 4

B. Reward Functions & Multiple Critics

C. Force Curriculum as Exploration Strategy … 2) Smoothness Regularization:

HoST uses an upward-force curriculum to support exploration from ground sitting, then reduces assistance as the robot maintains target height. Action rescaling and smoothness regularization constrain motion while policies are trained with PPO.

C. Force Curriculum as Exploration Strategy

  • An upward base force acts only when the trunk is near vertical, and decreases as the robot maintains a target height at episode end.The force magnitude is largely set at the start of training.Source: C. Force Curriculum as Exploration… · p. 4
  • Because random action noise alone makes the transition from falling to stable kneeling difficult to explore, external environmental assistance is used to accelerate exploration.The design is inspired by infants learning motor skills with external supports.Source: C. Force Curriculum as Exploration… · p. 4

1) Action Bound (Rescaler):

  • To avoid violent motions without restricting exploration, an action rescaler β scales policy outputs and implicitly bounds maximal actuator torques, decreasing like the vertical force.The motivating behaviors include violent ground hitting and rapid bouncing after RL training.Source: 1) Action Bound (Rescaler) · p. 4

2) Smoothness Regularization:

  • To prevent motion oscillation, L2C2 regularizes both the actor network πθ and the critics Vϕi using a bounded sampling distance between consecutive states.The regularization is incorporated into the multi-critic formulation.Source: 2) Smoothness Regularization · pp. 4–5
  • Figure 4 visualizes terrain-specific but overall consistent joint-space trajectories and stable, coordinated hand-foot motion that adapts to starting postures.The caption notes an exception to the terrain-pattern similarity for the wall.Source: Figure 4 · p. 5
  • The regularization defines an interpolated state using uniform noise between consecutive states, with distance metric D and weights λπ and λV.The interpolated state is ¯st = st +(st+1 −st)·u, where u ∼U(·).Source: 2) Smoothness Regularization · p. 5
  • The regularization objective is combined with ordinary PPO objectives to train the control policies.Source: 2) Smoothness Regularization · p. 5

E. Training in Simulation & Sim-to-Real Transfer … V. SIMULATION EXPERIMENTS

The paper trains standing-up policies in simulation across varied starting postures and uses domain randomization to support real-world deployment. The supplied passages describe the setup and implementation, but do not report simulation experiment results.

E. Training in Simulation & Sim-to-Real Transfer

1) Terrain Design:

  • Four terrain types—flat ground, trunk-supporting platform, trunk-supporting wall, and a benign slope—diversify simulated starting postures.The authors visualize the terrains alongside corresponding real-world scenes in Fig. 3.Source: 1) Terrain Design · p. 5

2) Domain Randomization:

  • Domain randomization varies body mass, base CoM offset, PD gains, torque offset, and initial pose to help bridge the simulation-to-reality gap.The randomization follows [28], and its parameters are detailed in Table II.Source: 2) Domain Randomization · p. 5
  • CoM-offset randomization is critical for robustness to real-world CoM position noise from insufficient torques or simulation-to-hardware model discrepancies.Source: 2) Domain Randomization · p. 5

F. Implementation Details

  • The PPO implementation uses 3-layer actor and 2-layer critic MLPs, with 500-step rollouts per episode.The implementation is based on.Source: F. Implementation Details · p. 5
  • Smoothness regularization uses λπ = 1 and λV = 0.1; the PD controller runs at 200 Hz in simulation and 500 Hz on hardware, while policies run at 50 Hz.The PD rates are used to ensure accurate tracking of PD targets.Source: F. Implementation Details · p. 5

A. Experimenrt Setup · 1) Evaluation Metrics.: · 2) Baselines:

The setup defines four measures of standing-up success and motion quality, then compares HoST with ablations targeting its critics, exploration, motion constraints, and historical inputs.

1) Evaluation Metrics.:

  • Success rate records stable standing above target height, while feet movement measures distance traveled after reaching that height.Success requires maintaining the target height for the rest of the episode; feet movement indicates stability in the standing pose.Source: 1) Evaluation Metrics · pp. 5–6
  • In simulation, HOST-History1 performs poorly under environmental disturbances, while HOST-Bound0.25 has high energy consumption, highlighting the importance of history and action-bound curriculum.Source: Figure 5 · p. 6
  • Simulation trade-offs show an inverse relationship between speed and smoothness, supporting constrained motion speed for real-world deployment.Source: Figure 6 · p. 6
  • Motion smoothness aggregates joint-angle changes across consecutive control steps, while energy measures consumption before reaching target height.These metrics assess whole-episode smoothness and avoidance of violent standing-up motion, respectively.Source: 1) Evaluation Metrics · p. 6

2) Baselines:

  • The ablations test whether multiple critics improve motor-skill learning and whether random-noise or curiosity-based exploration, including RND [1], outperforms the force curriculum.The comparison includes a single-critic baseline and exploration baselines using random noise and curiosity-based rewards.Source: 2) Baselines · p. 6
  • Motion-constraint ablations remove action bounds β or smoothness regularization L2C2 to test their influence on motion smoothness.Source: 2) Baselines · p. 6
  • Historical-state ablations vary the number of historical states to assess their effect on standing-up motion.Source: 2) Baselines · p. 6

B. Main Results · C. More Analyses

HoST outperforms a baseline in real-robot experiments, producing smooth, successful standing-up motions across tested scenes; analyses also show terrain-adaptive coordination, robustness, and speed–smoothness–energy trade-offs. Ablations identify the contributions of multiple critics, force curriculum, action bounds, smoothness regularization, and history length.

B. Main Results

  • HoST learns effective standing-up control across all simulated terrains.The main-results passage points to Table III but supplies no numerical values.Source: B. Main Results · p. 6
  • Multiple critics prevent the zero-success performance of the single-critic version, while force curriculum enables standing on terrains that otherwise defeat exploration from a fully fallen state.Curiosity-based exploration partly alleviates the no-curriculum challenge but remains unsatisfactory; the platform is the exception to failure without force curriculum.Source: B. Main Results · p. 6
  • Action bounds produce smoother motions and higher success rates than unbounded control, while smoothness regularization reduces oscillation and improves energy efficiency.Without bounds, learned motions are excessively violent; Bound0.25 performs well but restricts exploration and yields less natural motion.Source: B. Main Results · p. 6
  • Longer history improves performance over short history in contact-rich settings such as Wall, while slightly reducing smoothness and increasing energy use versus the default.The short-history policy underperforms in contact-rich scenarios; historical information also implicitly encodes contact dynamics relevant to robustness.Source: B. Main Results · p. 6, C. More Analyses · p. 7

C. More Analyses

  • Real-robot results show HoST surpasses the baseline, while policies produce smooth, successful motions across tested scenes and outdoor terrains.The real-robot comparison reports success rate and motion smoothness; outdoor tests include unseen grassland, wooden platforms, and stone roads, plus stone-platform and tree-leaning postures.Source: Table IV · p. 7, Figure 7 · p. 7, Figure 8 · p. 7
  • UMAP trajectories show distinct terrain-dependent motion patterns and coordinated hand–foot movement, while sim-to-real analysis finds domain randomization effective, CoM position especially influential, and joint-velocity discrepancies indicating torque gaps.Wall trajectories are more complex yet consistent; the hand–foot coordination reflects posture adaptation for balance and stability.Source: C. More Analyses · pp. 6–7
  • Policies maintain high success and efficient energy use under simulated disturbances; real-robot demonstrations show resilience to a 3 kg ball, ground obstacles, payloads up to 12 kg, large pushes, and a 15° slippery slope.Disturbances include sagittal CoM offsets and forces, initial joint-angle offsets, and random torque dropout; HOST-History1 performs poorly, while Bound0.25 uses more energy under disturbance.Source: C. More Analyses · p. 7, Figure 10 · p. 8, Table V · p. 8
  • Longer fall-to-standing times improve smoothness but reduce speed and increase energy use; Wall energy rises sharply at longer times despite low speed, unlike terrain-specific variations such as Slope’s higher balancing energy.The inverse speed–smoothness relationship supports constrained motion speed for real-world deployment.Source: C. More Analyses · pp. 7–8

VI. REAL ROBOT EXPERIMENTS · A. Main Results · B. Sim-to-real Analysis

On the Unitree G1, HoST achieves successful, smooth standing-up motions across laboratory and outdoor conditions; domain randomization reduces the sim-to-real gap, though joint-velocity discrepancies indicate remaining actuator-modeling challenges.

A. Main Results

  • Without L2C2 smoothness regularization, motions oscillate in every scene and often fail, while HoST produces smooth, stable motions, especially on the 10.5° slope.The comparison uses HOST-w/o-L2C2 as the baseline.Source: A. Main Results · p. 8
  • After successful standing, HoST’s control policies demonstrate stability against external disturbances.Source: Figure 11 · p. 8
  • HoST achieves a 100% success rate and high motion smoothness across all evaluated scenes.The evaluation includes laboratory and outdoor environments corresponding to simulation terrains.Source: A. Main Results · p. 8
  • HoST generalizes to untrained grassland, wooden platforms, and stone roads, and handles stone platforms and tree-leaning postures.The flat-ground terrains were not included in the training simulations.Source: A. Main Results · p. 8

B. Sim-to-real Analysis

  • Domain-randomization terms significantly reduce the sim-to-real gap, particularly for Center of Mass position.The analysis examines the effects of different domain-randomization terms.Source: B. Sim-to-real Analysis · p. 8
  • HoST learns across prone postures on the ground and handles side-lying postures.Source: Figure 12 · p. 9
  • Simulated and real-world knee and hip roll joint velocities differ, suggesting a torque gap, yet HoST’s joint paths remain consistent with simulation.The authors identify more accurate actuator modeling as necessary to bridge this gap, as also suggested for quadrupedal robots [15].Source: B. Sim-to-real Analysis · p. 9

C. Emergent Properties … VIII. LIMITATIONS AND FUTURE DIRECTIONS

HoST showed disturbance tolerance and adaptable standing behaviors in emergent tests, while the paper also reports posture-training and larger-robot challenges and identifies future work for real-world integration.

C. Emergent Properties

  • Controllers maintained stability under 3 kg ball impacts and obstructive objects, carried payloads up to 12 kg, and recovered upright after abrupt perturbations.The 12 kg payload is twice the mass of the robot’s trunk.Source: C. Emergent Properties · p. 9
  • On a 15° slippery slope, controllers adjusted posture and center of mass to stay stable, and resisted disturbances after standing.The tests covered dynamic balance on the incline and standing stability after successful stand-up.Source: C. Emergent Properties · p. 9

E. Extend HoST to Large-size Humanoid Robots

  • HoST extended successfully to H1 and H1-2, but required high-stiffness joints and showed upper-body posture deviations whose source remains unclear.The motions relied more on upper-body ground contact and high hip actuation; the observed sim-to-real gaps may stem from the framework or hardware limitations.Source: Figure 13 · p. 9, E. Extend HoST to Large-size Humanoid… · p. 9

D. Prone and Side-lying Postures on the Ground

  • HoST learned across prone and untuned side-lying postures, but prone–supine motion differences and hip constraints leave joint training unclear.The paper notes that harder hip-joint constraints are needed to prevent violent motions when training from the specified postures.Source: D. Prone and Side-lying Postures on… · p. 9

VII. CONCLUSION

  • HoST learns posture-adaptive standing-up motions from scratch across diverse terrains, using multi-critic reinforcement learning, smoothness regularization, and implicit speed constraints for G1 deployment.The conclusion reports smooth, stable, and robust standing-up motions in real-world scenarios.Source: §VII · p. 9

VIII. LIMITATIONS AND FUTURE DIRECTIONS

  • Limitations include outdoor seated-recovery collisions, degraded performance from mixed supine and prone training, and no demonstrated integration with existing humanoid systems.The paper proposes adding environmental perception to address collisions and envisions integration into current humanoid frameworks to extend real-world applications.Source: §VIII · p. 10

APPENDIX · A. More Experimental Details · B. More Implementation Details

The appendix specifies the G1 evaluation and robustness protocols, then details HoST’s curricula, rewards, PPO implementation, baselines, terrain range, and adaptations for other postures and robots.

A. More Experimental Details

  • G1 policies were tested for 1,250 episodes per terrain, with targets of 0.6 m on slopes and 0.7 m elsewhere; the robot has 23 actuated degrees of freedom.Each terrain evaluation comprised five repetitions of 250 episodes, with mean and standard deviation reported; onboard computation uses a Jetson Orin NX with IMU and joint-encoder feedback.Source: A. More Experimental Details · p. 12
  • Robustness tests applied CoM bias and sagittal force along the robot’s x-axis, joint-angle offsets to all joints, and random torque dropouts at 200 Hz.Dropped torques were set to zero.Source: A. More Experimental Details · p. 12
  • Smoothness uses second differences of joint positions, while energy uses torque and velocity under a stage-height condition, with policy steps of 0.02 s.The supplied metric expression is truncated; it identifies the stage-2 height threshold and defines joint torque, velocity, and the policy-step duration.Source: A. More Experimental Details · p. 12

B. More Implementation Details

  • The curriculum lowers vertical force from 200 N by 20 N and action bound from 1 by 0.02 after reaching the target head height, down to 0 N and 0.25.The righting and rising stages use height thresholds of 0.45 m and 0.65 m, respectively.Source: B. More Implementation Details · p. 12
  • Reward terms are grouped by critic, include tolerance rewards that decay outside bounds, and include an ankle-parallel reward based on ankle-keypoint height variance.The ankle keypoints are handcrafted without collision models; Table VI lists reward functions and groups, whose advantages are independently estimated by distinct critics.Source: B. More Implementation Details · p. 12, Table VI · p. 13
  • The actor is a 3-layer MLP with [512, 256, 128] hidden units, each critic a 2-layer [512, 256] MLP; PPO uses γ = 0.99, clip ratio 0.2, and entropy coefficient 0.01.Each iteration uses 50 steps per environment, five learning epochs, and four mini-batches per epoch; advantages are independently calculated and normalized within each reward group.Source: B. More Implementation Details · p. 12
  • Baselines vary critic count, force curriculum, action-bound curriculum, style rewards, and state-history length; tested platforms span 20–92 cm, with slopes of about 1°–14° and walls of 14°–84°.HOST-w/o-MuC uses one value network; HOST-w/o-Force-RND removes vertical-force curriculum and adds RND; HOST-Bound0.25 fixes β at 0.25; HOST-w/p-rstyle removes style rewards; HOST-History changes state-history length.Source: B. More Implementation Details · p. 12
  • For prone-posture learning, HoST tightens hip-deviation constraints and changes reward terms; extensions to H1 and H1-2 adjust task parameters and use 19 and 27 actuators, respectively.Hip and knee stiffness during hardware deployment is increased to 1.5 times simulation values for H1 and H1-2; G1 hip and knee stiffness are adjusted to 200 and 275 after a sim-to-real torque gap is observed.Source: B. More Implementation Details · p. 13

Related papers

Loading 2502.08378v2…