Source-linked AI summary
HC2Robot: Hyper-Connection Transformers with Predicted Foresight for Multi-Terrain Humanoid Parkour
Wenhao Xu, Yanheng Mai, Yifei Fu, Nan Zheng, Yanbo Zhou, Zirui Huang, Mingsheng Ling, Xinjue Wang, Renjing Xu
TL;DR
Humanoid parkour requires reasoning about future footholds, but simulator contact and support signals available to the training critic are unavailable to a deployable actor. HC2Robot combines a Hyper-Connection Transformer with a Predicted Foresight Module that predicts touchdown events from deployable observations. Across four simulated terrain classes, it achieves mean success and safe foothold rates of 97.60% and 97.94%, outperforming evaluated baselines.
Problem
Humanoid parkour requires coordinating perception, contact timing, and body motion before footholds are realized, while simulator-derived contact and support signals are unavailable to the deployed actor.
Source: §I · p. 1Method
HC2Robot pairs learnable multi-stream Hyper-Connection routing with a PFM that predicts each foot’s next touchdown timing, location, and support quality from deployable observations.
Source: §I · p. 1Results
97.60% mean SR and 97.94% mean SFR across four simulated terrain classes, compared with 94.17% and 94.75% for ParkourFormer, respectively.
Source: E. Main Results · p. 6Takeaways & Limitations
Abstract
from arXiv · showhide
Humanoid parkour requires robots to reason about future footholds and whole-body motions over highly discontinuous terrain. We revisit the information asymmetry in asymmetric actor-critic learning for perceptive humanoid locomotion: the critic exploits privileged simulator signals during training, whereas the deployable actor operates solely from onboard perception. We argue that the key is not exposing privileged information to the actor, but learning a deployable approximation of it to narrow the information gap between training and deployment. We present HC2Robot, a deployable perceptive humanoid parkour framework consisting of a Predicted Foresight Module (PFM) and a Hyper-Connection (HC) Transformer. PFM predicts from deployable observations the policy-conditioned timing, location, and support quality of each foot's next touchdown event, converting privileged supervision into a deployable latent representation that provides the actor with predicted touchdown information without privileged runtime sensing. The HC Transformer further enhances temporal reasoning through learnable multi-stream routing, while a future motion prediction head provides short-horizon motion priors for regularization. Across stairs, gaps, high platforms, and slopes, HC2Robot achieves a mean simulated traversal success rate of 97.60% and consistently higher safe foothold rates than strong perceptive locomotion baselines, supporting improved traversal performance and geometric foothold support under deployable sensing.
I. INTRODUCTION
Humanoid parkour requires coordinating perception, contact timing, and whole-body motion before footholds are reached, but deployable actors cannot use simulator-derived contact and support signals. HC2Robot combines predicted touchdown foresight with multi-stream temporal routing, improving traversal success and safe foothold rates over perceptive parkour baselines across four terrain classes.
I. INTRODUCTION
- Perceptive parkour extends beyond proprioception-only locomotion, which lacks foresight of upcoming terrain discontinuities.The introduction frames parkour as requiring perception, contact timing, and body motion to be coordinated before footholds are realized.Source: §I · p. 1
- PFM predicts each foot’s next touchdown timing, location, and support quality from deployable actor features, using simulator information only as training supervision.It uses a continuous time-to-event formulation without a fixed prediction offset or privileged signals in the deployment interface.Source: §I · p. 1
- The HC Transformer replaces a single residual pathway with learnable multi-stream aggregation and mixing while evaluating each attention and feed-forward sublayer once.This design aims to support richer cross-layer information exchange and temporal feature reuse.Source: §I · p. 1
- An auxiliary motion predictor provides two-step future AMP states for motion regularization.The motion prediction pathway remains deployment-compatible, with simulator information used only for training supervision.Source: §I · p. 1
- Across stairs, gaps, high platforms, and slopes, HC2Robot consistently improves traversal success and safe foothold rate over strong baselines.The framework overview also depicts simulated traversal comparisons and physical deployment.Source: §I · p. 1, Figure 1 · p. 2
II. RELATED WORK
Prior work has advanced deployable perceptive locomotion and humanoid parkour, while privileged-state methods address information unavailable to the deployed actor in different ways. HC2Robot focuses on predicting the next policy-conditioned touchdown event and pairing that foresight with a Hyper-Connection backbone.
A. Perceptive Locomotion and Humanoid Parkour
- Perceptive locomotion and humanoid parkour build on blind locomotion [3],, depth- and vision-conditioned policies –[7], and deployable parkour systems, [2], [8],, –, –.HC2Robot adopts this deployable-perception setting and focuses on predicted foothold events.Source: A. Perceptive Locomotion and Humanoid… · p. 2
B. Privileged-State Methods and Learning Paradigms
- Asymmetric actor–critic methods separate privileged critic inputs from the deployable policy, [18], while other approaches estimate terrain or dynamics, transfer privileged representations, or imitate privileged actions, [16], [26]–.PFM instead uses simulator descriptors as training labels to predict touchdown time, position, support attributes, and uncertainty without privileged actor inputs or teacher actions.Source: B. Privileged-State Methods and… · p. 2
C. Foothold Reasoning, Predictive Supervision, and Motion Priors
- Unlike terrain-region or hidden-dynamics prediction [6], [31], [32], PFM predicts the next touchdown event; unlike ParkourFormer, it uses foothold-specific labels and a censoring-aware event objective.HC2Robot keeps touchdown-event supervision distinct from the AMP motion-distribution prior.Source: C. Foothold Reasoning, Predictive… · p. 2
III. METHOD
- HC2Robot combines a Hyper-Connection Transformer for temporal feature reuse with PFM forecasts of each foot’s next touchdown from deployable observations.The framework uses asymmetric actor–critic training.Source: §III · p. 2
A. Asymmetric Observation and Foothold Supervision
The actor uses onboard proprioception and depth history, while privileged simulator variables are reserved for critic training and touchdown-event supervision. The support descriptor’s time series supplies PFM’s event times and attributes.
A. Asymmetric Observation and Foothold Supervision
- The actor combines a history of proprioceptive observations with a depth-history context token to form deployable features.The history has length L because contact transitions span multiple time steps.Source: A. Asymmetric Observation and Foothold… · p. 2
- During training, the critic receives privileged base velocity, center-of-mass position, and a two-foot support descriptor unavailable at deployment.The actor never directly observes these simulator variables and receives depth as its exteroceptive input.Source: A. Asymmetric Observation and Foothold… · p. 3
- The support descriptor records contact, nearby terrain coordinates, support ratio, terrain risk, load-weighted support, and a normal-force proxy for each foot.The support ratio is the fraction of rays terminating on terrain within the foot-sole grid.Source: A. Asymmetric Observation and Foothold… · p. 3
- The temporal trajectory of privileged support descriptors supplies PFM supervision: contact transitions define touchdown times, and descriptor values at those instants define event attributes.The descriptor also conditions the critic’s value function.Source: A. Asymmetric Observation and Foothold… · p. 3
B. HC-Transformer Backbone
The backbone replaces standard residual connections with multi-stream Hyper-Connection routing over deployable temporal history. It produces policy features by aggregating, processing, and remixing streams while evaluating each sublayer once.
B. HC-Transformer Backbone
- Two depth-conditioned query tokens attend to deployable history memory and are replicated across parallel streams to form the multi-stream state.The memory combines the reshaped proprioceptive history with the depth token.Source: B. HC-Transformer Backbone · p. 3
- Each attention and feed-forward sublayer aggregates streams, evaluates the sublayer once, then distributes updates and mixes residual paths.Learned α, β, and R parameters control aggregation, update distribution, and residual-path mixing.Source: B. HC-Transformer Backbone · p. 3
- After N layers, summed and normalized streams provide policy features; the critic uses analogous privileged-history routing without a foothold predictor.The critic also has a separate depth encoder.Source: B. HC-Transformer Backbone · p. 3
C. Predicted Foresight and Late Fusion
PFM predicts each foot’s next touchdown event from deployable features, including its timing, location, and support attributes. Its forecast is summarized and fused with the motion representation at the policy head.
C. Predicted Foresight and Late Fusion
- PFM treats each foot’s next touchdown as an event to predict, rather than regressing privileged state at a fixed future step.A touchdown is the first strictly future 0→1 contact transition, with lift-off required first if the foot is in stance.Source: C. Predicted Foresight and Late Fusion · pp. 3–4
- Each forecast estimates time-to-touchdown, base-frame touchdown position, and touchdown support attributes; episode termination before touchdown right-censors the event.The support attributes include support ratio, terrain risk, load support, and a normal-force proxy.Source: C. Predicted Foresight and Late Fusion · p. 4
- PFM models positive time-to-touchdown with a log-normal distribution, touchdown position with a diagonal Gaussian, and four touchdown attributes with uncertainty estimates.Auxiliary one-step estimates of base velocity and center-of-mass position regularize the PFM encoder and provide short-horizon kinematic context.Source: C. Predicted Foresight and Late Fusion · p. 4
- The control summary includes median time-to-touchdown and a confidence derived from predicted variance; the foresight feature is concatenated with auxiliary estimates.The confidence decreases with greater predicted uncertainty.Source: C. Predicted Foresight and Late Fusion · p. 4
- The foresight latent is fused with the motion representation at the policy head, and deployment uses the deterministic policy mean.The actor interface includes deployable observations and a 22-D forecast made of two 10-D event summaries and two confidence values.Source: C. Predicted Foresight and Late Fusion · p. 4, Table I · p. 4
D. Training and Deployment
Training uses separate objectives for actor–critic, PFM, and AMP discriminator parameters. Deployment retains the actor, including PFM, while removing the critic, simulator targets, and discriminator.
D. Training and Deployment
- Actor–critic, PFM, and AMP discriminator parameters are updated sequentially with separate objectives.The actor–critic objective combines clipped PPO with masked MSE on two future motion states.Source: D. Training and Deployment · p. 4
- PFM supervision uses valid rollout transitions and treats censored frames as survival likelihood, while touchdown-event regression applies only to observed touchdowns.Regression uses a predicted log-variance term; auxiliary regression covers one-step base velocity and center-of-mass position.Source: D. Training and Deployment · pp. 4–5
- AMP distinguishes detached policy trajectories from reference trajectories, and its detached style reward enters PPO returns.The discriminator objective includes gradient-penalty and weight-regularization terms.Source: D. Training and Deployment · p. 5
- Deployment retains the actor, including PFM, and removes the critic, simulator targets, and AMP discriminator.Detaching features blocks PFM-loss gradients to the actor encoder and PPO gradients to PFM.Source: D. Training and Deployment · p. 5
IV. EXPERIMENTS
The experiments cover simulation protocol, deployable PFM prediction accuracy, comparative traversal and foothold-support results, component ablations, and zero-shot real-world transfer on individual and mixed terrains.
IV. EXPERIMENTS
- The evaluation includes simulation, PFM prediction accuracy, traversal and foothold-support comparisons, ablations, and zero-shot real-world transfer across individual and mixed terrains.The supplied overview identifies these evaluation components but gives no results.Source: §IV · p. 5
A. Experimental Protocol
The protocol compares strategies under shared training and testing conditions across randomized terrain classes. Evaluation uses completion and geometric foothold-support measures, with SFR explicitly treated as a proxy rather than a direct measure of contact or stability.
A. Experimental Protocol
- All strategies trained for 30,000 iterations in 4096 IsaacLab humanoid environments, with stairs, gaps, high platforms, and slopes randomized each iteration.Methods shared the command distribution, terrain curriculum, and evaluation settings; testing used only the deployable actor and onboard observations.Source: A. Experimental Protocol · p. 5
- Results average five seeds with 1,000 trials per terrain class; Table II lists training ranges and held-out mixed-terrain validation settings.Table II is captioned as covering training and mixed-course terrain parameters.Source: A. Experimental Protocol · p. 5, Table II · p. 5
B. Metrics
- Success rate (SR) is the fraction of episodes completing a terrain segment without unsafe termination, including falls, illegal contacts, boundary exits, low base height, and timeouts as failures.Source: B. Metrics · p. 5
- Safe foothold rate (SFR) estimates geometric support by sampling a yaw-aligned 0.225 × 0.10 m foot patch at 0.025 m spacing, yielding 50 terrain rays; ray misses count as unsupported.Source: B. Metrics · p. 5
- Table IV reports simulation SR and SFR as mean ± standard deviation over five random seeds.Source: Table IV · p. 6
- The horizontal SFR patch ignores foot pitch and roll and may penalize continuous slopes, so SFR complements SR rather than directly measuring physical contact or dynamic stability.Source: B. Metrics · p. 6
C. PFM Prediction Accuracy
PFM prediction accuracy is assessed offline on held-out simulation episodes using deployable observation history. Overall mean errors are reported for touchdown timing, position, and support ratio.
C. PFM Prediction Accuracy
- PFM is scored on held-out simulation episodes using only deployable observation history, comparing point predictions with simulator-derived targets.Source: C. PFM Prediction Accuracy · p. 6
- 0.79 control steps, 0.019 m, and 2.23 percentage points are the overall mean timing, position, and support-ratio errors, respectively.Source: C. PFM Prediction Accuracy · p. 6
- Errors average frame–foot samples with an observed future touchdown; right-censored samples are excluded, and the overall row averages four terrain-class means within each seed.Source: C. PFM Prediction Accuracy · p. 6
D. Compared Methods
HC2Robot is compared with perceptive parkour baselines and evaluated across four simulated terrain classes, followed by component and training-dynamics analyses. The reported results favor HC2Robot on traversal and foothold support, while the fixed-horizon comparison evaluates the complete event-based design against a combined alternative.
D. Compared Methods
- The comparison includes Hiking-in-the-Wild and ParkourFormer, with model size and inference latency summarized in Table V.The supplied passages do not report Table V’s values.Source: D. Compared Methods · p. 6, Table V · p. 6
E. Main Results
- 97.60% mean SR and 97.94% mean SFR exceed ParkourFormer’s 94.17% and 94.75%, respectively, with HC2Robot highest on both metrics across all four terrains.The largest reported gains over ParkourFormer are 7.66 SR points on high platforms and 5.31 SFR points on slopes; the gap SFR gain is 2.80 points.Source: E. Main Results · p. 6
F. Auxiliary Diagnostics and Ablations
- Removing PFM lowers mean SR/SFR from 97.60%/97.94% to 92.50%/93.46%, while Auxiliary-only reaches 91.50%/90.02%.The Auxiliary-only comparison supports a contribution from supervised touchdown-event features beyond shared one-step predictions, but does not attribute the difference to one event target; removing AMP gives the lowest aggregate means and slope SFR.Source: F. Auxiliary Diagnostics and Ablations · pp. 6–7, Table VI · p. 7
- HC2Robot exceeds Fixed-horizon in SR and SFR on every terrain, with gains reaching 9.08 SR points on high platforms and 6.07 SFR points on gaps.The control uses a fixed prediction offset and omits event-time loss and input, so the comparison evaluates that combined alternative rather than isolating either change.Source: F. Auxiliary Diagnostics and Ablations · p. 7, Table VI · p. 7
- Among the displayed single-run training traces, the full model has similar left/right PFM confidence and the lowest terminal foot-slip speed.The w/o Foothold priv. trace is an earlier supervision-removal diagnostic, distinct from the Auxiliary-only Table VI control.Source: F. Auxiliary Diagnostics and Ablations · p. 7, Figure 6 · p. 7
G. Real-World Deployment
HC2Robot is deployed zero-shot on a Unitree G1 using onboard proprioception and egocentric depth. Physical trials cover individual training terrain classes and mixed courses; the conclusion reports transfer to both settings.
G. Real-World Deployment
- HC2Robot runs zero-shot on the Unitree G1 at 50 Hz using proprioception and an egocentric depth camera.The deployment passage specifies onboard inference from these observations.Source: G. Real-World Deployment · p. 7, §V · p. 8
- Physical trials cover realizations of the training terrain classes and mixed courses; Table VII reports complete-trial success rates, and Fig. 7 shows traversal examples.Each setting uses 20 trials; mixed courses test generalization to terrain arrangements and transitions rather than new terrain primitives.Source: Table VII · p. 7, Figure 7 · p. 7, G. Real-World Deployment · p. 8
V. CONCLUSION
- The conclusion reports zero-shot Unitree G1 trials demonstrating transfer to individual obstacles and mixed courses.The supplied passages do not give the Table VII success-rate values.Source: §V · p. 8, G. Real-World Deployment · p. 8