PaperlayerView exact revision on arXiv

Source-linked AI summary

Now You See That: Learning End-to-End Humanoid Locomotion from Raw Pixels

Wandong Sun, Yongbo Su, Leoric Huang, Alex Zhang, Dwyane Wei, Mu San, Daniel Tian, Ellie Cao, Baoshi Cao, Yang Liu, Finn Yan, Ethan Xie, Zongwu Xie

arXiv:2602.06382v2cs.RO

TL;DR

Vision-based humanoid locomotion is challenged by sim-to-real perception noise and conflicting objectives across diverse terrains. The paper combines realistic depth simulation, vision-aware distillation, and terrain-specific multi-critic and multi-discriminator learning. It reports 98.9% average success with 5.8% power degradation, while stair descent remains less robust than ascent.

  • Problem

    Sim-to-real perception noise degrades fine-grained humanoid locomotion, while conflicting objectives hinder a unified policy across diverse terrains.

  • Method

    The framework combines realistic depth simulation, vision-aware behavior distillation, and terrain-specific reward shaping with multi-critic and multi-discriminator learning.

  • Results

    98.9% average success rate with 5.8% PDR is reported, alongside 27.9% higher success and 25.1% lower PDR than Humanoid Parkour Learning.

  • Takeaways & Limitations

    The policy demonstrates robust cross-terrain locomotion across extreme obstacles and fine-grained bidirectional staircase traversal on humanoid platforms.

  • Takeaways & Limitations

    Stair descent remains less robust than ascent because gravitational amplification of minor errors and target-foot occlusion affect the task.

Abstract

from arXiv · show

Achieving robust vision-based humanoid locomotion remains challenging due to two fundamental issues: the sim-to-real gap introduces significant perception noise that degrades performance on fine-grained tasks, and training a unified policy across diverse terrains is hindered by conflicting learning objectives. To address these challenges, we present an end-to-end framework for vision-driven humanoid locomotion. For robust sim-to-real transfer, we develop a high-fidelity depth sensor simulation that captures stereo matching artifacts and calibration uncertainties inherent in real-world sensing. We further propose a vision-aware behavior distillation approach that combines latent space alignment with noise-invariant auxiliary tasks, enabling effective knowledge transfer from privileged height maps to noisy depth observations. For versatile terrain adaptation, we introduce terrain-specific reward shaping integrated with multi-critic and multi-discriminator learning, where dedicated networks capture the distinct dynamics and motion priors of each terrain type. We validate our approach on two humanoid platforms equipped with different stereo depth cameras. The resulting policy demonstrates robust performance across diverse environments, seamlessly handling extreme challenges such as high platforms and wide gaps, as well as fine-grained tasks including bidirectional long-term staircase traversal.

I. INTRODUCTION

Vision-based humanoid locomotion must handle unstable dynamics, noisy visual sensing, and conflicting objectives across diverse terrains. The framework addresses these challenges with realistic depth simulation, vision-aware distillation, and terrain-specific multi-network learning, validated across extreme and fine-grained tasks.

  • Humanoid locomotion requires precise perception-action coordination because bipedal robots operate in inherently unstable regimes.
  • Perception noise from the sim-to-real gap degrades centimeter-level fine locomotion, while heterogeneous terrain objectives make unified policy learning difficult.
  • Experiments on two humanoid platforms with different stereo depth cameras cover high platforms, wide gaps, and long-term stair traversal in both directions.
  • The framework simulates stereo artifacts, depth-dependent noise, optical distortions, and calibration uncertainties for more robust depth-based transfer.
  • Vision-aware behavior distillation transfers locomotion knowledge from privileged observations to noisy depth inputs through latent-space shaping and noise-invariant auxiliary tasks.
  • Terrain-specific reward shaping, dedicated critics, and discriminators capture distinct dynamics and motion priors within one unified policy.

III. METHOD

The method uses a two-stage pipeline: privileged height-scan training followed by depth-based vision-aware distillation. Its augmentation models stereo reconstruction artifacts and structured, depth-dependent noise to support sim-to-real transfer.

  • Method overview: Privileged reinforcement learning trains a teacher with height scans, followed by distillation into a deployment policy operating on augmented depth images.The two stages combine behavior distillation with comprehensive depth augmentation.
  • Augmentation pipeline: The augmentation pipeline combines stereo fusion, random convolution, Gaussian and Perlin noise, scale randomization, pixel failures, clipping, and cropping.These operations are summarized in the vision augmentation pipeline.
  • Stereo depth fusion: Stereo fusion renders left-right viewpoints and applies disparity consistency checks, marking inconsistent pixels invalid to reproduce sensor hole patterns.The consistency threshold τ ranges from 0.05 to 0.20.
  • Depth-dependent noise: Quadratic depth noise adds distance-dependent perturbations with randomly sampled coefficients, capturing nonlinear noise growth and inter-device variation.The coefficients c0, c1, and c2 are sampled independently from U(−0.03, 0.03).
  • Structured noise patterns: Multi-octave Perlin noise models spatially correlated interference with five octaves and persistence 0.5, using depth-dependent amplitude.The Perlin component is combined with the quadratic amplitude model.

4) Optical Distortions:

The method models optical, calibration, and sensing imperfections while using terrain-specific critics and discriminators for heterogeneous locomotion dynamics. Privileged terrain representations are transferred to depth perception through distillation.

  • Optical distortions: Randomized convolution models lens aberrations as local geometric distortions centered around the original measurement.The convolution weights are sampled from U(−0.05, 0.05) around the identity kernel.
  • Calibration and failures: Calibration randomization scales depth, perturbs camera intrinsics and extrinsics, and models zero- and max-pixel failures with probability 0.001 each.Depth scaling is sampled from U(0.90, 1.10).
  • Observation processing: Depth values are clipped to [0.3, 2.0] m, normalized to [0, 1], cropped from 30 × 40 to 24 × 32 pixels, and delayed by U[2, 4] frames.Cropping reduces peripheral distortion artifacts while delay models processing latency.
  • Terrain specialization: Terrains are divided into K = 3 categories—stairs and platforms, gap crossing, and rough terrain—with specialized reward components.The categories include ascending and descending stairs and platforms.
  • Multi-critic architecture: Shared critic backbones use separate terrain-specific output heads, while selected critics and discriminators provide category-appropriate value and style rewards.The privileged policy receives 693 height samples from a 1.6 m × 1.0 m window at 0.05 m resolution.

4) Motion Priors:

The framework uses terrain-conditioned motion priors and DAgger-style distillation to transfer privileged locomotion behavior into a depth-based deployment policy.

  • Motion priors: Adversarial Motion Priors guide natural locomotion through torso-centric observations.Separate motion datasets provide style guidance for each terrain category.
  • Terrain-conditioned style: Terrain-specific discriminators provide style rewards tailored to the motion characteristics required by each terrain type.This complements the terrain-specific critics used for value estimation.
  • Policy distillation: DAgger-style distillation lets the student interact with the environment while the teacher supervises its own state distribution.The deployment policy is trained from teacher actions during student-environment interaction.
  • Observation transfer: The student minimizes action discrepancy while receiving augmented depth images and proprioception, whereas the teacher receives clean height scans.The two policies therefore operate on different observation modalities during transfer.

2) Denoising Objective:

The denoising objective enforces consistent depth representations between clean and augmented inputs while regularizing encoder features toward a standard normal distribution. These losses support replacing the privileged height-scan embedding with depth-encoder features for vision-based deployment.

  • Denoising Objective: The depth encoder is trained for consistency between clean depth and its augmented counterpart.This encourages robust feature extraction under simulated sensor noise.
  • Feature Regularization: Feature regularization matches the batch-wise encoder distribution to a standard normal prior to avoid representation collapse.The encoder outputs z are modeled with a diagonal Gaussian estimated from empirical batch statistics.
  • Loss Combination: The total distillation loss balances the denoising, feature-regularization, and other distillation components.The supplied text specifies λdenoise = λkl = 0.1.
  • Deployment: The depth encoder output replaces the height-scan embedding, enabling deployment from visual depth observations.The evaluation includes real-world deployment with an Orbbec Gemini 336L depth camera.

A. Experimental Setup

The experiments evaluate realistic depth transfer and unified terrain control across stairs, gaps, and platforms using realistic depth noise and multiple baselines. Success rate, average power, and power degradation quantify traversal reliability, efficiency, and robustness.

  • Benchmark: The Real-World Depth Transfer Benchmark evaluates policies in simulation with CycleGAN-augmented depth images derived from real and simulated observations.This setup enables large-scale quantitative comparison under realistic perception conditions.
  • Terrain Configurations: The evaluation covers ascending and descending stairs, a 0.45 m gap, and a 0.40 m platform.Stairs use 15 cm steps with 30 cm depth; the gap and platform require accurate depth perception and precise stepping.
  • Evaluation Protocol: Each terrain configuration is tested in 1024 parallel environments for 100 episodes.The setup is intended to provide statistically robust performance estimates.
  • Compared Methods: Baselines and ablations include standard domain randomization, partial or absent augmentation, shared critics and discriminators, direct reinforcement learning, and behavior cloning only.These comparisons isolate perception augmentation, terrain-specific learning, and privileged distillation.
  • Metrics: Performance is measured by success rate, average mechanical power, and power degradation ratio under realistic sensor noise.Lower average power and lower power degradation ratio indicate more efficient and more noise-robust locomotion.
  • Real-World Deployment: Real-world deployment sequences demonstrate ascending and descending stair traversal without real-world fine-tuning.The policy uses anticipatory leg lifting when ascending and controlled foot placement when descending.

B. Main Results

On the realistic-depth benchmark, the proposed method achieves high success, low power consumption, and low power degradation while outperforming the evaluated baselines and ablations. Ablations attribute gains to realistic sensor modeling and terrain-specific multi-critic and multi-discriminator learning.

  • Overall Results: 98.9% average success rate, 27.7 ×101 W average power, and 5.8% PDR are achieved across RDT-Bench terrains.The method is reported to outperform all baselines across every terrain and metric.
  • Perception Transfer: 98.9% success with 5.8% PDR contrasts with 43.0% success and 70.9% PDR for no augmentation under realistic sensor artifacts.The full augmentation pipeline is reported to maintain consistent, efficient control despite perception noise.
  • Comparison with Prior Work: 27.9% higher success rate and 25.1% lower PDR are reported relative to Humanoid Parkour Learning.The baseline achieves 71.0% success with 30.9% PDR.
  • Architecture Contributions: The multi-critic and multi-discriminator approach reaches 98.9% success and 5.8% PDR versus 82.0% success and 20.8% PDR for a single shared critic and discriminator.The reported explanation is that terrain-specific value functions and discriminators learn distinct motion priors for each terrain.
  • Depth Augmentation Ablation: Stereo fusion provides the largest augmentation improvement by simulating characteristic hole patterns that cause perception failures on real hardware.Depth-dependent noise and calibration uncertainties provide moderate gains, while smaller-success-rate components also contribute to control smoothness.
  • Terrain Learning Ablation: +15.1% to +22.5% success-rate gains and -13.5% to -18.2% PDR changes are reported for multi-critic/discriminator learning across terrain categories.The comparison is against single-critic/discriminator architectures.
  • Representation Analysis: The depth encoder’s t-SNE embeddings form distinct clusters for six terrain types despite realistic sensor noise.This visualization indicates terrain-specific representation learning in the learned latent space.

3) Distillation Objectives:

The distillation objectives improve robustness when transferring privileged locomotion knowledge to noisy depth-based deployment, supporting onboard operation across challenging real-world terrains. Performance remains less robust for stair descent than ascent, revealing a specific deployment limitation.

  • Distillation ablation: 5.5% SR improvement comes from denoising, while KL regularization adds 2.8% SR improvement by stabilizing latent representations.Behavior cloning alone achieves 86.0% SR with 17.7% PDR, showing the auxiliary objectives contribute to robust deployment.
  • Latent representations: Distinct latent clusters form for six terrain types, indicating terrain-specific geometric features remain separable despite realistic sensor noise.The clusters cover ascending and descending stairs, gaps, ascending and descending platforms, and rough terrain.
  • Real-world evaluation: The policy runs onboard at 50 Hz without simulation fine-tuning and is evaluated on stairs, platforms, extended staircases, and wide gaps.The scenarios include 15 cm stairs, approximately 40 cm platforms, 30+ consecutive steps, and gaps exceeding 45 cm.
  • Real-world results: 97.8% overall success rate (88/90 trials) is achieved in real-world deployment, with perfect performance on five of six test conditions.The extended staircase test demonstrates long-horizon stability over 30+ consecutive steps without accumulated drift.
  • Failure analysis: Stair descent is less robust than ascent because gravitational acceleration amplifies errors and step edges occlude target footholds where stereo artifacts are pronounced.The reported asymmetry motivates higher-resolution near-field sensing or predictive foot placement strategies.

APPENDIX

The appendix documents the network architectures, distillation configuration, and CycleGAN-based depth translation used to evaluate realistic sim-to-real artifacts. Translation quality metrics indicate that the generated images match real sensor statistics while preserving geometric structure.

  • Network architectures: The teacher and student share policy architectures but use different exteroceptive encoders: an MLP for privileged height scans and a CNN for depth images.During distillation, the student CNN learns latent representations matching the teacher MLP output.
  • Distillation configuration: Distillation transfers locomotion knowledge from privileged height scans to depth images, with behavior cloning as the primary learning signal and balanced auxiliary losses.The configuration uses cosine annealing with warm-up for stable training dynamics.
  • Depth translation: CycleGAN translates simulated depth images toward real sensor outputs using approximately 200,000 real depth frames, exclusively for evaluation.The real-world recordings cover approximately two hours of humanoid traversal across various terrains.
  • Translation quality: FID = 23.4 for Sim→Real translation, while SSIM = 0.89 and PSNR = 28.3 dB indicate realistic artifacts with preserved geometric structure.The metrics are computed on a held-out test set of approximately 20,000 frames.

D. Terrain Specific Rewards

Terrain-specific rewards adapt velocity tracking to the demands of different obstacles. Exponential tracking prioritizes precise regulation on stairs, platforms, and rough terrain, whereas directional tracking permits extra speed for dynamic gap crossings.

  • Terrain-specific reward design: Terrain-specific reward functions encourage locomotion strategies tailored to different obstacle types within the multi-critic framework.The reward design reflects distinct terrain requirements rather than applying one velocity objective uniformly.
  • Exponential Velocity Tracking: Exponential Velocity Tracking is used for stairs, platforms, and rough terrain, where precise velocity regulation supports stable foot placement.Its smooth exponential kernel encourages accurate velocity following.
  • Exponential Velocity Tracking: The velocity-tracking formulation compares commanded velocity with the robot’s base linear velocity in the body frame, using σ to control tracking precision.The temperature parameter determines how sharply deviations are penalized.
  • Directional Velocity Tracking: Directional Velocity Tracking is used for gap terrain, rewarding movement along the commanded direction without penalizing speeds above the command magnitude.This permits dynamic crossing motions that build momentum for gap traversal.

2) Feet Contact Height Reward:

The feet contact height reward penalizes irregular support surfaces on stairs and platforms, while remaining inactive for gaps and rough terrain where that criterion is unsuitable. Cross-platform results provide initial evidence of transfer, but broader validation remains limited.

  • Feet Contact Height Reward: The feet contact height reward penalizes irregular height variation around contacting feet to discourage placement on edges or uneven surfaces.Lower clipped-height variation corresponds to flatter contact surfaces and supports stable stair and platform traversal.
  • Reward scope: The reward is not applied to gaps or rough terrain because dynamic leaps make contact analysis less relevant and rough surfaces make height variation inherent.This omission prevents penalizing terrain characteristics that are required or unavoidable in those settings.
  • Cross-platform validation: Unitree G1 achieves perfect stair-ascent success without platform-specific adaptation across different kinematic and sensor configurations.The deployment uses an Intel RealSense D435i instead of the primary platform’s Orbbec Gemini 336L.
  • Cross-platform validation: Cross-platform validation remains limited to one terrain type, leaving gaps, platforms, and descending scenarios for future benchmarking.The result therefore provides initial rather than comprehensive evidence of hardware generalization.
  • Depth augmentation: Depth augmentation combines stereo-camera views into outputs containing realistic fusion holes, depth-dependent noise, and structured patterns while preserving terrain geometry.Depth values are normalized to [0, 2] m for visualization.
Loading 2602.06382v2…