PaperlayerView exact revision on arXiv

Source-linked AI summary

CAP: Continuously Adaptive Perception-Blind Humanoid Locomotion via Learned Denoising

Hongjin Chen, Zijun Xu, Shihao Ma, Yi Zhao, Xilai Liu, Ke Ma, Wei Zhang, Chunyang Xie, Pengfei Li, Jieru Zhao, Wenchao Ding

arXiv:2609.11553v1cs.RO

TL;DR

Humanoid locomotion needs forward-looking depth for complex terrain, but deployment corruption exposes limits in clean-depth perceptive policies and switching-based blind/perceptive systems. CAP uses co-active denoising and proprioceptive pathways with degradation training; it matches or improves perceptive baselines when depth is informative, degrades more smoothly as perception worsens, and demonstrates robust Unitree G1 locomotion.

  • Problem

    Forward-looking depth is needed for obstacle anticipation, yet real deployments suffer partial and intermittent corruption that existing perceptive and switching-based policies do not fully exploit.

  • Method

    CAP uses a denoising world-model encoder, a co-active proprioceptive variational encoder, and coupled depth-noise curriculum and feature-dropout training.

  • Results

    CAP achieves the best Avg. SR and Avg. Ret. at clean and noisy anchors, with 99.6/97.9% Avg. SR across Clean/Noisy, while degrading smoothly across perception-quality sweeps.

  • Takeaways & Limitations

    CAP turns partial perception failure into graceful degradation within one policy rather than a discrete perceptive-to-blind handoff.

  • Takeaways & Limitations

    Large out-of-distribution geometric errors may make the denoised percept unreliable, and complete perception loss on exteroception-required terrain still requires recovery from failed steps or fall-prone states.

Abstract

from arXiv · show

Humanoid locomotion across complex terrain demands forward-looking exteroception to anticipate obstacles, yet this signal is unreliable in real-world deployment, failing partially and intermittently. Existing perceptive policies often assume that depth observations remain clean and in-distribution, while recent attempts to unify perceptive and blind control typically route or switch between separate sub-policies, leaving recoverable information in partially corrupted depth unexploited. We instead propose CAP, a single-stage humanoid locomotion policy that recovers this signal with a perceptive world-model encoder trained as a learned denoiser to reconstruct clean depth from a corrupted input, together with a co-active proprioceptive variational encoder that supplies depth-free body-state information. A coupled training recipe pairs a depth-noise curriculum on the world-model input with world-model feature dropout on the policy-facing latent, exposing the policy to failures across the entire perception-quality spectrum. In simulation, CAP matches or improves upon perceptive baselines when depth remains informative, and degrades more smoothly than a binary-switching baseline as perception worsens. On the Unitree G1, controlled trials and indoor-outdoor deployments demonstrate perception-robust locomotion under intermittent occlusion, real-sensor corruption, and outdoor depth artifacts.

1 Introduction

CAP addresses unreliable forward-looking depth with a single policy that denoises partial corruption and continuously combines perceptive and proprioceptive information. Its training exposes the policy to the full perception-quality spectrum, with evaluation spanning simulation and Unitree G1 trials.

  • 1 Introduction: CAP combines a denoising world-model encoder with a proprioceptive variational encoder in one continuously adaptive humanoid locomotion policy.The world-model reconstructs clean depth from corrupted input, while the proprioceptive pathway supplies depth-free body-state information.
  • 1 Introduction: Partial depth corruption often preserves recoverable geometric cues that prior policies neither denoise nor exploit.
  • 1 Introduction: CAP continuously fuses denoised perceptive and proprioceptive latents, avoiding an immediate handoff to a separate blind branch.
  • 1 Introduction: A depth-noise curriculum paired with world-model feature dropout trains the policy across clean, partially corrupted, and absent depth.The recipe exercises both the world-model input and its policy-facing latent.
  • 1 Introduction: Evaluation covers simulation perception-quality sweeps and Unitree G1 tests involving controlled occlusion, sensor corruption, and outdoor depth artifacts.

2 Related Work

Related work spans blind policies, perceptive pipelines, and systems that arbitrate between perceptive and blind control. These approaches differ in anticipation, explicit terrain representation, and how they handle unreliable exteroception.

  • 2 Related Work: Blind policies tolerate exteroceptive corruption but cannot anticipate gaps, high steps, and hurdles ahead.
  • 2 Related Work: Perceptive policies add exteroception, but existing approaches often assume clean, in-distribution depth and degrade when that assumption fails.
  • 2 Related Work: LiDAR pipelines commonly construct 2.5D elevation maps, yet state-estimation drift and motion distortion can undermine them during highly dynamic motion.
  • 2 Related Work: Recent unifying methods learn arbitration or routing between perceptive and proprioceptive sub-policies rather than continuously fusing co-active pathways.Figure 2 contrasts these paradigms with CAP’s continuously fused denoising world-model and proprioceptive pathways.

3 Method

CAP uses two co-active encoders and shared-gate actor–critic control to combine denoised depth with depth-free proprioception across perception failures.

  • Architecture Overview: CAP couples a perceptive world-model encoder with a proprioceptive variational encoder in a shared-gate mixture-of-experts actor–critic.The actor uses deployed-robot signals, while the critic additionally receives privileged velocity, disturbance, and heightmap signals.
  • Dual-Pathway Encoders: The proprioceptive VAE runs at 50 Hz to estimate short-horizon body state, complementing the world model’s 10 Hz terrain representation.It encodes proprioceptive history and predicts next-step proprioception without using depth.
  • Dual-Pathway Encoders: Both pathways remain co-active: the world model preserves terrain cues under partial corruption, while proprioception supports control when depth is absent.Neither pathway alone is sufficient because the world model can overshoot during sustained perception loss and proprioception cannot anticipate exteroception-required terrains.
  • World-Model Encoder: The world-model encoder ingests corrupted depth and proprioception, reconstructing rotation-stabilized clean depth rather than its observed input.This objective encourages both denoising and transformation from the body-tilted camera frame to a horizon-aligned reference frame.
  • World-Model Encoder: The world model uses recurrent temporal prediction to bridge brief corruption, but its predictions can overshoot under sustained perception loss or out-of-distribution terrain.The prior predicts forward when observations become unreliable, using dynamics learned from the training terrain distribution.
  • Training Recipe: A depth-noise curriculum and matched world-model feature dropout expose the policy to input corruption and latent prediction errors across perception conditions.Dropout replaces the world-model latent with absent, corrupted, or misaligned variants, while severity controls both mechanisms.

4 Experiments

CAP is evaluated across simulated perception-quality sweeps, anchor conditions, ablations, and real-world Unitree G1 trials. It maintains strong clean and partially corrupted-depth performance, degrades smoothly as perception worsens, and transfers to intermittent occlusion, sensor corruption, and outdoor artifacts.

  • 4.1 Anchor-Cell Comparison and Ablations: 99.6/97.9% Avg. SR across Clean/Noisy anchors is achieved by CAP, the best result at both perception conditions.Table 1 compares clean depth with heavily corrupted but still partially informative depth; CAP also achieves the best Avg. Ret. at both anchors.
  • 4.3 Perception-Quality Sweep: CAP degrades smoothly across the eight-stage perception sweep, unlike Binary-switch’s abrupt drop when its gate selects the blind branch.On Stair, CAP remains close to the blind reference through Degraded and Severe conditions, while Binary-switch remains below it under Severe failure.
  • 4.4 Real-World Experiments: 39/40 trials are completed under Clean and Partial-occlusion conditions across Stair, Platform, Gap, and Mixed terrain on the Unitree G1.Under Full cover, CAP succeeds in 5/5 Stair trials but 0/5 Platform, Gap, and Mixed trials.
  • 4.4 Real-World Experiments: A mixed-terrain cover–uncover episode shows CAP continuing from stairs through a gap and stair descent after perception is restored.The episode tests continuity through transient exteroceptive loss and reacquisition; paired reconstructions preserve traversable structure.
  • 4.4 Real-World Experiments: Flash corruption and outdoor vegetation artifacts are tested on G1, with world-model reconstructions preserving main traversable structure.Outdoor trials include platform, grass, and stair scenes, extending evaluation beyond simulator noise channels.

5 Conclusion, Limitations and Future Directions

CAP enables continuous adaptation to perception quality through denoising and proprioceptive pathways, while experiments show graceful degradation within one policy. The authors note unresolved safety issues under severe distribution shift and complete perception loss.

  • Conclusion: CAP combines a denoising world-model encoder, a co-active proprioceptive variational encoder, and degradation training to adapt continuously within one policy.The approach replaces a discrete perceptive-to-blind handoff with continuous adaptation to perception quality.
  • Conclusion: Simulation and Unitree G1 experiments show graceful degradation under partial perception failure rather than a discrete perceptive-to-blind handoff.
  • Limitations and Future Directions: Large out-of-distribution geometric errors, such as camera-extrinsic drift, may make the denoised percept unreliable.
  • Limitations and Future Directions: Under complete perception loss on exteroception-required terrain, blind degradation must be paired with recovery from failed steps or fall-prone states.The paper identifies reliability estimation, risk-aware action selection, and safe recovery as future directions.

A.1 Network Architecture

CAP uses separate world-model and proprioceptive pathways operating at different rates, with the actor consuming their latents alongside proprioception. The implementation trains perception robustness through corrupted-depth reconstruction, temporal latent reuse, and randomized camera and depth conditions.

  • Network Architecture: The actor combines current proprioception with world-model and VAE latents in a shared-gate mixture-of-experts with four actor experts.The critic augments this representation with privileged velocity, disturbance, and heightmap inputs.
  • Network Architecture: The world-model runs at 10 Hz while the VAE runs at the 50 Hz control rate, allowing cached world-model features between visual updates.The world-model refreshes every five policy steps, and prior rollout advances its state between refreshes.
  • Network Architecture: The world-model reconstructs clean simulator depth from corrupted, body-tilted depth rendered at the same camera position and heading.The encoder input and reconstruction target are deliberately decoupled by camera pose normalization in the target.
  • Network Architecture: Training randomizes depth corruption and camera geometry, with five corruption channels and severity interpolated over a 0–10 range.Evaluation uses fixed clean, noisy, and full-failure profiles rather than the adaptive training curriculum.
  • Simulation Setup: Simulation evaluates four terrain categories across 4096 environments, three seeds, a 0.8 m/s command, and fixed high-difficulty terrain.Success requires reaching the goal before timeout, deviation, collapse, or envelope-exit conditions.

B.2 Baseline Implementation Details

The baseline study implements binary branch selection within CAP’s framework and compares it with Hiking and PIE under shared evaluation conditions. The binary baseline switches auxiliary latent slots using filtered world-model reconstruction loss and is otherwise trained without CAP’s robustness components.

  • Binary-switch Baseline: Binary-switch masks exactly one of the world-model or proprioceptive latent slots while always retaining current proprioception and command inputs.Deployment switching is driven by a low-pass-filtered world-model reconstruction loss.
  • Binary-switch Baseline: The primary Binary-switch baseline uses nominal sensor noise and omits the depth-noise curriculum, world-model feature dropout, and heightmap-augmented critic.
  • Other Baselines: Hiking uses an end-to-end convolutional depth encoder without reconstruction, whereas PIE uses recurrent proprioception, convolutional depth, and heightmap reconstruction.Both are reproduced under the study’s depth-noise curriculum for matched comparison.
  • Evaluation Controls: All baselines share CAP’s reward function, terrain suite, and evaluation protocol so success differences reflect architecture and training recipe rather than evaluation distribution.

B.3 Matched Binary-switch Comparison

A matched Binary-switch comparison fixes the switching threshold through validation and evaluates success under a shared protocol. CAP achieves higher success across all reported conditions, while the results suggest that retaining co-active perceptive features avoids discarding recoverable terrain information.

  • Matched Comparison: CAP achieves higher success rates across all four reported conditions than the matched Binary-switch variant.The comparison uses success rates under the shared evaluation protocol summarized in Table A7.
  • Threshold Selection: Validation selects the largest tested switching threshold, 16β0, while the proprioception-only branch remains active for less than 1% of test time.
  • Interpretation: The results suggest that corruption-aware training reduces the need for a hard handoff because degraded perceptive features can retain useful terrain information.Switching away from the perceptive branch can discard information that remains recoverable under partial corruption.

B.4 Auxiliary Ablations

The auxiliary ablations show that the heightmap-augmented critic contributes more visibly to performance than WM-feature dropout at the Clean and Noisy anchor cells. WM-feature dropout instead targets recovery from out-of-distribution or biased world-model features, as illustrated by the full-cover case study.

  • Ablation findings: Removing WM-feature dropout causes only small in-distribution changes, with success remaining close to CAP and Avg. Ret. comparable or slightly higher at both anchors.These Clean and Noisy cells do not expose the out-of-distribution world-model errors targeted by the regularizer.
  • Ablation findings: Removing the heightmap-augmented critic lowers success on exteroception-required terrains, especially under Noisy perception, and decreases Avg. Ret. in both anchor cells.The critic provides a clean value-estimation signal independent of the actor-side depth path exposed to corruption.
  • Recovery motivation: With full camera cover during a stair-to-plane transition, WM-feature dropout enables recovery after one extra clearance, whereas removing it delays recovery for two steps.The comparison is reported in the full-cover stair-to-plane case study rather than the Clean and Noisy anchor cells.

B.5 Full-Cover Stair-to-Plane Recovery Case Study

The full-cover case study tests recovery when the robot leaves a four-step staircase for flat ground while depth remains unavailable. CAP with WM-feature dropout resumes plane walking sooner than the version without dropout, although the case specifically evaluates full-cover recovery rather than ordinary perceptive generalization.

  • Case-study setup: The robot climbs four ascending steps under full camera cover before stepping onto flat ground, creating a transition without an immediately corrected visual observation.Because depth is unavailable, the world model cannot immediately correct its recurrent state from the new visual observation.
  • Recovery outcome: With WM-feature dropout, CAP returns to plane walking after one extra high-clearance step, while removing dropout leaves stair-like foot timing for two steps.The comparison targets the regime where the world-model feature remains biased toward stair climbing after the transition.
  • Interpretation and scope: This case study supports WM-feature dropout as a regularizer against over-reliance on erroneous world-model features during full-cover recovery.Under clean or partially occluded perception, depth remains available for the world model to adapt across the transition.

C.2 Additional Real-World Results

Additional deployment episodes extend CAP’s real-world evaluation across controlled indoor perturbation trials and outdoor scenes with depth artifacts. The episodes cover stairs, platforms, mixed terrain, platform edges, and grass or vegetation.

  • Indoor trials: Indoor trials cover stair, platform, and mixed-terrain traversal under perception perturbations with synchronized raw-depth insets.These panels provide controlled deployment episodes beyond the main real-world results.
  • Outdoor deployment: Outdoor deployments transfer CAP to platform edges, stairs, and grass or vegetation scenes with depth artifacts.These environments extend beyond the simulator’s terrain and noise distribution.
  • Additional evidence: The supplementary video contains the corresponding full deployment episodes.
Loading 2609.11553v1…