Source-linked AI summary
What the Elevation Map Cannot See: Semantic-Aware Locomotion and Execution-Aware Navigation for Humanoid Robot
Shunyu Yao, Songyang Liu, Dinghao Chen, Yuanyuan Lei, Shuai Li
TL;DR
Humanoid VLN benchmarks can overlook hazards poorly represented in elevation maps and assume commanded and realized motion remain aligned. The paper introduces a benchmark and closed-loop VLN–locomotion framework; its semantic-aware student reduces contact with elevation-subtle hazards, while anti-deviation improves navigation success.
Problem
Physics-based VLN benchmarks assume hazards are detectable from depth or elevation maps and that realized motion matches high-level commands, although hazard avoidance can shift the robot’s trajectory.
Source: §1 · p. 2Method
The paper benchmarks hazards of varying geometric observability and execution deviations, and couples VLN target poses with semantic-aware locomotion and realized-state feedback.
Source: §1 · p. 2Results
The depth-only student contacts elevation-ambiguous and elevation-invisible hazards roughly 2–3× more often than the semantic-aware student, which closes 51–61% and 74–100% of the oracle hazard-contact-rate gap, respectively.
Source: §5 · p. 9Takeaways & Limitations
The benchmark evaluates humanoid navigation jointly through route completion and hazard avoidance, with navigation tested in simulation and locomotion tested in simulation and reality.
Source: §1 · p. 2On uneven ground under clean conditions, Ours reaches 50.6% SR versus Oracle’s 93.6%, and the comparison does not isolate perception error from distillation or control limitations.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
Navigation for humanoid robots is critical, yet large-scale evaluation on physical hardware is often impractical due to cost and safety concerns, making simulation benchmarks essential. Existing VLN benchmarks achieve physically executable navigation, but still assume (1) all hazards are observable from elevation maps; (2) realized motions closely match desired motions. In real environments, however, fallen bottles may be ambiguous in elevation maps, while phones and water spills may be difficult to differentiate; hazard avoidance by the locomotion policy can cause the robot's actual trajectory to deviate from the path intended by the VLN policy. Such command-execution mismatch can accumulate and lead the robot toward unintended locations. To expose these failure modes, we introduce a benchmark that models both elevation-subtle hazards and execution deviations, together with a closed-loop VLN + locomotion control framework that continuously realigns high-level navigation with the robot's actual state. We evaluate navigation in simulation and further validate the locomotion policy on a physical Unitree G1 humanoid robot. Results show that semantic input reduces contact with hazards poorly represented in elevation maps, while anti-deviation improves navigation success. These findings highlight the need to evaluate humanoid navigation jointly in terms of route completion and hazard avoidance.
1 Introduction
Humanoid VLN benchmarks model physical motion but can miss hazards poorly represented in elevation maps and assume commanded motion matches realized motion. This work introduces a benchmark and closed-loop framework for evaluating hazard safety and navigation recovery together.
1 Introduction
- Existing physics-based VLN benchmarks assume hazards are observable from elevation maps and realized motion matches commands, although safety-driven avoidance can shift the robot’s trajectory.Repeated pose and heading errors can move observations away from those expected by the VLN policy.Source: §1 · p. 2
- The benchmark distinguishes commanded from realized motion and tests whether agents reach language-specified destinations safely while recovering navigation state after local deviations.It includes hazards with different levels of geometric observability, including hazards weakly represented or absent in elevation maps.Source: §1 · p. 2
- The framework couples a high-level VLN module with semantic-aware whole-body locomotion and uses realized-state feedback to correct avoidance-induced deviations.The VLN module generates relative target poses from instructions and visual observations; locomotion tracks derived velocity commands with terrain and hazard awareness.Source: §1 · p. 2
- The benchmark adds geometrically invisible hazards and physical disturbances, with Hazard Contact Rate among its metrics; navigation is tested in simulation and locomotion in simulation and reality.Large-scale physical evaluation is costly, time-consuming, and potentially unsafe, motivating repeatable simulation evaluation before deployment.Source: §1 · p. 2
2 Related Works
Prior humanoid navigation benchmarks model physical execution, while locomotion research emphasizes control and terrain geometry rather than semantic navigation hazards. This work targets hazards that geometry alone cannot distinguish and couples hazard avoidance with feedback on realized motion.
2.1 Humanoid Navigation Benchmark
- Existing humanoid navigation benchmarks advance physical evaluation but cover a more limited range of perturbations and obstacles than the richer settings introduced here.The proposed settings jointly diagnose semantic hazards and safety-induced execution deviations.Source: §2.1 · p. 2
2.2 Humanoid Locomotion and Whole-Body Control
- Humanoid control methods address motion imitation, velocity tracking, whole-body control, or terrain-aware traversal, but geometry alone cannot distinguish surfaces with different semantic risks.The paper’s focus is humanoid foot-level hazard avoidance using aligned geometric and semantic maps, coupled with realized-motion feedback during language-guided navigation.Source: §2.2 · pp. 2–3
3 Methodology
The methodology formulates humanoid navigation as a hierarchy: language-guided waypoints are executed by a hazard-aware locomotion policy, with realized-state feedback correcting deviations. It develops semantic locomotion through privileged training, robustness training, and RGB-D distillation, and adds hazard-specific rewards and anti-deviation control.
3.1 Problem Formulation
- The high-level VLN policy proposes relative target poses, while the low-level policy tracks derived velocity commands using RGB-D, terrain awareness, and proprioception.The objective is to reach the language-specified goal without hazard contact or falls while keeping realized poses close to commanded targets.Source: §3.1 · p. 3
3.2 Semantic Branch for Privileged Locomotion
- An added semantic branch represents steppability alongside terrain geometry, allowing a pretrained terrain-traversal policy to use ground-truth elevation and semantic maps.The curriculum first trains semantic hazard avoidance on challenging terrain, then transfers the capability to a waist-camera student.Source: §3.2.1 · p. 4, §3.2 · p. 4, Figure 4 · p. 4
3.2.1 Augmenting the geometric locomotion policy
- PPO training adds visible, ambiguous, and elevation-invisible hazards, using step-over clearance and premature-descent rewards to build on pretrained terrain traversal.Source: §3.2.1 · p. 4
3.2.2 Training a robust semantic teacher
- The semantic teacher is trained under randomized dynamics and payload, proprioceptive noise, pushes, and external forces while retaining ground-truth maps.Source: §3.2.2 · p. 4
3.2.3 Distillation for Deployable RGB-D Locomotion
- The deployable student learns from the ground-truth-map teacher using onboard RGB-D and proprioception, because elevation maps can underrepresent small hazards or miss their safety relevance.Its semantic branch uses RGB and frozen DINOv2-S features; its geometric branch uses depth, with stereo-derived cleaner depth at deployment.Source: §3.2.3 · p. 4
3.3 Reward Design for Stepping Over Hazards
- Two complementary rewards encourage foot clearance during swing and penalize descent before the foot clears an overlapping hazard.Both rewards are zero outside hazard overlap, leaving the nominal gait untouched; clearance is measured relative to the contact-termination boundary.Source: §3.3 · p. 5, Figure 7 · p. 5
3.4 Anti-deviation for waypoint tracking in navigation
- Localization feedback compares the realized pose with the waypoint’s intended target pose and adjusts velocity commands to steer back after hazard-avoidance deviations.Without this correction, waypoint-derived velocity is held open-loop for a fixed interval, carrying deviations into the next VLN decision.Source: §3.4 · pp. 5–6
4.1 Experiment Setup
The setup evaluates humanoid locomotion in simulation and on hardware, while navigation is tested only in simulation. The benchmark varies terrain, hazards, and physical or sensing perturbations.
4.1 Experiment Setup
- Locomotion is evaluated in simulation and on a Unitree G1, whereas navigation uses physically simulated humanoid execution in simulation only.Physical trials compare Elevation-only, Sonic with human-expert velocity commands, and Ours across three flooring types at fixed terrain difficulty.Source: §4.1 · p. 6, §4.1.1 · p. 6
4.1.1 Locomotion
- Simulation locomotion uses procedurally generated corridor episodes across uneven ground, stairs, stakes, and stonebridge terrain.Perturbations cover dynamics, payload, external disturbances, sensing, visual appearance, and camera mounting and timing.Source: §4.1.1 · p. 6
4.1.2 Shared Hazard Types
- The benchmark shares six hazard types between locomotion and navigation, organized by their visibility in elevation maps.Table 4 identifies the shared hazard categories and their elevation visibility.Source: §4.1.2 · p. 6, Table 4 · p. 6
4.1.3 Navigation
- Navigation is evaluated on training and unseen validation splits with scripted obstacles, 40% on the desired route and 60% within 1.0 m of it.Added hazards span geometrically visible, ambiguous, and invisible cases, challenging avoidance on the route and nearby detours.Source: §4.1.3 · p. 7
4.2 Locomotion Result
Semantic-aware locomotion improves success and progress over Elevation-only in simulation, particularly for hazards weakly represented in elevation maps. Physical trials and disturbance tests further evaluate locomotion beyond simulation.
4.2.1 Simulation Results
- Across difficulty levels and perturbation settings, Ours achieves higher SR and Progress than Elevation-only, with broadly comparable VTE.Under clean conditions, SR gains are 23.6, 28.5, and 25.8 percentage points on L1, L2, and L3, while Progress rises by 9.3–11.9 percentage points; some settings show slightly higher VTE.Source: §4.2.1 · p. 7
- On L3 at perturbation scale 1.5, Ours records 56.7% Progress but 22.7% SR.Harder terrain and stronger perturbations reduce both measures; partial traversal therefore does not necessarily mean completing the route without failure.Source: §4.2.1 · p. 7
- 41.2, 40.3, 34.5, and 21.4 percentage-point SR gains over Elevation-only occur on uneven ground under clean conditions and perturbation scales 0.5, 1.0, and 1.5, respectively.Ours closes 74–100% of the Elevation-only–Oracle HCR gap for elevation-invisible hazards, 51–61% for ambiguous hazards, and 12–24% for visible hazards; these are class-specific HCR ratios, not overall success-rate recovery.Source: §4.2.1 · p. 7, Table 6 · p. 8
4.2.2 Physical results
- Even with expert-provided velocity commands, Sonic leaves the corridor in 80–100% of physical trials while avoiding hazards.The physical comparison uses three flooring types; Table 7 reports clean success, corridor departures, and hazard-contact rates.Source: §4.2.2 · p. 7, Table 7 · p. 8
- Ours shows robustness to the tested common physical disturbances, consistent with the simulation results.Table 8 compares the policy with no disturbance and under three physical-disturbance conditions.Source: §4.2.2 · p. 8, Table 8 · p. 8
4.3 Navigation
Navigation ablations test semantic input and anti-deviation separately. Semantic input improves safe completion, while anti-deviation improves successful and safe completion despite slightly lower average progress.
4.3 Navigation
- Across four evaluated settings, semantic input raises Safe SR by 3.34–12.26 percentage points and reduces hazard intrusion and FR.The controlled comparison removes the semantic branch while keeping other components unchanged.Source: §4.3 · p. 8
- Anti-deviation raises Safe SR from 11.98% to 23.68%, SR from 19.50% to 29.80%, and SPL from 0.169 to 0.256.Collision rate falls from 35.93% to 12.26% and hazard intrusion from 18.38% to 12.81%, while average Progress falls from 46.12% to 43.20%; FR remains broadly similar at 8.64% versus 10.86%.Source: §4.3 · p. 8
4.4 Limitations
Semantic-aware locomotion still falls short of the oracle on uneven-ground route completion and visible-hazard avoidance. The reported results also do not isolate perception error from distillation or control limitations.
4.4 Limitations
- On uneven ground under clean conditions, Ours reaches 50.6% SR versus Oracle’s 93.6%, and its elevation-visible HCR is 24.7% versus 2.2%.The 43.0-percentage-point SR gap shows that improved handling of ambiguous and invisible hazards does not ensure reliable route completion.Source: §4.4 · p. 8
- The comparison does not isolate perception error from distillation or control limitations because Oracle and Ours differ in observation interfaces and teacher–student roles.Table 9 and Table 10 identify semantic-branch and anti-deviation ablations, but the supplied passages do not give their results.Source: §4.4 · pp. 8–9, Table 9 · p. 9, Table 10 · p. 9
- Under perturbation scale 1.5, Ours’ FR rises to 11.9%, versus 5.8% for Oracle and 3.1% for Elevation-only.The increase may reflect the camera’s sensitivity to lighting changes.Source: §4.4 · p. 9
5 Conclusion
The paper introduces a benchmark and framework for elevation-map blind spots and command–execution deviations. Semantic-aware locomotion reduces hazard contact, while closed-loop anti-deviation improves simulated navigation; future work targets additional hazard effects and hardware validation.
5 Conclusion
- On elevation-ambiguous and elevation-invisible hazards, the depth-only student contacts hazards roughly 2–3× more often than the semantic-aware student.The semantic-aware student closes 51–61% of the oracle hazard-contact-rate gap for ambiguous hazards and 74–100% for invisible hazards, versus 12–24% for geometrically visible hazards.Source: §5 · p. 9
- Anti-deviation raises simulated navigation completion from 19.5% to 29.8% SR while reducing hazard intrusion from 18.38% to 12.81%.The conclusion attributes this comparison to closed-loop execution correction.Source: §5 · p. 9
- On the physical Unitree G1, the same policy traverses cluttered floors with water spills, phones, and fallen bottles where the depth-only baseline fails.Source: §5 · p. 9
- Future work includes modeling physical hazard effects such as slipping and unstable contact, retaining previously observed hazards, and validating the complete navigation framework on hardware.The authors also propose incorporating hazard awareness into target-pose selection for anti-deviation.Source: §5 · p. 9