PaperlayerView exact revision on arXiv

Source-linked AI summary

TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model

Anqi Li, Yuxin Chen, Zhaobo Li, Zhuo Cao, Junli Ren, Masayoshi Tomizuka, Dhruv Shah

arXiv:2609.09158v1cs.ROcs.AI

TL;DR

Humanoid navigation in cluttered indoor environments requires whole-body reasoning beyond planar path planning. TANGO learns language-conditioned 29-DoF action prediction from simulated collision-free traversal data and achieves strong navigation and cluttered-scene traversal, including zero-shot real-world deployment, while remaining subject to tracker and RGB-only limitations.

  • Problem

    Planar navigation methods do not capture the continuous whole-body feasibility needed for humanoids to traverse cluttered 3D environments.

  • Method

    TANGO predicts 29-DoF whole-body action chunks from language and RGB observations, trained with simulated collision-free traversal data and executed through a low-level tracker.

  • Results

    TANGO achieves state-of-the-art vision-language navigation, outperforms evaluated modular baselines in cluttered traversal, and transfers zero-shot to a Unitree G1 robot.

  • Takeaways & Limitations

    The results support directly predicting whole-body actions for language-guided humanoid navigation and indicate compatibility with alternative low-level tracking.

  • Takeaways & Limitations

    Deployment is constrained by low-level tracker capability and RGB-only perception, especially in stairs, visually ambiguous, cluttered, or low-light environments.

Abstract

from arXiv · show

We study the problem of navigating cluttered indoor environments with a humanoid robot. Unlike conventional methods that model navigation as a 2D path planning problem, humanoid traversal in cluttered environments requires continuous geometry-aware whole-body adaptation, including coordinated arm placement, torso adjustment, and gait modulation for collision-free movement through complex 3D spaces. We introduce TANGO, the first whole-body vision-language navigation framework for language-conditioned humanoid traversal in cluttered environments. Given a natural-language instruction and egocentric RGB observations, TANGO directly predicts 29-DoF joint-space actions for downstream whole-body control. We train TANGO entirely in simulation by synthesizing diverse collision-free traversal behaviors via global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking. This pipeline provides dynamically feasible action supervision for learning language-conditioned whole-body policies. In extensive simulation experiments, TANGO demonstrates state-of-the-art performance in vision-language navigation, while outperforming strong modular baselines in navigating challenging scenes requiring obstacle negotiation. Lastly, we deploy TANGO zero-shot on a Unitree G1 humanoid robot, and observe robust language-guided traversal in cluttered real-world scenes without training on any real-world navigation data.

1 Introduction

TANGO addresses the gap between planar vision-language navigation and the continuous whole-body feasibility required for humanoid traversal in cluttered 3D environments. It directly predicts whole-body actions and is evaluated on obstacle negotiation and real-world traversal.

  • Motivation: Humanoid navigation couples route decisions with continuously changing body geometry, requiring collision-free coordination of arms, torso, and legs.A route that appears valid at the planning level can remain infeasible for the embodied robot.
  • Evaluation: TANGO demonstrates robust traversal in cluttered scenes with overhead and floor obstacles, narrow passages, and a 30-meter zero-shot real-world route.The reported demonstrations are summarized in Figure 1.
  • Research gap: Existing VLN methods usually predict 2D waypoints or discrete actions, while existing approaches do not jointly reason about navigation intent and whole-body traversability.Their low-dimensional action spaces cannot explicitly represent whole-body feasibility in spatially constrained scenes.
  • Approach: TANGO is a unified whole-body VLA framework that directly predicts 29-DoF joint-space actions from language instructions and egocentric RGB observations.This avoids separate navigation and control modules.
  • Approach: TANGO’s simulation pipeline synthesizes collision-free traversal trajectories to provide dynamically feasible supervision for language-conditioned whole-body policies.The framework targets semantic task diversity and physically plausible traversal behavior.

2 Related Works

Prior work covers planar vision-language navigation, short-horizon humanoid traversal, and large-scale humanoid control, but these lines of work leave long-horizon, collision-aware whole-body navigation insufficiently addressed.

  • Large Models for Vision-Language Navigation: Large multimodality navigation models and fine-tuned VLA systems generally treat navigation as planar trajectory planning, omitting the physical gap in deployment.These methods support diversified environments but do not explicitly model whole-body traversal feasibility.
  • Cluttered Environment Traversal: Humanoid parkour methods demonstrate challenging obstacle traversal but mainly focus on short-horizon interactions with scene objects.TANGO instead targets long-horizon navigation with collision avoidance.
  • Humanoid Whole-Body Control: Recent humanoid motion-tracking advances enable large-scale learning of humanoid control policies.This work forms the broader control foundation for whole-body learning.
  • Humanoid Whole-Body Control: Decoupled systems simplify loco-manipulation by predicting upper-body motions alongside high-level lower-body commands, but limit the whole-body coordination needed for cluttered-scene traversal.TANGO contrasts this design with end-to-end whole-body motion prediction used as reference trajectories for a low-level tracker.

3 Method

TANGO builds a whole-body navigation system around simulated data generation, language-conditioned action prediction, and tracked execution. Its method augments scenes, synthesizes and filters collision-free motions, then streams predicted action chunks to a low-level tracker.

  • Problem formulation: TANGO formulates whole-body VLN as predicting action chunks from language, RGB histories, and proprioceptive state.Each action contains 29-DoF joint targets and a base representation, then streams to a low-level tracker.
  • Environment Augmentation: Scene augmentation creates challenging traversal settings with lateral, ground-level, and overhead obstacles across 578 retained source scenes.The obstacles respectively create narrow passages, require stepping, and enforce upper-body clearance.
  • Collision-Free Motion Generation: The PET pipeline plans collision-aware paths, edits motions for obstacle negotiation, and tracks them to filter out physically infeasible or colliding trajectories.Editing adds behaviors such as arm clearance, stepping over obstacles, and crouching.
  • Collision-Free Motion Generation: The resulting dataset contains 64,633 trajectories generated with 211 total GPU-hours for PET processing and rendering.PET and rendering require 86 and 125 RTX PRO 6000 GPU-hours, respectively.
  • TANGO Architecture: TANGO combines a vision-language backbone, a flow-based diffusion action expert with real-time chunking, and a motion tracker for continuous whole-body control.The model jointly trains the backbone and action expert while streaming predicted chunks to the tracker.
  • System-1 & System-0: The action expert predicts stabilized whole-body reference chunks, using relative base yaw, planar displacement and heading deltas, and training-time RTC for streaming execution.The recovered chunks are sent to the SONIC tracker, which produces high-frequency joint commands from proprioception.
  • Joint Training Objectives: Joint training combines VideoQA cross-entropy and flow-matching losses, with the flow-matching term weighted by w_FM = 20.TANGO is trained end-to-end for one epoch at a learning rate of 1 × 10^-5.
  • Deployment: Deployment separates compute-intensive VLA inference from onboard high-frequency tracking across a server and Unitree G1 robot.The simulation and real-world deployment setup is designed to evaluate system effectiveness and robustness.

4 Experiments

TANGO is evaluated on benchmark, cluttered-scene, real-world, and ablation settings, with results emphasizing whole-body action generation for physically constrained navigation. It achieves strong navigation and safety performance in simulation, transfers zero-shot to real-world trials, and benefits from motion continuity and obstacle-aware supervision.

  • VLN Performance: TANGO achieves the highest SR and lowest reported NE on VLNVerse-seen and VLNVerse-unseen, with SPL and OSR comparable to RDP.Unlike the baselines, TANGO uses low-level physical control while the others are evaluated with teleportation.
  • Cluttered Environment Traversal Performance: TANGO reduces CR from 15.81% to 9.90% against the strongest baseline while improving SR by 1.87 points and SPL by 2.34 points.The comparison uses augmented VLNVerse-unseen scenes and evaluates RGB-only TANGO against modular baselines, including one with LiDAR-based perception.
  • Real-World Experiment: TANGO demonstrates zero-shot sim-to-real transfer through long-horizon navigation, narrow-path side-stepping, overhead-obstacle avoidance, and stepping over ground obstacles.The robot continuously adapts its body configuration while following natural-language instructions in physical environments.
  • Real-World Experiment: TANGO achieves the highest SR across short-horizon, long-horizon, and cluttered 3D real-world settings, with fewer collisions than the fine-tuned InternVLA-N1 + Unitree WBC baseline.Each setting contains three scenes with five trials per scene, totaling 15 trials per method per setting.
  • Ablation Studies: Planar policies experience a substantial performance drop under physical execution, whereas TANGO remains robust under low-level control constraints and outperforms the evaluated baselines.The Ours-2D versus TANGO comparison isolates the benefit of whole-body action prediction more directly than the zero-shot InternVLA-N1 rows.
  • Ablation Studies: Removing RTC reduces SR from 43.75% to 10.94% and increases CR from 9.90% to 14.60%, while removing motion editing reduces SR to 36.25% and increases CR to 20.60%.These ablations support motion continuity across action chunks and obstacle-aware motion supervision as important design components.

5 Conclusion

TANGO directly predicts 29-DoF joint-space actions for whole-body humanoid navigation, with experiments indicating strong navigation and cluttered-traversal performance. The authors identify low-level tracking and RGB-only perception as constraints for more complex deployment.

  • TANGO directly predicts 29-DoF joint-space actions through a whole-body vision-language navigation framework.The system uses a Qwen2.5VL-7B backbone with a flow-matching action expert, alongside training-time RTC and a high-frequency tracker for real-time execution.
  • Experiment results indicate state-of-the-art vision-language navigation capability and decent traversal performance in cluttered environments.Ablations support whole-body action prediction, obstacle-aware motion editing, and RTC, while also showing compatibility with an alternative low-level tracker.
  • TANGO demonstrates the feasibility of directly predicting 29-DoF actions as a step toward practical whole-body large planning models.The authors suggest adapting and scaling this foundation toward more general loco-manipulation models requiring coordinated use of the entire body.
  • Low-level tracking constrains deployment in more complex environments, while RGB-only input may limit understanding of cluttered, ambiguous, or low-light scenes.The authors identify stair walking as an example and suggest depth cameras and LiDAR as possible future additions.

A Training Details

The training procedure emphasizes rare whole-body behaviors through both data-level rebalancing and loss-level weighting.

  • Training data are rebalanced by upsampling action segments containing large turns, sideways motion, squat, stride, and mixed squat-stride behaviors.Stride and sideways motions receive the strongest boosting because they occur less frequently and have weaker action signals.
  • Loss-level weighting further emphasizes behaviorally important whole-body motions during training.The passage states that both data-level rebalancing and loss-level weighting are used to emphasize rare behaviors.
  • Stride and sideways motions are prioritized most strongly to compensate for lower frequency and weaker action signals.

B Deployment Details

The real-world system separates server-side VLA inference from onboard low-level tracking, combining RGB observations and proprioception for high-frequency humanoid control.

  • The VLA runs on an RTX PRO 6000 server, while the SONIC tracker runs on the onboard Jetson Orin NX.
  • Front and downward RGB observations plus proprioception are transmitted with approximately 20 ms latency.The observations come from RealSense D455 and D435i cameras.
  • VLA inference runs every 0.5 s, producing 15-action segments at 30 Hz that are resampled to 50 Hz before tracking.
  • The low-level control loop runs at approximately 200 Hz.

C.1 Environment Augmentation Details

The environment-augmentation pipeline inserts trajectory-conditioned obstacles into indoor scenes to elicit stride, sidle, and squat behaviors while preserving scene validity and aligning language prompts with traversal actions.

  • Obstacle placement: Obstacles are inserted along A* trajectories at locations where whole-body behaviors are likely required.The augmentation uses original scenes, navigation trajectories, and occupancy maps to create challenging traversal scenarios.
  • Validity checks: Candidate placements are aligned with local path directions and checked against occupancy maps and other trajectories to preserve scene validity.Obstacles are prevented from blocking unrelated paths or appearing too close to trajectory endpoints, while the source trajectory may be affected.
  • Target behaviors: Low obstacles encourage stride, side obstacles encourage sidle, and overhead obstacles encourage squat.Obstacle assets are selected according to the target behavior, including rugs or stools for stride and furniture or ceiling objects for other interactions.
  • Dataset composition: After filtering, 205 VLNVerse and 373 SAGE-3D scenes are retained, with augmented obstacles distributed as 44% stride, 15% sidle, and 41% squat.Obstacle poses are randomly sampled within traversable ranges rather than fixed by category.
  • Language augmentation: Obstacle-aware language prompts are added to align instructions with action labels and support learning collision-avoidance behaviors.The passage reports that these prompts help convergence and provide complementary cues in the RGB-only real-world setting.
  • Generation cost: Generating 64,633 trajectories requires 211 RTX PRO 6000 GPU-hours for motion generation and rendering.The reported cost excludes VLA training.

C.2 Motion Generation Details

The PET pipeline plans a reference gait, edits it for whole-body obstacle negotiation, and tracks it into dynamically feasible motion. Its editing stage preserves gait timing while applying localized, geometry-aware adaptations for squat, stride, and sidle obstacles.

  • Pipeline overview: PET generates a 29-DoF whole-body reference motion from an augmented scene and start–goal pair, then rerenders observations and regenerates language instructions.The pipeline comprises Plan, Edit, and Track stages.
  • Plan: A★ plans a safer planar route using obstacle-aware costs derived from 2D signed distance, while SONIC converts rewritten path commands into a smooth whole-body gait.Narrow passages trigger 90° body-heading offsets, decoupling travel and facing directions for crab-walking.
  • Track and edit boundary: The initial gait follows the planar route and side-walks through narrow passages but remains obstacle-agnostic above the floor until Edit adds 3D body adaptations.The Plan-stage gait therefore does not yet step over or duck under obstacles.
  • Edit: The reference gait preserves timing, phase, and style, while whole-body IK applies local edits only where the path corridor contains obstacles.The editor uses posture tracking, soft obstacle-avoidance objectives, and other whole-body IK terms rather than replacing the locomotion prior.
  • Edit: SDF-based repulsive guidance acts on upper-body links through bounded pseudo-forces, with low-pass filtering and coordinated center-of-mass offsets for coherent avoidance.Feet and ankles remain excluded so the reference gait and gait-adaptation module determine contact patterns.
  • Edit: Squat obstacles receive lookahead-based downward guidance and coordinated crouch posture changes, while stride obstacles retarget footsteps and add asymmetric vertical clearance arcs.These mechanisms respectively lower the upper body and help the trailing leg clear obstacles shortly after lift-off.

D Experiment Metrics

The evaluation uses standard vision-language navigation metrics for task success, stopping accuracy, path efficiency, and goal visitation, with collision metrics added for cluttered-scene safety. Metrics operate on trajectories produced from language instructions, sequential observations, and whole-body action chunks.

  • Metric set: The evaluation reports SR, SPL, NE, and OSR, adding CR to quantify physical safety during cluttered-scene traversal.Simulation tables report SR as a percentage, while Table 3 reports successful trials out of 15.
  • Navigation Error (NE): NE measures the average geodesic distance from the final stopping position to the target goal, with lower values indicating closer stopping.The final stopping position is taken from each executed robot-base trajectory.
  • Success Rate (SR): SR counts episodes whose final stopping position lies within the 3-meter success threshold of the target goal.The threshold δ is set to 3 meters following standard VLN evaluation.
  • Oracle Success Rate (OSR): OSR records whether an executed trajectory enters the goal region at least once, regardless of its final stopping position.This distinguishes reaching the goal neighborhood from ending there.
  • Success weighted by Path Length (SPL): SPL jointly measures task completion and path efficiency, assigning zero to failed episodes and penalizing unnecessarily long successful trajectories.The penalty uses the path-length ratio.
  • Collision metrics: CR is an episode-level safety metric for cluttered-scene traversal, whereas mean real-world collisions counts multiple collisions within each trial and is not a percentage.Lower CR indicates safer whole-body traversal.
Loading 2609.09158v1…