Source-linked AI summary
NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation
Xiangyu Miao, Junsong Wu, Jiyuan Shi, Weiji Xie, Jinrui Han, Xingyi Wang, Weinan Zhang, Chenjia Bai, Xuelong Li
TL;DR
Paired supervision linking the same whole-body behavior on flat ground and varied terrain is scarce, despite the need to adapt posture and contacts to local support conditions. NEXUS generates terrain-adapted motion pairs and uses teacher–student learning for perceptive control. Its evaluations report broad behavioral coverage, terrain adaptability, and tracking fidelity, with zero-shot teleoperation on unseen terrains.
Problem
Paired motion supervision linking the same behavior across flat ground and varied terrains is scarce, while terrain-mismatched teleoperation requires adapting posture and contacts to local support conditions.
Source: §I · pp. 1–2Method
NEXUS generates terrain-adapted references from flat-ground motions without per-motion or per-terrain tuning, then distills a privileged teacher into a student using proprioception and onboard depth.
Source: §I · p. 2Results
NEXUS evaluations demonstrate broad behavioral coverage, terrain adaptability, and tracking fidelity, with the highest contact preservation and valid time ratio in both motion groups.
Source: A. Kinematic Motion Adaptation · p. 6, §V · p. 8Takeaways & Limitations
Zero-shot deployment enables real-time whole-body teleoperation on diverse unseen terrains.
Source: §V · p. 8
Abstract
from arXiv · showhide
Whole-body teleoperation requires a humanoid robot to reproduce a human operator's behavior even when their terrains differ. This demands that the robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control. To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher-student learning to reproduce commanded behaviors across terrains. Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method. Project website: https://nexus-humanoid.github.io/
I. INTRODUCTION
NEXUS addresses the lack of scalable paired supervision for adapting whole-body teleoperation commands to terrain that differs from the operator’s. It combines terrain-aware motion adaptation with perceptive teacher–student control, aiming to preserve commanded behavior while adjusting posture and contacts.
I. INTRODUCTION
- NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, and validates zero-shot teleoperation on unseen terrains.Its contact-guided adaptation generates paired references without per-motion or per-terrain tuning.Source: §I · p. 2
A. Kinematic Terrain Adaptation
- NEXUS adapts both hand and foot contacts without per-motion or per-terrain tuning, producing temporally aligned reference pairs at scale.This contrasts with TCRS, which leaves non-leg joints unchanged.Source: A. Kinematic Terrain Adaptation · p. 2
B. Perceptive Whole-Body Humanoid Control
- Perceptive locomotion commands leave whole-body poses and timing unspecified, while motion-reference controllers generally remain terrain-agnostic.PMT learns from terrain-adapted supervision but has limited broader behavioral generalization and computationally expensive, inconsistent reference synthesis.Source: B. Perceptive Whole-Body Humanoid… · p. 2
A. Overview
- NEXUS constructs terrain-adapted references offline, trains a PPO teacher to track them, then distills the teacher into a perceptive student using DAgger.The student uses flat-ground commands, proprioceptive history, and onboard depth at deployment.Source: §I · p. 2, A. Overview · p. 3
B. Terrain-Conditioned Motion Adaptation
Terrain-conditioned adaptation starts from a flat-ground motion and a fixed terrain, then corrects contact poses and propagates those changes through the motion. A single configuration handles all motions and terrains.
B. Terrain-Conditioned Motion Adaptation
- The method detects ankle and wrist contacts from source-motion height and velocity, using separate entry and exit thresholds to select contact keyframes.Clips without contact keyframes are excluded.Source: B. Terrain-Conditioned Motion Adaptation · p. 3
- For valid terrain queries, it shifts pelvis height by the signed terrain offset with the greatest absolute value, then solves contact-keyframe IK with the pelvis fixed.Active hands and feet are fitted to the terrain surface using limb-length and bend constraints.Source: B. Terrain-Conditioned Motion Adaptation · p. 3
- After fitting each contacting limb, penetration correction raises targets to clear sampled collisions and re-solves the chain, restoring the prior pose if acceptance tolerances fail.The correction also addresses penetrating knees, elbows, and non-contact limbs.Source: B. Terrain-Conditioned Motion Adaptation · p. 4
- Contact-keyframe corrections are interpolated within their span and held at the nearest correction outside it, producing frame-aligned adapted and flat-ground references.The horizontally placed source root’s x/y coordinates and orientation remain unchanged.Source: B. Terrain-Conditioned Motion Adaptation · p. 4
C. Perceptive Policy Learning
NEXUS connects terrain-adapted tracking objectives to flat-ground teleoperation commands through paired references and robot observations.
C. Perceptive Policy Learning
- Paired references connect the terrain-specific tracking objective to the flat-ground commands available during teleoperation.NEXUS uses the adapted references for tracking while retaining flat-ground motion features as student input.Source: C. Perceptive Policy Learning · p. 4
- The student receives flat-ground motion features, proprioceptive history, and a depth image, while the teacher uses privileged motion and robot-state observations.The teacher’s grouped observations include root-relative body poses and velocities, joint states, terrain height, contacts, and control signals.Source: C. Perceptive Policy Learning · p. 4
- Sagittal-reflection symmetry training transforms left-right body, joint, contact, and observation signals, and behavioral cloning re-queries the frozen teacher on augmented samples.The paired teacher/student observations include original and mirrored samples.Source: C. Perceptive Policy Learning · p. 4
- Teacher rewards combine motion tracking, contact objectives, control regularization, and survival, penalizing contact-timing and clearance errors alongside control-limit violations.Tracking matches pose, velocity, and joint states to the adapted reference; regularization penalizes action changes, joint velocities, limit violations, and self-collisions.Source: C. Perceptive Policy Learning · pp. 4–5
- Termination uses root clearance and root-relative end-effector heights, making its criteria insensitive to horizontal drift alone.The listed checks include an end-effector height threshold of 0.25 m and a projected-gravity difference threshold of 0.8; adaptive sampling prioritizes high-failure terrain-motion-phase segments while retaining uniform sampling.Source: C. Perceptive Policy Learning · p. 5
IV. EXPERIMENTS
The experiments assess motion adaptation, perceptive-policy performance, component contributions, and zero-shot real-world transfer.
IV. EXPERIMENTS
- The evaluation covers adaptation scalability, geometric quality and motion fidelity; policy behavioral coverage, terrain adaptability and tracking fidelity; component contributions; and zero-shot real-world teleoperation.Training uses curated flat-ground motion subsets totaling nearly 1,000 hours with terrain-adapted variants.Source: §IV · p. 5
- Policies train in mjlab with 16,384 parallel environments per GPU, a 10-frame proprioceptive history, and a single depth image.Training uses four NVIDIA H100 GPUs and procedural slopes and stairs alongside flat ground.Source: §IV · p. 5
- Policy evaluation measures full-motion success rate, body tracking errors, root position error, and contact F1 using each policy’s own pre-failure window.Kinematic evaluation also reports validity, penetration, floating, contact preservation, pose deviation, and generation real-time factor.Source: §IV · p. 5
A. Kinematic Motion Adaptation
NEXUS’s motion adaptation improves terrain-contact quality while preserving non-contact pose fidelity, and scales across motion-terrain pairs without per-case tuning.
A. Kinematic Motion Adaptation
- The benchmark covers 55 full-length motions and 16 stair and slope terrains, with one fixed configuration per method across all motion-terrain pairs.Quality metrics are averaged over seeds and terrains, with equal weight per motion within each group.Source: A. Kinematic Motion Adaptation · p. 5
- NEXUS reduces mean generation RTF relative to TCRS by factors of 4.88 and 4.23 across the two motion groups.NEXUS and Root-only each complete 128/128 attempts; TCRS has six process errors in the timing subset.Source: A. Kinematic Motion Adaptation · p. 6
- NEXUS achieves the highest contact preservation and valid time ratio in both motion groups, with the lowest hand-contact penetration and floating.Fig. 3 shows baseline foot and hand contact failures and NEXUS improving contact while preserving the free leg’s extension.Source: A. Kinematic Motion Adaptation · p. 6, Figure 3 · p. 6
- NEXUS improves contact consistency with less non-contact joint deviation than TCRS, preserving the free-leg extension that TCRS retracts.Root-only penetration can compromise reference-based initialization, while TCRS has occasional pose distortions, generation errors, and higher computational cost.Source: A. Kinematic Motion Adaptation · p. 6, Figure 3 · p. 6
B. Whole-Body Control Generalization
Across the evaluated motion subsets and terrain conditions, NEXUS combines high completion with strong orientation and foot-contact tracking results.
B. Whole-Body Control Generalization
- On flat ground, NEXUS and SONIC track a broad motion range, while PMT fails on the 100STYLE and LaFAN1 subsets; corpus diversity remains important alongside terrain adaptation.The evaluated subsets cover mostly limited translation, directional locomotion, and longer sequences of diverse whole-body behaviors.Source: B. Whole-Body Control Generalization · pp. 6–7
- NEXUS achieves the highest motion-completion rates across three dataset subsets, including ties, and terrain difficulty levels, with lowest body orientation error and highest foot-contact F1.Tracking errors and contact F1 use each controller’s own pre-failure window, so early termination omits later motion segments.Source: Figure 4 · p. 7, B. Whole-Body Control Generalization · p. 7
- Terrain-agnostic controllers retain some success on mild terrain but deteriorate as support geometry departs further from flat ground; NEXUS performs more strongly across terrain levels.NEXUS trains on 10 cm steps and 16.7° slopes, differing from both tested terrain levels.Source: B. Whole-Body Control Generalization · pp. 6–7
C. Ablation Study
The ablations show that vision improves tracking and completion, especially on harder terrain levels, while teacher–student training substantially improves tracking accuracy.
C. Ablation Study
- 78.10% vs. 70.51% SR on the two harder terrain levels; vision improves global position tracking and completion, while teacher–student training substantially improves tracking accuracy.The ablation table compares vision and teacher–student variants; vision and no-vision students share a teacher and training settings.Source: C. Ablation Study · p. 7, Table II · p. 7
D. Zero-Shot Real-World Deployment
NEXUS was deployed without finetuning for real-time whole-body teleoperation across indoor and outdoor terrains. Demonstrations include challenging steps, slopes, and whole-body behaviors, while the conclusion notes a boundary in overhanging terrain.
D. Zero-Shot Real-World Deployment
- The robot ascends staircases with 15 cm steps and steps or jumps down from a 60 cm-high platform, although training terrains contain only 10 cm steps.Falls occurred during some platform drops and cartwheel attempts; no failures were observed during routine pose tracking.Source: D. Zero-Shot Real-World Deployment · p. 8
- NEXUS supports kneeling on one or both knees, crouching, squatting, single-leg kicking and balancing, jumps, and cartwheels; routine pose tracking had no observed failures, though some 60 cm platform drops and cartwheel attempts caused falls.On Unitree G1, NEXUS uses onboard Intel RealSense D435i depth perception without finetuning, and its control loop is configured at 50 Hz.Source: D. Zero-Shot Real-World Deployment · pp. 7–8
V. CONCLUSION
- The conclusion reports scalable paired-motion generation without per-motion or per-terrain tuning and zero-shot real-world teleoperation on diverse unseen terrains.It identifies overhanging terrain as unsupported and proposes broader geometric coverage and task-level planning as future work.Source: §V · p. 8
APPENDIX
The appendix details terrain-conditioned adaptation, from contact detection and terrain queries to bounded pose correction and temporal reconstruction of corrections.
A. Terrain-Conditioned Motion Adaptation Details
- Table III lists parameters for contact detection, terrain queries, and pose correction.The table is identified as the motion adaptation parameter table.Source: A. Terrain-Conditioned Motion… · p. 10
- Contact enters only when link height, horizontal speed, and absolute vertical speed are all below their entry thresholds, and persists until any exceeds its exit threshold.Contact states are initialized as inactive; Fig. 5(a) illustrates the hysteresis rule.Source: A. Terrain-Conditioned Motion… · p. 10
- Terrain queries cast vertical rays downward from above the terrain, excluding robot geometries; the first terrain intersection supplies height and local surface normal.The queries use MuJoCo’s mj_ray function.Source: A. Terrain-Conditioned Motion… · p. 10
- Bounded mid-link correction tests both bend-plane rotation directions, then accepts candidates using collision clearance, end-effector drift, and penetration checks.If no candidate passes, the preceding pose is restored; this local procedure does not guarantee a collision-free trajectory.Source: A. Terrain-Conditioned Motion… · p. 10
- Source-relative corrections are interpolated between contact keyframes, preserving zero corrections and extending the endpoint correction constantly beyond the keyframe span.Non-contact intervals receive corrections interpolated from surrounding contact frames.Source: A. Terrain-Conditioned Motion… · p. 10
B. Policy Training Details
The policy-training details cover adaptive sampling, tracking and contact objectives, actuation, and training configurations; the appendix also illustrates motion-adaptation procedures.
B. Policy Training Details
- Fig. 5 illustrates height-based contact hysteresis and PCHIP reconstruction of scalar corrections from contact keyframes, with constant extension beyond their span.Its curves are illustrative, not measured results.Source: Figure 5 · p. 10
- Tracking targets align the adapted reference with the robot root’s horizontal position and heading, compare body velocities in world coordinates, and use soft joint-position bounds.The teacher curriculum scales non-tracking weights from 0.5 to 1.0 times their listed values over training.Source: B. Policy Training Details · p. 10
- Tables IV and V list reward expressions and weights, and training-time domain randomization, while Table VI summarizes teacher and student hyperparameters and architectures.The listed randomizations are applied at startup and reset, through periodic pushes, and in student observations.Source: B. Policy Training Details · pp. 10–11, Table IV · p. 11, Table V · p. 11
- The contact-agreement timer gains one control step when states agree and loses one when they disagree; foot support measures sole samples more than 3 cm above terrain among contacting feet.Foot-support samples are spaced 2 cm apart.Source: B. Policy Training Details · p. 11
- Policy actions specify scaled offsets from nominal joint positions, which a PD controller tracks under componentwise torque limits.The paper follows BeyondMimic for PD gains, simulated joint armature, torque limits, and action scales.Source: B. Policy Training Details · p. 11
- Adaptive sampling divides sequences into 50-frame bins, caps failure rates at 200 times their mean, mixes them with uniform sampling, and samples earlier frames to practice transitions.The frame offset is sampled uniformly from 0–199 and clamped at the sequence start.Source: B. Policy Training Details · p. 11
C. Evaluation Details
The evaluation compares controllers across flat-ground motions and non-flat terrain conditions, with separate protocols for the main comparison and ablation study. Metrics are aggregated across motions, groups, terrain types, and levels using specified evaluation windows and reference labels.
C. Evaluation Details
- The main comparison evaluates 160 motions on flat ground and eight non-flat terrain conditions, averaging results over motions within three motion groups and then equally over groups and terrain conditions.The conditions cover ascending and descending stairs and slopes at two levels.Source: C. Evaluation Details · p. 11
- The main comparison uses pre-termination windows for pose errors and the same flat-ground reference contact labels for Contact F1 across controllers, including terrain trials.Table VI is identified as containing training hyperparameters and network architecture.Source: C. Evaluation Details · p. 12, Table VI · p. 12
- The ablation protocol evaluates each policy on 415 held-out locomotion clips across five levels of ascending and descending stairs and slopes, totaling 8,300 trials per policy.The clips total 3,601.72 seconds, and each policy is evaluated against 20.01 hours of reference motion; the middle terrain level matches the training scale.Source: C. Evaluation Details · p. 12
- For the ablation, reported means and sample standard deviations summarize five level means, with clips and four terrain types averaged equally within each level.All trials are retained; pose errors use each policy’s pre-termination window, Contact F1 uses adapted reference labels, and the common-window check truncates matched trials at the earliest of three policy terminations.Source: C. Evaluation Details · p. 12