Source-linked AI summary
LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
Haoru Xue, Xiaoyu Huang, Dantong Niu, Qiayuan Liao, Thomas Kragerud, Jan Tommy Gravdahl, Xue Bin Peng, Guanya Shi, Trevor Darrell, Koushil Sreenath, Shankar Sastry
TL;DR
Humanoid whole-body control lacks expressive vision-language action interfaces and suitable visual benchmarks. LeVERB introduces a synthetic-data benchmark and hierarchical latent-action policy, achieving zero-shot deployment in simulation and on real hardware while remaining limited in long-horizon planning.
Problem
Humanoid WBC lacks expressive action interfaces and sufficient visual demonstration data for vision-language control.
Method
LeVERB combines a high-level vision-language policy with a low-level controller through a structured latent action space learned from synthetic robot motion data.
Results
LeVERB achieves zero-shot closed-loop deployment on real humanoid hardware and outperforms a naive hierarchical VLA by 7.8 times in task success rate.
Takeaways & Limitations
LeVERB demonstrates zero-shot sim-to-real humanoid WBC using a latent vision-language interface.
Takeaways & Limitations
LeVERB has limited long-term history and planning, predicting only a few seconds into the future.
Abstract
from arXiv · showhide
Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times.
1 Introduction
LeVERB addresses the difficulty of applying vision-language-action models to humanoid whole-body control by replacing rigid action interfaces with a hierarchical latent-action architecture. It combines synthetic robot-specific data, a vision-language policy, and a reactive whole-body controller for zero-shot deployment.
- Humanoid robots remain underexplored for VLA control because they are high-dimensional nonlinear dynamic systems unlike quasi-static arm manipulators.
- Existing VLA-enabled WBC systems use explicit low-dimensional actions, limiting expressive whole-body motion and complex scene interactions.
- LeVERB couples a high-level vision-language policy with a low-level reactive controller that executes whole-body motions from latent actions.
- LeVERB trains on photorealistically rendered, randomized scenes containing retargeted human motions and semantically related language commands.
- A CVAE-based structured latent space aligns visual-language inputs with actions, after which a separate action module learns robot dynamics from the frozen latent space.
- LeVERB achieves zero-shot closed-loop deployment with expressive whole-body motions and scene interactions in simulation and on real humanoid hardware.
2 Related Work
Prior work combines language or vision with humanoid control, but existing approaches lack the full combination of visual grounding, latent whole-body control, photorealistic data, and real-robot validation targeted by LeVERB.
- Physics-based humanoid control has used motion tracking and latent policies conditioned on language or object interactions.
- Real-world humanoid systems commonly predict explicit commands, which can produce jittery or unnatural motions, while LangWBC lacks visual grounding and high-level reasoning.
- End-to-end VLAs can suffer high inference latency, conflicting with the high-frequency feedback needed for humanoid whole-body stability and agility.
- No prior work had demonstrated vision-language-driven whole-body control on real humanoid robots with a hierarchical latent architecture.
- Humanoid visual demonstration data remains scarce because collecting large-scale whole-body motions on physical robots is complex.
- LeVERB provides photorealistic renderings and physics-based simulation for whole-body motions intended for transfer to real hardware.
3 LeVERB Dataset and Benchmark
LeVERB-Bench generates scalable, photorealistic humanoid whole-body demonstrations by replaying retargeted motions in randomized simulation environments. The resulting dataset combines visual-language and language-only examples for closed-loop benchmarking and latent-action training.
- LeVERB-Bench is an efficient, scalable pipeline for synthetic visual-language humanoid WBC data generation and closed-loop benchmarking.
- Retargeted MoCap motions are replayed in simulation to collect photorealistic rollouts without requiring reliable dynamic control during data collection.
- The training pipeline uses retargeted synthetic motions to train LeVERB-VL and cache latent verbs for conditioning LeVERB-A.
- The pipeline randomizes scene backgrounds, object properties, task setups, camera views, and mirrored rollouts before assigning egocentric text commands.
- 154 trajectories randomized 100 times produce 17.1 hours of photorealistic motion rollouts spanning multiple task categories.
- An additional 2.7 hours of language-only data covers 500 diverse trajectories and uses spatial cues to preserve disambiguating context.
4 Dual Process Humanoid Control
LeVERB separates vision-language interpretation from high-frequency whole-body action generation through a hierarchical latent interface. Its training aligns semantic inputs with latent motion representations, then distills reactive control from trajectory-tracking teachers.
- Overall hierarchy: The policy consumes proprioceptive readings, two-view visual inputs, and textual instructions to predict a latent action distribution.The visual inputs come from egocentric and randomized third-person cameras, while the language input is the textual instruction.
- Overall hierarchy: LeVERB formulates humanoid WBC as a hierarchical System 2/System 1 policy, with LeVERB-VL producing latent commands and LeVERB-A producing dynamics-level actions.The latent vector serves as a one-way interface encoding complex whole-body motion objectives.
- LeVERB-VL training: LeVERB-VL uses a residual CVAE and kinematics encoder so semantic vision-language reasoning can be separated from fine-grained motion information.The residual latent combines the VLA prior with trajectory-derived information, while the decoder reconstructs future states from sampled latents.
- LeVERB-VL training: The LeVERB-VL objective combines trajectory reconstruction, distribution alignment, and adversarial classification.The adversarial component aligns latent representations from visual and language-only trajectories.
- LeVERB-A training: LeVERB-A is trained after freezing LeVERB-VL by distilling multiple PPO teacher policies into a Transformer student conditioned on latent commands.Teachers track retargeted kinematic trajectories with privileged proprioceptive observations, while the student is trained with DAgger and Huber loss.
- LeVERB-A training: During training, latent codes are sampled from the predicted distribution and held fixed for the System 1–2 resampling interval; deployment conditions LeVERB-A on the predicted mean.This training procedure samples from the latent distribution rather than always using its mean.
5 Experiment
LeVERB is evaluated on closed-loop whole-body tasks using ablations, with 20 runs per task/environment and unseen material-object combinations. It leads in 9 of 10 categories, while ablations reveal the importance of structured latent representations, cross-source alignment, visual-language planning, and latent sampling.
- Closed-Loop Evaluation: 20 runs per task/environment measure success rates on unseen combinations of materials and objects.The evaluation is designed to approximate real-world deployment while testing combinations absent from training.
- Ablations: The NE variant reaches 53%, as its finer-grained, less semantic latent space is more vulnerable to unseen scenes.Removing the kinematics encoder shifts motion-style encoding to the vision-language model.
- Overall Results: LeVERB reaches 58.9% average success and leads in 9 of 10 task categories.Table 2 compares LeVERB against its ablated variants across the benchmark.
- Ablations: Removing the discriminator causes a significant visual-navigation decline by separating latent distributions for vision-language and language-only demonstrations.This prevents motion skills learned from language-only demonstrations from transferring effectively to vision-language tasks.
- Ablations: The NS variant fails on almost all tasks because its unstructured latent space produces poor interpolation and frequent out-of-distribution inputs for the action module.NS disables sampling in both the vision-language and action modules.
- Ablations: The NVL baseline has minimal success on visual-language tasks, whereas text-only locomotion reaches 100%.The comparison indicates that visual feedback requires high-level planning capacity beyond the low-level policy alone.
- Real-World Deployment: On real hardware, LeVERB zero-shot replays latent verbs to complete visual navigation and sitting on a Unitree G1.The system handles unseen verb-object combinations and adjusts walking and turning according to the chair’s visual position.
6 Limitations
LeVERB is limited by its short temporal horizon and lack of fast vision feedback in the low-level controller. These constraints restrict long-duration, agile, and dexterous behaviors.
- Limitations: LeVERB lacks long-term history and planning, limiting tasks to a horizon of only a few seconds.The paper identifies longer-horizon task execution as a direct boundary of the current system.
- Limitations: The low-level controller lacks a fast vision feedback loop, constraining more agile or dexterous tasks.This is presented as a separate limitation from the short planning horizon.
7 Conclusion
The paper presents LeVERB as a latent-action model and benchmark for humanoid whole-body control, with zero-shot real deployment after synthetic-data training. It reports a 7.8-times advantage over a naive hierarchical VLA and identifies reinforcement-learning fine-tuning as a possible future direction.
- Conclusion: LeVERB is presented as the first vision-language latent action model for humanoid whole-body control and the first sim-to-real-ready photorealistic benchmark of its kind.The conclusion frames both the model and benchmark as first-of-kind contributions.
- Conclusion: LeVERB achieves zero-shot deployment to a real humanoid despite training only on a small synthetic dataset.The conclusion attributes this result to the dual-process, CVAE-based VLA.
- Conclusion: LeVERB outperforms a naive hierarchical VLA by 7.8 times in task success rate.This is the conclusion’s headline comparative result.
- Conclusion: The authors conjecture that post-training, especially reinforcement-learning fine-tuning, could improve performance by aligning the closed-loop latent-vocabulary distribution.This is stated as a future-work possibility rather than a demonstrated result.
A Detailed Visual-Language WBC Data Generation Workflow
The workflow randomizes scenes and household objects to generate semantically meaningful visual-language tasks for humanoid WBC data.
- Randomization: Scene-level randomization selects background-object colors and materials for synthetic environments.These options provide variation in the surrounding scene.
- Randomization: Object-level randomization varies colors and materials of labeled household objects such as chairs and desks.The object labels support instructions grounded in specific visual attributes.
- Task Creation: The workflow enables task instructions such as “walk towards the yellow desk” and “go sit on the red sofa.”These examples connect object appearance randomization with semantically meaningful language commands.
B Implementation Details for System 2
System 2 uses mixed, augmented trajectory data and a structured latent representation for vision-language-conditioned motion prediction. Training includes representation choices, Transformer-size selection, and an ablation supporting the latent input's role in reconstruction.
- Data mixture: System 2 combines 3,696 image-conditioned and 2,300 non-image trajectories, repeating them with varied language prompts and image renderings.The augmentation is intended to balance data sources and visual domains while reducing overfitting.
- Data processing: The 13-joint data pipeline represents root pose with position and rotation, body joints with positions, and predicts future delta actions.Root actions include delta position and rotation, while other joints use delta positions; optional training noise improves robustness.
- Hyperparameters: ViT-Base provides the selected Transformer-backbone trade-off between performance and computational efficiency, with a 256-dimensional latent space.The backbone ablation spans ViT-Tiny through ViT-Base, while visual and textual encoders produce 768-dimensional features.
- Training details: Equation 2 training uses β1 = 10^-1 and β2 = 5×10^-4, with schedulers ramping alignment and adversarial terms from 0 to 1 over 40% of training.The schedules are applied to stabilize training.
- Objective validation: Randomly sampled latents produce much higher trajectory-reconstruction loss than the learned latent input, supporting the CVAE objective's effectiveness.The result indicates that current-state information alone is insufficient to reconstruct future states.
D Details for LeVERB-A Teacher Policies
LeVERB-A teacher policies track reference whole-body motions using proprioception, reference-motion commands, and regularized rewards, then support a student policy conditioned on latent commands for real-time deployment.
- Teacher observations and actions: Teacher observations combine base velocities, joint states, previous actions, and next-frame reference joint and torso motion commands.Actions are joint-position commands with small stiffness and damping.
- Rewards and termination: Teacher rewards combine DeepMimic-style motion tracking, action-rate and soft-joint-limit penalties, and penalties for termination from large torso tracking error.Episodes terminate when torso position or orientation tracking error becomes too large.
- Domain randomization: Teacher training randomizes friction, restitution, calibration positions, armature, and velocity perturbations for zero-shot sim-to-real transfer.These randomizations target physics and calibration variation during training.
- Student policy: The student receives proprioception, gravity projected onto the base, previous actions, and latent commands from System 2.System 2 updates at 10 Hz while System 1 runs at 50 Hz, holding each sampled latent for five steps.
- Student architecture: The student uses a two-layer Transformer with four attention heads and 128 hidden dimensions, while observation history degrades performance in this setting.Latent commands and proprioceptive inputs are encoded as separate tokens.
- Hardware deployment: Deployment runs LeVERB-A at 50 Hz on an onboard CPU and LeVERB-VL at about 10 Hz on an external RTX 4090 PC, communicating through ROS2.The robot uses joint encoders, IMU sensing, a state estimator, and camera streaming.
G Evaluation Environment
Evaluation uses visually unseen task instances across randomized environments, instructions, textures, object properties, and camera viewpoints.
- Evaluation protocol: Each task category is evaluated on 20 random environments and instructions with completely randomized, previously unseen scene textures and object properties.Third-person camera angles are also locally randomized within scene constraints, and evaluation tasks are visually unseen during training.