Source-linked AI summary
ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control
Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen
TL;DR
Humanoid motion trackers lack exteroceptive feedback, leaving environmental reactions to higher-level planning. ViBe adapts them by learning task-relevant visual feedback through a multi-query extractor and low-rank adapters, trained directly with policy optimization. Across four tasks, the resulting perceptive controllers transfer zero-shot from simulation to hardware while retaining pretrained vision and motor modules.
Problem
Motion trackers are scalable motor priors but are blind to environmental constraints, while existing perceptive controllers commonly use task-specific encoders and teacher-student distillation.
Method
ViBe connects a frozen pretrained vision encoder to a frozen whole-body tracker through a task-conditioned multi-query extractor and low-rank adapters optimized with task rewards and reference motion.
Results
Zero-shot sim-to-real transfer spans four tasks—perceptive locomotion, object reorientation, loco-manipulation, and dodgeball—while retaining pretrained modules.
Takeaways & Limitations
Perceptive whole-body control can be obtained by learning only the intermediate visual-to-motor connection while retaining pretrained vision and motion priors.
Takeaways & Limitations
Dynamic distractors can trigger false responses, and the planner and controller are not trained in a closed loop.
Abstract
from arXiv · showhide
Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.
1 Introduction
ViBe adapts blind humanoid motion trackers into perceptive controllers by learning a task-relevant visual pathway between pretrained vision and motor priors. The framework uses low-rank adapters and direct policy optimization, supporting zero-shot sim-to-real transfer across four tasks.
- Motivation: Motion trackers provide scalable natural-motion priors but lack environmental observations needed to react to constraints and external dynamics.Their blindness leaves environment-dependent corrections to higher-level planning.
- Motivation: Hierarchical perception-and-planning systems can miss local corrections such as curb foot placement, grasp adjustment, and ball avoidance.They expect the plan to be sufficiently accurate while the lower-level tracker executes it.
- Related approaches: Existing perceptive systems commonly train task-specific encoders and trackers with teacher-student distillation, motivating a more general adaptation approach.Attention-based encoders suggest that robot state can select control-relevant feedback from dense observations.
- Contributions: ViBe bridges a frozen vision encoder and whole-body tracker through a task-conditioned extractor and low-rank adapters trained by reinforcement learning.The framework preserves both pretrained components while learning the intermediate visual-to-motor connection.
- Contributions: Zero-shot sim-to-real transfer spans perceptive locomotion, object reorientation, loco-manipulation, and dodgeball.The four-task evaluation includes controlled ablations.
2 Related Works
ViBe builds on pretrained visual representations and attention-based selection while avoiding task-specific privileged-observation prediction or auxiliary representation losses. Its novelty is learning only a compact extractor–adapter pathway between visual and whole-body priors.
- Humanoid whole-body control: Prior whole-body trackers scale motion data and model capacity to form broad motion priors, while ViBe adds perceptive adaptation to the retained tracker.Related perceptive-control systems span locomotion, terrain traversal, manipulation, and whole-body interaction.
- Visual representations and attention: ViBe applies state-guided attention to visual tokens shared across perceptive whole-body control tasks.The approach extends attention-based selection of control-relevant regions to pretrained visual representations.
- Humanoid whole-body control: ViBe learns only the compact extractor–adapter pathway while retaining both pretrained visual and motor priors intact.This post-training design connects the two components without retraining them.
- Visual representations and attention: The extractor is trained only through the downstream task objective rather than predicting privileged observations or minimizing auxiliary losses.This distinguishes ViBe from common teacher-student perceptive-control recipes.
3 Approach
ViBe formulates adaptation as policy optimization over a deployable partially observed control system, preserving a pretrained tracker while adding visual tokens, an extractor, and LoRA adapters. Reference-phase annealing provides a curriculum for mastering full reference motions.
- Formulation: ViBe preserves the pretrained tracker and learns only a compact extractor–adapter pathway for task-specific perceptive adaptation.The post-training formulation keeps the pretrained priors intact.
- Formulation: The deployable actor observes proprioceptive history, future reference motion, egocentric images, and optional task commands, then outputs PD targets.Rewards combine motion tracking with task-specific objectives such as object tracking, object goals, or avoidance.
- Formulation: PPO updates only the extractor and adapters while a frozen visual encoder supplies tokens and the base tracker remains initially unchanged through zero-initialized adapters.An asymmetric actor–critic gives privileged observations to the critic but not the deployable actor.
- Task-based Visual Extractor: The extractor uses global image context, proprioception, and optional task commands as queries over patch tokens from a frozen Theia-Tiny vision encoder.Each query selects an attention-weighted patch summary; concatenated summaries are projected to a fixed 128-dimensional percept.
- Task-based Visual Extractor: Rank-16 LoRA adapters condition the frozen tracker decoder on the extracted percept, restricting learning to task-specific corrections of the motion prior.The query signals guide patch selection while retaining direct proprioceptive and reference-command paths to the controller.
- Reference-Phase Annealing: Reference-phase annealing contracts reset coverage from the full reference toward frame zero, forcing episodes to begin at the start after K updates.The curriculum addresses specialization to short motion segments and supports mastering the entire reference.
4 Results
ViBe transfers motion-tracking priors to perceptive whole-body control across four tasks, while preserving reference motion and adapting contacts or trajectories to visual task demands. Hardware evaluations show strong visual adaptation, robustness to appearance shifts, planner-compatible Repose Cube solving, and design benefits from pretrained encoders, multi-query extraction, and phase annealing.
- Zero-shot transfer: Across four tasks, ViBe preserves reference motion while adapting foot placement, momentum, object interactions, and dodgeball behavior through separate task-specific policies.The tasks include perceptive walking and parkour, cube reorientation, omni-object loco-manipulation, and dodgeball.
- Privileged observation vs. visual learning: ViBe reaches 89.6%, 95.5%, 90.3%, and 94.7% of privileged-policy performance on perceptive walk, parkour, omni-object loco-manipulation, and dodgeball, respectively.The comparison is trained directly from visual feedback and uses no teacher action targets.
- Blind tracking vs. visual adaptation: With identical bases and reference clips, ViBe-SONIC traverses curbs and completes parkour jumps where the blind tracker fails.Visual adaptation changes foot placement and motion velocity, enabling terrain traversal and jump momentum.
- Visual robustness: Adapted policies tolerate direct sunlight, low light, and RGB distractors, but the trials do not establish invariance to unmodeled dynamic physical distractors.The appearance-shift evaluation covers perceptive walking and Repose Cube hardware rollouts.
- Solving tasks with a planner: A fixed routine of right flip, right flip, and front flip solves goal-oriented Repose Cube using ViBe policies for localization, contact placement, and recovery.The planner selects reference clips without a dynamics model or forward predictive rollouts.
- Ablations: DINOv3-S+ performs best among tested encoders, pretrained backbones vary by only 2.8% standard deviation, and each outperforms the scratch-trained CNN.The ablation supports extractor flexibility across pretrained visual backbones.
- Ablations: All three query rows achieve 93.1% success at 50k updates, while removing CLS or proprioception reduces success by 15%; phase annealing raises success from 88.5% to 93.3%.Removing the task-command query instead marginally increases success to 93.4% on Repose Cube.
5 Conclusion and Limitations
ViBe adapts pre-trained humanoid whole-body trackers into perceptive controllers by learning task-relevant visual feedback while retaining the pre-trained modules. Across four tasks, the framework transfers from simulation to hardware, but dynamic distractors and open-loop planner-controller training remain limitations.
- ViBe achieves zero-shot sim-to-real transfer across four tasks while retaining its pre-trained vision encoder and whole-body tracker.The tasks span non-flat terrain traversal, omni-object loco-manipulation, dodgeball, and Repose Cube.
- Learning only the intermediate connection from scratch enables parameter-efficient perceptive whole-body control.ViBe uses attention-based perceptive feedback and low-rank adapters while retaining the pre-trained modules.
- Dynamic distractors can trigger false responses, including unnecessarily dodging when the policy mistakes a thrower’s head for the ball.The authors identify additional domain randomization as future work for learning the necessary invariances.
- Because the planner and controller are not trained in a closed loop, selected references may fall outside the adapter’s training distribution.The frozen tracker preserves reasonable tracking, but task-optimal performance would benefit from a closed-loop reference generator.