PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation

Shaofeng Yin, Yanjie Ze, Hong-Xing Yu, C. Karen Liu, Jiajun Wu

arXiv:2509.20322v2cs.ROcs.CVcs.LG

TL;DR

Humanoid loco-manipulation needs egocentric perception and whole-body control, while prior approaches face limits in deployment or task generalization. VisualMimic combines a general keypoint tracker with a task-specific visual policy and reports zero-shot transfer across diverse real-world tasks, including outdoor settings.

  • Problem

    Existing approaches either depend on external motion-capture systems or have limited generalization across diverse visual humanoid interactions.

    Source: §I · p. 2

  • Method

    VisualMimic pairs a task-agnostic keypoint tracker learned from human motion with a task-specific visuomotor policy trained through hierarchical sim-to-real learning.

    Source: §III · p. 3, C. Clipping Action Space to Human… · p. 6

  • Results

    The visuomotor policies transfer successfully to real humanoid robots across four loco-manipulation tasks and show robustness in outdoor conditions.

    Source: §I · p. 2, bustly to the real world? · p. 7

  • Takeaways & Limitations

    The framework supports whole-body object interaction from visual and proprioceptive inputs in both simulation and real-world deployments.

    Source: §V · p. 9

    Deformable-object and human-collaboration tasks remain unexplored, while scaling to long-horizon tasks and diverse real-world environments may require advances in domain randomization and adaptive control.

    Source: §V · p. 9

Abstract

from arXiv · show

Humanoid loco-manipulation in unstructured environments demands tight integration of egocentric perception and whole-body control. However, existing approaches either depend on external motion capture systems or fail to generalize across diverse tasks. We introduce VisualMimic, a visual sim-to-real framework that unifies egocentric vision with hierarchical whole-body control for humanoid robots. VisualMimic combines a task-agnostic low-level keypoint tracker -- trained from human motion data via a teacher-student scheme -- with a task-specific high-level policy that generates keypoint commands from visual and proprioceptive input. To ensure stable training, we inject noise into the low-level policy and clip high-level actions using human motion statistics. VisualMimic enables zero-shot transfer of visuomotor policies trained in simulation to real humanoid robots, accomplishing a wide range of loco-manipulation tasks such as box lifting, pushing, football dribbling, and kicking. Beyond controlled laboratory settings, our policies also generalize robustly to outdoor environments. Videos are available at: https://visualmimic.github.io .

I. INTRODUCTION

Humanoid loco-manipulation calls for egocentric perception and whole-body dexterity, but existing approaches have limited task diversity or rely on constrained data and settings. VisualMimic combines a reusable keypoint tracker with task-specific visual control and reports zero-shot transfer to diverse real-world tasks.

I. INTRODUCTION

  • Prior vision-based RL had largely addressed simple interactions, while imitation learning faced scarce demonstrations and external motion-capture methods restricted deployment to controlled laboratories [3],, [5]–, [8], [9].Locomotion-focused approaches addressed terrain traversal rather than object interaction [1], [2].Source: §I · p. 2
  • Its hierarchy pairs a task-agnostic tracker for balanced control with a task-specific policy that generates compact keypoint commands from egocentric vision.The keypoint interface covers the root, hands, feet, and head.Source: §I · p. 2
  • A teacher motion tracker distilled into a keypoint tracker preserves human-like behavior that simplified keypoint commands alone may not capture.The shared tracker is trained from curated human motion data and remains task-agnostic.Source: §I · p. 2
  • A privileged state-based teacher is distilled into a visuomotor policy using egocentric vision and robot proprioception, avoiding external object-state estimation at deployment.The paper also uses heavy depth-image masking in simulation to address the visual sim-to-real gap.Source: §I · p. 2
  • To stabilize training, the method adds noise during low-level training and clips high-level actions to the feasible human motion space.The stated concern is exploration beyond the human motion space represented in training data.Source: §I · p. 2
  • VisualMimic zero-shot transfers to real humanoid robots across box lifting, pushing, football dribbling, and kicking, including outdoor settings.The reported demonstrations include lifting a 0.5-kilogram box to 1 meter and pushing a 3.8-kilogram box.Source: §I · p. 2

II. RELATED WORK

Prior humanoid loco-manipulation work has traded task versatility against scalable data, broad sim-to-real deployment, or reliable visual interaction. VisualMimic uses a hierarchical design, with a general tracker distilled from a motion teacher and a task-specific policy trained for visual control.

A. Learning Humanoid Loco-Manipulation

  • Prior approaches include teleoperation imitation limited by scarce high-quality data and sim-to-real RL with strong motor skills but less task diversity; some hierarchical methods remain simulation-only or rely on privileged states [5]–, [12],, [3], [8], [9], –[17].Other visuomotor work focused on upper-body manipulation, perceptive locomotion, or static interactions, while PDC demonstrated whole-body loco-manipulation only in simulation [8]–[10], [18]–[23].Source: A. Learning Humanoid Loco-Manipulation · pp. 2–3

III. METHOD

  • VisualMimic combines a task-agnostic keypoint tracker learned from human motion with a task-specific visuomotor policy, and only the high-level policy needs training per task.Both policies are trained in simulation and transfer zero-shot to the real robot.Source: §III · p. 3

A. General Keypoint Tracker

  • The keypoint tracker uses teacher–student distillation because simplified keypoint commands alone yield weaker motion capture and less human-like behavior than motion tracking.The privileged teacher accesses future reference motions; the student is trained with DAgger using proprioception and keypoint commands.Source: A. General Keypoint Tracker · p. 3
  • The teacher tracker uses PPO and a reward that encourages accurate motion tracking while penalizing jitter and foot slippage.The method tracks body positions and root velocities in the world frame and retargets AMASS and OMOMO motions using GMR.Source: A. General Keypoint Tracker · pp. 3, 5

B. Task-Specific Keypoint Generator

The task-specific keypoint generator is trained in two stages: a state-informed teacher learns task behavior, then a vision-based student is distilled to drive the human-motion-trained tracker. Task rewards guide object interaction, with additional terms regularizing behavior.

B. Task-Specific Keypoint Generator

  • The design uses a compact set of task rewards without requiring paired human-object interaction data or additional human-like-motion constraints.The low-level tracker supplies human-motion priors, leaving the generator to focus on task completion.Source: B. Task-Specific Keypoint Generator · p. 5
  • A teacher generator trained with object states is distilled into a student that uses depth images for object information.The teacher is trained with PPO; the student uses depth images and proprioceptive inputs, with DAgger used for distillation.Source: B. Task-Specific Keypoint Generator · pp. 5–6
  • Task-specific rewards encourage end-effector contact, forward object motion, and sufficient force.Approach rewards target-point contact, forward progress rewards new object motion, and the force reward favors reaching a desired force threshold.Source: B. Task-Specific Keypoint Generator · pp. 5–6
  • Look-at-object and drift terms regularize task behavior by encouraging the robot to face the object and penalizing lateral deviation.The drift term uses the object's lateral position.Source: B. Task-Specific Keypoint Generator · p. 6
  • The student keypoint generator combines depth-image features with proprioception and is distilled using DAgger.A CNN encodes depth input before its output is concatenated with proprioceptive features and passed to an MLP.Source: B. Task-Specific Keypoint Generator · p. 6

C. Clipping Action Space to Human Motion Space

Exploration beyond the feasible Human Motion Space (HMS) makes keypoint-generator training unstable. VisualMimic addresses this with noisy low-level commands and a high-level action clip based on the tracker’s input statistics.

C. Clipping Action Space to Human Motion Space

  • Exploration beyond the Human Motion Space (HMS), the feasible range of human-motion-derived keypoint commands, destabilizes keypoint-generator training.The instability persists despite using a compact command space.Source: C. Clipping Action Space to Human… · p. 6
  • Multiplicative noise during low-level training broadens the tracker’s command robustness and the range of the Human Motion Space.Each command dimension is scaled by a factor sampled uniformly from 0.5 to 1.5, corresponding to a 50% relative noise level.Source: C. Clipping Action Space to Human… · p. 6
  • The high-level policy’s output is clipped to µ ± 1.64σ per input dimension, a range covering approximately 90% of the probability mass.The bounds are estimated from the low-level policy input normalizer; the paper reports that clipping significantly stabilizes generator training.Source: C. Clipping Action Space to Human… · p. 6

D. Real World Deployment

Real-world deployment addresses noisy depth input and provides a binary pause-or-execute command for safer task operation. The supplied figure depicts varied loco-manipulation tasks in simulation.

D. Real World Deployment

  • Random masking during training approximates noisy real-world depth images, while camera-view orientation is randomized by up to ±5°.Real depth images are spatially and temporally filtered; without masking, the robot exhibits unstable deployment behavior.Source: D. Real World Deployment · p. 6
  • A binary command lets the robot pause or execute a task, starts it in pause, and is trained with separate task and standing-pause rewards.The two command states are sampled with 50% probability each.Source: D. Real World Deployment · p. 6
  • The simulation figure shows four task examples: Balance Ball, Push Cube, Reach Box, and Large Kick.The examples are arranged left to right in that order.Source: Figure 5 · p. 7

IV. EXPERIMENTS AND ANALYSIS

The experiments are organized around task performance, human-like behavior, and real-world transfer, alongside questions about dexterity, visual input, and design choices.

IV. EXPERIMENTS AND ANALYSIS

  • The experiments ask whether VisualMimic trains humanoids for diverse tasks in a human-like manner and whether its policies transfer robustly to the real world.Source: §IV · p. 7
  • The evaluation also asks how effectively the framework demonstrates whole-body dexterity and uses vision for object interaction.Source: §IV · p. 7
  • A further question is whether each VisualMimic design choice is necessary for system success.Source: §IV · p. 7

A. Q1: Does VisualMimic enable effective training of hu-

VisualMimic completes its simulated loco-manipulation tasks, including box pushing, kicking, lifting, and football dribbling. Teacher–student training also supports human-like box-kicking motion.

A. Q1: Does VisualMimic enable effective training of hu-

  • The framework completes all simulated tasks, including pushing and kicking boxes, lifting, reaching, football dribbling, balancing, and tabletop cube pushing.The reported examples include pushing a 3.8-kilogram box and dribbling a football at an average of 135 meters per minute.Source: A. Q1: Does VisualMimic enable… · p. 7
  • With teacher–student training, the humanoid kicks a box with human-like motion; without it, the motion is non-human-like.Source: Figure 6 · p. 7

B. Q2: Can policies trained with VisualMimic transfer ro-

Policies trained in simulation transfer to the Unitree G1 across four real-world tasks. Outdoor trials show robustness to changing lighting, uneven ground, and environmental distractions.

B. Q2: Can policies trained with VisualMimic transfer ro-

  • All four policies transfer successfully to the Unitree G1: lifting a box, dribbling a football, kicking a box, and pushing a large box.The robot uses its onboard RealSense D435i camera.Source: bustly to the real world? · p. 7
  • In outdoor experiments, the policies remain robust amid fluctuating lighting, irregular ground surfaces, and environmental distractions.Source: bustly to the real world? · p. 7
  • The real-world behaviors include stable box holding, forward football dribbling with a human-like pose, coordinated box kicking, and steady pushing.Source: bustly to the real world? · p. 7

C. Q3: How effectively does our framework demonstrate whole-body dexterity?

VisualMimic exhibits whole-body object interaction and adapts its pushing strategy to conditions. An upper-body-only command variant struggles to engage the feet in Balance Ball.

C. Q3: How effectively does our framework demonstrate whole-body dexterity?

  • The robot uses its entire body, including its lower limbs, to interact with objects and flexibly adapts body-part use within a task.Source: C. Q3: How effectively does our… · p. 7
  • On low-friction ground it bends to push with both hands, while on high friction it leans forward to push with its shoulder; a lower box prompts one-handed pushing.Source: Figure 7 · p. 8
  • With only upper-body keypoints as commands, the variant performs poorly on Balance Ball and tends to kick the ball away instead of placing a foot on it.Source: C. Q3: How effectively does our… · p. 8

D. Q4: How well does VisualMimic utilize vision for object interaction?

VisualMimic’s vision-enabled policy outperforms a version distilled without visual input. Ablations also show that its training and interface choices matter across simulated tasks.

D. Q4: How well does VisualMimic utilize vision for object interaction?

the success of the system?

  • None of the tested baselines matches VisualMimic; removing noise degrades Kick Box and Kick Ball performance and causes Push Box failure, while removing clipping collapses returns across tasks.The reported ablations also find reduced performance with a DoF interface and substantial drops with local-frame tracking.Source: the success of the system? · p. 8, Table III · p. 9
  • Direct visual reinforcement learning fails entirely on Kick Box and Kick Ball.Source: the success of the system? · p. 9
  • Removing teacher–student training for the low-level tracker produces highly unnatural behavior during keypoint-generator training.The passage identifies the comparison in Fig. 6.Source: the success of the system? · p. 9

V. CONCLUSIONS AND LIMITATIONS

VisualMimic combines a high-level keypoint generator with a low-level general keypoint tracker to support robust simulation-to-real deployment for humanoid loco-manipulation. Its tested scope leaves complex interactions and scaling to longer-horizon, more diverse real-world settings unresolved.

V. CONCLUSIONS AND LIMITATIONS

  • Experiments demonstrate robust deployment of VisualMimic from simulation to the real robot across diverse loco-manipulation tasks.The framework combines a high-level keypoint generator with a low-level general keypoint tracker.Source: §V · p. 9
  • Interactions involving deformable objects or human collaboration remain unexplored.Source: §V · p. 9
  • Scaling to long-horizon tasks and diverse real-world environments may require further advances in domain randomization and adaptive control.Source: §V · p. 9

Related papers

Loading 2509.20322v2…