VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

1 Introduction

• Existing teleoperation, motion-capture, and egocentric-video data each lack a part of the paired vision-language-kinematics tuple needed for scalable policy learning.Teleoperation is costly to scale; motion-capture lacks robot egocentric observations, and egocentric video lacks robot-compatible kinematics or actions.Source: §1 · p. 2
• The proposed data-generation system pairs visual observations rendered in 3DGS worlds with robot kinematic behaviors suitable for whole-body tracking.It uses prescribed language commands, privileged scene information, oracle-conditioned motion synthesis, and hindsight rendering to produce the synchronized data without human demonstrations.Source: §1 · p. 2
• The system generates 48,000 trajectories automatically in metric-scale reconstructed indoor environments, trains a VLK policy, and uses a contact-aware tracker to execute its predictions on the physical humanoid.The policy predicts short-horizon whole-body kinematics and binary wrist-object contact labels from an egocentric image, instruction, and current robot state.Source: §1 · pp. 2–3
• Evaluation on navigation and object transport in simulation and on the physical G1 demonstrates effective supervision for sim-to-real perception-based loco-manipulation.The task categories exercise visual grounding, locomotion, picking up, carrying, and placing objects.Source: §1 · p. 3

Vision-language and perception-based policies for humanoids

• WholeBodyVLA [11] and Ψ0 use egocentric video and real humanoid teleoperation, whereas this work generates paired supervision in reconstructed scenes for explicit future G1 kinematic prediction.The paper presents its approach as complementary to policies relying on existing robot demonstrations or human videos as primary supervision.Source: §2 · p. 3

Synthetic data and reconstructed scenes for robot learning

• Prior synthetic-data work uses digital twins and reconstructed scenes for robot learning [18] [19] [20] [21] [22]; this work combines 3DGS scenes with humanoid motion synthesis for paired observations and interaction behaviors.Source: §2 · p. 3

Humanoid motion synthesis and whole-body tracking

• Building on human retargeting and whole-body tracking methods [23] [24] [25] [26] [27] [28] [29], the paper adapts human-object interaction synthesis [33] [34] to Unitree G1 trajectories paired with renderings and language.Source: §2 · p. 3

3 Method

• The system trains a high-level policy to predict short-horizon G1 trajectories from the egocentric view, instruction, and current kinematic state, then uses a tracker to produce executable actions.This decomposition makes whole-body kinematics the policy target and shifts the data requirement to paired views, instructions, and G1 kinematics.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Synthetic Data Generation in Reconstructed Scenes

• Metric-scale indoor scenes are reconstructed and annotated with object boxes and walkable regions; task waypoints condition G1 motion synthesis, and replay renders paired egocentric observations.Navigation motions use G1 data, while interaction synthesis produces coupled robot-object trajectories conditioned on scene waypoints and interaction constraints.Source: §3.1 · pp. 4–5

3.2 Vision-Language-Kinematics Policy

• The VLK policy predicts a 30-frame, one-second future trajectory from the current RGB view, instruction, and G1 state, and is fine-tuned with an x0-prediction flow-matching objective.The training objective reconstructs the clean future trajectory from a noisy one and adds auxiliary losses.Source: §3.2 · p. 5

3.3 Whole-Body Motion Tracking

• A contact-aware tracker converts predicted reference states and current robot proprioception into joint-level PD targets, using wrist-contact labels to stabilize object transport.The tracker does not receive egocentric observations or language; the VLK policy handles perception-conditioned replanning.Source: §3.3 · p. 6

3.4 Deployment

• Real-world deployment uses Triton inference on an RTX 5090 with 31 ms end-to-end latency, overlapping adjacent action chunks by 10 timeframes and selecting sharp frames to mitigate motion blur.Source: §3.4 · p. 6

4.1 Experimental Setup

• The evaluation uses four lab and four apartment layouts, reconstructed from scans, with physical furniture layouts varied across trials.Real-world evaluation covers six task modes with 20 trials per mode, for 120 trials total; simulation evaluates 1,000 rollouts per mode on held-out data.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Synthetic Data Generation

• The pipeline samples task-relevant objects, initial poses, waypoints, and instructions, synthesizes G1-compatible trajectories, filters scene penetrations, and renders paired observations.Each sequence pairs egocentric observations and language instructions with G1 whole-body kinematics.Source: §4.2 · p. 6
• Object-directed navigation and floor-level box manipulation achieve high success in simulation and real-world evaluation, while surface-level interactions are less reliable.Limited coverage of box manipulation across support-surface heights contributes to less reliable grasps and placements.Source: §4.2 · p. 7

4.4 Effect of Synthesized Data Volume

• Pick (Surface) success rises from 0% at 10% data to 46% with the full dataset, while object-directed navigation performs well even with limited data.Increasing synthesized training data consistently improves performance across evaluated modes.Source: §4.4 · pp. 7–8, Figure 4 · p. 7

4.5 Effect of Visual Domain Randomization

• Removing visual domain randomization reduces walking success from 90% to 41% under visual perturbations in the lab scene.The evaluated configurations vary lighting and camera randomization, separately and together.Source: §4.5 · p. 8, Table 2 · p. 8

5 Conclusion

• The system generates Unitree G1-compatible navigation and box-manipulation trajectories, renders egocentric observations, and trains a VLK policy for tracker-based execution.Simulation and physical G1 experiments demonstrate object-directed navigation and box manipulation.Source: §5 · p. 8
• Interaction synthesis is limited by OMOMO’s coverage of large objects, making generated behavior better suited to bimanual box-like transport than grasping small objects.The contact-aware tracker stabilizes large-object manipulation but does not address precise grasping; small-object manipulation would require richer interaction data and a controller designed for precise grasping.Source: §5 · p. 9

A.1 Scene Annotation

• Scene annotation uses semantic oriented 3D bounding boxes and walkable-region polygons to define task objects, feasible initial poses, and navigation waypoints.The annotations are added with an interactive tool using a point cloud extracted from the reconstructed scene.Source: §A.1 · p. 13

A.2 Waypoint, Language Generation in 3D Scenes

• Task-specific waypoint rules sample collision-free navigation paths and set box-interaction waypoints and contact timing according to object and support-surface height.For walk-toward-object tasks, the target must be visible from the initial viewpoint; language instructions are generated from task metadata with templates.Source: §A.2 · pp. 13–14

A.3 Interaction Synthesis Module

• The interaction module adapts CHOIS’s conditional diffusion formulation to Unitree G1 trajectories and adds a G1-specific forward-kinematics loss.It retargets OMOMO motions to G1; conditioning includes instructions, initial states, scene waypoints, object geometry, wrist poses, and contact labels.Source: §A.3 · pp. 14–15
• The G1 motion representation encodes joint angles with sine and cosine to avoid discontinuities from angle wrap-around, and combines robot and object trajectories.The robot trajectory includes global joint positions and rotations alongside joint angles.Source: §A.3 · p. 14

A.4 Post-Processing for Generated Motion

• Post-processing fixes robot feet during predicted contact phases using inverse kinematics and optimizes wrist poses toward input poses in the object frame.These steps address foot sliding and hand-object contact artifacts, respectively.Source: §A.4 · p. 15

A.5 Domain Randomization in Rendering

• Egocentric rendering randomizes camera parameters, lighting, and image appearance to model calibration differences and visual variation.Augmentations include brightness, contrast, saturation, hue, Gaussian noise, and blur.Source: §A.5 · p. 15

A.6 Dataset Examples

• Each dataset example pairs an egocentric RGB observation and language instruction with its corresponding G1 kinematic trajectory.The sequences are generated in annotated 3D scenes.Source: §A.6 · p. 15

• The supplementary details include evaluation metrics and deployment considerations such as motion-blur handling and system profiling.Source: B Experiment Details · pp. 15–16
• Table 3 lists the domain-randomization parameters used for egocentric rendering.Source: Table 3 · p. 16
• The synthetic dataset examples pair an egocentric RGB observation and language instruction with a G1 kinematic trajectory.Source: Figure 6 · p. 16

• The full VLK model is fine-tuned from pretrained π0.5 after adapting its action space to the G1 kinematic representation.Source: §B.1 · p. 16
• Flow matching trains the policy to predict a clean future trajectory from a noisy trajectory, conditioned on the observation, instruction, and current G1 state.Source: §B.1 · p. 16
• Auxiliary objectives address foot-floor contact prediction, accumulated root-trajectory consistency, forward-kinematics accuracy, and foot-skating regularization.Source: §B.1 · p. 17
• Foot-contact labels use focal loss because contact events are sparse, while the accumulated root loss penalizes drift after integrating predicted relative displacements.Source: §B.1 · p. 17
• The foot-skating regularizer penalizes horizontal foot velocity when the predicted contact label indicates stance.Source: §B.1 · p. 17

B.2 Metric for Evaluation in Simulation

• Simulation evaluation holds out 10% of synthesized trajectories, initializes the humanoid from each held-out trajectory, and tests completion of its language-conditioned task.Source: §B.2 · p. 17
• Success is task-specific: navigation requires stopping within 0.5 m without collision, turning requires a clear instructed turn, pickup requires a stable 20-frame grasp, and put-down requires placing the box at the target.Source: §B.2 · p. 17

B.3 Motion Blur Handling During Deployment

• Training adds mild synthetic motion blur to some RGB observations, while deployment selects the sharpest frame from a 0.3 s image buffer using variance of the Laplacian.Source: §B.3 · p. 18

• Three concurrent processes—a state estimator, whole-body tracker, and VLK inference client—communicate through shared memory and inter-process queues, while the client sends observations and state to a remote GPU server.Source: §B.4 · p. 18, Figure 7 · p. 19
• The tracker runs at 50 Hz and averages 4.3 ms per tick, with no missed deadlines observed over a full task rollout.Source: §B.4 · p. 18
• VLK predicts one-second, 30-frame chunks with 10-frame overlap; end-to-end replanning averages ∼63 ms, and chunk merging adds ∼3.4 ms on arrival.Source: §B.4 · p. 18, Table 4 · p. 19