WholeBodyVLA: Towards Unified Latent VLA for Whole-Body Loco-Manipulation Control

1 INTRODUCTION

• Autonomous humanoid loco–manipulation remains limited, particularly in manipulation-aware locomotion that approaches, orients, and stabilizes the robot for intended manipulation.Serializing locomotion and manipulation can accumulate errors, while end-to-end imitation requires difficult-to-obtain whole-body data.Source: §1 · p. 2
• Humanoid loco–manipulation data are scarce and expensive to collect through MoCap or teleoperation, motivating learning from low-cost, action-free egocentric videos.Such videos expose locomotion direction, end-effector trajectories, object affordances, and physical interaction cues.Source: §1 · p. 2
• Unified latent learning turns videos into discrete latent actions for VLA supervision, while a separate LMO RL policy uses discrete commands for accurate, stable locomotion.The contributions combine video-based learning with an LMO policy tailored to advancing, turning, and squatting.Source: §1 · pp. 2–3
• WholeBodyVLA surpasses prior baselines by 21.3% and 24.0% in comprehensive AgiBot X2 experiments.The framework supports real-world, large-space end-to-end loco–manipulation.Source: §1 · p. 3

2.1 HUMANOID WHOLE-BODY CONTROL

• Velocity-tracking whole-body controllers can leave start–stop semantics implicit, fragment gaits, and provide little episode-level supervision for braking or heading fidelity.Prior systems also model upper-body influence as task-agnostic noise or motion clips, which sparsely reflects real task inertial couplings.Source: §2.1 · pp. 3–4
• Modular planners can leave unstable or task-infeasible configurations after locomotion, while existing humanoid VLAs emphasize locomotion or manipulation rather than both.The Boston Dynamics demonstration is described as limited-workspace and reliant on expensive MoCap collections.Source: §2.1 · p. 4

2.2 VISION-LANGUAGE-ACTION MODELS

• Most VLA systems focus on upper-body manipulation, while latent action learning uses discrete tokens from frame-to-frame visual changes to supervise policies with action-free videos.The paper applies latent action learning to locomotion and manipulation together, addressing the greater difficulty of obtaining humanoid loco–manipulation data.Source: §2.2 · p. 4

3 WHOLEBODYVLA

• WholeBodyVLA combines a VLA with unified latent learning and an LMO RL policy to support long-range, large-area humanoid tasks.At runtime, the VLA encodes egocentric images and language into latent tokens, decoded into dual-arm actions and locomotion commands.Source: §3 · p. 4, Figure 2 · p. 5

3.1 UNIFIED LATENT ACTION MODEL

• Separate manipulation and locomotion LAMs address their different camera-motion patterns, then jointly supervise the VLA to predict both types of latent actions.A single LAM on mixed data can confuse arm motion with camera motion; the LAMs encode frame-to-frame changes as discrete codebook tokens.Source: §3.1 · pp. 4–5
• A single-operator egocentric video pipeline collects goal-directed locomotion, including advancing, turning, and squatting, without MoCap or teleoperation.The collected movements are directed toward potential manipulation goals to align locomotion data with loco–manipulation learning.Source: §3.1 · p. 6

3.2 LOCO–MANIPULATION–ORIENTED RL POLICY

• The LMO policy replaces velocity tracking with discrete forward, lateral, and turning commands plus a stance-height target, with smoothed transitions to reduce oscillations.The command interface gives explicit start–stop semantics and is formulated for goal-conditioned lower-body control while maintaining balance.Source: §3.2 · p. 6
• A two-stage curriculum first acquires a basic gait, then refines directional precision and stability under structured arm-motion disturbances and stationary-command penalties.The policy is designed to produce stable, repeatable gaits and reliable whole-body coordination.Source: §3.2 · pp. 6–7

4 EXPERIMENTS

• The experiments are organized around benchmark performance, action-free video learning, LMO contributions, and generalization to extended scenarios.The setup includes Agibot X2 hardware, task suites, baselines, and training protocol.Source: §4 · p. 7, §4.1 · p. 7

4.1 SETUP

• The Agibot X2 evaluation uses three tasks, compares modular and VLA baselines, and trains methods on shared teleoperation trajectories.The tasks are bag packing, box loading, and cart pushing; WholeBodyVLA uses separate manipulation and locomotion LAMs before VLA training and robot fine-tuning.Source: §4.1 · p. 7, §4 · p. 7

4.2 MAIN RESULTS

• WholeBodyVLA achieves consistently higher success rates than modular and end-to-end baselines across bag packing, box loading, and cart pushing.The task suites test lateral stepping and bimanual manipulation, turning and squatting while placing objects, and sustained forward pushing.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 HOW DOES ACTION-FREE VIDEOS CONTRIBUTE TO LOCO–MANIPULATION?

• Full latent pretraining improves success rate by 38.7% over WholeBodyVLA w/o LAM, and full pretraining outperforms manipulation-only latent learning by 14.7%.With more than 50% human-video pretraining, models match variants using less than 25% human videos with only 25 teleoperation trajectories, versus 200 trajectories for similar performance; stronger pretraining also improves manipulation generalization.Source: §4.3 · p. 8

4.4 HOW DOES LMO CONTRIBUTE TO LOCO–MANIPULATION?

• The velocity-based RL variant has a 24% lower success rate, with 91.7% of the gap arising from locomotion-heavy second subgoals.It also fails substantially more often on extended locomotion-dominated scenarios; controlled MuJoCo ablations show that removing directional accuracy reward, Stage II, or Stage I degrades turning, trajectory accuracy and squatting sway, or gait stability, respectively.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 DOES WHOLEBODYVLA GENERALIZE TO LONG-HORIZON AND EXTENDED LOCO-MANIPULATION SCENARIOS?

• WholeBodyVLA remains superior across five extended tasks, including uneven terrain, long-horizon sequences, visual navigation, wiping, and vacuum cleaning.Figure 3 also presents generalization tests with varied robot start poses and scene appearance, alongside data-scaling curves.Source: §4.5 · pp. 9–10, Figure 3 · p. 9

5 CONCLUSION AND FUTURE WORK

• WholeBodyVLA combines unified latent learning and an LMO RL policy to support large-space humanoid loco-manipulation.The authors say the components address teleoperation-data scarcity and decision-execution misalignment from redundant velocity-tracking objectives.Source: §5 · p. 10
• WholeBodyVLA still faces challenges in handling long-horizon and dexterous tasks.Future work targets lightweight mapping and memory for extended planning, and active perception for robustness in cluttered or dynamic environments.Source: §5 · p. 10

A.1 DETAILS OF LAM AND VLA ARCHITECTURE

• The LAM encodes consecutive frames into a continuous vector, quantizes it to a learned codebook, and trains a decoder to reconstruct the future frame.Reconstruction uses mean-squared error alongside a VQ-VAE objective.Source: §A.1 · p. 13
• The VLA predicts manipulation and locomotion latent actions from visual observations and task language, modeling their interaction in a unified latent space.The two LAMs represent manipulation and locomotion, respectively.Source: §A.1 · p. 13
• A lightweight decoder maps predicted latent actions into upper-body joint angles and a locomotion command for the low-level RL controller.The decoder uses robot state to produce robot-specific control commands.Source: §A.1 · p. 13
• A single operator records egocentric video while performing diverse locomotion actions, including advancing, turning, and squatting near potentially manipulable objects.The collection pipeline uses head-mounted cameras; its videos and an in-place manipulation dataset pretrain the LAM.Source: §A.1 · p. 13, Figure 4 · p. 14

A.2 DETAILS OF THE LOCO–MANIPULATION–ORIENTED RL POLICY

• The LMO policy uses discrete start–stop flags for forward, lateral, and yaw motion plus a target stance height, with proprioceptive inputs for closed-loop balance.Smoothed intent flags produce velocity references with predictable transitions and implicitly bounded accelerations.Source: §A.2 · p. 14
• Stage I establishes a stable basic gait, while Stage II targets precision and robustness through fixed cruising speed, yaw-drift penalties, and structured arm-motion disturbances.Stationary episodes also receive a penalty that discourages spurious leg motion.Source: §A.2 · pp. 14–15

• Locomotion LAM training uses about 300 hours of collected egocentric video, while manipulation LAM training uses the AgiBot World bimanual dataset.For a shared LAM, mixed data are balanced-sampled in every batch.Source: B TRAINING AND DEPLOYMENT DETAILS · p. 15
• LAM training runs for 30,000 steps and VLA training from Prismatic-7B for 20,000 steps, followed by 10,000-step LoRA fine-tuning on task data.For the Section 4.2 experiments, one model is fine-tuned on all three tasks rather than separately by task.Source: B TRAINING AND DEPLOYMENT DETAILS · p. 15
• At deployment, the VLA runs at approximately 10 Hz on an RTX 4090 workstation, while the LMO policy runs at 50 Hz on a NanoPi onboard computer.ZeroMQ over Ethernet streams commands between the VLA and robot.Source: B TRAINING AND DEPLOYMENT DETAILS · p. 15

• Bag packing tests dual-arm grasping, lateral stepping, and squat placement; box loading tests squatting, turning, and placement; cart pushing tests sustained movement under a 50 kg load.The cart task requires pushing several meters without lateral drift.Source: §B.1 · pp. 15–16
• The Agibot X2 has 7-DoF arms, a 1-DoF waist, 6-DoF legs, and a head-mounted Intel RealSense D435i RGB-D camera.A Meta Quest Pro headset provides upper-body teleoperation, while a joystick issues locomotion commands.Source: §B.1 · pp. 16–17
• Each of the three tasks is executed 50 times to collect diverse trajectories for training.Source: §B.1 · p. 17

B.2 BASELINES AND ABLATIONS

• The modular baseline uses a human teleoperator for locomotion and hands control to WholeBodyVLA for manipulation, providing a near-oracle upper bound for modular pipelines.GR00T with LMO predicts locomotion commands executed by the same LMO controller, isolating its high-level reasoning from low-level locomotion stability.Source: §B.2 · p. 17
• OpenVLA-OFT shares WholeBodyVLA’s Prismatic-7B initialization and predicts upper-body actions and locomotion commands for execution by the same LMO controller.The comparison is intended to reduce architectural and model-size confounds.Source: §B.2 · p. 17
• Five ablations remove the RL controller, replace LMO with velocity-based RL, remove LAM, use manipulation-only LAM, or use a shared LAM without modality separation.Source: §B.2 · p. 18

B.3 PROTOCOLS FOR REAL-ROBOT EXPERIMENT

• Two blinded independent judges label subgoal success, trials are evaluated sequentially over 25 runs per task, and the reported score averages across subgoals.A first-subgoal failure makes the second subgoal count as a failure.Source: §B.3 · p. 18

B.4 DETAILS OF SIMULATION SETUP IN SECTION 4.4

• MuJoCo locomotion tests measure settling-phase position and yaw errors after five seconds of commanded walking, lateral stepping, or turning.Each trial then has a 10-second zero-command settling phase.Source: §B.4 · p. 18
• MuJoCo stability tests replay perturbed upper-body trajectories in standing and squatting postures, measuring Center-of-Mass Sway as RMS deviation of the horizontal CoM projection.Lower CoMS indicates improved balance; arms follow replayed trajectories during these tests.Source: §B.4 · pp. 18–19

C.1 GENERALIZATION EXPERIMENT DETAILS

• The 12 experiments cover start-pose, scene, and additional locomotion or long-horizon generalization, with success rates summarized in Fig. 3.Fig. 3 includes averages for start-pose tasks (1)–(4), scene tasks (5)–(7), and a radar chart for tasks (8)–(12).Source: §C.1 · pp. 19–20
• Start-pose tests vary distance, lateral offset, heading, or table height while the task scene is otherwise held fixed.Evaluation includes distances through 1.5 m, lateral offsets through 75 cm, headings up to ±60°, and unseen table heights of 55 cm, 40 cm, and 20 cm.Source: §C.1 · p. 19
• Scene-generalization tests change the snack bag, table appearance, or bag position while retaining the underlying manipulation task.For unseen objects and tables, object positions match training; the unseen bag-position test evaluates two positions outside the 30 cm × 30 cm collection region.Source: §C.1 · p. 19
• Additional tests cover terrain traversal, long-horizon drawer manipulation, visual navigation, vacuum cleaning, and stain wiping.The terrain task spans five surfaces; the navigation and cleaning tests use fixed start configurations, while stain positions vary within a 30 cm × 30 cm region.Source: §C.1 · pp. 19–20
• WholeBodyVLA outperforms GR00T and OpenVLA-OFT across Bag Packing, Box Loading, and Cart Pushing under object and load variations.The scene layout remains fixed while manipulated objects or loads change.Source: Figure 6 · p. 20

C.2 ADDITIONAL VISUAL GENERALIZATION UNDER OBJECT AND LOAD VARIATIONS

• WholeBodyVLA achieves the highest success rates across object- and load-perturbed Bag Packing, Box Loading, and Cart Pushing.Perturbations include a different bag, plastic containers in place of box contents, and a 60 kg barbell load on the cart.Source: §C.2 · p. 20
• Removing proprioceptive state from the action decoder increases variance and slightly reduces performance, especially under visual variations, but completion rates remain comparable.The comparison covers the original setting and visual-variation settings in Tables 6 and 7.Source: Table 6 · p. 20, §C.2 · p. 21, Table 7 · p. 21

C.3 FAILURE MODE ANALYSIS

• The failure analysis uses 50 failed rollouts per primitive in start-pose generalization and groups them into locomotion and pick/place errors with finer causes.The reported broader limitation is small, systematic stance and orientation errors during approach rather than catastrophic behavior.Source: §C.3 · pp. 21–22
• For horizontal primitives, most failures involve locomotion and pass through object-or-basket-unreachable outcomes; squat failures are more evenly split between locomotion and manipulation.Squat locomotion errors chiefly involve final height or contact during descent, while pick/place errors involve arm trajectories or grasp alignment.Source: §C.3 · p. 22

C.4 TASK EXECUTION TIME

• Execution time is wall-clock duration from instruction to successful subgoal, averaged over five successful trials and aggregated by task.Table 8 summarizes the task-level averages.Source: §C.4 · p. 22, Table 8 · p. 22

C.5 SHARED LATENT ACTION SPACE ACROSS EMBODIMENTS

• The three-stage pipeline trains visual inverse-dynamics LAMs, uses their codes to supervise the VLA, then associates latents with robot targets and locomotion commands.Because the codes depend on frame-to-frame visual motion rather than embodiment-specific joint values, retrieved human and robot clips for the same latent action show aligned semantics.Source: §C.5 · pp. 22–23

C.6 SHARED LATENT ACTION SPACE ACROSS EMBODIMENTS

• Relative Reconstruction Gain measures LAM reconstruction error reduction over a copy-the-previous-frame baseline, with higher values indicating more predictive latent codes.The authors present it as a proxy for LAM quality that does not rely solely on downstream policy evaluation.Source: §C.6 · p. 23
• Separately trained LAMs outperform the shared alternative on downstream reconstruction in both manipulation and locomotion splits.The authors suggest joint training faces conflicts between locomotion-specific and manipulation-specific objectives.Source: §C.6 · p. 23, Table 9 · p. 23