PaperlayerView exact revision on arXiv

Source-linked AI summary

$\omega$-0: A Latent Predictive World Action Model for Concurrent Humanoid Loco-Manipulation

Zhe Li, Zhenzhe Zhang, Yangyang Wei, Wenjie Zhang, Xichen Yuan, Peiyuan Zhi, Gen Li, Xinying Guo, Fengjie Gao, Jianfei Yang, Shanghang Zhang

arXiv:2608.06375v2cs.RO

TL;DR

Humanoid household tasks require coordinated movement and manipulation, while existing policies often decompose those behaviors. ω-0 couples latent future-observation prediction with whole-body action generation, and experiments report improvements over representative baselines across real-world household tasks.

  • Problem

    Humanoid household tasks require simultaneous movement, posture adjustment, balance, and object manipulation, while existing policies often decompose locomotion and manipulation.

    Source: §1 · p. 2

  • Method

    ω-0 jointly predicts compact future-observation embeddings and controller-compatible whole-body action latents for coordinated humanoid loco-manipulation.

    Source: §1 · p. 2, §7 · pp. 17–18

  • Results

    ω-0 consistently improves over representative imitation-learning, VLA, humanoid, and WAM baselines, and one model executes 11 diverse real-world household tasks.

    Source: §1 · p. 3, §7 · p. 18

  • Takeaways & Limitations

    The experiments highlight latent predictive world-action modeling as an effective framework for scalable concurrent humanoid loco-manipulation.

    Source: §7 · p. 18

Abstract

from arXiv · show

Humanoid household tasks often require concurrent loco-manipulation, where the robot must move, adjust posture, maintain balance, and manipulate objects as a single coordinated behavior. Yet existing humanoid policies typically decompose locomotion and manipulation, while recent world-action models remain either arm-centric or video-centered. We present $\omega$-0, a latent predictive whole-body world-action model for real-world humanoid concurrent loco-manipulation. Given a language instruction, current visual observation, and robot proprioceptive state, $\omega$-0 directly predicts controller-compatible whole-body action latents for real-robot execution. Rather than reconstructing future videos, $\omega$-0 learns compact future observation embeddings as a lightweight predictive objective, coupling latent visual foresight with diffusion-based whole-body action generation. The model supports egocentric RGB, exocentric RGB, and exocentric depth inputs, and leverages controller-based simulation replay to ground human/public visual-motion priors into robot-executable action latents. We further collect $\omega$-HOME, a 40+ hour real-world household humanoid dataset with synchronized multi-view observations, whole-body SMPL motions, robot states, and action latents. Real-world experiments on 11 household tasks demonstrate that a single $\omega$-0 model can produce smooth manipulate-while-moving behaviors and consistently outperform representative imitation learning, VLA, humanoid, and WAM baselines.

1 Introduction

ω-0 targets humanoid household tasks that require continuous whole-body coordination, rather than separate locomotion and manipulation. It combines latent future-observation prediction with whole-body action generation and evaluates one model across real-world household tasks.

1 Introduction

  • Humanoid household tasks require the lower body, torso, arms, and hands to adapt together while the robot manipulates during movement.Examples include wiping a table while stepping and leaning, mopping while moving the base, and reaching or bending while maintaining balance.Source: §1 · p. 2
  • Existing policies often separate locomotion, balance, and manipulation, a decomposition that can limit tasks requiring simultaneous stepping, torso adjustment, reaching, and contact maintenance.Arm-centric VLA policies and mobile-manipulation systems also commonly focus on arm actions or treat the base and arms as separate components.Source: §1 · p. 2
  • Video-centered world-action models can amplify noisy or inconsistent predicted videos into abrupt or unstable whole-body actions, while pixel fidelity need not improve control.The paper instead asks whether future prediction can serve as a compact signal for whole-body action learning.Source: §1 · p. 2
  • ω-0 jointly predicts compact future-observation embeddings and diffusion-generated whole-body action latents that a low-level controller executes.The action branch produces coordinated behaviors without dividing execution into separate locomotion and manipulation phases.Source: §1 · pp. 2–3
  • SONIC-based simulation replay grounds human and public visual-motion priors in robot-executable action latents and proprioceptive supervision.Motions that cannot be reliably tracked are filtered out before the remaining trajectories provide supervision.Source: §1 · p. 3
  • ω-0 accepts egocentric RGB, exocentric RGB, or exocentric depth, using third-person observations as complementary training supervision while supporting first-person execution.View tokens distinguish camera perspectives; exocentric RGB-D provides supervision about whole-body motion, object-scene relations, and task progress.Source: §1 · p. 3
  • A single ω-0 model executes 11 real-world household tasks with smooth whole-body coordination during stepping, reaching, contact-rich manipulation, and object interaction.The tasks span tabletop manipulation, cleaning, laundry handling, object transfer, appliance interaction, and mobile manipulation; the paper also reports promising generalization to held-out objects, scenes, and human-data transfer settings.Source: §1 · p. 3
  • ω-HOME is a 40-hour real-world household dataset with synchronized multi-view videos, whole-body SMPL motions, robot states, and action latents.The dataset supports training and evaluation across 11 household loco-manipulation tasks.Source: §1 · p. 3

2 Related Work

Related work spans physics-based whole-body tracking, humanoid vision-language-action policies, and world-action models. ω-0 differs from video-centered humanoid WAMs by using future visual prediction as a latent auxiliary objective while directly generating controller-compatible whole-body action latents.

2.1 Whole-Body Humanoid Control

  • SONIC scales natural humanoid whole-body tracking to large motion corpora and provides a controller interface for real-world execution (Luo et al., 2025).Earlier work establishes reinforcement-learning-based motion tracking as a standard paradigm; GMT (Chen et al., 2025b) and UniTracker (Yin et al., 2025) address motion coverage and generalizable tracking, respectively.Source: §2.1 · p. 3

2.2 Humanoid VLAs

  • Humanoid VLAs extend language-conditioned control toward loco-manipulation, but primarily focus on direct observation-to-action learning or execution interfaces rather than future visual modeling.Examples include staged human-video pretraining and real-world post-training in Ψ-0 (Wei et al., 2026), whole-body native action spaces in OpenHLM (Hu et al., 2026), and action-free egocentric videos in WholeBodyVLA (Jiang et al., 2025).Source: §2.2 · p. 4

2.3 Humanoid WAMs

  • ω-0 uses lightweight future-observation embedding prediction as auxiliary supervision and directly denoises controller-compatible whole-body action latents, rather than relying on a video world model or video-to-action inversion.DiT4DiT (Ma et al., 2026) uses separate video and action DiTs with a decoupled upper- and lower-body interface, while MotionWAM (Zheng et al., 2026) conditions action generation on video-model denoising features.Source: §2.3 · p. 4

3 Method

ω-0 maps language, visual observations, and robot state to controller-compatible whole-body action latents, coupling action generation with lightweight prediction of future visual latents. Its three training stages build action-aware representations, align visual prediction with action generation, and fine-tune and deploy the model for real-world control.

3.1 Overview

  • ω-0 predicts controller-compatible whole-body action chunks from language, visual observations, and robot state, while future visual latents provide auxiliary task-progress cues.The predicted action latents are executed by SONIC in a receding-horizon manner; the model does not require a large video generator or test-time video-to-action inversion.Source: §3.1 · p. 4

3.2 Stage 1: Whole-Body Action VLM Pretraining

  • A whole-body FAST tokenizer converts continuous action trajectories into discrete tokens that a VLM learns to predict autoregressively from language and visual observations.The resulting VLM representation supplies an action-aware semantic prior for later stages.Source: §3.2 · p. 5

3.3 Stage 2: Human-to-Humanoid Action-Latent Pretraining

  • Stage 2 jointly trains future visual latent prediction and diffusion-based whole-body action generation, using SONIC simulation replay to turn human motions into robot-executable supervision.Trajectories that SONIC cannot reliably execute, including physically infeasible or highly dynamic motions, are filtered out.Source: §3.3 · pp. 6–7

3.4 Stage 3: Fine-tuning on Real-World Data

  • Real-world fine-tuning uses synchronized observations, robot states, and action latents, with training-time real-time chunking to promote continuity between adjacent action chunks.A randomly sampled clean action prefix anchors generation of the remaining future latents, reducing discontinuities and improving execution smoothness.Source: §3.4 · p. 7

3.5 Real-World Deployment

  • At deployment, ω-0 conditions on language, head-camera observations, and proprioception, then uses cached prior actions as a clean prefix before passing generated latents to SONIC.The robot state includes joint positions and torso orientation represented continuously in 6D rotation form.Source: §3.5 · p. 8

4 ω-HOME Dataset

ω-HOME is a real-world household humanoid dataset designed to support whole-body loco-manipulation learning and evaluation. It combines synchronized multi-view observations, motion references, robot states, and controller-compatible action latents across household tasks and temporal horizons.

4 ω-HOME Dataset

  • ω-HOME contains 40.3 hours, 4,827 episodes, and 24 household tasks recorded at 30 Hz, with synchronized language, multi-view video, robot states, whole-body motion references, and action latents.The dataset addresses the scarcity of real-world data for household tasks requiring coordinated locomotion, posture adjustment, dexterous manipulation, and interaction with furniture or tools.Source: §4 · p. 8

4.1 Tasks and Modalities

  • Each episode pairs deployment-matched egocentric RGB with third-person RGB-D, proprioception, whole-body motion references, and action latents for policy learning and future visual latent prediction.The exocentric stream supplies complementary supervision about global body motion, object-scene relations, and task progress.Source: §4.1 · p. 8

4.2 Dataset Statistics and Usage

  • The dataset spans short manipulation skills and longer loco-manipulation activities, and its remaining trajectories are combined with public demonstrations for pretraining without overlap with the 11 downstream evaluation tasks.Incorporating this additional real-world humanoid data significantly improves real-world performance after fine-tuning.Source: §4.2 · pp. 8–9

4.3 Teleoperation Details

  • SONIC-based teleoperation combines headset and foot-tracker motion cues, handheld hand commands, and synchronized onboard and room-camera RGB-D capture for whole-body demonstrations.The robot is equipped with Inspire dexterous hands.Source: §4.3 · p. 9, Figure 4 · p. 9

5 Implementation and Experiments

The implementation combines public human-motion and video datasets with simulation replay and real-world humanoid demonstrations. Motion formats are standardized, unreliable replay trajectories are filtered, and a single model is fine-tuned across the household task set.

5.1 Datasets and Data Processing

  • ARCTIC, Xperience-10M, and Motion-X supply complementary egocentric and exocentric visual-motion data, with irrelevant or physically unsuitable Xperience-10M sequences manually filtered.ARCTIC includes both egocentric and exocentric video, while Motion-X contributes third-person human motion videos.Source: §5.1 · pp. 9–10
  • All dataset motion annotations are converted to a unified SMPL representation so the tokenizer, VLM pretraining, and SONIC replay pipeline can use a consistent motion space.The source datasets may use different body parameterizations, including SMPL-H and SMPL-X.Source: §5.1 · p. 10
  • SONIC simulation replay converts public motions into robot states and controller-compatible action latents, discarding trajectories that cannot be reliably tracked.The resulting state-action latent pairs support future-aware action DiT post-training.Source: §5.1 · p. 10
  • Real-world training mixes approximately 200 demonstrations per task across 11 household tasks, totaling 2,220 trajectories, to fine-tune one model for evaluation on all 11 tasks.The mixed-task data cover coordinated stepping, torso motion, reaching, dexterous hand motion, and object interaction.Source: §5.1 · p. 10

6 Implementation Details

This section details ω-0’s training and evaluation protocol, real-world baseline comparisons, main experiments, ablations, and deployment and generalization results. It also examines held-out generalization across objects, scenes, and human-to-robot transfer, execution under environmental variations, and the use of human demonstrations grounded in the robot action-latent space.

6 Implementation Details

  • Training uses a frozen V-JEPA image encoder and pretrained T5 language encoder; a FAST tokenizer converts unified SMPL motions to discrete action tokens for Stage-1 VLM pretraining.In Stages 2 and 3, the V-JEPA encoder and action VLM remain frozen while the future-aware query module, state encoder, and action DiT are optimized; all stages run on eight H100 GPUs.Source: §6 · p. 10

6.1 Evaluation Metrics

  • Evaluation uses 10 trials per task for a single multi-task policy and reports success rate, average completed-subtask score, and ordered task progress.Success requires completing every objective; score credits completed subtasks even after an earlier error, whereas progress measures how far execution follows the intended sequence before failure or irreversible deviation.Source: §6.1 · pp. 10–11

6.2 Baselines

  • Baselines are trained or fine-tuned on the real-world dataset as single multi-task policies and evaluated under the same protocol, spanning imitation, VLA, humanoid, and world-action-model approaches.The comparisons include action-chunking and diffusion policies, arm-centric control with separate locomotion, and WAM designs centered on manipulation or video generation rather than a unified controller-compatible whole-body latent interface.Source: §6.2 · pp. 11–12

6.3 Task Suite

  • The 11-task domestic suite stresses coordinated upper- and lower-body control, including contact-rich manipulation, locomotion, posture adjustment, balance, and continuous perception-action feedback.It spans tabletop manipulation, object transfer, cleaning, laundry, and mobile manipulation.Source: §6.3 · p. 12, §6.4 · p. 12, Table 1 · p. 12

6.4 Main Experiment

  • ω-0 consistently outperforms the evaluated baselines across success, subtask completion, and task progress, with joint future-visual and whole-body action-latent learning supporting more coherent behavior.The omni-view variant performs better overall than the egocentric-only variant, particularly on locomotion-heavy tasks; adding non-overlapping ω-HOME pretraining data also improves performance after fine-tuning.Source: §6.4 · pp. 12–14, Table 2 · p. 13, Table 3 · p. 13

6.5 Ablation Studies

  • Ablations show that robot state, the whole-body VLM prefix, future visual latent prediction, RTC training, and V-JEPA current-image encoding each support real-world action generation.State conditioning grounds actions in the robot’s physical configuration; the VLM prefix adds task-conditioned action semantics; future latent prediction supplies task-progress and scene-evolution cues; RTC reduces hesitation and discontinuity; and V-JEPA provides more action-relevant single-frame features than Wan despite weaker offline future-latent reconstruction.Source: §6.5 · pp. 15–16

6.6 Real-World Evaluation

  • On the G1, ω-0 autonomously executes long-horizon tasks requiring posture adjustment, object interaction, and dexterous manipulation, without teleoperation, motion replay, or scripted intervention.Examples include collecting trash from different heights and walking to retrieve and deliver a drink.Source: §6.6 · p. 17

6.7 Generalization to Held-Out Task

  • ω-0 generalizes to novel object instances and appearances, where it can still localize, approach, and manipulate targets rather than simply memorizing training demonstrations.The section also describes evaluations using new room layouts and household configurations, but the supplied passage does not report their outcome.Source: §6.7 · p. 17
  • Successful execution under environmental variations indicates ω-0 learns reusable visual-action priors rather than fitting only a fixed scene layout.Source: §6.7 · p. 17
  • After fine-tuning with human demonstrations and grounding them in the robot action-latent space, ω-0 completes target tasks on the real humanoid.The demonstrations come from a different embodiment.Source: §6.7 · p. 17
  • Removing video latent prediction degrades performance across cross-object, cross-scene, and human-data-transfer settings.The ablation removes the video query branch while keeping the VLM prefix, robot-state conditioning, and action diffusion head unchanged.Source: §6.7 · p. 17

7 Conclusion

ω-0 couples future visual-latent prediction with whole-body action generation for coordinated household loco-manipulation. The conclusion reports improved performance over representative baselines, while the appendices document the action format, data processing, and deployment procedures.

7 Conclusion

  • ω-0 jointly coordinates the lower body, torso, arms, and hands to manipulate while moving, coupling future visual-latent prediction with diffusion-based whole-body action generation.It predicts whole-body action latents for autonomous household task execution.Source: §7 · p. 18
  • ω-0 consistently improves over representative imitation learning, VLA, humanoid, and WAM baselines in real-world experiments and ablations.Table 5 examines future video latent prediction and generalization, with maximum scores of 13 for cross-object generalization, 6 for cross-scene generalization, and 3 for human-data transfer; the caption does not report the table’s results.Source: §7 · p. 18, Table 5 · p. 18

Appendix

  • The appendix documents the paper’s data format and preprocessing, inference and RTC deployment, real-world evaluation details, task gallery, and per-task baseline comparisons.It also lists per-task success rate, task progress, and task score comparisons.Source: Appendix · p. 21

A.1 Action Latent and State Format

  • ω-HOME action trajectories use 66 dimensions: a 64-dimensional whole-body controller latent and two scalar hand commands, with 1 denoting closed and 0 open.The robot state includes body and hand joint positions and torso orientation, for a 47-dimensional state vector.Source: §A.1 · p. 22

A.2 Normalization

  • Action latents use mean-standard-deviation normalization, while robot states and hand commands use min-max normalization with the same statistics during training and inference.This scheme standardizes controller latents while preserving the bounded semantics of hand commands and state inputs.Source: §A.2 · p. 22

A.3 Public Motion Data Processing

  • Public motions are converted to SMPL and a z-up coordinate convention, then zero-yaw normalized by subtracting each sequence’s initial root yaw from every frame.This preserves relative turning while aligning initial headings before simulation replay and action-latent extraction.Source: §A.3 · p. 22

B.1 Receding-Horizon Deployment

  • Deployment uses receding-horizon inference, executing the first 8 actions of each predicted 25-action chunk before refreshing the egocentric image and robot state.A forward pass takes approximately 0.14 seconds, corresponding to more than 7 Hz inference; future visual-latent prediction is not required for real-time control.Source: §B.1 · p. 23

B.2 RTC-Style Warm Start and Overlap Blending

  • RTC-style warm-starting reuses an unexecuted future segment as the next chunk’s prefix, and overlap blending linearly joins adjacent chunks to reduce discontinuities.After stitching, normalized action latents are denormalized and sent to the low-level whole-body controller.Source: §B.2 · p. 23

C Real-World Evaluation Details

The real-world evaluation details define progress annotations as binary milestones and provide per-trial records for ω-0Ego across household tasks.

C.1 Progress Annotation Protocol

  • Each task is evaluated over 10 real-robot trials, with binary stages marking milestones such as grasping, transport, placement, cleaning, or closing an articulated object.A 1 means the stage was completed in that trial; a 0 means it was not.Source: §C.1 · p. 24

C.2 Per-Trial Progress Tables of ω-0Ego

  • The paper provides per-trial progress annotation tables for ω-0Ego across six household tasks: put_apple_and_close_drawer, wipe_table, pick_garbage, clean_bed, mop_floor, and pick_and_place.Source: Table 6 · p. 24, Table 7 · p. 24, Table 8 · p. 24

E Per-Task Baseline Comparison

Across 11 real-world household tasks, ω-0Ego and ω-0Omni perform strongly on both short- and long-horizon tasks, while baseline performance declines on more demanding coordination tasks.

E Per-Task Baseline Comparison

  • ω-0Ego and ω-0Omni consistently perform strongly across short-horizon manipulation and long-horizon loco-manipulation tasks.The comparison covers seven models and reports per-task success rate, task progress, and average task score.Source: E Per-Task Baseline Comparison · p. 26
  • Baseline performance drops more on articulated-object interaction, sequential collection, dual-hand coordination, and extended whole-body long-distance locomotion than on simpler pick-and-place tasks.The omni-view variant improves several challenging tasks, with the passage suggesting complementary views provide useful information about whole-body task progress and scene understanding.Source: E Per-Task Baseline Comparison · p. 26

E.1 Per-Task Success Rate

The per-task figures report success and partial progress across methods; the supplied passages state that ω-0 variants lead on success for most tasks, especially demanding interactions.

E.1 Per-Task Success Rate

  • ω-0Ego and ω-0Omni achieve higher success rates on most tasks, especially contact-rich cleaning, articulated-object operation, and long-horizon retrieval.Figure 9 defines success as completing all predefined progress stages.Source: §E.1 · p. 26

E.2 Per-Task Task Progress

  • Figure 10 reports normalized fractions of completed progress stages, capturing partial completion when a rollout does not achieve full success.This metric is particularly informative for long-horizon tasks, where early stages may be completed but later placement or appliance-closing stages may fail.Source: §E.2 · p. 26

E.3 Per-Task Score

The per-task score compares how far seven models progress through household tasks, complementing all-stages-complete success rates. ω-0 improves intermediate task completion across diverse scenarios.

E.3 Per-Task Score

  • ω-0 improves intermediate task completion across diverse household scenarios, beyond its gains in fully completed trials.Average score counts completed progress stages, whereas success rate counts trials completing every stage.Source: §E.3 · p. 26
  • Figure 9 compares success rates across seven models on 11 real-world tasks; success requires completing all progress stages.Source: Figure 9 · p. 27
  • Figure 10 compares normalized task progress across seven models, measuring the fraction of completed progress stages.Source: Figure 10 · p. 27
  • Figure 11 compares the average number of completed progress stages for each task across seven models.This score reflects partial task completion, unlike success rate.Source: §E.3 · p. 26, Figure 11 · p. 27

D Task Gallery

The task gallery documents real-world evaluation tasks spanning household manipulation, cleaning, tool use, articulated-object interaction, laundry, retrieval, and mobile manipulation. Individual task cards specify prompts, completion stages, and the capabilities stressed.

D Task Gallery

  • The evaluation spans tabletop manipulation, contact-rich cleaning, tool use, articulated-object interaction, laundry, retrieval, and long-horizon mobile manipulation.The task cards include wiping a table, mopping, handling washing-machine doors and clothes, and retrieving objects from a refrigerator.Source: D Task Gallery · pp. 28, 29, 30, 31, 33
  • Cleaning cards require sustained tool contact, with mopping and wiping tasks scored on tool control, effective contact, and stain removal.Source: D Task Gallery · pp. 29–30
  • Several cards assess multi-step appliance and articulated-object interaction, including retrieving laundry or fruit and then closing a washing-machine or refrigerator door.Source: D Task Gallery · pp. 31, 33
  • Manipulation cards cover object transport and placement, including putting an apple in a drawer and closing it with a leg.Source: D Task Gallery · pp. 30, 32

D.2 Full ω-HOME Task Cards

The full ω-HOME task cards illustrate household behaviors beyond the evaluation suite, including cleaning, sorting, collection, furniture interaction, appliance use, clothing tasks, and human-robot collaboration. Their prompts and process descriptions specify what each task entails.

D.2 Full ω-HOME Task Cards

  • The collection suite spans bathroom cleaning, semantic sorting, object collection, furniture interaction, refrigerator operation, clothes handling, and human-robot collaboration.Source: §D.2 · pp. 34, 35, 37
  • Some cards test categorization and collection, including sorting fruit from office supplies and retrieving fruit from multiple closet cells while holding a basket.Source: §D.2 · pp. 35–36
  • Furniture tasks include jointly moving a table with a human and pushing a chair beneath a table.Source: §D.2 · pp. 37–38
  • Appliance cards include placing a drink in a refrigerator drawer and closing both the drawer and outer door, as well as loading a drink into the upper compartment.Source: §D.2 · pp. 38–39
  • Other cards cover bathroom surface cleaning, hanging clothes, and placing a fruit-filled basket on a cabinet.Source: §D.2 · pp. 37, 39

Related papers

Loading 2608.06375v2…