PaperlayerView exact revision on arXiv

Source-linked AI summary

Humanoid World Action Model With Joint State--Action Generation

Yan Yang, Jikun Rong, Minzhao Zhu, Zheyi Zhao, Qirui Hu, Zihan Lan, Weixin Mao, Yinhao Li, Zhen Fu, Hua Chen

arXiv:2610.12026v1cs.ROcs.AI

TL;DR

Hierarchical humanoid policies produce reference actions that low-level execution may realize differently, making it difficult to connect actions to visual outcomes. HWAM jointly generates actions and post-execution states, with visual-transition training paths. It achieves the highest observed success rate on the three evaluated OLI tasks.

  • Problem

    In hierarchical humanoid control, realized motion can differ from policy references, so visual prediction must account for both execution and scene change to link actions with outcomes.

    Source: §1 · p. 2

  • Method

    HWAM jointly generates reference actions and aligned post-execution proprioceptive states, using Policy, FDM, and IDM paths to connect them with visual transitions.

    Source: §1 · pp. 2–3

  • Results

    HWAM achieves the highest observed success rate on the three evaluated LimX OLI tasks.

    Source: §5 · p. 9

  • Takeaways & Limitations

    On the mobile manipulation task, the ablation reports higher task success when the joint target is added under FDM/IDM/Policy training.

    Source: §5 · p. 9

    The joint target does not explicitly learn a low-level dynamics model or impose a separate physical-consistency constraint, and is not assumed useful in isolation.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Humanoid robots are a promising platform for general-purpose manipulation. Recent Vision-Language-Action (VLA) policies learn actions directly from multimodal observations, while World Action Models (WAMs) further incorporate future visual prediction to improve action generation. However, in hierarchical humanoid systems, VLA and WAM policies output reference actions that are subsequently realized through whole-body control, robot dynamics, balance, and contact. This hierarchy creates an action--execution gap: the reference produced by the policy can differ from the motion realized by the robot. Without explicitly modeling the realized body state, future visual prediction must jointly explain scene evolution and discrepancies between reference actions and executed motion, making it difficult to associate an action with its physical outcome. We propose HWAM, a Humanoid World Action Model with joint state--action generation, which makes the robot's post-execution proprioceptive state an explicit prediction target. By jointly generating reference actions and their realized body states, HWAM directly incorporates supervision of executed motion into action learning. HWAM is trained through three complementary conditional paths. The Policy path jointly denoises state--action trajectories conditioned only on current observations, matching deployment conditions. Forward Dynamics Modeling (FDM) predicts future visual observations conditioned on actions and post-execution states, while Inverse Dynamics Modeling (IDM) reconstructs the joint trajectory from visual transitions. Together, these paths connect policy references, realized body motion, and visual outcomes. HWAM achieves the highest success rate among evaluated baselines on three real-robot tasks on the LimX OLI humanoid. On Candy Picking, HWAM achieves a 70.6% success rate, compared with 43.3% for Fast-WAM.

1 INTRODUCTION

Hierarchical humanoid control turns policy outputs into reference actions that low-level controllers adapt before execution, creating a gap between references and realized motion. HWAM jointly generates post-execution states and actions, then connects them to visual outcomes through complementary training paths.

1 INTRODUCTION

  • Hierarchical control treats policy actions as kinematic motion references that whole-body tracking and joint servoing adapt using proprioceptive feedback before execution.The action is not a torque command and is not assumed to be realizable point by point.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The action–execution gap arises because low-level tracking adapts reference motion to body state, contact, and dynamics, so realized motion can differ from the policy reference.Without modeling the intermediate realized state, future visual prediction must explain both physical realization and scene change.Source: §1 · p. 2
  • Dataset statistics show larger discrepancies between commanded joint references and measured proprioceptive states in mobile humanoid manipulation than in stationary bimanual manipulation, even at steady state.Source: §1 · p. 2
  • HWAM aligns each action with its temporally corresponding post-execution proprioceptive state and jointly denoises the two channels with a shared trajectory generator.The state estimate is available throughout denoising, giving action generation access to the evolving state estimate.Source: §1 · pp. 2–3
  • Three training paths connect action references, realized body motion, and visual outcomes: Policy jointly generates state–action trajectories, FDM predicts future visuals, and IDM predicts trajectories from visual transitions.The Policy path is conditioned on current observations and does not generate future video.Source: §1 · p. 3
  • The evaluation examines action–execution gap observability, the joint target with bidirectional visual grounding, and real-robot manipulation on LimX OLI and clothes-folding on ALOHA.Source: §1 · p. 3

2 RELATED WORK

Related work spans multimodal VLA policies, predictive and joint-generative WAMs, and humanoid body-state representations. HWAM builds on these lines by treating post-execution states as part of the generated trajectory.

2.1 VISION-LANGUAGE-ACTION MODELS

  • VLA models unify visual perception, language understanding, and robot control, with examples spanning embodied language models, action-tokenized control, and open-source learning from demonstrations.Recent systems also explore heterogeneous co-training and cross-embodiment generalization.Source: §2.1 · p. 3

2.2 WORLD ACTION MODELS AND JOINT GENERATIVE POLICIES

  • WAMs incorporate future prediction into policy learning, including cascaded and joint visual-action generation, unified conditional modes, and joint trajectory modeling.Related approaches include video-action diffusion, separate forward and inverse dynamics modes, and models that predict actions without explicit future-video synthesis at inference.Source: §2.2 · pp. 3–4

2.3 BODY-STATE REPRESENTATIONS FOR HUMANOID CONTROL

  • Humanoid body-state methods use shared or predictive representations to support coordinated behavior, motion transfer, and whole-body control.Prior examples include hierarchical control, shared state–action representations, and predictive body-part features fused with visual-language information.Source: §2.3 · p. 4

3 METHOD

HWAM pairs high-level motion references with the proprioceptive states measured after low-level execution, making realized body motion part of its prediction target. Its shared state–action generator and video expert are trained through policy, forward-dynamics, and inverse-dynamics paths that connect trajectories with visual transitions.

3.1 HIERARCHICAL CONTROL INTERFACE AND OVERVIEW

  • The high-level policy outputs kinematic reference actions, which a low-level whole-body tracking controller adapts using proprioceptive feedback.The action is a motion reference, while the paired state records the body's response after execution.Source: §3.1 · p. 4

3.2 JOINT POST-EXECUTION STATE–ACTION CHUNK TARGET

  • Each action reference is paired with the proprioceptive state measured at the next sample, and the two are concatenated into a joint prediction chunk.The shared representation lets partially denoised state channels inform action generation, while state supervision can influence the action representation.Source: §3.2 · pp. 4–5

3.3 ARCHITECTURE

  • Separate video and state–action experts exchange information through path-dependent masked cross-attention, supporting trajectory-to-video, vision-to-trajectory, and policy learning.The state–action expert jointly denoises state and action channels in a shared trajectory representation.Source: §3.3 · p. 5

3.4 BIDIRECTIONAL STATE–ACTION–VISION GROUNDING

  • Policy generates joint state–action trajectories from current observations, FDM predicts future video from those trajectories, and IDM predicts trajectories from visual transitions.The paths share experts and parameters but differ in targets and visible conditions; Policy hides future-video tokens to match deployment.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Experiments assess the action–execution gap, real-robot task success, open-loop prediction, and a controlled ablation on Plush-Toy Picking. The evaluation combines LimX OLI tasks with action–state measurements on OLI and ALOHA.

4.1 ACTION–EXECUTION GAP IN ROBOT DATA

  • OLI shows 4.420° direct and 2.885° steady-state action–state MAE, while ALOHA shows 0.648° and 0.402°, respectively.Direct MAE compares commanded joint references with aligned post-execution states; steady-state MAE uses approximately constant references.Source: §4.1 · p. 7

4.2 EXPERIMENTAL SETUP

  • The OLI evaluation covers Candy Picking, Object Collection, and Plush-Toy Picking, with 1,783, 4,220, and 404 training episodes, respectively, and compares HWAM with five policies.Candy Picking also has 422 held-out validation episodes; OLI success rates are measured in physical robot trials.Source: §4.2 · pp. 7–8

4.3 REAL-ROBOT EVALUATION

  • HWAM records the highest observed success rate on all three OLI tasks, improving over π0.5 by 8.1, 1.7, and 12.6 percentage points on Candy Picking, Object Collection, and Plush-Toy Picking, respectively.Table 1 summarizes the real-robot results across the three OLI tasks.Source: Table 1 · p. 8, §4.3 · p. 8

4.4 OPEN-LOOP PREDICTION DIAGNOSTICS

  • 3.816° is HWAM’s averaged open-loop action MAE over 32 positions, versus 4.024° for action-only FDM/IDM/Policy; HWAM and VGM/Policy have similar state MAE.The diagnostics use held-out Candy Picking episodes and average errors over the first 31 shared joints.Source: §4.4 · p. 8
  • With ground-truth conditions, conditioning future-video prediction on actions and post-execution states raises PSNR from 20.559 to 23.336 versus the action-only model; predicted-trajectory conditions show no video-quality advantage.The predicted-condition PSNR ranges from 20.345 to 20.844 in 12 held-out episodes immediately before grasp closure.Source: §4.4 · p. 8

4.5 CONTROLLED ABLATION ON PLUSH-TOY PICKING

  • Adding the joint target lowers success from 60.0% to 45.5% with VGM/Policy but raises it from 55.0% to 73.3% with FDM/IDM/Policy, a 32.8-point difference between effects.The four-configuration Plush-Toy Picking ablation holds other settings fixed and does not isolate individual paths or architectural choices.Source: §4.5 · p. 9, Table 2 · p. 9

5 CONCLUSION

HWAM jointly generates high-level actions with aligned post-execution states, connecting action references to their physical and visual consequences. Its evaluated OLI results and mobile-task ablation support the combined formulation, while predicted states remain generated variables rather than controller inputs.

5 CONCLUSION

  • HWAM jointly generates high-level actions and temporally aligned post-execution proprioceptive states, with FDM, IDM, and Policy paths connecting them to visual transitions.The Policy path generates state–action trajectories conditioned on current observations; predicted states are not sent to the controller.Source: §5 · p. 9

AI USE STATEMENT

The authors report using generative AI tools during implementation, writing, feedback, and literature review, while taking responsibility for the final work.

AI USE STATEMENT

  • Generative AI assisted with implementation, manuscript work, methodological feedback, result interpretation, and literature identification; the authors take responsibility for the final content.The statement includes experimental results, conclusions, and AI-assisted text or other artifacts.Source: AI USE STATEMENT · p. 10

A ACTION–EXECUTION GAP ANALYSIS

Both platforms show nonzero direct and steady-state error between commanded joint references and measured post-execution states.

A ACTION–EXECUTION GAP ANALYSIS

  • On the LimX OLI mobile humanoid and stationary ALOHA bimanual platform, commanded joint references have non-zero gaps from aligned post-execution states, including residual errors under approximately constant references.Direct MAE pairs the joint reference with the measured post-execution joint state; steady-state MAE uses that pairing under approximately constant joint references.Source: A ACTION–EXECUTION GAP ANALYSIS · p. 13, Table 3 · p. 13

B TASK SUITE AND DATA

Table 4 summarizes the OLI task suite and training data; the policies use synchronized camera images, language instructions, and proprioception.

B TASK SUITE AND DATA

  • Table 4 summarizes the three OLI tasks, their training data, and completion criteria.Its episode counts refer to demonstrations used for training.Source: Table 4 · p. 13, B TASK SUITE AND DATA · p. 13
  • Training sets contain 1,783 Candy Picking, 4,220 Object Collection, and 404 Plush-Toy Picking demonstrations.Candy Picking also has 422 held-out episodes for offline validation, excluded from training.Source: B TASK SUITE AND DATA · p. 13
  • OLI policies receive synchronized head- and wrist-camera RGB, a language instruction, and the current proprioceptive state.Task success is measured in physical robot trials; held-out demonstrations are used for prediction diagnostics.Source: B TASK SUITE AND DATA · p. 13

C.1 FUTURE-VIDEO PREDICTION DIAGNOSTICS

The diagnostics assess future-video prediction, trial uncertainty, and Candy Picking breakdowns; end-to-end video quality does not track closed-loop task performance.

C.1 FUTURE-VIDEO PREDICTION DIAGNOSTICS

  • End-to-end video prediction shows no HWAM quality advantage: PSNR ranges from 20.345 to 20.844, with HWAM at 20.501.Evaluation uses 12 held-out Candy Picking episodes immediately before grasp closure; all conditions are predicted rather than ground-truth future trajectories.Source: §C.1 · p. 13, Table 5 · p. 14
  • With ground-truth trajectory conditions, HWAM reaches PSNR 23.336 versus 20.559 for the action-only model.The comparison is on held-out Candy Picking episodes.Source: §C.1 · p. 14, Table 6 · p. 14

C.2 TRIAL COUNTS AND UNCERTAINTY

  • Table 7 reports available Candy Picking trial counts and 95% Wilson confidence intervals, assuming independent Bernoulli trials.These intervals do not measure variation across training seeds.Source: §C.2 · p. 14, Table 7 · p. 14

C.3 MOTION AND WORKSPACE BREAKDOWNS

  • Table 8 breaks Candy Picking success rates down by motion direction and workspace region; these are not held-out spatial-split evaluations.The table identifies CH/CV as cross-horizontal/vertical and X-P/X-S as primary/secondary diagonal.Source: §C.3 · p. 14, Table 8 · p. 15

D EVALUATION ON ALOHA CLOTHES FOLDING

HWAM is evaluated on ALOHA Clothes Folding with task-specific training, and it achieves the highest reported task-completion score among evaluated methods.

D EVALUATION ON ALOHA CLOTHES FOLDING

  • The task requires coordinated grasping and successive folds of a spread garment, using 640 task-specific demonstrations.Observations combine a high-mounted RGB camera, two wrist cameras, proprioception, and a language instruction; this is not zero-shot transfer from OLI.Source: D EVALUATION ON ALOHA CLOTHES FOLDING · p. 15
  • Figure 6 follows the garment-folding sequence from the initial grasp at 0.4 seconds through three intermediate folds to final release at 49.2 seconds.Source: Figure 6 · p. 15
  • The ALOHA configuration retains joint state–action prediction and uniform FDM/IDM/Policy sampling.Video and state–action experts are fine-tuned for six epochs while the video VAE remains frozen; evaluation uses a 0–100 score with partial credit.Source: D EVALUATION ON ALOHA CLOTHES FOLDING · p. 15, Table 9 · p. 15
  • HWAM achieves the highest reported ALOHA Clothes Folding task-completion score among evaluated methods.Table 9 scores completion on a 0–100 scale and gives partial credit for completed folding stages.Source: D EVALUATION ON ALOHA CLOTHES FOLDING · p. 15, Table 9 · p. 15
Loading 2610.12026v1…