PaperlayerView exact revision on arXiv

Source-linked AI summary

SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation

Juyi Sheng, Hua Wang, Mengyuan Liu

arXiv:2610.02120v1cs.RO

TL;DR

Existing WAMs predict videos or visual latents that represent interaction geometry implicitly and may retain information unrelated to control. SkeleWAM models robot–object structure as a sparse 3D skeleton and jointly learns actions and future skeletons. On LIBERO-Plus, it achieves 85.9% overall success with 57.1M parameters, 3.7 percentage points above Cosmos-Policy.

  • Problem

    Existing WAMs use video or visual-latent predictions that represent interaction geometry implicitly, leaving open whether sparse robot–object structure can support effective world action learning with lower model complexity.

    Source: §1 · p. 1

  • Method

    SkeleWAM represents robot joints, object centers, and interaction points as a sparse 3D state and jointly trains action generation with future-skeleton prediction.

    Source: §1 · p. 1

  • Results

    85.9% overall success with 57.1M parameters on LIBERO-Plus outperforms Cosmos-Policy by 3.7 percentage points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments on LIBERO-Plus and five real-world tasks support explicit robot–object geometry as an effective, parameter-efficient state space for world action learning.

    Source: §5 · p. 8

    Layout-perturbation performance remains limited: success is 66.6% versus 84.1% for π0.5, and remains 69.0% with simulator-derived coordinates.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.

1. Introduction

SkeleWAM addresses the implicit geometry and appearance information in visual WAM states with a sparse 3D robot–object skeleton. It jointly learns actions and future skeletons, and reports strong LIBERO-Plus results with a compact model.

1. Introduction

  • The skeleton uses robot-joint positions and object centers plus interaction points to represent configuration and contact-relevant geometry explicitly.Object centers encode spatial location; interaction points identify locations relevant to contact and manipulation.Source: §1 · p. 1
  • SkeleWAM builds geometric tokens online from RGB-D observations and proprioception, jointly training action generation and future-skeleton prediction; inference uses the current skeleton and language instruction.Future prediction provides auxiliary geometric supervision during training without visual reconstruction; it is omitted at inference.Source: §1 · pp. 1–2
  • Medoid Action Consensus selects the sampled trajectory with the smallest average distance to other candidates, without a reward model or trajectory averaging.It is introduced as an auxiliary inference strategy to reduce sensitivity to stochastic sampling.Source: §1 · p. 2
  • 85.9% overall success with 57.1M parameters on LIBERO-Plus outperforms Cosmos-Policy by 3.7 percentage points.Under camera perturbations, success is 93.4%, 17.6 points above Cosmos-Policy.Source: §1 · p. 2
  • The work evaluates the full LIBERO-Plus benchmark and reports systematic ablations of skeleton composition, future-skeleton supervision, visual estimation, and MAC.The primary setting estimates object landmarks from RGB-D; simulator coordinates are reported as a diagnostic reference.Source: §1 · p. 2

2. Related Work

Visuomotor and VLA policies have broadened manipulation capabilities, while WAMs add future-state prediction to action learning. SkeleWAM differs by representing current and future interaction geometry explicitly as sparse 3D skeletons.

2.1. Visuomotor and Vision-Language-Action Models

  • Visuomotor and VLA models learn action mappings from visual inputs, but do not explicitly model how manipulation scenes evolve under robot actions.The passage describes ACT action chunks, image- and point-cloud-based action distributions, and language-conditioned multitask control in VLA models.Source: §2.1 · p. 2

2.2. World Action Models

  • WAMs add future-state prediction to action learning, but video and latent states represent interaction geometry implicitly; SkeleWAM instead predicts sparse 3D skeleton states.Its future geometric prediction supervises representation learning in training and is not required for action generation at inference.Source: §2.2 · p. 3

3. Method

SkeleWAM represents manipulation scenes with sparse 3D skeletons and jointly learns action generation and future skeleton prediction. At inference, it generates actions from the current skeleton and uses Medoid Action Consensus to select among sampled action chunks.

3.1. Problem Formulation

  • Given the language instruction, RGB-D observation, and robot proprioception, SkeleWAM generates an action chunk and jointly learns future skeleton prediction as auxiliary supervision.Future skeleton prediction is omitted at inference.Source: §3.1 · p. 3

3.2. Skeleton World Representation

  • The scene skeleton combines robot joint and end-effector positions with object centers and task-relevant interaction points, organized with consistent node identities and connectivity over time.Robot keypoints use forward kinematics; object landmarks are estimated from RGB-D observations, with coordinates expressed in a shared robot-centric frame.Source: §3.2 · p. 3

3.3. Skeleton World Action Model

  • Separate input adapters map current skeleton, noisy action, and noisy future-skeleton inputs to tokens; a world expert and an action expert process their respective prediction tasks.The experts share current-skeleton context, and the action and future-skeleton prediction groups do not attend to each other.Source: §3.3 · pp. 3–4

3.4. Training Objective

  • Flow matching trains both action generation and future-skeleton prediction, with the skeleton loss supervising future geometry and the action loss propagating through current-skeleton context.The skeleton supervision loss is weighted by λs.Source: §3.4 · p. 4

3.5. Inference and Medoid Action Consensus

  • At inference, SkeleWAM samples action chunks and MAC selects the candidate with the smallest average dissimilarity, then executes the first He actions before observing and replanning.MAC compares the first h steps over continuous motion dimensions and retains the complete selected chunk; it requires neither a reward nor a value model.Source: §3.5 · pp. 4–5

4. Experiments

Experiments evaluate zero-shot LIBERO-Plus performance, real-world manipulation, and design ablations. SkeleWAM performs strongly in the reported settings, while layout perturbations remain difficult.

4.1. Simulation Setup

  • The evaluation measures zero-shot task success across 10,030 LIBERO-Plus variants spanning seven perturbation categories, without fine-tuning on evaluation variants.The primary setting estimates object landmarks from RGB-D observations; simulator-derived coordinates are reported only as a privileged-input diagnostic reference.Source: §4.1 · p. 5

4.2. Simulation Results

  • 85.9% overall success with 57.1M parameters puts SkeleWAM 3.7 percentage points above Cosmos-Policy on LIBERO-Plus.Under camera perturbations, SkeleWAM reaches 93.4%, 17.6 points above Cosmos-Policy; layout success is 66.6%, versus 84.1% for π0.5.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3. Real-World Experiments

  • 89% average success exceeds Cosmos-Policy, Fast-WAM, and π0.5 by 2, 6, and 13 percentage points, respectively, across five real-world tasks.SkeleWAM achieves the highest or joint-highest success rate on four of the five tasks.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Ablation Studies

  • 85.9% success with both object centers and interaction points exceeds centers-only and interaction-points-only representations by 3.6 and 8.2 percentage points, respectively.The ablation holds robot keypoints unchanged; centers provide spatial references, while interaction points identify manipulation-relevant regions.Source: §4.4 · p. 7, Table 3 · p. 8
  • Removing future-skeleton supervision reduces success from 85.9% to 80.1%, while changing cross-branch attention yields smaller differences of 0.3 or 1.1 points from 85.9%.The future-prediction branch is omitted at inference, and the attention ablation indicates the larger benefit comes from co-training through the shared world expert.Source: §4.4 · p. 7, Table 3 · p. 8
  • 85.9% success with a 16-action execution horizon exceeds the 5- and 32-action settings by 5.2 and 4.1 percentage points, respectively.The study attributes the intermediate horizon's balance to feedback from updated observations and temporally consistent action execution.Source: §4.4 · p. 7, Table 4 · p. 8
  • 85.9% success with three MAC candidates exceeds single-trajectory sampling's 84.2%, while five and ten candidates reach 85.2% and 84.1%.Across tested consensus windows, success ranges from 85.3% to 85.9%, with a maximum difference of 0.6 percentage points and the best result at h = 10.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5. Conclusion

SkeleWAM uses sparse 3D robot–object skeletons and future skeleton prediction for geometric supervision, with MAC as an auxiliary trajectory-selection strategy. Experiments support this representation as an effective, parameter-efficient state space, while large layout changes remain challenging.

5. Conclusion

  • SkeleWAM represents scenes with sparse 3D robot joints, object centers, and interaction points, using future skeleton prediction as training supervision without visual reconstruction or additional world prediction at inference.Source: §5 · p. 8
  • MAC selects a representative trajectory from stochastic action samples without relying on a reward model.Source: §5 · p. 8
  • Experiments on LIBERO-Plus and five real-world manipulation tasks support explicit robot–object geometry as an effective, parameter-efficient state space for world action learning.SkeleWAM shows strong zero-shot robustness with substantially fewer parameters than compared VLA and world-model-based methods.Source: §5 · p. 8
  • Combining object centers with interaction points and predicting future skeleton states during training both benefit performance in the reported ablations.Source: §5 · p. 8
  • Performance gaps under large layout changes indicate that sparse geometry alone does not fully capture the variation needed for spatial generalization.Proposed future directions include uncertainty-aware perception, adaptive interaction landmarks, and richer relational structure.Source: §5 · p. 8

Related papers

Loading 2610.02120v1…