PaperlayerView exact revision on arXiv

Source-linked AI summary

Skeleton-Grounded Latent Action Model for Robot World Models

Qi Lv, Jiayi Lin, Yuxin Jiang, Jianzong Wang, Xiang Deng, Mike Shou Zheng

arXiv:2610.31474v1cs.RO

TL;DR

Reconstruction-driven latent actions can mix robot motion with scene changes, complicating their correspondence with robot controls. SkeLAM uses skeleton supervision and cross-modal reconstruction to separate kinematic and residual components, then conditions generation on robot actions. On DROID, it improves end-effector ADE and LPIPS over DreamDojo, with additional evidence from motion probing, transfer, and policy evaluation.

  • Problem

    Reconstruction-driven latent actions can combine robot motion with object responses and other scene changes, complicating their mapping to robot controls.

    Source: §1 · p. 1

  • Method

    SkeLAM learns skeleton-grounded kinematic and residual latents, uses cross-modal swap reconstruction and branch dropout, and adapts robot actions to condition generation through the kinematic pathway.

    Source: §1 · pp. 1–2

  • Results

    19.5% lower end-effector ADE and 11.1% lower LPIPS than DreamDojo on DROID.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Motion probing supports kinematic specialization, qualitative rollouts show zero-shot human-to-robot motion transfer, and policy evaluation yields success-rate estimates consistent with real executions.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

1 INTRODUCTION

Reconstruction-driven latent actions can mix robot motion with scene responses, complicating their mapping to robot controls. SkeLAM separates motion-grounded and residual transition information, then uses the kinematic component for action-conditioned generation.

1 INTRODUCTION

  • Reconstruction objectives can entangle robot motion with object responses and other scene changes, making latent actions harder to map to robot controls.The residual retains complementary transition information useful for modeling interaction outcomes.Source: §1 · p. 1
  • Skeleton supervision grounds a kinematic latent in robot motion while a residual retains complementary information for RGB reconstruction and dynamics learning.The paired RGB and skeleton transitions are used to learn these components from RGB observations.Source: §1 · pp. 1–2
  • Cross-modal swap reconstruction substitutes the paired skeleton kinematic latent into RGB reconstruction, encouraging compatible motion conditioning in the RGB decoder.Separate world-model conditioning pathways and branch dropout prepare generation for robot-action conditions with a zero residual.Source: §1 · p. 2
  • A lightweight adapter maps robot actions to the kinematic space, after which world-model fine-tuning adapts generation to those predicted conditions.At inference, generation uses the current observation and robot actions.Source: §1 · p. 2
  • SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% on DROID compared with DreamDojo.The evaluation also reports motion probing, zero-shot human-to-robot motion transfer, and agreement between predicted and measured policy success rates.Source: §1 · p. 2
  • The contributions combine motion-grounded action adaptation with residual visual dynamics, cross-modal reconstruction, and evaluation of motion, visual fidelity, transfer, and policy success.The reported evidence includes DROID results and agreement between predicted and measured policy success rates.Source: §1 · p. 2

2 RELATED WORK

Related work studies robot world models, video-derived latent actions, and structured action representations. SkeLAM uses skeleton supervision to ground video-derived kinematic latents while retaining complementary visual dynamics.

2 RELATED WORK

  • Robot world models support interactive simulation and policy evaluation, while visual prediction is used for foresight-guided policy learning and multi-view data augmentation.Related work also includes point-cloud latent prediction, skeleton or kinematic conditioning, and spatial grounding.Source: §2 · p. 2
  • Video-derived latent actions support world modeling and policy learning, but reconstruction-based learning is sensitive to distracting dynamics and ambiguous action correspondence.Prior approaches use optical flow, shared RGB–depth features, or other representation and evaluation strategies to address related questions.Source: §2 · p. 2
  • Structured action representations include quantized skills, flow-matching policies, spatial and kinematic priors, and joint visual-action models.Related robot world models also learn over robot–object skeletons or cotrain visual and hand-skeleton dynamics; SkeLAM grounds video-derived kinematic latents with skeleton supervision.Source: §2 · p. 3

3 METHODOLOGY

SkeLAM learns latent actions with a kinematic component grounded in robot motion and a residual component retaining complementary visual dynamics. It then conditions a world model on these components and adapts robot actions to the kinematic pathway for action-conditioned generation.

3 METHODOLOGY

  • SkeLAM separates kinematic and residual latent components, then uses an action adapter and world-model fine-tuning to condition generation on robot controls.The kinematic component is grounded in robot motion, while the residual retains complementary visual dynamics.Source: §3 · p. 3, §3.2 · p. 5

3.1 SKELETON-GROUNDED LATENT ACTIONS

  • Cross-modal swap reconstruction replaces the RGB kinematic input with the paired skeleton posterior mean while retaining the RGB residual sample.Grounding also reconstructs the next skeleton from the RGB kinematic posterior mean through a frozen skeleton decoder and aligns it with the paired skeleton motion representation.Source: §3.1 · p. 4

3.2 ACTION-CONDITIONED WORLD MODELING

  • Separate conditioning projections and branch dropout train the world model with video-derived latents, including kinematic-only conditions that match action-conditioned inference.At inference, the residual latent and its mask are zero, so the current observation and actions specify the rollout.Source: §3.2 · pp. 5–6

4 EXPERIMENT

The experiments evaluate SkeLAM’s action-conditioned world modeling, motion representation, zero-shot motion transfer, and policy evaluation. On DROID, results assess motion accuracy and visual fidelity alongside downstream transfer and rollout agreement.

4.1 EXPERIMENTAL SETUP

  • 400 DROID clips provide the main evaluation, with training also incorporating AgiBot; reported metrics assess trajectory agreement and visual fidelity.Each clip contains one conditioning frame and 12 future frames; Cosmos-Predict2.5-2B is the world-model backbone.Source: §4.1 · p. 6

4.2 ACTION-CONDITIONED WORLD MODELING

  • SkeLAM ranks first on seven of eight reported action-conditioned metrics; versus DreamDojo, it reduces ADE by 19.5% and LPIPS by 11.1%.It also reduces ADE by 23.7% and LPIPS by 17.4% versus CD-LAM; Ctrl-World and OSCAR have higher direction F1.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Removing skeleton supervision, cross-modal swap, residual input, or action adaptation worsens at least one reported action-conditioned metric.Without skeleton supervision, ADE rises from 41.15 to 64.59; without swap, it rises to 66.74, while removing action adaptation raises it to 65.84.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 MOTION REPRESENTATION ANALYSIS

  • The 32D kinematic component approaches the 64D concatenated representation’s probing performance and outperforms the residual component on all three metrics.Concatenation raises EEF R2 from 0.3200 to 0.3277 and magnitude macro-F1 from 0.7467 to 0.7482.Source: §4.3 · p. 8, Table 3 · p. 9
  • Paired skeleton substitution raises reconstruction MAE by 0.53%, versus 11.67% for mismatched substitution; zeroing kinematics raises it by 5.78%, versus 0.61% for residuals.SSIM follows the same ordering for paired and mismatched substitutions.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 ZERO-SHOT HUMAN-TO-ROBOT MOTION TRANSFER

  • With model parameters fixed, human-video latent prompts from UniSkill produce distinct robot end-effector motions corresponding to the demonstrations.DreamDojo shows limited end-effector displacement in both examples.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 WORLD MODEL FOR ROBOT POLICY EVALUATION

  • Across seven policies and 105 episodes, SkeLAM has 33.33% success recall and 63.02% balanced accuracy, versus 0.00% and 48.44% for Cosmos-Predict2.5.Its policy success-rate MAE is 4.76 percentage points versus 7.62, with positive Spearman and Pearson correlations; success recall remains modest.Source: §4.5 · p. 9, Table 4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

SkeLAM structures latent actions around robot motion while retaining complementary visual dynamics. Experiments show improved motion accuracy and visual fidelity, qualitative human-to-robot transfer, and policy success-rate estimates consistent with real executions.

5 CONCLUSION

  • SkeLAM separates a motion-grounded kinematic component for robot action adaptation from a residual component retaining complementary visual dynamics.Cross-modal swap reconstruction and branch dropout support use of the learned kinematic pathway for action-conditioned generation.Source: §5 · p. 10
  • The authors identify closed-loop policy evaluation as future work, including how prediction errors affect later decisions and outcomes over longer horizons.The reported protocol uses recorded-action replay.Source: §5 · p. 10

A DATA AND PREPROCESSING

DROID and AgiBot World supply data for latent-action learning, world-model training, and action adaptation. RGB and skeleton observations are paired, while preprocessing and model initialization are specified across the training stages.

A DATA AND PREPROCESSING

  • DROID supplies 60,000 Stage 1 training clips and 45,358 clips for each Stage 2A, 2B, 3A, and 3B; AgiBot World is also used across training stages.DROID evaluation samples 400 trajectories; AgiBot evaluation uses 400 held-out clips and 800 arm trajectories.Source: A DATA AND PREPROCESSING · p. 15
  • RGB and skeleton observations are matched by episode, camera, and frame index, with overlapping episode windows kept in one partition and normalization fitted on training data only.Stage 1 letterboxes both modalities to 240×320; Stage 2/3 uses a 4:3 center crop resized to 480×640.Source: A DATA AND PREPROCESSING · p. 15
  • Each DROID window samples 13 frames at 15 FPS over 0.8 seconds, pairing consecutive observations with executed state-delta proxies.Translation is in the robot base frame, rotation is a relative SO(3) rotation vector, and gripper openness is the destination value; deltas are between consecutive sampled states.Source: A DATA AND PREPROCESSING · p. 15

B MODEL AND TRAINING DETAILS

  • The RGB and skeleton Transformer-VAE branches start from random weights, while the world model starts from official Cosmos-Predict2.5 2B weights; the Stage 1 teacher, video tokenizer, and text encoder remain frozen in Stage 2/3.Both branches use 16 × 16 patches, width 1,024, and 16 encoder and decoder blocks.Source: B MODEL AND TRAINING DETAILS · p. 15

B.1 LATENT-ACTION TRAINING

Latent-action training progressively adds skeleton grounding, paired-latent alignment, and cross-modal swap reconstruction to RGB and skeleton reconstruction. The stages use distinct loss combinations and selected checkpoints.

B.1 LATENT-ACTION TRAINING

  • Stages 1A/B/C progress from RGB and skeleton reconstruction to grounding and alignment, then add swap reconstruction; the skeleton branch is frozen in Stages 1B/C.Stage 1B minimizes RGB reconstruction, grounding, alignment, and variance losses; Stage 1C adds the weighted swap loss.Source: §B.1 · pp. 15–16
  • Swap reconstruction uses the frozen skeleton mean with the same residual sample as ordinary RGB reconstruction, and adds no KL term.It replaces the RGB kinematic latent with the paired skeleton counterpart during reconstruction.Source: §B.1 · p. 15
  • Cross-covariance uses centered RGB components and excludes the frozen skeleton latent, discouraging linear redundancy without enforcing independence.Stage 1C applies swap-loss weight 0.5 for 500 updates and 1.0 thereafter; 20% of updates use only RGB loss, including its KL terms.Source: §B.1 · p. 16
  • Stages 1B/C use AdamW with constant learning rates of 5 × 10^-5 and 2 × 10^-5, respectively; selected checkpoints are 20K and 5K updates.Configured limits are 40K/10K updates, and only model weights are restored at phase transitions.Source: §B.1 · p. 16

B.2 WORLD-MODEL TRAINING AND ACTION ADAPTATION

World-model training first fits conditioning and the full DiT, then learns an action adapter and fine-tunes generation with action-predicted kinematic latents. Branch dropout and a residual curriculum prepare conditioning for deployment.

B.2 WORLD-MODEL TRAINING AND ACTION ADAPTATION

  • Stage 2A performs 5K conditioning-projector updates, followed by 20K full-DiT and conditioning updates in Stage 2B.Stage 3A fits only the adapter for 2K updates with the world model fixed; Stage 3B performs 30K action-conditioned generation updates.Source: §B.2 · p. 16
  • Branch dropout masks standardized latents before projection, with a mode sampled per clip and shared across its 12 transitions.After global Stage 2 step 1,000, the probabilities for both branches, neither branch, residual only, and kinematic only are 0.50, 0.10, 0.05, and 0.35.Source: §B.2 · p. 16
  • The per-transition adapter maps 7-dimensional standardized actions to 32-dimensional RGB kinematic posterior means using an MLP with two 256-unit hidden layers.Its regression loss uses the valid-transition indicator.Source: §B.2 · pp. 16–17
  • Stage 3B always conditions on action-predicted kinematics while paired residuals are retained with probability 0.5 initially, decay to zero, and are then absent.DROID inference generates 12 future frames from one observed frame and 12 action transitions, always with zero residual and mask.Source: §B.2 · p. 17

C.1 VIDEO AND MOTION METRICS

The evaluation defines video-fidelity and end-effector-motion measures, then describes probes, interventions, human-transfer examples, and open-loop policy evaluation. The supplied passages specify evaluation procedures and scope rather than comparative outcome values.

C.1 VIDEO AND MOTION METRICS

  • Video-fidelity metrics include the conditioning frame, while motion metrics use the 12 future frames; LPIPS uses TorchMetrics AlexNet and tLPIPS measures adjacent-frame perceptual change.MSE, MAE, PSNR, SSIM, and LPIPS are averaged across frames and clips as specified.Source: §C.1 · p. 17
  • ADE requires valid arm trajectories with in-image reference positions and complete finite prediction tracking at every evaluated frame; invalid trajectories remain in the coverage denominator.Methods can therefore have different valid subsets, with generated-video and valid-video counts reported separately.Source: §C.1 · p. 17

C.2 BASELINES, PROBES, AND INTERVENTIONS

  • Interventions compare RGB kinematic, residual, and skeleton kinematic latents while holding decoder, current frame, and target fixed; they support motion specialization, not complete disentanglement.The tests include correct and wrong swaps and zeroed-kinematic or zeroed-residual inputs.Source: §C.2 · p. 18

C.3 HUMAN TRANSFER AND POLICY EVALUATION

  • Human-transfer examples assess visible motion correspondence rather than task completion or reliable transfer across arbitrary viewpoints and embodiments.Open-loop evaluation pairs seven policies across 15 sessions (105 episodes), with nine real successes and 96 failures; the authors note that these counts limit ranking evidence.Source: §C.3 · p. 18

D SUPPLEMENTARY RESULTS AND ABLATIONS

Supplementary results report metric-specific comparisons across DROID and AgiBot World, describe adaptation-budget trends, and outline the component ablations.

Additional metrics and coverage

Component ablations

  • Ablations vary skeleton supervision, swap reconstruction, residual input, and action adaptation; decoder interventions instead keep weights fixed.The removal variants change training, while the latter two retain the full model’s action interface and inference settings, including zero residual.Source: D SUPPLEMENTARY RESULTS AND ABLATIONS · p. 18

Adaptation budgets

  • DROID LPIPS decreases with adaptation updates, while motion metrics on both DROID and AgiBot World are nonmonotonic.Table 9 compares Stage 3B adaptation budgets; budgets exclude pretraining and the common 2K adapter fit, and inference uses zero residual.Source: Table 9 · p. 19

E ADDITIONAL QUALITATIVE COMPARISONS

Figures 8 and 9 add tabletop and kitchen DROID rollouts, comparing recorded observations with generated videos across four time steps and highlighting final-row end-effector motion.

Additional qualitative comparisons

Tabletop scene

Kitchen scene

Loading 2610.31474v1…