PaperlayerView exact revision on arXiv

Source-linked AI summary

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

Chunghyun Park, Beomjun Kim, Seungcheol Park, Heeseung Kwon, Yashu Shukla, Seunghoon Sim, Jinwoo Shin, Minsu Cho

arXiv:2610.02840v1cs.ROcs.CV

TL;DR

Existing WAMs often lack an explicit representation of 3D interaction geometry, which matters for dexterous manipulation. PointWAM jointly forecasts scene and hand trajectories as 3D points and retargets the hand forecast to robot actions. It improves DexJoCo success with human-video pre-training and scene-trajectory supervision, and surpasses the prior best multi-task result.

  • Problem

    Many WAMs represent the world as RGB frames or latent features, limiting their representation of 3D spatial structure and actor-environment relations central to dexterous manipulation.

    Source: §1 · p. 2

  • Method

    PointWAM jointly forecasts scene-point and hand-keypoint trajectories in a shared 3D space-time frame, then retargets forecast hand motion to robot actions.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    PointWAM raises average DexJoCo success from 12.1% to 69.0% and surpasses the strongest baseline by 11.7 percentage points.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Scene-trajectory supervision adds 10.9 percentage points over forecasting the hands alone, and human-video pre-training improves average DexJoCo success by 56.9 percentage points over training from scratch.

    Source: §1 · p. 2

Abstract

from arXiv · show

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.

1 INTRODUCTION

Existing world-action approaches often do not explicitly represent the 3D structure and interaction between actor and environment. PointWAM forecasts scene and hand trajectories together, and the paper reports gains from human-video pre-training and scene-trajectory supervision.

1 INTRODUCTION

  • Existing WAMs often represent the world as RGB frames or latent features, while point-based policies omit scene-motion forecasting and action-conditioned 3D world models do not predict actions.These approaches also differ in their ability to represent 3D structure and actor-environment relations, and point-based policies may require selecting task-relevant objects.Source: §1 · p. 2
  • Shared 3D modeling supports pre-training on human demonstrations without selecting task-specific objects or scene keypoints in advance.The paper also notes that decoupling hand-point trajectories from the scene makes forecast hand motion amenable to robot-hand retargeting.Source: §1 · p. 2
  • PointWAM jointly forecasts scene points and hand keypoints as 3D trajectories in a shared space-time frame, then retargets forecast hand motion to robot actions.The paper reports that scene-trajectory supervision improves average DexJoCo success by 10.9 percentage points over the same model without it.Source: §1 · p. 2
  • 11.7 percentage points: with human-video pre-training, PointWAM surpasses the prior best multi-task DexJoCo success rate, and it outperforms robot-pre-trained VLAs on a real robot by up to 25 percentage points.Pre-training on EgoDex and VITRA improves average DexJoCo success by 56.9 percentage points over training from scratch.Source: §1 · p. 2

2 RELATED WORK

Related work spans visual-representation WAMs, point-based action policies, and 3D scene-motion models. PointWAM combines scene and hand trajectory forecasting with action retargeting for dexterous hands.

2 RELATED WORK

  • Many WAMs forecast latent visual representations and predict end-effector poses, rather than explicitly modeling scene and hand motion as 3D trajectories.Conventional VLAs map visual observations directly to motor commands, whereas WAMs forecast world changes and predict actions.Source: §2 · p. 2
  • Unlike the closest 3D scene-motion models, PointWAM pre-trains on human demonstrations for dexterous hands, jointly forecasts scene and hand trajectories, and retargets hand trajectories to robot actions without a separate planner.EgoWAM targets parallel-jaw grippers; PointWorld requires a separate planner, while PointZero predicts no actions.Source: §2 · p. 3
  • Point-based policies can forecast hand keypoints from human videos, but may require selected objects and do not explicitly forecast scene motion; 3PoinTr forecasts scene tracks but uses a separate robot-trained action policy.PointWAM forecasts every observed scene point and hand keypoint with one backbone, then retargets hand trajectories to robot actions.Source: §2 · p. 3

3 POINT WORLD ACTION MODEL

PointWAM predicts scene and hand trajectories from points, keypoints, and an instruction, then maps forecast hand motion and robot state to actions. Its forecaster is trained on human and robot demonstrations, while the retargeter is trained on robot demonstrations.

3.1 PROBLEM STATEMENT AND OVERVIEW

  • The task is to predict a robot action chunk from an instruction, scene point cloud, and current robot state; scene and hand trajectories are expressed as displacements from the current time.Scene trajectories supervise world modeling, while hand trajectories provide an intermediate action representation.Source: §3.1 · pp. 3–4

3.2 3D TRAJECTORY FORECASTER

  • The forecaster encodes scene and hand features on a shared voxel grid, processes multimodal tokens, and uses two heads to forecast trajectories for scene points and hand keypoints.The scene head forecasts every observed point without task-specific target selection, and joint supervision encourages the backbone to model how scene and hand points co-evolve.Source: §3.2 · p. 4

3.3 ACTION RETARGETER

  • The action retargeter uses forecast hand trajectories and robot state to predict an action chunk, with a transformer decoder mapping its context and learnable queries to end-effector poses and finger-joint targets.Scene forecasts supervise the shared backbone but are not inputs to the retargeter.Source: §3.3 · p. 4

3.4 LEARNING FROM HUMAN AND ROBOT DEMONSTRATIONS

  • The forecaster is trained on human and robot demonstrations, whereas the action retargeter is trained only on robot demonstrations that contain the commands it predicts.Human-video hand and scene trajectories are estimated from annotations or tracks lifted to 3D; the resulting 1.15M episodes supervise the same forecaster. Training first pre-trains the backbone and trajectory heads on human data, then fine-tunes them and trains the retargeter on robot data.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments compare PointWAM with prior methods on simulated benchmarks and a real robot, then examine human-video pre-training, scaling, and model components. PointWAM leads on DexJoCo, while the component analyses show gains associated with 3D scene forecasting and representation choices.

4.1 EXPERIMENTAL SETUP

  • Pre-training uses 1.15M EgoDex and VITRA human episodes without robot actions, followed by fine-tuning on each benchmark’s robot demonstrations and real-robot data.Evaluation includes DexJoCo, RoboDojo-Precision, and two OpenArm tasks: single-arm banana pick-and-place and bimanual square-peg insertion.Source: §4.1 · p. 5, §4.1.1 · pp. 5–6

4.2 COMPARISON WITH PREVIOUS APPROACHES

  • 69.0% average SR across ten DexJoCo tasks; PointWAM leads GR00T N1.6 by 11.7 percentage points and succeeds on eight tasks.It also ranks first with one policy per task, with its margin over the strongest baseline widening from 5.3 to 22.2 percentage points under full randomization.Source: Table 1 · p. 6, §4.2 · p. 7

4.1.2 BASELINES

  • Comparisons include published DP-T and π0.5 results and same-protocol evaluations of GR00T N1.6, Fast-WAM, and PointACT; only Fast-WAM forecasts future scenes, as images.On the real robot, GR00T N1.6 and π0.5 are fine-tuned on the same data.Source: §4.1.2 · pp. 6–7

4.2 COMPARISON WITH PREVIOUS APPROACHES

  • 8.8 average score and 4.8% SR on RoboDojo-Precision; PointWAM exceeds GR00T N1.6’s 6.2 score and 2.3% SR.Fast-WAM and PointACT remain near zero, while GR00T leads on three of eight tasks; PointWAM’s human-hand pre-training shows initial transfer to grippers.Source: Table 2 · p. 7, §4.2 · p. 7
  • 58.3% SR on unseen-box banana PnP and 50.0% on square-peg insertion; PointWAM outperforms both baselines across the real-robot trials.The box resembles the table in color; PointWAM inserts in 12 of 21 trials, versus 4 of 19 and 6 of 18 for the baselines.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 HUMAN-DEMONSTRATION PRE-TRAINING AND SCALING

  • 60.1% to 69.0% average SR as human pre-training grows from 324K to 1.15M episodes; Volt-H reaches 69.0%, versus Volt-S at 52.3%.With EgoDex pre-training, PointWAM rises from 12.1% to 60.1%, while baseline changes are at most 1.4 percentage points.Source: Figure 7 · p. 8, §4.3 · pp. 8–9

4.4 COMPONENT ANALYSIS

  • 64.4% SR with Mosaic3D and scene-trajectory supervision exceeds lifted image features (59.0%), no world modeling (53.5%), and 2D latent forecasting (60.1%).The 3D trajectory model outperforms forecasting future image latents from image features.Source: Table 3 · p. 9, §4.4 · p. 9
  • 64.4% SR with anatomical keypoints and 1 cm voxels exceeds sampled hand surfaces (43.6% and 44.6%) and 0.5 cm and 2 cm voxels (59.9% and 52.2%).The authors use 1 cm voxels in all experiments.Source: Table 3 · p. 9, §4.4 · p. 9

5 CONCLUSION

PointWAM combines human-video pre-training with joint scene-and-hand trajectory forecasting, improving DexJoCo success and outperforming the strongest baseline. The related-work discussion distinguishes its shared 3D trajectory forecasts from methods that forecast other futures, use geometry only as input, or select scene points externally.

5 CONCLUSION

  • 69.0% average DexJoCo success, versus 12.1% without human-video pre-training, and 11.7 percentage points above the strongest baseline; scene forecasting adds 10.9 points over hands alone.The conclusion also notes that PointWAM surpasses the prior state of the art on ten DexJoCo tasks.Source: §5 · p. 9
  • PointWAM’s point-only view can miss details finer than point spacing and surfaces depth sensing cannot capture, including printed text and transparent objects.The conclusion proposes richer point appearance and physical properties such as mass and friction as future work.Source: §5 · p. 9
  • The authors conclude that shared 3D point trajectories let robots learn dexterous manipulation largely from human videos.Human videos provide trajectories without selecting task-specific points; the forecast hand motion is retargeted to robot actions.Source: §5 · p. 9

A EXTENDED RELATED WORK

  • Unlike methods forecasting geometric, image, or latent futures, PointWAM represents scene and hand motion with the same 3D point trajectories and retargets only the hands to actions.Dexterous World Model conditions interaction video on supplied hand trajectories rather than forecasting them for robot control.Source: A EXTENDED RELATED WORK · p. 14
  • Prior 3D-policy methods use geometry as input or auxiliary features, whereas PointWAM supervises its backbone with future scene and hand trajectories from human videos without robot actions.PointACT admits pre-trained point features only in its action expert; CordViP adds contact and hand–arm correspondence objectives.Source: A EXTENDED RELATED WORK · p. 14
  • Compared with video-to-robot methods that transfer hand poses, poses, or image-space tracks, PointWAM supervises every observed scene point and the hand without externally selected scene points.General Flow, MolmoMotion, and µ0 supervise selected query points; their selection is decided outside the model.Source: A EXTENDED RELATED WORK · p. 14

B IMPLEMENTATION DETAILS

The implementation converts human and robot demonstrations into aligned point trajectories, tokenizes point-cloud inputs, and trains a trajectory forecaster with an action retargeter. Preprocessing, normalization, and loss weighting define the data and supervision used by these components.

B.1 HUMAN-DEMONSTRATION PREPROCESSING

  • Human hand poses are mapped to ten robot-hand keypoints in a z-up frame; the little finger is omitted because the Allegro hand has no counterpart.EgoDex annotates 25 joints per hand and VITRA reconstructs 21; absent hands are marked invalid rather than imputed.Source: §B.1 · p. 15
  • Human clips contain 31 resampled frames over 1 s, while scene tracks are lifted from image tracks using scaled depth and retained only for valid first-frame depth.Clips start every 0.5 s; tracks are deduplicated on a 1.5 cm grid, capped at 12,000 points, and correspond only within each clip.Source: §B.1 · p. 15

B.2 ROBOT-DEMONSTRATION PREPROCESSING

  • DexJoCo scene trajectories are recovered by replaying demonstrations and assigning non-robot points to rigid objects or parts; robot hands use ten keypoints, including fingertips on contact pads.For RoboDojo-Precision, gripper-specific points and fused camera depth are used; absent object poses and segmentation mean fine-tuning supervises only hand keypoints and actions, with the scene head frozen.Source: §B.2 · pp. 15–16

B.3 3D TRAJECTORY FORECASTER

  • Each 5 cm voxel patch token concatenates stem features in fixed spatial order, preserving within-patch voxel positions that pooling would discard.Occupied voxels are processed by a sparse-convolutional stem before projection to the token width.Source: §B.3 · p. 16
  • The Volt-H backbone uses projected instruction tokens and axial 3D rotary embeddings for vision tokens; decoded features are fused with per-point stem features before scene and hand heads forecast trajectories.The backbone uses bidirectional attention within each observation, and only output vision tokens are decoded.Source: §B.3 · p. 16

B.4 ACTION RETARGETER

  • The retargeter predicts the action chunk in parallel from unmasked action-query self-attention, then maps robot demonstrations to actions without explicitly enforcing joint limits or collision constraints.Its recorded-action targets already respect joint limits and avoid collisions because a robot executed them.Source: §B.4 · p. 16

B.5 NORMALIZATION AND LOSS WEIGHTING

  • Scene and hand displacements are standardized separately using pre-training statistics, while robot action targets are expressed relative to robot state and standardized per horizon step.Scene displacement coordinates are clipped to [−0.5, 0.5] m before standardization; the action map is inverted at inference.Source: §B.5 · pp. 16–17
  • Hand loss averages over valid keypoint-step pairs, while scene weights exclude invalid targets and softly emphasize motion between adjacent frames; all loss coefficients equal one.Static scene points retain a small positive weight, and each squared error is normalized to a per-coordinate mean.Source: §B.5 · p. 17

C.1 TRAINING SETTINGS

The main model is pre-trained on human demonstrations and fine-tuned on DexJoCo, while ablations use a smaller backbone and the scaling study varies backbone size. The largest configuration is Volt-H.

C.1 TRAINING SETTINGS

  • The Volt-H multi-task model is pre-trained for 60k iterations on 1,148,324 EgoDex and VITRA episodes, then fine-tuned for 60k iterations on DexJoCo with frozen 3D and text encoders.Source: §C.1 · p. 17
  • Component ablations use Volt-L12 with 175M trained parameters, pre-training on EgoDex for 40k iterations and fine-tuning for 60k; every variant changes both stages.The reference checkpoint is the 35k pre-training snapshot with the lowest held-out hand-trajectory error.Source: §C.1 · p. 17
  • The scaling study varies backbone width and depth across Volt-S, Volt-B, Volt-L, and Volt-H, whose fine-tuning parameter counts are 39M, 106M, 326M, and 655M, respectively.All configurations share frozen encoders with 592M parameters; Table 1 uses Volt-H.Source: §C.1 · p. 18

C.2 DEXJOCO EVALUATION PROTOCOL

DexJoCo inference runs asynchronously and uses snapshot selection based on the first seed’s ten-task average. The evaluation setup makes inference latency relevant to executed behavior.

C.2 DEXJOCO EVALUATION PROTOCOL

  • 55 ms per forward pass for the Volt-L12 reference, including 37 ms in the frozen encoder, exceeds one 20 ms simulator control step.The evaluator requests a new chunk when fewer than 24 of 30 actions remain and holds the robot pose if the buffer empties.Source: §C.2 · p. 18
  • A 30-step chunk covers 0.6 s in DexJoCo; the corresponding 31-frame human clips span 1 s, so predicted-step durations differ between pre-training and fine-tuning.A predicted step spans about 33 ms in pre-training and 20 ms in fine-tuning.Source: §C.2 · p. 18
  • The Volt-H multi-task checkpoint is selected among 20k, 40k, and 60k fine-tuning snapshots using the first seed’s ten-task average, which selects 60k.All component ablations use 60k; single-task policies use their final snapshot after 20k iterations, or 30k for bimanual hanoi and microwave.Source: §C.2 · p. 18

C.3 ROBODOJO-PRECISION EVALUATION PROTOCOL

RoboDojo-Precision evaluation uses a synchronous official evaluator and fixed seed-0 layouts. Results average success and partial progress equally across eight tasks, with one training seed and one evaluation seed per method.

C.3 ROBODOJO-PRECISION EVALUATION PROTOCOL

  • 50 episodes per task use the seed-0 layouts in order, with episodes ending at success or task-specific step limits.Limits range from 300 steps for insert key and deposit coin to 1,900 for fasten screws (12–76 seconds at 25 Hz).Source: §C.3 · p. 18
  • The evaluator runs synchronously, executing each full action chunk before the next query, so inference latency does not affect outcomes.Source: §C.3 · p. 18
  • Success rate and partial-progress score are averaged equally across the eight tasks.Source: §C.3 · p. 18
  • Each method uses one training seed and one evaluation seed; PointWAM is evaluated at its 60k fine-tuning snapshot without further selection.The published benchmark instead trains three seeds per policy and evaluates each for 50 trials; public leaderboard entries are excluded because they train on all RoboDojo tasks.Source: §C.3 · p. 18

C.4 REAL-WORLD EXPERIMENTS DETAILS

The real-robot experiments test pick-and-place under ambiguous color cues and precise bimanual peg insertion. The appendix also specifies fine-tuning and human-video pre-training setups for GR00T N1.6, Fast-WAM, and PointACT.

C.4 REAL-WORLD EXPERIMENTS DETAILS

  • Banana pick-and-place uses 54 demonstrations and 24 trials per method, including 12 trials with an unseen box whose color resembles the tabletop.Success requires the object to lie entirely inside the basket at episode end.Source: §C.4 · p. 18
  • Square-peg insertion tests precise grasping and bimanual coordination across 24 trials; success requires the peg to remain in the hole at episode end.The 0–3 partial-progress score distinguishes lifting the peg, lifting both objects, and insertion success.Source: §C.4 · p. 18

C.5.1 GR00T N1.6

  • GR00T N1.6 is fine-tuned from its released 3B checkpoint, while EgoDex pre-training regresses each next hand state as the action.The stated setup uses separate task policies for DexJoCo single-task training and one policy across tasks for multi-task DexJoCo and RoboDojo.Source: §C.5.1 · p. 19

C.5.2 FAST-WAM

  • Fast-WAM uses the pre-trained Wan2.2-5B model for 100K training steps, with three camera views for benchmark fine-tuning.Its EgoDex pre-training uses single-view videos with the same hand-state actions and budget as GR00T N1.6.Source: §C.5.2 · p. 19

C.5.3 POINTACT

  • PointACT keeps its Qwen2.5-VL-3B-Instruct backbone frozen and initializes its point encoder from a pretrained Concerto checkpoint.It is pre-trained on EgoDex with the same hand-state actions and budget as GR00T N1.6.Source: §C.5.3 · p. 19

D.1 SINGLE-TASK DEXJOCO RESULTS

PointWAM leads the single-task DexJoCo results under both object placement and full randomization, with a larger margin under full randomization. Additional ablations show trade-offs from patch size and encoder fine-tuning.

D.1 SINGLE-TASK DEXJOCO RESULTS

  • In object placement, PointWAM averages 61.8%, 5.3 points above π0.5 at 56.5%, and is best on five of ten tasks.It trails other methods on several tasks, reaching 24.0% on bimanual photograph.Source: §D.1 · p. 19, Table 5 · p. 20
  • Under full randomization, PointWAM averages 62.9% and is best on all ten tasks, with a 22.2-point margin over the strongest baseline.Every baseline loses 9.4 to 32.8 percentage points; randomization varies table height, camera pose, lighting, and texture.Source: §D.1 · p. 19, Table 6 · p. 20

D.2 PATCH SIZE AND ENCODER FINE-TUNING

  • With 1 cm voxels, reducing Kv from 5 to 3 raises SR from 64.4% to 67.1% but costs 1.63× training time and 1.66× peak memory; Kv = 7 yields 44.5%.Fine-tuning the Mosaic3D encoder lowers SR from 64.4% to 54.9%, so it remains frozen in the other experiments.Source: Table 7 · p. 20, §D.2 · p. 21

E ADDITIONAL QUALITATIVE RESULTS

The qualitative results show selected successful and failed rollouts across DexJoCo, RoboDojo-Precision, and real-robot tasks. DexJoCo failures often occur after a grasp, when the rollout stops short of the final task step.

E ADDITIONAL QUALITATIVE RESULTS

  • DexJoCo figures show the first success and first failure among eight evaluation-seed rollouts per task; water plant has no failure row because all 16 rollouts succeeded.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 21, Figure 9 · p. 22, Figure 10 · p. 23
  • Many DexJoCo failures grasp the object but stop before the final step, such as holding a hammer above a nail or a peg beside its hole.Other examples include glasses next to the box and a camera held without pressing the shutter.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 21
  • RoboDojo-Precision shows one successful rollout for each task PointWAM solves at least once and one failure for each of the other three tasks.Source: Figure 11 · p. 24
  • The real-robot figure compares GR00T N1.6, π0.5, and PointWAM on two tasks, with one banana trial each for an in-domain banana and an unseen box.Source: Figure 12 · p. 25
Loading 2610.02840v1…