PaperlayerView exact revision on arXiv

Source-linked AI summary

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song

arXiv:2610.22363v1cs.CVcs.RO

TL;DR

Egocentric manipulation methods have largely learned 3D reconstruction, action-conditioned prediction, and goal-directed planning separately. P3-WAM trains these modes in one video–action model, and unified training improves the three capabilities, with reported gains over specialized baselines in reconstruction and prediction and stronger planning scores.

  • Problem

    Egocentric reconstruction, action-conditioned prediction, and world-action planning have developed largely separately, despite their shared reliance on actions and their consequences.

    Source: §1 · p. 2

  • Method

    P3-WAM unifies video-based 3D hand-and-camera reconstruction, action-conditioned future prediction, and goal-directed joint action–future planning in one model.

    Source: §1 · p. 2

  • Results

    Unified training benefits reconstruction, action-conditioned video prediction, and planning, with better baseline comparisons in reconstruction and prediction and higher VLM scores for planning process and instruction adherence.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The experiments indicate that a shared model can connect perception, prediction, and planning for egocentric dexterous manipulation, with robot demonstrations illustrating transfer of generated human-hand motion.

    Source: §1 · p. 2, §5 · p. 9

    Monocular video leaves global 3D hand position inherently ambiguous, and the robot experiments rely on open-loop execution.

    Source: §5 · p. 9

Abstract

from arXiv · show

We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/

1 INTRODUCTION

Egocentric manipulation research treats reconstruction, action-conditioned prediction, and goal-directed planning as related but largely separate capabilities. P3-WAM unifies these modes in one model, using randomized modality visibility to train them together and reporting gains over single-mode training and evaluated baselines.

1 INTRODUCTION

  • Prior egocentric manipulation work develops 3D interaction reconstruction, action-conditioned world prediction, and joint action–future planning largely in isolation or only partially coupled.These capabilities recover hand and camera motion, predict outcomes under prescribed trajectories, and jointly generate actions and future observations for goal-directed control.Source: §1 · p. 2
  • P3-WAM unifies reconstruction, action-conditioned prediction, and joint action–future planning for egocentric dexterous manipulation.The modes recover 3D motion from observed interactions, anticipate visual consequences of prescribed actions, and jointly generate goal-directed hand trajectories and visual futures.Source: §1 · p. 2
  • The model uses a pretrained video diffusion transformer, structured hand-and-camera action tokens, a mixture-of-transformers architecture, and randomized visibility of video and actions.The visibility patterns make modalities context or prediction targets for perception, prediction, and planning; successive visual subgoals specify target arrangements across long sequences.Source: §1 · p. 2
  • Unified training improves all three modes over single-mode training, while P3-WAM outperforms evaluated baselines in perception and prediction and earns higher VLM scores for planning process and instruction adherence.Robot demonstrations illustrate transfer of generated human-hand motion to a dexterous robot.Source: §1 · p. 2

2 RELATED WORK

Related work spans action-conditioned video prediction, egocentric hand reconstruction, and shared multimodal models. P3-WAM connects these lines by jointly modeling hand interaction, prediction, and planning with visual goals.

Video World Models and World Action Models

  • Action-conditioned world models predict images or video, while world action models generate actions alongside anticipated visual dynamics; several approaches use pretrained video priors or explicit scene and camera geometry.The cited approaches include causal video prediction with inverse dynamics, joint video–action flow matching, and predictive visual information for control without a complete iterative rollout at deployment.Source: §2 · pp. 2–3

Monocular Hand Reconstruction

  • Egocentric hand reconstruction combines hand-motion estimation with camera-motion recovery, while video-diffusion methods also use generative features for hand-motion recovery.P3-WAM trains reconstruction jointly with future prediction and planning; the passage reports improved reconstruction in its single-mode comparison.Source: §2 · p. 3

Unified Multimodal Models

  • Shared multimodal models connect tasks by varying which modalities are observed and generated, and some extend this formulation with language, geometric inputs, or visual goals.P3-WAM applies joint reconstruction, prediction, and planning to articulated hand interaction, planning hand and camera trajectories with video and supporting visual goals.Source: §2 · p. 3

3 METHOD

P3-WAM defines perception, prediction, and planning as different observation-and-generation patterns over video and hand-and-camera actions. Its coupled video–action model uses structured action representations, shared transformer weights, and visibility-aware training to support these modes and visual-goal-conditioned inference.

3.1 UNIFIED TASK FORMULATION

  • PERCEIVE predicts hand and camera motion from observed video, PREDICT generates future video from hand and camera trajectories, and PLAN generates video and trajectories guided by instruction and an optional visual goal.The task formulation defines the action sequence from hand states and camera states, with camera rotation and translation expressed relative to the first frame.Source: §3.1 · pp. 3–4

3.2 JOINT VIDEO–ACTION MODELING

  • P3-WAM couples RGB and action token streams through shared transformer base projections and bidirectional attention, with action-specific low-rank updates and normalization.Observed hand skeletons and camera-ray maps provide spatially aligned controls, while compact action tokens carry corresponding states into joint attention.Source: §3.2 · p. 4

3.3 ACTION REPRESENTATION

  • The action branch represents each frame with camera, left-hand, and right-hand tokens, and predicts missing hand and camera states at learned query positions in one forward pass.Hand outputs include articulated pose and shape, camera-space root position, and presence; the MANO layer reconstructs camera-space joints.Source: §3.3 · pp. 4–5

3.4 OPTIMIZATION

  • Video flow-matching loss is restricted to unobserved video latents, while the total objective adds action losses for parameters, spatial and temporal motion, presence, and camera states.The video-loss passage defines the unobserved latent set and diffusion training weight.Source: §3.4 · p. 5

3.5 INFERENCE

  • At inference, PERCEIVE predicts actions in one pass, whereas PREDICT and PLAN iteratively denoise unobserved RGB latents; PLAN retains the generated video and final-step action trajectory.For long tasks, each planning window uses a starting observation, instruction, and goal image; successive visual subgoals specify target arrangements, and the cited example without goal images accumulates placement errors.Source: §3.5 · p. 5

4 EXPERIMENTS

Across matched training conditions and specialized-baseline comparisons, P3-WAM shows gains in perception, prediction, and planning; qualitative examples and robot demonstrations illustrate generated motion and execution. The conclusion notes that monocular global hand position is ambiguous and robot experiments use open-loop execution.

4.1 DOES UNIFIED TRAINING BENEFIT EACH MODE?

  • Under matched evaluations, unified training improves PERCEIVE and PREDICT versus single-mode training, and improves PLAN VLM scores across all five dimensions versus PLAN-only training.PERCEIVE camera-space, root-relative, and PA errors fall by 23.46%, 14.23%, and 5.69%; PREDICT PSNR rises from 17.356 to 18.138 dB and FVD falls from 1105.95 to 1009.84. PLAN scores improve most in temporal/identity consistency (2.15 to 3.57) and goal transition (1.98 to 3.29).Source: §4.1 · p. 6

4.2 PERCEIVE EVALUATION

  • P3-WAM achieves the lowest presence-aware reconstruction error on all four datasets, with average camera-space MPJPE-p of 63.05 mm versus 145.34 mm for HaMeR and 231.54 mm for Dyn-HaMR.Root-relative error is 22.54 mm versus 46.61 mm for HaMeR and 45.25 mm for Dyn-HaMR; PA error is 11.79 mm versus 16.75 and 32.94 mm, respectively. It also has the highest hand-visibility F1 on all four datasets.Source: §4.2 · p. 6, Table 3 · p. 7
  • Qualitative PERCEIVE comparisons show more consistent hand placement and articulation, while baselines miss or misplace hands and confuse left and right hands.The authors attribute this consistency to a generative prior over plausible motions combined with full-video context, which helps resolve ambiguity across frames.Source: §4.2 · p. 6, Figure 5 · p. 7

4.3 PREDICT EVALUATION

  • P3-WAM achieves the best full-frame results on all four datasets, with average PSNR of 19.020 dB versus 15.927 and FVD of 621.73 versus 1217.56 for Wan2.2-Fun-Control.Hand-region LPIPS decreases from 0.4852 to 0.4099 and hand-box IoU increases from 0.6178 to 0.6709; P3-WAM leads on all five hand-region metrics in EgoDex, HOT3D, and EgoLive.Source: §4.3 · p. 8, Table 4 · p. 8
  • In qualitative PREDICT comparisons, P3-WAM preserves manipulated objects while following prescribed hand–camera motion, whereas baselines show object, hand, or scene changes.The comparisons suggest explicit hand and camera conditioning helps preserve the scene as the interaction progresses.Source: Figure 6 · p. 7, §4.3 · p. 8

4.4 PLAN EVALUATION

  • PLAN videos score higher than all four baseline settings in action process, instruction adherence, physical interaction, and goal transition under a method-blind VLM evaluation.The rubric scores five dimensions on a 1–5 scale and assesses both the intermediate action sequence and progress toward the goal.Source: §4.4 · p. 8, Table 5 · p. 9
  • In goal-conditioned examples, P3-WAM reaches the plate target before supplied goal frames and completes the jar task, while baselines leave the plate misplaced or fail to finish the lid-and-scoop actions.At t = 77, the final generated frame before supplied goal frames at t = 78–81, both baselines leave the plate near the mat’s middle.Source: §4.4 · p. 8, Figure 7 · p. 9
  • Robot demonstrations retarget PLAN-generated human-hand motion to dexterous robot commands and execute them on a physical robot; the experiments remain open-loop.The paper notes that monocular video makes global 3D hand position inherently ambiguous and suggests tactile or depth sensing could provide stronger geometric constraints.Source: §4.4 · p. 9, Figure 8 · p. 9, §5 · p. 9

A ADDITIONAL METHOD DETAILS

The method adapts a shared video transformer for visual and action tokens, with masking that lets the model observe or predict either modality. Action objectives supervise hand, camera, and temporal motion outputs.

A.1 SHARED WEIGHTS AND MODALITY-SPECIFIC ADAPTATION

  • The RGB and action branches share base attention and FFN weights, while the action branch adds low-rank updates and separate normalization parameters.The high- and low-noise experts have separate base weights and action adaptations; the video VAE, text encoder, and action codec are frozen.Source: §A.1 · p. 14

A.2 ATTENTION AND TOKEN ENCODING

  • Self-attention queries attend to a common set of RGB and action keys and values, with role, handedness, and frame embeddings distinguishing action tokens.Unavailable actions use learned query tokens, while observed RGB tokens remain clean and receive zero-noise timesteps.Source: §A.2 · p. 14

A.3 MASKED INPUT CONSTRUCTION

  • Visibility masks define PERCEIVE, PREDICT, and PLAN inputs, while temporal sampling mixes whole-clip modes with independently sampled modes across latent positions.Observed video latents stay clean; unavailable actions use learned queries, and video flow matching supervises unobserved RGB positions.Source: §A.3 · p. 14

A.4 ACTION OBJECTIVES

  • Action objectives supervise MANO parameters, 3D and image-plane joints, temporal hand motion, hand presence, and camera rotation, translation, and field of view.Camera poses are relative to the first frame; hand losses include rotation geodesic distance and smooth-L1 penalties averaged over valid annotations.Source: §A.4 · p. 15

B ACTION CODEC DETAILS

The action codec compresses camera and hand states into three tokens per frame and is frozen during video–action training. Its held-out reconstruction errors are reported across four datasets.

Codec architecture and training

  • Each frame is represented by three 64-dimensional tokens for camera, left hand, and right hand, using separate camera and shared hand encoders.The codec is trained with masked reconstruction losses for joints, image coordinates, depth, visibility, and camera states, then frozen for video–action training.Source: B ACTION CODEC DETAILS · p. 15

Held-out reconstruction

  • 0.868–2.600 mm camera-space joint errors and 0.323–0.882 pixel image-space errors are achieved by the multi-domain codec on held-out validation data.The reported measurements characterize compression of annotated actions before video-model prediction.Source: B ACTION CODEC DETAILS · p. 15, Table 6 · p. 16

C IMPLEMENTATION AND TRAINING DETAILS

The final training recipe fine-tunes a pretrained video backbone with action adaptations and samples across the three task modes. Evaluation uses aligned source observations and specified image and hand-region metrics.

C IMPLEMENTATION AND TRAINING DETAILS

  • The final unified recipe trains the backbone transformer, camera adapter, action adaptations, and MANO head on 16 H100 GPUs for approximately 450 hours.Training clips contain 81 frames at 16 fps and 848 × 480 resolution; the video VAE, text encoder, and pretrained action codec remain frozen.Source: C IMPLEMENTATION AND TRAINING DETAILS · p. 16
  • Unified training assigns 20%, 50%, and 30% of latent positions to PERCEIVE, PREDICT, and PLAN in expectation, using whole-clip or independently sampled modes.Single-mode and unified ablations use the same architecture, initialization, data mixture, and total optimizer updates, but not equal exposure to each complete mode.Source: C IMPLEMENTATION AND TRAINING DETAILS · p. 16

D DATASET AND EVALUATION PROTOCOLS

  • Training uses EgoDex, EgoLive, and ARCTIC; evaluation methods receive temporally aligned source observations and dataset means are macro-averaged with equal weights.The evaluation protocol specifies detection, joint association, and image metrics, with PLAN protocol and statistical analysis detailed separately.Source: D DATASET AND EVALUATION PROTOCOLS · p. 16

D.1 PREDICT CONDITIONING

  • PREDICT baselines receive different native motion controls, while image and hand-region evaluation uses paired CLIP-I, FVD, crop metrics, and IoU.Hand-region crops use WiLoR detections; the Macro score averages four dataset-level scores equally, and FVD is computed separately by dataset and method.Source: §D.1 · p. 16, §D.2 · p. 17

D.3 PERCEIVE TARGETS AND METRICS

PERCEIVE evaluation defines eligible hands, matching, and penalties for reconstruction metrics across four datasets. PLAN can condition on an initial image and instruction, with an optional fixed goal-image latent.

D.3 PERCEIVE TARGETS AND METRICS

  • PERCEIVE reports presence-aware MPJPE-p in camera-space, root-relative, and Procrustes-aligned coordinates, with eligibility and missing-hand penalties specified for comparison.Targets come from fitted MANO joints for EgoDex-MANO and provided annotations for ARCTIC, HOT3D, and EgoLive; dataset means are equally weighted.Source: §D.3 · p. 17

D.4 GOAL-IMAGE CONDITIONING

  • Both PLAN settings use the instruction and initial image; the default additionally fixes a terminal latent encoded from repeated copies of the goal image.Hand and camera trajectories are generated for all frames; frames 2–77 are freely generated between the observed initial and terminal segments, and the VLM evaluates the complete video.Source: §D.4 · p. 18, Figure 9 · p. 18

E HUMAN AND VLM EVALUATION

Table 7 reports the PLAN-only and unified-training human and VLM comparison, including 95% confidence intervals for gains.

E HUMAN AND VLM EVALUATION

  • Table 7 gives the complete human and VLM comparison, while Table 2 shows the VLM means.The reported unified-training gains include 95% confidence intervals.Source: E HUMAN AND VLM EVALUATION · p. 18

E.1 HUMAN EVALUATION

The evaluation combines human and VLM ratings of planning videos, comparisons with video-generation baselines, and real-world robot trials.

E.1 HUMAN EVALUATION

  • Human scores used 21 complete pairs across 15 tasks, while VLM scores used 188 pairs across 17 tasks.Each model generated 188 videos across 17 task categories for the human study; the VLM agreement analysis reports correlations across five dimensions.Source: §E.1 · pp. 18–19, Table 7 · p. 19

E.2 SCORING RUBRIC

  • The shared five-dimension rubric scores the manipulation sequence and progress toward the goal, including required intermediate actions, visible contact, and hand–object consistency.Action process and instruction adherence are the primary criteria.Source: §E.2 · p. 19

E.3 VLM EVALUATION

  • The method-blind VLM scores one anonymous video at a time, using 22 sampled frames and marking insufficient visual evidence as unobservable.The frame sample includes 16 uniformly spaced frames and the final eight consecutive frames, deduplicated.Source: §E.3 · p. 19

E.4 COMPARISON WITH VIDEO-GENERATION BASELINES

  • Ours shows consistent gains in action process, instruction adherence, physical interaction, and goal transition against every baseline; action-process and instruction-adherence intervals are positive in all four comparisons.The four-dimension lead remains when the two conditions with ambiguous instructions or reference images are excluded.Source: §E.4 · p. 20, Table 9 · p. 20

E.5 REAL-WORLD EXECUTION EVALUATION

  • Two real-world experiments test whether generated interactions and action ordering support robot execution by replaying retargeted trajectories without further modification.The evaluation checks reproduction of the intended manipulation process and the specified task outcome.Source: §E.5 · pp. 20–21

E.6 QUESTIONNAIRE INTERFACE

  • The questionnaire interface presents the video and five rating controls.Source: §E.6 · p. 21

F.1 PERCEIVE JOINT ERRORS AND HAND VISIBILITY

Additional evaluations examine reconstruction and visibility, consistency between planning actions and video, action guidance, backbone comparisons, and qualitative results.

F.1 PERCEIVE JOINT ERRORS AND HAND VISIBILITY

  • 47.77 mm mean camera-space error versus HaMeR’s 95.17 mm, and 99.17% visibility F1 versus 92.14% for HaMeR and 89.95% for Dyn-HaMR.Joint errors are reported without missing-hand penalties, while visibility metrics are scored separately.Source: §F.1 · p. 21, Table 10 · p. 21

F.2 RECONSTRUCTION AGAINST ORIGINAL EGODEX JOINTS

  • 30.07 mm camera-space and 19.02 mm root-relative errors beat Dyn-HaMR’s 114.58 and 27.45 mm under presence-aware scoring; PA error is 10.98 mm versus HaMeR’s 13.70 mm.The model has the lowest error in all three coordinate settings, with and without presence penalties.Source: §F.2 · p. 24, Table 11 · p. 24

F.3 PLAN VIDEO–ACTION CONSISTENCY

  • Camera-space discrepancies are 4.79 mm for EgoDex and 4.99 mm for generated images; wrist-relative discrepancies are 2.31 and 2.76 mm, respectively.The results remain similar over frames 2–77, before the goal-conditioned terminal interval.Source: §F.3 · p. 24, Table 12 · p. 24

F.4 ACTION-GUIDANCE SENSITIVITY

  • 20.644→25.327 dB full-image PSNR and 16.764→23.724 dB hand PSNR accompany γ increasing from 0 to 1; at γ=1.5, hand PSNR reaches 23.899 dB.The additional guidance gain is 0.175 dB; the full-image PSNR at γ=1.5 is 25.271 dB.Source: §F.4 · p. 24

F.5 BACKBONE COMPARISON WITH EGOSIM

  • 18.020 dB PSNR, 0.6319 SSIM, and 0.4251 LPIPS beat EgoSim’s 15.384 dB, 0.5615, and 0.4974 with the matched Wan2.1 backbone.The Wan2.2 model reaches 19.020 dB PSNR, 0.6547 SSIM, and 0.3891 LPIPS.Source: §F.5 · p. 25, Table 14 · p. 25

G ADDITIONAL QUALITATIVE RESULTS

  • Qualitative examples span generated videos and reconstructed hands across four datasets, goal-conditioned planning, successive visual targets for longer tasks, and physical execution.For goal-conditioned examples, motion before the terminal observed interval shows the generated transition.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 26

G.1 SINGLE-MODE VERSUS UNIFIED TRAINING

  • Figure 12 qualitatively compares single-mode and unified training in hand placement, scene perspective, and object arrangement across PERCEIVE, PREDICT, and PLAN.The examples show the unified model placing the reconstructed palm closer to the observed hand, matching table extent and viewpoint more closely, and positioning the narrow support beside the fixed broad post.Source: §G.1 · p. 26, Figure 12 · p. 26

G.2 GOAL-CONDITIONING ABLATION AND BASELINE COMPARISONS

The section contrasts goal-conditioned planning with no-goal behavior and states the scope and ethical boundaries for use.

G.2 GOAL-CONDITIONING ABLATION AND BASELINE COMPARISONS

  • Without a goal image, the model maintains support shape and puzzle-piece identity in examples; visual goals specify arrangements beyond what the instruction conveys.The multiwindow rollout comparison uses prescribed reference starts and guidance 2 for goal-conditioned windows, versus generated starts and text guidance 6 without goals, so it is separate from a matched condition ablation.Source: §G.2 · p. 27

H.1 SOURCE ASSETS AND INTENDED USE

  • Benchmark coverage reflects source-dataset activities, cameras, hands, and environments; broader populations and embodiments require additional evaluation.Source: §H.1 · p. 28

H.2 ETHICAL CONSIDERATIONS

  • Motion generated from visual goals requires embodiment-specific validation and supervised execution before physical-robot use.Egocentric data use and sharing should follow source-dataset access and consent terms, and generated interaction videos should be labeled as synthetic.Source: §H.2 · p. 28

Related papers

Loading 2610.22363v1…