DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models

• Action-conditioned video can look plausible while predicting an incorrect object pose or letting geometry and surface features drift through occlusion.Small errors in pose or local geometry can break hand-object contact relationships.Source: §1 · p. 2
• Metric 3D prediction makes physical evolution measurable before rendering, but generic point trajectories leave hand and object structure implicit.Hand joints have a kinematic hierarchy, while object points share coherent motion with local articulation or deformation.Source: §1 · p. 2
• DeXplicit predicts object-centric motion and deformation in metric 3D before video synthesis, using the predicted state as a scaffold for rendering.Appearance is propagated through predicted 3D motion and projected into future views, leaving uncertain or newly visible content for the renderer to complete.Source: §1 · p. 2
• 39.9%, 28.0%, and 27.1% lower long-horizon moved-point error than PointWorld on HOT3D, OakInk2, and PhysTwin, respectively.Under the same renderer, HOT3D object IoU+ rises from 0.373 with PointWorld predictions to 0.549 with DeXplicit predictions.Source: §1 · p. 2

• World models link visual prediction to embodied control, while robot-interaction models use action-conditioned prediction and scalable video priors.Examples include UniPi and GR-1 connecting video prediction to robot policies, and IRASim and Ctrl-World emphasizing action-conditioned interaction.Source: §2 · p. 2
• Dexterous and egocentric world models use hand, camera, silhouette, mask, or structured action inputs, while DeXplicit focuses on measurable object dynamics.DeXplicit couples metric forecasts of transport and deformation to visual synthesis.Source: §2 · pp. 2–3
• Learned simulators and point-based models represent spatial interactions or trajectories, while object-pose and rigid-motion methods emphasize coherent motion.DeXplicit combines shared object pose for transport with an embedded graph for residual deformation, without restricting all objects to rigid motion.Source: §2 · p. 3
• Controllable video methods condition on spatial, motion, viewpoint, hand-object, or contact cues; DeXplicit couples initial appearance to persistent object points.Depth, visibility, and forecast confidence determine valid appearance support.Source: §2 · p. 3

• The task is to predict the resulting metric 3D state and future observations from an initial scene and proposed hand trajectory.The prediction is intended to preserve the object's response, identity, and appearance.Source: §3 · p. 3
• Inputs include the initial image, metric scene points, an anatomically indexed hand trajectory, and calibrated cameras.Each object point also has an object label and a canonical coordinate in the object's reference shape.Source: §3 · p. 3
• The pipeline uses a dynamics predictor, deterministic projection, and conditional video renderer to produce future visual controls, masks, and images.The projection builds controls and a soft mask from predicted geometry, confidence, appearance, hand poses, and cameras.Source: §3 · p. 3

4.1 Structured Interaction Dynamics

• Joint-aware action encoding represents hand anatomy, while object-centric prediction models shared transport and local deformation.Object features are pooled with canonical coordinates; object-time features predict a shared pose and local transforms for articulation or deformation.Source: §4.1 · pp. 3–4
• Bounded pose correction adjusts inaccurate initial object poses, and uncertainty in predicted point trajectories supplies confidence for visual synthesis.The pose adjustment is bounded to 5 cm and 15° per component.Source: §4.1 · pp. 4–5

4.2 Metric-aligned Refinement

• Metric-aligned refinement directly reduces object-point position errors using a moved-point objective and hand-derived motion references for pose correction.The motion mask selects points whose reference displacement exceeds 5 mm; candidate pose references include either or both hands, static hypotheses, and the current prediction.Source: §4.2 · p. 5

4.3 Confidence-gated Visual Readout

• Appearance features attached to object points are projected into future views, with coverage and confidence guiding the renderer to retain reliable content or synthesize uncertain regions.The appearance stream uses initial-image features; projected scene and hand geometry also condition the renderer.Source: §4.3 · p. 6
• The renderer's conditioning branches are trained with a frozen video backbone, then adapted to a causal backbone using endpoint reconstruction, adversarial supervision, and on-policy distillation.Distillation targets robustness to generated history.Source: §4.3 · p. 6

5.2 Quantitative and Qualitative Results

• Shared transport preserves coherent motion, while local deformation captures articulation and cloth shape change in the qualitative forecasts.The experiments include rigid, articulated, and deformable interactions, with PointWorld and DEXPLICIT forecasts overlaid on the same reference image.Source: §5 · p. 6, Figure 4 · p. 7

5.1 Experimental Setup

• The setup evaluates one dynamics model on HOT3D, OakInk2, and PhysTwin with matched source windows and reference targets, using 3D and video metrics.The datasets span rigid, articulated, and deformable interactions; the video metrics include perceptual quality, hand alignment, object overlap, and position error.Source: §5.1 · p. 6

5.2 Quantitative and Qualitative Results

• The video panel reports strongest perceptual quality and object overlap for DEXPLICIT, while Figure 5 highlights retained object orientation and identity during rotation and placement.Baselines with higher PCK20 can have much lower object overlap, so hand agreement alone does not capture interaction fidelity.Source: §5.2 · p. 7, Figure 5 · p. 9
• Causal-DEXPLICIT generates four frames in 0.54 seconds and 32 frames in 4.90 seconds, with higher object overlap than similarly fast autoregressive baselines.The video evaluation spans perceptual quality, hand alignment, object fidelity, and latency.Source: §5.2 · p. 7, Table 2 · p. 8
• Moved-point error falls 42–48% at the early horizon and 27–40% at the long horizon across three datasets, though OakInk2 all-point error is higher.The comparison uses identical scene and action inputs; all-point error also improves on HOT3D and PhysTwin.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 Ablation Studies

• Ablations show structured transport gives the largest dynamics gain, while scene tracks, hand conditioning, and appearance transport progressively improve video metrics.The experiments compare dynamics and video-conditioning ablations.Source: §5.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8

6 Conclusion

• The conclusion presents explicit 3D prediction before visual synthesis as the principle for producing geometrically grounded future states and efficient causal rendering.The authors say this reduces the video generator’s burden to learn action-conditioned dynamics implicitly.Source: §6 · p. 9

Reproducibility Statement

• The reproducibility statement locates evaluation protocols and model, training, and inference details in the appendices.It also points to additional controlled ablations in Appendix E.Source: Reproducibility Statement · p. 9

ETHICS STATEMENT

• The reported experiments concern visual prediction for manipulation and do not establish suitability for safety-critical robotic control.The ethics statement cautions that plausible generated futures may misrepresent physical outcomes.Source: ETHICS STATEMENT · p. 9

AI Assistance

• AI tools assisted with software implementation, experiments, and evaluation; the authors reviewed the assisted code and take responsibility for the final content and results.Source: AI Assistance · p. 10

A Evaluation Details

• The evaluation uses matched initial geometry, hand actions, and cameras for the 3D forecasting comparison, while video systems retain method-specific controls and inference procedures.Causal generation produces eight four-frame blocks using two denoising steps and generated history, with future reference images excluded.Source: A Evaluation Details · p. 13

Causal post-training

• The post-training continuations start from the same condition-aligned Fast Adapter and use the same training data and update budget; the critic is the controlled difference.The critic operates on generated-history sequences with adversarial weight 0.02.Source: §A.1 · pp. 13–14
• Adding the video critic improves all five measures, while distillation alone does not improve interaction fidelity.Figure 6 shows better preservation of object shape and placement than ordinary flow-matching continuation.Source: Table 6 · p. 14, §A.1 · p. 14, Figure 6 · p. 14
• The table captions identify the comparison as held-out variants under the same two-step inference protocol, with PCK20 and coverage reported as percentages.The complete-metrics table also specifies aligned hand error in millimeters.Source: Table 6 · p. 14, Table 7 · p. 15

B Additional Quantitative Results

• 2.43% is the final-slot PCK10 score for DexWM’s native hand probe.Source: Table 8 · p. 15
• The probe is evaluated on the native 224 × 392 grid.Source: Table 8 · p. 15
• The native hand probe uses 12 temporal slots.Source: Table 8 · p. 15

C.1 Dynamics Architecture and Optimization

• The shared dynamics model takes initial metric scene features and a 35-state hand sequence, encoding scene and action tokens with point and object-time Transformers.Scene coordinates are in meters, and the model predicts states 1–34 jointly.Source: §C.1 · p. 16
• Object-time features drive global pose and dense motion predictions, while graph-based local deformation supplies residual object motion.The deformation head projects six temporal controls into a spline basis and combines spatial modes with object-time and hand-relation features.Source: §C.1 · pp. 16–17
• One dynamics model is trained across HOT3D, OakInk2, and PhysTwin with resampling to balance the larger datasets.The objective combines full-horizon, early, and endpoint moved-point errors with weights 0.45, 0.20, and 0.35.Source: §C.1 · p. 17, §C.2 · p. 17

C.2 Geometric Objectives and Pose Refinement

• Metric-aligned refinement targets object-point position errors and adds bounded object-frame pose corrections guided by hand-derived motion references.The correction twist is bounded by 0.05 m and 15°; the model can mask unavailable hand candidates.Source: §C.2 · pp. 17–18
• Gauge fixing transfers the best-fitting rigid component of graph motion into the global pose while preserving predicted world positions.Source: §C.2 · p. 18

C.3 Video Generation

• Appearance transport projects persistent object-point descriptors into future views, using calibrated projection, depth selection, and hand occluders to determine supported features.The appearance stream marks unsupported cells and supplies additional VACE channels.Source: §C.3 · p. 18

D Causal Adaptation Details

• Figure 7 summarizes transfer from a bidirectional renderer, alignment to a frozen few-step causal model, and subsequent on-policy adaptation.The causal post-training matches student and teacher velocities at both deployed denoising steps while the teacher reads cleaner cache context.Source: Figure 7 · p. 19, D Causal Adaptation Details · p. 19

E.1 Action Encoding and Structured Object Motion

• Endpoint error on ever-moved points falls from 122.76 to 87.45 mm, a 28.8% reduction with structured object motion.The configuration combines object-centric SE(3) transport, an object-time Transformer, and auxiliary losses.Source: §E.1 · p. 21
• Moved-point L2 decreases from 44.6 mm without an action encoder to 36.4 mm with MANO-relative action tokens.A joint Transformer alone reaches 42.4 mm in the shorter-horizon HOT3D study.Source: §E.1 · p. 21

E.2 Geometry, Hand Conditioning, and Appearance Transport

• The geometry ablation compares hand-pointmap conditioning with added projected scene tracks under equal training budgets on 150 windows from 75 clips.The separate 1.3B study uses a VACE backbone with rendered geometry controls.Source: §E.2 · p. 21
• On 48 held-out windows with no training-recording overlap, appearance transport improves pixel reconstruction and perceptual similarity over the preceding hand-conditioned model.This comparison is separate from the main 14B and causal video evaluations.Source: §E.2 · p. 21

F Additional Qualitative Results

• Figures 8 and 9 present six distinct HOT3D windows, each with more than 30 mm maximum annotated object translation, selected using perceptual quality, annotated motion, and complete sequences.The samples are from distinct source recordings and do not repeat either main qualitative figure.Source: F Additional Qualitative Results · p. 22
• Across spoon transfer, spatula sweep, and flask lifting and tilting, DeXplicit retains manipulated-object appearance while tracking changing placement and orientation.Rows show future frames 1, 16, and 32, with matched times and crops across methods within each case.Source: Figure 8 · p. 22
• Toy-house reorientation and coffee-pot rotation or lifting illustrate object identity and spatial response across rotation, changing visibility, and translation.The displayed future frames are 1, 16, and 32, using the same crop for every method in each sample.Source: Figure 9 · p. 23

G Limitations

• Errors in the initial metric 3D scene or future hand trajectory may propagate into both physical-state prediction and video synthesis.Obtaining these inputs reliably may require dedicated perception systems.Source: G Limitations · p. 23
• Appearance transport covers features visible initially; newly revealed or heavily occluded regions must be synthesized, and large disocclusions can reduce visual fidelity.A confidence-gated visual readout handles unsupported regions but does not remove the stated disocclusion limitation.Source: G Limitations · p. 23