PaperlayerView exact revision on arXiv

Source-linked AI summary

Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures

Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, Xingang Pan

arXiv:2602.09600v2cs.CV

TL;DR

Generating egocentric interactions from a scene image under free-space hand gestures is challenging because existing mask conditioning encounters a distribution gap. Hand2World uses projected 3D hand controls and explicit camera geometry, and reports 76% lower FVD and 42% lower camera trajectory error than state-of-the-art baselines across three datasets.

  • Problem

    Existing monocular interaction generators condition on visible 2D masks, while free-space gestures yield complete masks, leaving a distribution gap for this target setting.

    Source: §1 · p. 2, §3.3 · p. 4

  • Method

    Hand2World is an autoregressive monocular generator that conditions on projected 3D hand meshes and explicit Plücker-ray camera geometry, with automated annotation and causal distillation.

    Source: §1 · p. 2

  • Results

    Across three egocentric datasets, Hand2World improves perceptual quality and 3D/viewpoint consistency, with 76% lower FVD and 42% lower camera trajectory error than state-of-the-art baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework supports controllable long-horizon rollouts on egocentric interaction benchmarks.

    Source: §5 · p. 8

    Mask-conditioned methods face a distribution shift between partially occluded training masks and complete free-space masks, which can produce phantom occluders and incorrect depth ordering.

    Source: §1 · p. 2

Abstract

from arXiv · show

Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.

1. Introduction

Hand2World targets photorealistic egocentric interaction videos from a single scene image and free-space gestures, with coherent hand–object interaction, occlusion, camera motion, and arbitrary-length generation. It combines occlusion-invariant 3D hand conditioning, explicit camera geometry, automated monocular annotation, and causal distillation.

1. Introduction

  • The task is to generate photorealistic video from a scene image and gesture sequence, with plausible object responses, correct hand occlusion, coherent backgrounds under head motion, and arbitrary-length streaming.Hands should enter the depicted scene and manipulate objects as if the gestures were physically performed there.Source: §1 · pp. 1–2
  • Prior monocular methods conditioned on 2D hand masks [2] [17] [26] can fail under fully visible free-space gestures, while implicit camera inference risks background drift and existing pipelines lack causal streaming interaction.The paper attributes mask failures to training masks that are often occluded, unlike fully visible free-space gestures; multi-view systems [15] [28] require synchronized rigs or scanned assets.Source: §1 · p. 2
  • Hand2World conditions on projected complete 3D hand meshes as silhouette-and-wireframe signals, enabling occlusion reasoning from scene context, and uses per-pixel Plücker-ray embeddings to stabilize geometry under head motion.The mesh signal remains consistent regardless of occlusion state, and the camera embeddings disentangle viewpoint changes from hand motion.Source: §1 · p. 2
  • A fully automated monocular pipeline extracts temporally consistent hand meshes and approximate camera trajectories, and distillation from a bidirectional diffusion teacher enables causal, streaming-friendly generation of arbitrary length [14] [36].The annotation pipeline avoids manual labels and multi-view rigs.Source: §1 · p. 2
  • 76% reduction in FVD and 42% reduction in camera trajectory error over state-of-the-art baselines are reported across three egocentric datasets.The paper says these results improve perceptual quality and 3D/viewpoint consistency.Source: §1 · p. 2

2. Related Work

Prior work provides world modeling, hand-interaction synthesis, and controllable video generation, but leaves gaps in free-space hand control, camera geometry, and scalable generation. Hand2World combines these capabilities with monocular hand modeling and autoregressive generation.

2.1. Generative World Models

  • World models predict environmental dynamics for planning and simulation, while existing approaches may lack fine-grained hand control or require specialized hardware.Examples include playable environments from Genie [4], GameNGen [29], and Oasis, and physical-AI scaling with Cosmos [1].Source: §2.1 · p. 3

2.2. Hand-Object Interaction Synthesis

  • Hand2World targets all six capabilities: free-space control, explicit camera control, arbitrary-length generation, scene dynamics, and monocular scalability for training and deployment.It uses occlusion-invariant 3D hand conditioning for free-space control and explicit camera geometry for viewpoint changes.Source: §2.2 · p. 3

2.3. Controllable Video Diffusion

  • ControlNet [38] and video extensions [31] standardize spatial conditioning, while Hand2World uses Plücker-ray embeddings [11] to ground egocentric viewpoint changes.Implicit camera learning [33] can produce floating-background artifacts by failing to distinguish head motion from hand and object motion.Source: §2.3 · p. 3

2.4. Hand Modeling and Autoregressive Generation

  • Monocular hand mesh estimation [22] [37] extracts 3D geometry from unlabeled video, and autoregressive distillation [36] with self-forcing [14] enables causal arbitrary-length generation.Source: §2.4 · p. 3

3. Method

Hand2World generates egocentric interaction videos from a scene image and free-space hand gestures, combining visibility-independent hand controls, explicit camera geometry, and causal generation. Its goal is plausible object dynamics, scene-consistent occlusion and depth, and backgrounds that follow camera motion.

3.1. Overview

  • Hand2World combines projected 3D hand controls, per-pixel Plücker-ray camera embeddings, and a distilled causal generator, building on Wan2.1-1.3B-Control [30].The framework targets the mismatch between occluded training hands and fully visible free-space gestures, as well as background drift from viewpoint changes.Source: §3.1 · pp. 3–4

3.2. Problem Formulation

  • Given a scene image, framewise hand gestures, and a camera trajectory, the generator aims for plausible object dynamics, scene-consistent depth and occlusion, and camera-consistent backgrounds.The inputs are a reference image, hand configurations, and viewpoint changes.Source: §3.2 · p. 4

3.3. Occlusion-Invariant Hand Conditioning

  • Projected 3D hand meshes provide visibility-independent controls: filled silhouettes encode extent and coarse shape, while wireframes expose articulation and preserve hand identity under overlap.Because the controls derive from meshes rather than observed pixels, the generator infers visibility and depth ordering from scene context despite occlusion differences between training and inference.Source: §3.3 · p. 4

3.4. Explicit Camera Control

  • Per-pixel Plücker-ray embeddings derived from camera pose, translation, and intrinsics provide a spatially grounded viewpoint signal to separate head motion from hand motion and reduce background drift.The camera trajectory is represented by per-frame parameters (Rt, tt, Kt).Source: §3.4 · p. 4

3.5. Monocular Auto-Annotation Pipeline

  • An automated monocular-video pipeline detects hands, stabilizes detections with temporal heuristics, estimates MANO parameters with HaMeR [22], and recovers camera trajectories normalized to the first frame.Depth supports camera-pose recovery but is not provided to the generator.Source: §3.5 · p. 4

3.6. Dual-Pathway Video Diffusion

  • The diffusion backbone concatenates hand controls and the scene image as latent channels, while an adapter additively injects downsampled Plücker-ray features into the token stream.Training first fits the camera adapter with the backbone frozen, then jointly fine-tunes the adapter and transformer LoRA modules to stabilize camera conditioning while preserving pretrained priors.Source: §3.6 · pp. 5–6

3.7. Autoregressive Generator Distillation

  • The bidirectional teacher is distilled into a causal autoregressive generator using ODE pretraining, distribution matching, and self-forcing, then generates arbitrary-length videos in KV-cached blocks.Replacing teacher-provided context with student predictions during training mitigates exposure bias; block-wise inference maintains coherence across block boundaries.Source: §3.7 · p. 6

4. Experiments

Experiments assess perceptual realism and temporal coherence, hand and camera controllability, and 3D/viewpoint consistency under ego-motion, including long rollouts. They evaluate both the bidirectional teacher and its distilled autoregressive generator.

4.1. Experimental Setup

  • Evaluation uses ARCTIC as the primary benchmark and compares four methods plus unmodified Wan2.1-1.3B-Control on identical test splits.Metrics cover realism and semantic alignment, frame fidelity, temporal coherence, and depth and camera consistency; mask-based baselines receive SAM3-derived masks. Training uses two stages, followed by causal distillation.Source: §4.1 · p. 6

4.2. Quantitative Results

  • FVD falls from 908.32 (InterDyn) to 218.76 (Hand2World), while DINO rises from 0.80 to 0.88; Hand2World also achieves lower Depth-ERR and Cam-ERR than all baselines.The comparison is on ARCTIC; the distilled Hand2World-AR remains close to the teacher at the 81-frame horizon while enabling causal long-horizon generation.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Ablation Study

  • Removing the camera adapter raises FVD to 815.14 and Cam-ERR to 0.13, with viewpoint changes becoming entangled with scene dynamics and causing background drift.The ablation identifies explicit camera conditioning as the dominant factor; Plücker-ray injection helps generated camera trajectories follow the ground truth.Source: §4.3 · p. 7, Table 3 · p. 7, Figure 4 · p. 7
  • Wireframe augmentation and temporal stabilization provide complementary gains: wireframes clarify occluded finger articulation, while temporal filtering reduces detection gaps that cause flicker or discontinuous controls.The study ablates both components alongside camera conditioning on ARCTIC; broader three-dataset ablations are reported in the supplement.Source: §4.3 · p. 7, Table 3 · p. 7, Figure 5 · p. 7

4.4. Autoregressive Generation

  • Hand2World-AR closely matches the teacher at 81 frames and degrades gradually through 324-frame rollouts, maintaining coherent camera motion and plausible interactions at 8.9 FPS on one A100.The causal generator supports streaming, arbitrary-length synthesis.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5. Qualitative Results

  • Long-horizon free-space gesture rollouts remain stable under head-induced viewpoint changes, preserve hand-object occlusions, and support sequential interactions with multiple objects.Examples include grasping and removing a thin book while preserving a partly occluded box, then opening, closing, and repositioning the box while maintaining its identity and temporal continuity.Source: §4.5 · p. 8, Figure 6 · p. 8

5. Conclusion

Hand2World generates controllable egocentric interaction videos from a scene image and free-space hand gestures, combining occlusion-invariant hand conditioning, explicit camera control, monocular auto-annotation, and autoregressive distillation. Its free-space controls do not enforce physical contact constraints, so infeasible gestures can produce implausible interactions.

5. Conclusion

  • Hand2World combines occlusion-invariant 3D hand projections, Plücker-ray camera control, monocular auto-annotation, and autoregressive distillation for controllable long-horizon egocentric video generation.The framework decouples hand articulation from visibility and improves viewpoint consistency; it achieves strong performance on egocentric interaction benchmarks.Source: §5 · p. 8
  • Free-space gestures lack physical contact constraints, so infeasible motions such as penetrating solid objects may produce implausible interactions.The conclusion suggests force-feedback devices as a way to help enforce physical plausibility.Source: §5 · p. 8

Supplementary Material … A.4. LoRA Fine-tuning

The supplement documents implementation details and extends the paper with annotation, distillation, evaluation, runtime, and qualitative results.

Overview

  • The supplement covers backbone, conditioning, training and inference, annotation, distillation, datasets and metrics, extended comparisons, runtime, and qualitative results.It includes challenging long-horizon cases and free-space gesture generation under camera motion.Source: Overview · p. 11

A.1. Backbone and Latent Representation

  • Hand2World builds on Wan2.1-1.3B-Control [30], a flow-matching video diffusion transformer with a VAE that downsamples space 8× and time 4× to 16 latent channels.Experiments use an empty text prompt unless otherwise specified, isolating hand and camera conditioning.Source: §A.1 · p. 11

A.2. Dual-Pathway Input Conditioning

  • The dual-pathway conditioning encodes the rendered hand-control video as an input.The passage introduces the noisy video latent and rendered hand-control video encoding but does not specify the encoding operation.Source: §A.2 · p. 11
  • The reference scene image is encoded only at the first latent index, with the remaining indices padded with zeros; the transformer input uses channel concatenation.Source: §A.2 · p. 11

A.3. Camera Control Injection

  • Camera control packs four consecutive per-frame Plücker-ray maps into 24 channels per latent-time index, repeating the last available frame for edge padding.The four-frame grouping corresponds to the VAE's 4× temporal compression.Source: §A.3 · p. 11
  • A lightweight camera adapter maps packed rays to patch-token embeddings, which are added to patch tokens and empirically stabilize viewpoint conditioning versus channel concatenation.The adapter uses PixelUnshuffle(8), a patch-aligned Conv2D layer, and a residual block.Source: §A.3 · p. 11

A.4. LoRA Fine-tuning

  • LoRA [13] with rank 256 and α=256 adapts query, key, and value projections and feed-forward layers across all transformer blocks.This enables efficient adaptation while preserving the pretrained Wan2.1-1.3B-Control prior.Source: §A.4 · p. 11

A.5. Training Schedule and Hyperparameters … D.3. Quantitative Comparison with Baselines

Hand2World combines geometry-aware hand and camera conditioning with causal distillation for long-horizon egocentric interaction generation. Across three benchmarks, the teacher ranks first on every metric, while the distilled generator remains close to it and outperforms prior methods.

A.5. Training Schedule and Hyperparameters; A.6. Inference Settings

  • Training uses two stages: camera-adapter-only pretraining followed by joint camera-adapter and LoRA fine-tuning; inference uses a bidirectional teacher and a three-frame-block AR generator.Training uses 8× NVIDIA A100-80GB GPUs, ∼10K pretraining steps and ∼100K fine-tuning iterations; the teacher samples 81-frame 480p sequences, while the AR generator uses B=3 frames per block and Ns=4 denoising steps to generate at 384p.Source: §A.5 · p. 12, §A.6 · p. 12

B. Auto-Annotation Pipeline; B.1. Hand Control Signal Extraction; B.2. Camera Pose Estimation and Pl¨ucker-Ray Embeddings

  • Because paired hand-geometry and camera-motion signals are typically unavailable in in-the-wild egocentric videos, the method recovers both from monocular video and uses them as conditioning pseudo-labels.Hand controls are extracted through detection, temporal stabilization, mesh reconstruction, and rendering; HaMeR [22] estimates MANO [24] parameters per frame, and meshes are rendered as color-coded filled silhouettes with wireframe overlays.Source: B. Auto-Annotation Pipeline · p. 12, §B.1 · p. 12

C. Autoregressive Distillation Details; C.1. ODE Pretraining; C.2. Asymmetric Distribution Matching; C.3. Self-Forcing

  • The causal generator is distilled from a bidirectional diffusion teacher through ODE trajectory pretraining, distribution matching, and self-forcing.The approach follows CausVid [36] with self-forcing [14]; the frozen teacher supervises DMD [35], the camera adapter is frozen during distillation, and self-forcing anneals its probability from 0 to 0.5 to improve long-horizon stability.Source: C. Autoregressive Distillation Details · p. 12, §C.1 · p. 12, §C.2 · p. 13

C.4. Inference with KV Caching; D. Evaluation Protocol and Additional Results; D.1. Datasets; D.2. Metrics

  • KV caching reuses prior-block key/value states for approximately constant throughput per block, and evaluation uses 81-frame clips with realism, fidelity, temporal-coherence, and 3D-consistency metrics.The evaluation covers HOT3D, ARCTIC, and HOI4D; reported metrics include FVD, DINO, PSNR, SSIM, LPIPS, Flow-ERR, Depth-ERR, and Cam-ERR, with generated and ground-truth clips resized to a common resolution before measurement.Source: §C.4 · p. 13, §D.1 · p. 13, §D.2 · pp. 13–14

D.3. Quantitative Comparison with Baselines

  • Hand2World ranks first across all metrics on all three datasets, and Hand2World-AR stays close to the teacher at 81 frames.On HOT3D, FVD rises from 106.20 to 112.90 and Cam-ERR changes from 0.13 to 0.14; across ARCTIC and HOI4D, the student remains within 0.01 Cam-ERR of the teacher.Source: §D.3 · p. 14
  • On ARCTIC, Hand2World achieves FVD 218.76 and Cam-ERR 0.07, versus the best baseline FVD of 908.32 and baseline Cam-ERR values of 0.12–0.14.InterDyn has the best baseline FVD but lacks explicit camera modeling; Hand2World projects full 3D hand meshes and adds Plücker-ray camera embeddings to preserve articulation under occlusion and separate viewpoint from hand motion.Source: §D.3 · p. 14
  • On HOT3D, Hand2World reduces FVD to 106.20 and Cam-ERR to 0.13, compared with baseline Cam-ERR values of 0.33–0.38.The dataset stresses viewpoint control under large head motion; Flow-ERR and Depth-ERR also improve, consistent with more stable background geometry and temporal dynamics.Source: §D.3 · p. 14
  • On HOI4D, Hand2World achieves FVD 251.05, LPIPS 0.19, Cam-ERR 0.04, and Depth-ERR 7.98.The benchmark spans 610 rooms; CosHand reaches PSNR 18.60 but has FVD 1090.31, while InterDyn and Mask2IV have Depth-ERR 14.19 and 13.96, respectively.Source: §D.3 · p. 14

D.4. Quantitative Ablation Studies

The ablations identify camera conditioning as the main contributor to quality and 3D consistency, with wireframe rendering and short-gap interpolation providing additional gains. The section also reports that autoregressive generation remains close to its teacher at 81 frames and degrades gradually with longer rollouts.

Camera conditioning

  • 3.7×, 4.1×, and 2.5× higher FVD follows removal of the camera adapter on ARCTIC, HOT3D, and HOI4D, respectively.FVD rises from 218.76 to 815.14 on ARCTIC, from 106.20 to 438.82 on HOT3D, and from 251.05 to 632.58 on HOI4D. Cam-ERR, depth, flow, and LPIPS also worsen; the camera pathway helps disentangle viewpoint change from interaction and stabilize background geometry under head motion.Source: §D.4 · p. 14, Table 7 · p. 15

Wireframe rendering

  • 218.76 to 223.29 on ARCTIC, 106.20 to 108.05 on HOT3D, and 251.05 to 255.15 on HOI4D are FVD increases without wireframe rendering.DINO similarity and PSNR also decrease slightly on HOT3D; wireframe edges expose articulation cues, especially for self-occluded poses whose silhouettes do not distinguish finger states. HOT3D Depth-ERR is marginally lower without wireframe (15.35 vs. 15.37), a difference reported as within noise.Source: §D.4 · pp. 14–15, Table 7 · p. 15

Temporal stabilization and filtering

  • 12.33, 4.99, and 14.42 higher FVD result from disabling short-gap interpolation on ARCTIC, HOT3D, and HOI4D, respectively.FVD changes by at most 0.5% when edge filtering or overlap filtering is removed; these components suppress occasional boundary jitter and duplicate detections during large-scale auto-annotation.Source: Table 7 · p. 15, §D.4 · p. 15

Autoregressive generation

  • Hand2World-AR matches the teacher closely at 81 frames and degrades gradually as rollout length increases beyond the 81-frame training horizon.Table 8 evaluates autoregressive generation across ARCTIC, HOT3D, and HOI4D.Source: §D.4 · p. 16, Table 8 · p. 16

E. Runtime Analysis

Runtime is benchmarked on 324-frame sequences at 544×384 resolution using one A100-80GB GPU; the autoregressive generator is the pipeline’s throughput bottleneck, with KV caching keeping throughput approximately constant as rollout length increases.

E. Runtime Analysis

F. Qualitative Results … F.4. Large viewpoint changes

Across qualitative tests, Hand2World more faithfully tracks object interactions and large head pans, while baselines exhibit state, viewpoint, or hand-pose failures.

F. Qualitative Results

  • The qualitative comparisons highlight three recurring failures: inconsistent object state or interaction, viewpoint instability under head motion, and hand poses misaligned with the input gesture.Examples include unstable contacts, background drift, and hands out of temporal synchronization with intended motion.Source: F. Qualitative Results · p. 16

F.1. Box opening and closing

  • During box opening and closing, Hand2World tracks the ground-truth state transition, preserving lid angle, interior appearance, and stable background geometry.The sequence requires generating an unseen interior during opening and maintaining hinge geometry across frames.Source: §F.1 · p. 16, Figure 7 · p. 18, Figure 8 · p. 19

F.2. Laptop opening and closing

  • Hand2World generates a coherent laptop opening with visible keyboard structure, then restores a clean closed state consistent with ground truth.Several baselines fail to open the laptop recognizably, and others drift or degrade later in the sequence.Source: §F.2 · p. 17, Figure 9 · p. 20, Figure 10 · p. 21

F.3. Object appearance fidelity under interaction

  • Hand2World alone correctly closes the patterned storage box while preserving its surface pattern and geometry.The high-frequency texture exposes distortion; CosHand and InterDyn lose the pattern and fail to close the box, while Wan2.1-1.3B-Control distorts its geometry.Source: §F.3 · p. 17, Figure 11 · p. 22
  • Hand2World maintains cup identity across depth changes, renders a faithful handle grasp, and plausibly reconstructs background as the cup retreats.Baselines produce blurred or artifact-laden cups, fail to match the intended near–far motion, or degrade as the cup approaches.Source: §F.3 · p. 17, Figure 12 · p. 23

F.4. Large viewpoint changes

  • Hand2World follows the rightward-then-leftward camera pan, revealing the whiteboard and returning toward the desk in close agreement with ground truth.CosHand and InterDyn stay near the initial view, while other baselines blur or degrade severely.Source: §F.4 · p. 17, Figure 13 · p. 24

G. Additional Qualitative Results on Challenging Cases … H.2. Free-space gestures under motion blur

Hand2World-AR maintains coherent long-horizon interactions under challenging viewpoint and scale changes, while Hand2World produces coherent free-space box-opening sequences under viewpoint reversal and severe motion blur.

G. Additional Qualitative Results on Challenging Cases

G.1. Challenging Cases

  • At approximately 4× the training clip length (160 frames), Hand2World-AR handles unseen page content, extreme camera rotation, and large object-scale variation while maintaining coherent interactions and viewpoint-consistent backgrounds.The examples include opening and flipping a book, plus cup and container interactions as objects move close to and far from the camera.Source: §G.1 · p. 25, Figure 14 · p. 25

H. Free-Space Gesture Generation

  • Free-space gestures create a train–test distribution shift for mask-based methods because complete hand masks at inference differ from the partial masks seen during training.The examples combine this shift with viewpoint change and motion blur.Source: H. Free-Space Gesture Generation · p. 25

H.1. Free-space gestures under viewpoint change

  • During a rightward-then-leftward camera pan, Hand2World alone produces a coherent box-opening trajectory, natural hand appearance, and hands aligned with the box.The box’s visible interior appears at Frames 60–80; competing methods have inconsistent or unclear opening trajectories, spatial drift, unnatural poses, or distorted skin tones.Source: §H.1 · p. 25, Figure 15 · p. 26

H.2. Free-space gestures under motion blur

  • Rapid head rotation blurs the driving stream around Frames 40–80, requiring the model to recover hand articulation while also handling viewpoint change.This compounds degraded input quality with the free-space distribution shift.Source: §H.2 · p. 26, Figure 16 · p. 27
  • Under severe motion blur, Hand2World progressively opens the box, reveals a plausible interior by Frame 60, and preserves natural hands and a stable background.CosHand’s opening progression becomes inconsistent; InterDyn and Mask2IV keep the box largely closed and let the hands drift, while Wan2.1-1.3B-Control keeps it closed and distorts hand color.Source: §H.2 · p. 26, Figure 16 · p. 27

Related papers

Loading 2602.09600v2…