PaperlayerView exact revision on arXiv

Source-linked AI summary

Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures

Yuxi Wang, Wenqi Ouyang, Tianyi Wei, Yi Dong, Zhiqi Shen, Xingang Pan

arXiv:2602.09600v2cs.CV

TL;DR

Hand2World addresses egocentric video generation from a single scene image under free-space hand gestures, where mask distribution shift, camera–hand ambiguity, and arbitrary-length synthesis are central challenges. It uses occlusion-invariant 3D hand controls, explicit camera geometry, and autoregressive generation, improving perceptual and viewpoint consistency across three benchmarks.

  • Problem

    Egocentric generation from monocular videos must support free-space hand control despite mask distribution shift, ambiguity between hand and camera motion, and the need for arbitrary-length synthesis.

  • Method

    Hand2World conditions generation on projected complete 3D hand meshes, injects Plücker-ray camera geometry, and combines automated monocular annotation with autoregressive diffusion distillation.

  • Results

    Across three egocentric datasets, Hand2World substantially improves perceptual quality and 3D/viewpoint consistency, reducing FVD by 76% and camera trajectory error by 42% versus state-of-the-art baselines.

  • Takeaways & Limitations

    The framework supports explicit camera control, free-space interaction generation, and efficient long-horizon rollouts from a single scene image.

  • Takeaways & Limitations

    Mask-conditioned approaches remain vulnerable to complete free-space masks, which can produce phantom occluders and incorrect depth ordering.

Abstract

from arXiv · show

Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.

1. Introduction

Hand2World targets photorealistic egocentric interaction videos from a single scene image and free-space gestures, addressing interaction plausibility, occlusion, camera motion, and arbitrary-length synthesis. It combines occlusion-invariant 3D hand conditioning, explicit camera geometry, automated monocular annotation, and autoregressive generation, with improvements in perceptual and viewpoint consistency.

  • Interactive world models support environment simulation for augmented reality, teleoperation, and embodied AI, while free-space hand gestures provide a natural control signal.
  • The task synthesizes photorealistic videos from a scene image and egocentric gesture sequence, with hands manipulating objects under plausible visibility, depth, and head-motion dynamics.
  • Mask-based conditioning breaks under free-space gestures because complete inference masks differ from partially occluded training masks, causing phantom occluders and incorrect depth ordering.
  • The framework combines explicit camera control, automated monocular annotation, and autoregressive synthesis to support scalable, arbitrary-length egocentric generation.
  • Hand2World uses projected complete 3D hand meshes as occlusion-invariant controls, letting scene context determine visibility rather than encoding it in the control signal.
  • 76% reduction in FVD and 42% reduction in camera trajectory error are reported over state-of-the-art baselines across three egocentric datasets.

2. Related Work

Related work spans world models, hand–object interaction synthesis, camera control, and arbitrary-length video generation. Hand2World is positioned as a monocular system combining free-space control, explicit camera modeling, generative dynamics, autoregressive synthesis, and scalable training and inference.

  • World models learn environmental dynamics for planning and simulation across driving, robotics, and interactive gaming, but existing systems may lack fine-grained hand control or require specialized hardware.
  • Free-Space Gesture Control: Mask-based monocular methods suffer free-space distribution shifts, while multi-view methods provide robust control but require expensive capture rigs; reenactment methods require driving videos.
  • Camera Control: Monocular methods lacking explicit camera modeling can conflate hand and viewpoint motion, producing floating backgrounds, whereas explicit camera modeling targets 3D-consistent viewpoint changes.
  • Scalability and Monocular Inference: Multi-view approaches depend on synchronized camera rigs or scanned assets, while monocular methods use abundant single-view video and require only monocular input at inference.
  • Scalability and Monocular Inference: Hand2World combines free-space gesture control, explicit camera control, autoregressive generation, generative scene dynamics, and monocular scalability for training and deployment.

3. Method

Hand2World generates egocentric interaction videos from a scene image, hand gestures, and camera trajectories while targeting plausible object dynamics, consistent occlusions, and background motion. Its method combines occlusion-invariant projected 3D hand controls, explicit camera-ray conditioning, automated monocular annotations, and causal distillation for arbitrary-length generation.

  • Problem formulation: Hand2World conditions generation on a reference scene image, hand configurations, and camera trajectories to synthesize egocentric interaction videos.The target includes hands entering the scene, plausible object dynamics, geometry-consistent occlusions, and camera-consistent background evolution.
  • Occlusion-invariant hand conditioning: Projected 3D hand meshes provide an occlusion-invariant control signal by encoding complete geometry rather than observed visibility.The signal is rendered from hand meshes using camera intrinsics, preserving format consistency across contact-heavy occlusions and free-space gestures.
  • Occlusion-invariant hand conditioning: The composite hand control combines a filled silhouette for spatial extent with a wireframe exposing finger configuration and joint topology.Left and right hands are color-coded, and the wireframe supplies articulation cues under self-occlusion.
  • Camera conditioning: Per-pixel Plücker-ray embeddings explicitly represent camera motion so viewpoint changes are separated from hand motion and background drift is reduced.The camera pathway uses a lightweight adapter and additive injection rather than channel-wise concatenation.
  • Training and generation: An automated monocular pipeline recovers hand and camera conditioning from in-the-wild video, while bidirectional diffusion is distilled into a causal autoregressive generator.The causal generator is designed for streaming-friendly, long-horizon synthesis, and inference uses monocular estimators with KV-cached block-wise generation.

4. Experiments

Experiments evaluate perceptual quality, controllability, viewpoint consistency, ablations, and long-horizon autoregressive generation. Hand2World improves benchmark metrics and maintains stable interactions under camera motion, while Hand2World-AR supports efficient extended rollouts.

  • Evaluation setup: Evaluation measures perceptual realism, temporal coherence, hand and camera controllability, 3D/viewpoint consistency, and long-horizon autoregressive behavior.Both the bidirectional teacher and distilled causal variant are evaluated.
  • Quantitative results: Hand2World reduces FVD from 908.32 to 218.76 and increases DINO similarity from 0.80 to 0.88 versus InterDyn on ARCTIC.The comparison uses InterDyn as the strongest baseline in FVD.
  • Quantitative results: Hand2World achieves the strongest overall results on ARCTIC, while Hand2World-AR remains close and enables causal long-horizon generation.Table 2 covers distributional realism, semantic alignment, frame fidelity, temporal coherence, and 3D/viewpoint consistency.
  • Ablation study: Removing explicit camera conditioning increases FVD to 815.14 and Cam-ERR to 0.13, producing background drift and unstable parallax.The camera adapter is identified as the dominant ablation factor.
  • Ablation study: Wireframe augmentation preserves plausible finger configurations, while temporal interpolation and filtering reduce flicker and discontinuous motion from detection errors.These components provide complementary gains beyond camera conditioning.
  • Long-horizon generation: Hand2World-AR quality degrades gradually as rollout length increases, yet it maintains coherent camera motion and plausible interactions while reaching 8.9 FPS on one A100.At 81 frames it closely matches the teacher; longer-horizon quality declines as autoregressive errors accumulate.
  • Qualitative results: Long-horizon free-space gesture rollouts remain stable under head-induced viewpoint changes and preserve hand–object occlusions during contact.Qualitative examples include sequential manipulation of a book and box, plus opening, closing, and carrying a hinged box.

5. Conclusion

Hand2World generates egocentric interaction videos from a scene image and free-space gestures using occlusion-invariant hand conditioning, explicit camera control, automatic annotation, and autoregressive distillation. It enables controllable long-horizon rollouts, but free-space gestures can specify infeasible motions without physical contact constraints.

  • Contributions: Hand2World conditions on projected 3D hand meshes and Plücker-ray camera embeddings to improve visibility handling and viewpoint consistency.The framework combines monocular auto-annotation with autoregressive distillation.
  • Conclusion: The framework enables controllable long-horizon rollouts and achieves strong performance on egocentric interaction benchmarks.These capabilities follow from combining the conditioning, annotation, and distillation components.
  • Limitations: Free-space gestures lack physical contact constraints, so users may specify infeasible motions such as penetrating solid objects.The paper identifies force-feedback devices as a possible way to enforce physical plausibility.

Overview

The supplementary overview describes the model backbone, conditioning, annotation pipeline, training and inference settings, distillation, evaluation, runtime, and qualitative results. Camera information is temporally packed and injected into transformer patch tokens alongside video and hand-control latents.

  • Supplementary contents: The supplementary material covers conditioning and training, monocular hand-mesh and camera annotation, autoregressive distillation, datasets, metrics, runtime, and qualitative comparisons.These topics are organized across Sections A–G.
  • Backbone and conditioning: The model builds on Wan2.1-1.3B-Control, using a flow-matching video diffusion transformer in a VAE latent space with an empty text prompt.The VAE uses 8× spatial and 4× temporal downsampling and produces 16 latent channels.
  • Input conditioning: The input combines noisy video latents, rendered hand-control information, and a reference scene image encoded only at the first latent index.The remaining scene-image latent positions are padded with zeros.
  • Camera conditioning: Four consecutive per-frame Plücker-ray maps are packed into a 24-channel tensor because the VAE compresses time by 4×.The last available frame is repeated when the sequence ends before a full group of four frames.
  • Camera conditioning: A lightweight camera adapter spatially compresses packed ray maps, aligns them with patch tokens, and adds the resulting embeddings to those tokens.This injection empirically provides more stable viewpoint conditioning than channel concatenation.
  • Efficient adaptation: LoRA modules with rank 256 adapt transformer projections and feed-forward layers while preserving the pretrained Wan2.1-1.3B-Control prior.The modules are applied to query, key, value projections and feed-forward layers across transformer blocks.

A.5. Training Schedule and Hyperparameters

The training pipeline recovers hand geometry and camera motion from monocular video, then distills a bidirectional diffusion teacher into a causal autoregressive generator for long rollouts.

  • Hand and camera conditioning: Hand controls are extracted through detection, temporal stabilization, mesh reconstruction, and rendering of silhouette-plus-wireframe signals.The rendered representation preserves hand identity under overlap while leaving visibility and occlusion to be inferred from scene context.
  • Hand and camera conditioning: Camera parameters are estimated in streaming mode with overlapping chunks aligned by SIM3 transformations.Training uses larger, more accurate chunks, while inference switches to smaller chunks for speed.
  • Autoregressive training: The bidirectional diffusion teacher is distilled into a causal autoregressive generator using CausVid and self-forcing.Self-forcing replaces ground-truth history with student-generated history with probability p, annealed from 0 to 0.5, improving long-horizon rollout stability.
  • Autoregressive training: The autoregressive generator produces video block-wise, caches previous key/value states, and maintains approximately constant throughput per block as rollout length increases.This design supports arbitrary-length generation beyond the training horizon.
  • Evaluation setup: Evaluation uses three egocentric datasets, 81-frame clips, and metrics covering realism, fidelity, temporal coherence, depth, and camera consistency.Cam-ERR and Depth-ERR measure viewpoint and depth consistency using per-pixel Plücker-ray embeddings.
  • Evaluation setup: Hand2World ranks first across all reported metrics on all three datasets, while Hand2World-AR remains close to the bidirectional teacher and outperforms prior methods.On ARCTIC, the teacher achieves an FVD of 218.76, compared with 908.32 for the strongest baseline.

D.4. Quantitative Ablation Studies

Ablations show that explicit camera conditioning is the dominant contributor to performance, while wireframe rendering and temporal interpolation provide additional gains and long-horizon generation remains stable.

  • Camera conditioning: Removing the camera adapter raises FVD from 218.76 to 815.14 on ARCTIC, from 106.20 to 438.82 on HOT3D, and from 251.05 to 632.58 on HOI4D.These changes correspond to 3.7×, 4.1×, and 2.5× higher FVD, respectively.
  • Camera conditioning: Removing the camera adapter also increases Cam-ERR from 0.07 to 0.13 on ARCTIC, from 0.13 to 0.29 on HOT3D, and from 0.04 to 0.07 on HOI4D.Depth-ERR, Flow-ERR, and LPIPS likewise deteriorate in the reported ablations.
  • Hand representation: Wireframe rendering provides consistent but modest gains by supplying articulation cues beyond the hand silhouette, especially for self-occluded poses.On HOT3D, Depth-ERR is marginally lower without the wireframe, 15.35 versus 15.37, within noise.
  • Annotation stabilization: Disabling short-gap interpolation increases FVD from 218.76 to 231.09 on ARCTIC, from 106.20 to 111.19 on HOT3D, and from 251.05 to 265.47 on HOI4D.Removing edge or overlap filtering changes FVD by at most 0.5% across datasets.
  • Long-horizon generation: Hand2World-AR matches the teacher closely at 81 frames and degrades gradually as rollout length increases.The ablation evaluates autoregressive generation beyond the 81-frame training horizon.

E. Runtime Analysis

Runtime evaluation measures the complete pipeline on long sequences, with parallel preprocessing and an autoregressive generator whose cached computation keeps throughput approximately constant as rollouts grow.

  • Benchmark setup: Inference speed is benchmarked on 324-frame sequences at 544×384 resolution using a single A100-80GB GPU.Hand reconstruction and camera estimation run in parallel during the benchmark.
  • Runtime behavior: The autoregressive generator uses key/value caching so throughput remains approximately constant as rollout length increases.The runtime breakdown identifies the AR generator as the throughput bottleneck.

F. Qualitative Results

Qualitative evaluations stress unseen object states, fine hand articulation, appearance preservation, depth changes, and non-monotonic camera motion; Hand2World maintains coherent interactions across these cases.

  • Failure modes: The qualitative comparisons organize failures into state or interaction inconsistency, viewpoint inconsistency under head motion, and hand-pose misalignment.These failure modes correspond to incorrect object states or contacts, background drift or rotation errors, and gesture deviations or temporal desynchronization.
  • Articulated object manipulation: Bimanual box manipulation tests unseen interior generation, lid-angle tracking, and hinge consistency across opening and closing.The box interior is absent from the reference image, requiring generative completion with temporally consistent state tracking.
  • Articulated object manipulation: For laptop manipulation, thin-lid articulation exposes angular errors while opening requires generating new keyboard and screen content.Several baselines fail to reach a recognizable open state or exhibit viewpoint drift and later visual artifacts.
  • Appearance fidelity: Hand2World is the only method that closes the patterned storage box while preserving its high-frequency surface pattern and box geometry.The textured surface exposes spatial distortion and temporal inconsistency that other methods fail to avoid.
  • Depth-varying interaction: During cup grasping across near–far depth changes, Hand2World preserves cup identity, renders the handle grasp faithfully, and reconstructs plausible background behind the cup.The comparison stresses precise handle control, scale-dependent appearance, and occlusion of the background.
  • Camera motion: A large rightward-then-leftward head pan tests novel-view generation and return consistency across previously seen regions.Baselines without explicit camera conditioning keep the background nearly static, while others develop blur, artifacts, or washed-out scenes.

G. Additional Qualitative Results on Challenging Cases

The challenging cases test Hand2World-AR on long-horizon interaction, viewpoint reversals, free-space distribution shift, and severe motion blur. Across these settings, the method maintains more coherent object manipulation, hand appearance, and spatial alignment than competing methods.

  • Long-horizon rollouts: At approximately 4× the training clip length, Hand2World-AR handles unseen book-page content, extreme viewpoint rotation, and large object-scale variation.The rollouts cover book opening and flipping, plus cup and container interactions as objects move toward and away from the camera.
  • Viewpoint change: Non-monotonic camera motion tests whether generated hands and objects remain aligned when the viewpoint reverses direction.The box-opening sequence combines a rightward pan with a leftward reversal and evaluates spatial drift under the mask distribution shift.
  • Viewpoint change: Hand2World is the only compared method that produces coherent box opening, natural hand appearance, and spatial alignment throughout viewpoint reversal.It shows a visible interior at Frames 60–80, whereas competing methods produce inconsistent opening, drifting hands, or unnatural hand colors.
  • Motion blur: Severe motion blur around Frames 40–80 compounds the free-space distribution shift by degrading the hand-articulation cues needed for viewpoint-aware generation.The blurred-input setting requires reliable hand cues while the camera is changing, making it a particularly challenging test case.
  • Motion blur: Under blurred input, competing methods lose box consistency or keep it closed while hands drift from the intended opening motion.CosHand changes box orientation inconsistently, while InterDyn and Mask2IV largely fail to open the box and decouple the hands from the motion.
Loading 2602.09600v2…