Source-linked AI summary
Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control
Linxi Xie, Lisong C. Sun, Ashley Neall, Tong Wu, Shengqu Cai, Gordon Wetzstein
TL;DR
Current XR-oriented video world models accept coarse controls that do not capture dexterous hand–object interaction. The paper introduces head- and joint-level hand-conditioned video generation, distills it into a causal interactive system, and reports improved task performance and perceived control, while acknowledging substantial system limitations.
Problem
Current video world models accept limited control signals, leaving joint-level hand-pose conditioning for dexterous interaction unresolved.
Method
The authors evaluate diffusion-transformer conditioning strategies, combine 2D ControlNet-style and 3D joint conditioning, then distill a bidirectional teacher into a causal interactive model.
Results
71.2% task accuracy was achieved by the hand-controlled model on average, compared with 3.0% for the text-prompt baseline.
Takeaways & Limitations
The system demonstrates a first interactive generated-reality system with improved task performance and increased perceived control in user studies.
Takeaways & Limitations
Resolution, latency, stereo rendering, image quality, computing efficiency, and long-horizon stability remain substantially behind modern virtual-reality systems.
Abstract
from arXiv · showhide
Extended reality (XR) demands generative models that respond to users' tracked real-world motion, yet current video world models accept only coarse control signals such as text or keyboard input, limiting their utility for embodied interaction. We introduce a human-centric video world model that is conditioned on both tracked head pose and joint-level hand poses. For this purpose, we evaluate existing diffusion transformer conditioning strategies and propose an effective mechanism for 3D head and hand control, enabling dexterous hand--object interactions. We train a bidirectional video diffusion model teacher using this strategy and distill it into a causal, interactive system that generates egocentric virtual environments. We evaluate this generated reality system with human subjects and demonstrate improved task performance as well as a significantly higher level of perceived amount of control over the performed actions compared with relevant baselines.
1. Introduction
XR content creation is costly and current video world models provide insufficiently precise control for embodied interaction. This work develops human-centric head- and hand-conditioned generation and distills it into an interactive system with improved task performance and perceived control.
- XR content creation remains difficult, laborious, and expensive because it requires specialized expertise, complex tools, and high production costs.
- Current video world models mainly accept keyboard controls or text prompts, while camera- and full-body-pose methods lack precision for dexterous wrist and finger interactions.
- The study identifies 2D ControlNet-style conditioning combined with 3D joint-level hand representations as the most effective hand-pose strategy.
- The authors distill a bidirectional head- and hand-conditioned teacher into a causal real-time architecture for interactive content generation.
- The resulting system supports improved task accuracy and increased perceived control in user studies.
2. Related Work
Prior video-generation systems support realistic sequence modeling and increasingly causal prediction, but interactive egocentric control requires coordinated camera and fine-grained hand conditioning. The paper addresses this gap with joint head and hand control mechanisms.
- Bidirectional diffusion transformers generate realistic, temporally coherent video but require full sequences, limiting interactive use; autoregressive models enable sequential causal prediction.
- Camera motion and hand motion are both essential control signals because they jointly determine perception and interaction in egocentric virtual environments.
- The generated reality pipeline tracks head and hand poses, combines 2D rendered hand-skeleton and 3D hand-parameter features, and injects them with head features into a diffusion transformer.
- Existing pose-conditioned methods often treat hands as part of full-body pose, limiting the granularity of hand control.
3. Conditional Video Generation with Tracked Head and Hands
The paper develops a tracked head-and-hand conditioning framework for interactive egocentric video generation. It combines 2D and 3D hand representations with camera control and trains a joint conditioning architecture for human-centric XR.
- Hand Pose Representation: Joint-level hand poses are challenging because 2D skeleton videos lack depth information and suffer from self-occlusion ambiguity.The paper represents hands with 3D hand pose parameters alongside spatially aligned 2D skeleton videos.
- Hand Pose Conditioning: The method systematically evaluates four DiT conditioning strategies: token concatenation, token addition, AdaLN, and cross-attention fusion.The comparison concerns both how hand poses are represented and how conditioning information is injected into the generative model.
- Hybrid 2D–3D Hand Pose Conditioning: The proposed hybrid 2D–3D scheme concatenates raw-video and skeleton-video latents, then injects 3D hand pose features through token addition.This combines ControlNet-style spatial grounding with the depth and articulation information of hand pose parameters.
- Joint Camera and Hand Control: Interactive XR generation uses the user’s viewpoint and both hands as control signals for realistic egocentric video generation.The framework is designed around natural user interactions in head-mounted-display settings.
- Joint Camera and Hand Control: The joint architecture projects 6-DoF camera poses into Plücker embeddings and adds video, hand, and camera representations before DiT processing.Both hand and camera signals are jointly optimized for coherent alignment between user actions and viewpoint changes.
- Joint Camera and Hand Control: Camera and hand encoders are trained iteratively because jointly training them from scratch is unstable when hand and camera motion are ambiguous.The encoders are first trained independently and later trained jointly.
4. Experiments
The experiments compare hand-pose representations and conditioning strategies, then evaluate joint hand–camera control. The proposed hybrid 2D–3D strategy and JointCtrl achieve strong hand accuracy, visual quality, and coordinated control.
- Experimental setup: The study evaluates hand-pose conditioning on HOT3D using video quality, hand-pose accuracy, and camera-pose accuracy metrics.The hand metrics include Procrustes-aligned errors over joints and vertices, while camera metrics include rotation and translation error.
- Hand-pose conditioning: Among individual injection strategies, token addition achieves the best hand-pose accuracy, while cross-attention and AdaLN perform worse than the unconditioned baseline.The authors attribute this weaker performance to the limited HOT3D scale and high-dimensional hand-pose parameters.
- Hand-pose conditioning: The hybrid strategy combining 2D skeleton projection and 3D hand-pose parameters achieves the best hand accuracy while maintaining competitive video quality.It also produces more stable and anatomically faithful hand reconstructions qualitatively.
- Hand-pose conditioning: The hybrid method approaches the WiLoR-based lower bounds of MPJPE 9.42, MPVPE 7.74, and L2 landmark error 9.08 on HOT3D.Consistent improvements over 2D-only conditioning are also observed on GigaHands.
- Joint head–hand control: JointCtrl achieves the best overall performance across video quality, hand-pose, and camera-pose metrics compared with camera-only and hand-only baselines.CameraCtrl specializes in camera accuracy, HandCtrl in hand accuracy, whereas JointCtrl coordinates both modalities; qualitative results show improved boundary reconstruction and simultaneous control.
5. The Generated Reality System
The generated reality system streams a causal video model conditioned on users’ tracked head and hand poses into a VR headset. In user studies, explicit hand conditioning improves task accuracy and perceived control over head-only or text-conditioned baselines.
- System: The system uses tracked head and hand poses to condition video generation for interactive egocentric virtual environments.The model output is streamed directly to the user’s VR headset.
- System: A bidirectional teacher is distilled into a causal 5B student that generates 12-frame chunks and supports image-to-video and text-to-video settings.The resulting system provides a closed-loop generative experience.
- System: The VR implementation captures Quest 3 head and hand poses, streams them to a server, and returns generated video chunks for interactive viewing at 11 FPS.Unity connects the Quest 3, tracking buffer, server-hosted model, and streamed output.
- User study: Eleven subjects performed button-pushing, jar-opening, and steering-wheel tasks within 8 seconds under hand-and-head-conditioned and head-only conditions.The study used a commercial VR headset and allowed two practice runs before recording results.
- User study: 71.2% average task accuracy with hand control exceeds the baseline’s 3.0% under identical conditions.The baseline uses text prompts alone for task completion, whereas the proposed condition provides explicit hand control.
- User study: The hand-controlled model receives a mean perceived-control score of 4.21 versus 1.74 for the baseline on a 7-point Likert scale.Participants therefore report greater control over hand pose and movements with explicit hand conditioning.
6. Discussion
The paper presents an interactive generated reality system as an initial step toward human-centric world simulation, while acknowledging substantial system limitations and outlining future work to address them.
- The work presents efficient mechanisms for conditioning video diffusion models on tracked head and joint-level hand data.
- System resolution, latency, stereo rendering, image quality, and computing efficiency lag behind modern virtual reality systems.
- Autoregressive rollout drift significantly degrades image quality after a few seconds.
- Future work targets retinal-resolution stereo, imperceptible latency below 20 ms, and long rollouts on headset-embedded wearable computers.
- Generated reality could support immersive learning and exploration without laborious modeling of 3D virtual environments.
A. Experiment Details
The experiments use continual training across motion-conditioning stages to stabilize optimization and improve alignment, while estimating the evaluation pipeline's inherent error from validation annotations and reconstructions.
- The experiments use the Wan2.2 14B mixture-of-experts model with separate high-noise and low-noise DiT experts.
- The motion encoder is transferred from high-noise to low-noise training after convergence, improving training stability and motion alignment under limited HOT3D data.
- Hybrid conditioning initializes the DiT from skeleton-video LoRA weights to refine articulation and depth cues from hand pose parameters.
- Joint hand–camera conditioning initializes from the hybrid model, supporting stable convergence, motion consistency, and decoupling of the two conditionings.
- The evaluation pipeline's lower bound is estimated by applying the same metrics to HOT3D validation annotations, WiLoR hand poses, and GLOMAP camera trajectories.
B.1. Alternative Datasets
On the larger GigaHands dataset, hybrid conditioning continues to improve hand-pose accuracy over 2D-only conditioning, while the model transfers hand interactions to unseen scenes.
- Hybrid conditioning reduces MPJPE by 10%, MPVPE by 11%, and 2D error by 34% relative to 2D-only conditioning.
- The GigaHands results indicate scalability to larger, more complex data and richer hand motions.
- Despite training in a controlled studio environment, the model transfers hand interaction capabilities to diverse unseen scenes.
C. User Study Details
The user study uses short, simple tasks and measures both binary task completion and participants' perceived controllability, with baseline-versus-method comparisons captured qualitatively.
- Short, simple tasks enable objective binary completion measures while isolating controllability from generation complexity and long-horizon drift.
- Participants rate perceived system controllability on a 1–7 scale from no control to full control.
- Figure 8 visualizes qualitative comparisons between the baseline and the proposed method captured during the user study.
- A separate participant blind-reviews generated videos for binary failure or success in task completion.
D. Limitations
The system has important long-horizon and latency limitations despite supporting complex hand–object interactions. Its 1.4-second latency is not sufficient for fully immersive XR systems, though the authors identify possible improvements.
- The causal system struggles with longer-range hand–object–object dependencies.
- The causal model exhibits mode-seeking behavior and oversaturation over long horizons, drawbacks associated with DMD distillation.
- 1.4-second latency is insufficient for fully immersive XR systems.The authors state that this latency may improve through better hardware, alternative distillation methods, and system optimization.