PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoForge: Goal-Directed Egocentric World Simulator

Yifan Shen, Jiateng Liu, Xinzhuo Li, Yuanzhe Liu, Bingxuan Li, Houze Yang, Wenqi Jia, Yijiang Li, Tianjiao Yu, James Matthew Rehg, Xu Cao, Ismini Lourentzou

arXiv:2603.20169v1cs.CVcs.MM

TL;DR

Egocentric video simulation must handle rapid viewpoint changes, hand–object interactions, and latent intent without relying on dense motion supervision. EgoForge generates goal-directed first-person rollouts from minimal static inputs using geometry-aware grounding and VideoDiffusionNFT reward-guided refinement. It consistently outperforms strong baselines across semantic alignment, geometric stability, and motion fidelity, with smart-glasses validation.

  • Problem

    Egocentric video generation remains challenging because viewpoint changes, hand–object interactions, and goal-directed evolution depend on latent human intent, while existing methods may require dense supervision.

  • Method

    EgoForge conditions egocentric rollout generation on an egocentric observation, high-level instruction, and optional exocentric reference, adding geometry-aware grounding and VideoDiffusionNFT trajectory refinement.

  • Results

    EgoForge consistently outperforms strong baselines across metrics, including +13.5% DINO-Score, +10.1% CLIP-Score, 43% lower FVD, and 51% lower flow MSE.

  • Takeaways & Limitations

    The framework produces egocentric rollouts with improved instruction alignment, realistic motion, stable scene evolution, and robust performance in smart-glasses scenarios.

  • Takeaways & Limitations

    At inference, EgoForge assumes an initial egocentric frame or short clip and an optional exocentric reference, while requiring the rollout to satisfy goal, temporal, and physical consistency.

Abstract

from arXiv · show

Generative world models have shown promise for simulating dynamic environments, yet egocentric video remains challenging due to rapid viewpoint changes, frequent hand-object interactions, and goal-directed procedures whose evolution depends on latent human intent. Existing approaches either focus on hand-centric instructional synthesis with limited scene evolution, perform static view translation without modeling action dynamics, or rely on dense supervision, such as camera trajectories, long video prefixes, synchronized multicamera capture, etc. In this work, we introduce EgoForge, an egocentric goal-directed world simulator that generates coherent, first-person video rollouts from minimal static inputs: a single egocentric image, a high-level instruction, and an optional auxiliary exocentric view. To improve intent alignment and temporal consistency, we propose VideoDiffusionNFT, a trajectory-level reward-guided refinement that optimizes goal completion, temporal causality, scene consistency, and perceptual fidelity during diffusion sampling. Extensive experiments show EgoForge achieves consistent gains in semantic alignment, geometric stability, and motion fidelity over strong baselines, and robust performance in real-world smart-glasses experiments.

1 INTRODUCTION

Egocentric world simulation must model rapid viewpoint changes, hand–object interactions, and goal-directed behavior while remaining controllable and physically consistent. EgoForge addresses these challenges from minimal visual context using geometry-aware generation and reward-guided trajectory refinement.

  • Egocentric video is difficult because first-person streams combine rapid viewpoint changes, frequent hand–object interactions, and behaviors governed by latent human intent.
  • XR applications require human-centric world models that generate predictive, controllable, and physically consistent simulations for interaction and decision-making.
  • Existing methods often require dense motion annotations, camera trajectories, long video prefixes, or synchronized multi-view recordings that are costly and unreliable at inference.
  • EgoForge generates goal-directed first-person rollouts from a single egocentric observation, a high-level instruction, and an optional exocentric reference image.
  • VideoDiffusionNFT refines diffusion trajectories using rewards for goal completion, temporal causality, scene stability, and perceptual fidelity.

2 RELATED WORK

Prior work spans egocentric recognition, view translation, video generation, and predictive world models, but often depends on explicit motion supervision or limits scene evolution. EgoForge targets goal-directed egocentric simulation with minimal static observations.

  • EgoForge: EgoForge generates goal-directed first-person rollouts from minimal inputs rather than camera trajectories, pose signals, or synchronized multi-view streams.
  • Egocentric Generation: Prior egocentric generation either uses continuous exocentric video, performs static view translation, or synthesizes hand-centric instructional motion with largely static scene context.
  • Egocentric Vision: Egocentric vision benchmarks support analysis of daily tasks and multi-view activity, while models also forecast hand motions, hand–object interactions, and gaze.
  • Egocentric Vision: Cross-view translation and joint egocentric video-motion synthesis commonly depend on camera trajectories or synchronized multicamera recordings, which are difficult to acquire and do not capture high-level intent.
  • Video Generation: Latent diffusion, score-based modeling, and transformer video diffusion underpin generators with improved temporal coherence, longer duration, and realism.
  • World Models: World models simulate environment dynamics for agent policy optimization or high-fidelity world simulation and game-engine applications.

3. Method

EgoForge generates goal-directed egocentric rollouts by conditioning a diffusion-transformer video generator on visual observations, instructions, and optional exocentric context. Geometry alignment and VideoDiffusionNFT refine spatial stability, temporal coherence, and reward-based trajectory quality.

  • Egocentric Generator: EgoForge synthesizes plausible future egocentric video from an initial observation, instruction, and optional exocentric reference without camera trajectories, pose signals, or synchronized multiview streams.The conditioning context comprises the egocentric input, instruction, and exocentric reference.
  • Egocentric Generator: The diffusion-transformer generates video in pretrained autoencoder latent space, injecting noisy latents, ego features, and conditioning features through adaptive normalization and cross-attention.The model uses a learned timestep embedding and velocity-prediction denoising objective.
  • Geometry Weak Supervision: Geometry Weak Supervision aligns intermediate diffusion representations with pretrained VGGT geometry features using projected features, angular consistency, and scale consistency losses.The combined objective balances angular and scale alignment across selected transformer layers.
  • VideoDiffusionNFT Alignment: VideoDiffusionNFT converts rollout rewards into positive and negative optimality signals, then guides intermediate diffusion states toward higher-reward trajectories.The guidance direction combines the current policy with positive and negative policy velocity fields, with β controlling guidance strength.
  • VideoDiffusionNFT Alignment: The refinement evaluates goal completion, scene consistency, temporal causality, and perceptual fidelity using rollout rewards and vision–language model scoring.These rewards measure task outcome, environment preservation, physically coherent motion, and visual quality.

4. Experiments

Experiments evaluate EgoForge on the X-Ego benchmark through quantitative comparisons, user studies, qualitative rollouts, and module and reward ablations. Across these evaluations, EgoForge shows stronger semantic alignment, visual fidelity, temporal coherence, and scene consistency than baselines and reduced variants.

  • Benchmark and evaluation: X-Ego is a large-scale benchmark with dense annotations for hand–object dynamics, object-state changes, and step-level action semantics.
  • Benchmark and evaluation: EgoForge is evaluated against general-purpose and egocentric-specific video models, with baseline fine-tuning used to reduce domain differences.
  • Quantitative results: +13.5% DINO-Score, +10.1% CLIP-Score, +9.7% SSIM, +17.8% PSNR, and 35% lower LPIPS are reported against the strongest baseline.
  • Quantitative results: 43% lower FVD and 51% lower flow MSE indicate stronger temporal modeling, more coherent motion, and more stable scene dynamics.
  • Module ablations: The full EgoForge model achieves the best overall results after comparing diffusion fine-tuning, geometry weak supervision, and VideoDiffusionNFT refinement.
  • Human evaluation: In a 20-annotator study of 25 video groups, EgoForge scores 4.75 for Alignment and 4.71 for Fidelity, including a +1.60 Alignment advantage over WAN2.2.
  • Module ablations: Geometry supervision improves spatial structure and realism, increasing DINO-Score by +2.1 and CLIP-Score by +1.9 while reducing LPIPS to 0.16 and flow MSE to 3.92.
  • Reward ablations: Removing the perceptual, temporal, goal, or environment rewards selectively harms visual quality, temporal consistency, semantic alignment, or background plausibility.

5 CONCLUSION

EgoForge produces temporally coherent first-person rollouts that follow instructions while preserving scene structure and viewpoint motion. Exocentric guidance can further improve spatial grounding, and smart-glasses deployment tests the simulator on real-world out-of-domain tasks.

  • Conclusion: Dense frame sequences show coherent dynamics, instruction-following, stable scene structure, and consistent egocentric viewpoint motion across activities and environments.
  • Conclusion: An auxiliary exocentric view improves spatial grounding and scene consistency by anchoring simulated trajectories to distinctive reference-environment details.
  • Conclusion: ARGO smartglasses experiments evaluate EgoForge on real-world out-of-domain tasks involving diverse multi-step actions.

5. Conclusion

EgoForge generates coherent, goal-directed first-person rollouts from sparse visual and instructional inputs, using VideoDiffusionNFT to align trajectories with multiple temporal and semantic objectives. Experiments and smart-glasses validation show improved alignment, motion, and scene stability over strong baselines.

  • EgoForge generates coherent first-person rollouts from an egocentric observation, high-level instruction, and optional exocentric reference.
  • VideoDiffusionNFT refines diffusion sampling at the trajectory level across goal completion, scene stability, temporal causality, and perceptual fidelity.
  • EgoForge consistently outperforms strong baselines and performs robustly in smart-glasses scenarios, improving instruction alignment, egocentric motion, and scene evolution.
  • Holistic rewards make multi-step goal completion difficult to optimize because failure at the end does not identify which earlier frame caused it.
  • Long video synthesis can suffer goals drift, with geometry and background elements deviating from conditioning despite per-step input conditioning.
  • Multimodal conditioning can produce shortcuts in which auxiliary visual inputs are followed while procedural text instructions are ignored.

B. VideoDiffusionNFT Reward Details

VideoDiffusionNFT uses reward components to evaluate whether generated manipulation trajectories complete tasks and achieve correct target states.

  • The 2.0-point goal completion reward combines task completion and visual alignment, with each component capped at 1.0.
  • Task completion evaluates dynamic fidelity by rewarding state changes and manipulations that transition the initial state to the final one.

C X-EGO DETAILS

The benchmark and reward design emphasize preserving the environment while producing physically and perceptually coherent manipulation videos. X-Ego supplies large-scale, richly annotated data for evaluating such goal-directed egocentric synthesis.

  • Scene consistency rewards temporal stability of static environmental features and penalizes unmotivated drift or target-image environmental contamination.
  • Temporal causality rewards physical and logical coherence, requiring visible causes such as grasping before corresponding object motion.
  • Perceptual fidelity combines PSNR, FVD, and LPIPS to assess pixel reconstruction, perceptual similarity, realism, and temporal motion.
  • X-Ego is a large-scale benchmark for synthesizing complex goal-directed scenes from sparse static context.
  • X-Ego uses uniformly 10-second clips from Nymeria and Ego-Exo4D, selecting stationary atomic hand-on-object manipulations.
  • Dense annotations describe fine-grained hand-object dynamics, object state changes, and step-level semantics grounded in video evidence.

E VISUALIZATION RESULTS

The visualization-results passage points to high-fidelity video generation evaluated with multiple complementary metrics.

  • EgoForge’s visual results are assessed through multiple complementary metrics for high-fidelity video generation.

D. Video Generation Evaluation

EgoForge is evaluated with metrics covering semantic alignment, visual fidelity, distributional realism, and temporal motion consistency.

  • DINO-Score measures frame-level semantic similarity using average cosine similarity between DINOv2 features.
  • SSIM evaluates visual fidelity by comparing luminance, contrast, and structural information between generated and ground-truth frames.
  • FVD measures distributional realism through Fréchet distance between real and generated video-feature distributions.
  • Optical-flow MSE penalizes discrepancies between generated and ground-truth motion fields.

E. Visualization Results

Visualization results show EgoForge generating long, coherent first-person rollouts across manipulation, deformable-object, navigation, throwing, and assembly tasks. The sequences preserve scene structure and exhibit plausible task-directed motion over extended horizons.

  • Visualization Results: EgoForge maintains a stable first-person viewpoint and preserves scene layout over extended video horizons.
  • Visualization Results: Generated rollouts sustain hand–object interactions within spatially coherent workspaces during manipulation-heavy tasks.Examples include cracking eggs, tearing adhesive tape, chopping onions, and installing drawer handles.
  • Visualization Results: For deformable and large objects, the model preserves object presence and the interaction region over extended sequences.The visualization includes tasks such as folding a blanket.
  • Visualization Results: Dynamic scenarios show temporal progression toward identifiable goals while maintaining environment geometry.Examples include rock climbing with consistent holds and basketball shooting with a consistent hoop.
  • Visualization Results: Figure 7 presents 26 frames from each generated video to highlight seamless transitions and stable dynamics in complex egocentric tasks.
  • Visualization Results: Figure 8 identifies the DigiLens ARGO smart glasses used in the real-world experiments.

F. Device for Real World Experiments

EgoForge is tested in real-world settings using DigiLens ARGO smart glasses, whose sensing and compute capabilities support egocentric data capture. The evaluation framework scores goal alignment, environment preservation, and temporal causality, while auxiliary prompts refine and segment video annotations.

  • Device: The DigiLens ARGO is an enterprise-grade standalone AR system used to capture egocentric video in real-world settings.It uses waveguide technology and is designed for industrial and enterprise applications.
  • Device: ARGO combines a 48MP autofocus camera, optical and electronic stabilization, pixel binning, low-light support, spatial audio, and Snapdragon XR2 compute.
  • Device: The real-world deployment targets out-of-domain robustness, using high-fidelity sensor data to ground visual cues and high-level semantic intent.
  • Evaluation: Goal alignment scores task completion and visual correspondence between the generated final state and the target image.Each component ranges from 0.0 to 1.0, for a total of 0–2.0 points.
  • Evaluation: Environment preservation scores consistency with the initial environment and resistance to contamination from the target image.Consistency and contamination are each scored from 0.0 to 1.0.
  • Evaluation: Temporal causality scores physical plausibility and causal logic in progression from the initial to final state.Each component ranges from 0.0 to 1.0, for a total of 0–2.0 points.
Loading 2603.20169v1…