PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoSim: Egocentric World Simulator for Embodied Interaction Generation

Jinkun Hao, Mingda Jia, Ruiyan Wang, Hongrui Zhu, Jiafei Cao, Xihui Liu, Ran Yi, Lizhuang Ma, Jiangmiao Pang, Xudong Xu

arXiv:2604.01001v2cs.CVcs.AI

TL;DR

Existing egocentric simulators lack either explicit 3D grounding or persistent world-state updates, while aligned scene-interaction training data is difficult to acquire at scale. EgoSim uses geometry- and action-aware observation simulation with interaction-aware state updating, supported by scalable video processing and low-cost smartphone capture. Experiments show strong visual, spatial, generalization, and cross-embodiment performance across complex interactions.

  • Problem

    Existing egocentric simulators lack explicit 3D grounding or persistent world-state updating, and densely aligned scene-interaction training data is difficult to acquire at scale.

  • Method

    EgoSim models 3D scenes as updatable world states, generates action-conditioned observations, updates interactive-object geometry, and builds aligned training data from videos and uncalibrated smartphones.

  • Results

    EgoSim generates spatially consistent interactive videos, handles complex dexterous interactions, generalizes across scenes, and adapts to robotic manipulation.

  • Takeaways & Limitations

    The system supports continuous egocentric simulation with persistent scene transitions and cross-embodiment transfer to robotic manipulation.

  • Takeaways & Limitations

    The described baseline world-simulation methods support only coarse controls such as directional commands or camera poses.

Abstract

from arXiv · show

We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation. Existing egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and datasets will be open soon. The project page is at egosimulator.github.io.

1 Introduction

EgoSim targets egocentric simulation with explicit 3D grounding, persistent scene updates, and scalable training data. It is designed to support fine-grained embodied interactions and generalization across complex scenes and embodiments.

  • Motivation: Existing egocentric simulators struggle with 3D consistency, world-state updating across long sequences, and scalable aligned training data.Prior methods use implicit camera motion or explicit scene-action separation, but do not maintain consistent world states and rely on restricted multiview or synthetic data.
  • EgoSim: EgoSim models static 3D scenes as updatable world states for continuous egocentric interaction simulation.Its editable point-cloud representation supports persistent updates after generated interaction observations.
  • EgoSim: The simulator separates geometry-conditioned observation generation from interaction-aware state updating.Generated observations are used to update the point clouds of objects interacting with human or robotic embodiments.
  • Data: A scalable pipeline extracts scene geometry, camera trajectories, and embodiment actions from large-scale monocular egocentric videos.The pipeline addresses the need for aligned scene-interaction training data beyond restricted multiview or synthetic sources.
  • Results: Experiments report realistic, spatially consistent videos, complex dexterous manipulations, strong generalization, and adaptation to robotic-arm manipulation.Reported examples include deformable-object handling, small-object assembly, and promising performance after 100 fine-tuning steps on AgiBot-World data.

3 Method

EgoSim forms a closed loop between action-conditioned observation generation and persistent 3D state updates. Its modules combine scene geometry, camera motion, hand actions, object tracking, and incremental point-cloud fusion.

  • Framework: EgoSim alternates visual observation generation with 3D state updates to maintain temporal and physical consistency.The loop comprises Geometry-action-aware Observation Simulation and Interaction-aware State Updating.
  • Problem Formulation: At stage k, the action is separated into camera trajectory C_k and hand interaction sequence H_k, while S_k represents the maintained 3D scene state.This formulation explicitly distinguishes environment state, interaction action, and visual observation.
  • Observation Simulation: The observation model renders the background from S_{k-1} along C_k and synthesizes hand-action residuals ∆O(H_k).The geometry-action-aware model conditions generation on explicit scene geometry and projected action representations.
  • State Updating: The state updater extracts the latest physical layout from O_k and applies it back to the persistent 3D scene.This prevents manipulated objects such as opened doors from reverting during later renderings.
  • Observation Simulation: The framework uses spatially aligned background, hand, and mask latents to train a diffusion-transformer observation model with an inpainting prior.The inpainting initialization encourages preservation of known backgrounds and generation in action-conditioned regions.
  • State Updating: State reconstruction, interactive-object tracking, and incremental fusion produce editable scene states from generated observation sequences.Interactive objects are identified with a vision-language model and SAM3, while consecutive states are aligned and merged with TSDF fusion.

4 Scalable Data Pipeline for Interactive World Simulators

EgoSim addresses aligned-data collection through an automated pipeline for human and robotic videos, supplemented by EgoCap for low-cost smartphone capture. The resulting representations align scene geometry, camera motion, and embodiment actions.

  • Automated Data Pipeline: Training requires aligned quadruplets containing a static 3D scene, camera trajectory, embodiment actions, and interaction video.The automated pipeline extracts these components from large-scale monocular egocentric and robotic videos.
  • Automated Data Pipeline: The pipeline initializes static scene point clouds by removing hands from an initial frame and unprojecting monocular depth into 3D.SAM3 masks hand regions, Qwen-Image-Editing removes them, and DepthAnything3 supplies depth.
  • Automated Data Pipeline: DepthAnything3 estimates per-frame camera parameters that render the initial point cloud into a geometry-consistent reference video.The estimated parameters include rotation, translation, and intrinsic matrices.
  • Automated Data Pipeline: Human hand poses and robotic end-effector trajectories are converted into a universal 3D action representation and projected onto the camera plane.Human actions use HaMeR pose estimates, while robotic actions use joint trajectories and robot URDF files.
  • EgoCap: EgoCap replaces expensive calibrated multicamera arrays with uncalibrated smartphones for low-cost viewpoint-aligned data collection.The pipeline builds a 3D Gaussian Splatting map, relocalizes interaction recordings against it, and refines the recovered trajectory before aligned rendering.

5 Experiments

EgoSim is evaluated for visual quality, spatial controllability, continuous generation, real-world adaptation, cross-embodiment simulation, and 3D-state updating. Across these settings, it maintains realistic interactions and spatially coherent scene representations.

  • Experimental Setup: EgoSim is evaluated on EgoDex and EgoVid using video-quality and spatial-controllability metrics, with baselines extended to 61 frames.The metrics include PSNR, SSIM, LPIPS, Depth-ERR, and Cam-ERR.
  • Qualitative Results: EgoSim preserves physical state changes, hand-object identity, temporal coherence, unobserved backgrounds, and egocentric viewpoint transitions better than the compared baselines.The comparison covers tabletop scooping and in-the-wild watercolor-painting scenarios.
  • Real-World Adaptation: With 30 training and 20 testing clips, EgoCap enables physically plausible interactions and consistent backgrounds on unseen supermarket scenes after 50 fine-tuning steps.The clips were captured in a supermarket environment focusing on shelf interactions.
  • Quantitative Comparison: 25.056 PSNR and 0.896 SSIM are achieved on EgoDex, while EgoSim maintains the highest visual fidelity on challenging in-the-wild EgoVid scenes.These results are reported for the single-clip setting.
  • Quantitative Comparison: 8.888 Depth-ERR is achieved by EgoSim on EgoDex versus 44.345 for InterDyn, while Cam-ERR is reduced by over an order of magnitude.The authors attribute this spatial improvement to rendering point clouds along the true camera trajectory.
  • Ablation Study: Removing camera-trajectory rendering lowers PSNR to 23.380, whereas accurate visibility masks yield the best PSNR of 25.056.The ablations test explicit camera motion and mask constraints.
  • Ablation Study: The complete interaction-aware state-update pipeline maintains a clean, coherent 3D representation, while removing filtering, TSDF fusion, or spatial masks introduces reconstruction artifacts.The ablation visualizes post-interaction point clouds from a novel third-person view.

6 Conclusion

EgoSim is presented as a closed-loop egocentric simulator that combines explicit 3D grounding with persistent state updates. Its experiments support complex interactions, scalable data construction, real-world capture, and robotic transfer, while monocular reconstruction remains a limitation.

  • Conclusion: EgoSim anchors egocentric video generation to an updatable 3D point cloud and maintains persistent world states across long-horizon interactions.The system includes an Updatable 3D Memory and a scalable web-video data framework complemented by EgoCap.
  • Conclusion: EgoSim performs well in complex multi-object interactions and transfers to bimanual robotic manipulation.The conclusion also highlights aligned triplet extraction from web-scale videos and low-cost real-world capture.
  • Limitations: Monocular depth and camera-pose estimation can fail in heavily occluded or highly dynamic environments, producing imperfect initial point clouds.The authors identify robust multi-view priors and physics-based contact constraints as future directions.

Supplementary Material

The supplementary material extends evaluation beyond the main experiments with 3D consistency, fine-grained interaction accuracy, additional comparisons, and continuous state-update visualizations.

  • Additional Evaluation: The TRUMANS evaluation uses ground-truth camera poses and depth maps to assess 3D consistency beyond image-level quality.This provides a 3D-aware assessment of whether updated world states remain consistent with the actual scene.
  • Fine-Grained Metrics: EgoSim achieves better 3D PA-MPJPE than InterDyn, better PCK in dynamic EgoVid scenes, and higher 6 DoF accuracy across all reported metrics.The metrics evaluate hand pose and object-level pose accuracy.
  • More Comparisons: Across four diverse scenarios, EgoSim generates more realistic hand-object interactions and better preserves scene consistency than Mask2IV, InterDyn, and CosHand.These are additional qualitative comparisons.
  • Continuous Generation: Continuous-generation examples show interacted objects correctly updated and registered in the global scene after each generation stage.Examples include closing cup lids and retaining bread placements on plates.

2 Application

EgoSim supports robot manipulation, correct and incorrect action rollouts, persistent object dynamics, and human-to-robot behavior transfer. These applications show its use as an action-conditioned visual dynamics model across embodiments and viewpoints.

  • Robot Manipulation: EgoSim is evaluated for real-robot tabletop manipulation on Agibot G1 after Pi 0.5 fine-tuning from 150 teleoperated samples.The baseline success rate is 53.3% (8/15) in 15 trials.
  • Action-Conditioned Simulation: EgoSim follows human or robot skeleton actions while generating physically plausible successful and faulty embodiment interactions.Faulty interactions can be driven by randomized or incorrect input trajectories, including corresponding object dynamics.
  • Real-World Scene Simulation: EgoSim can generate interactions in novel supermarket scenes from a single EgoCap scan after tuning on fewer than ten local manipulation clips.The resulting system functions as a scalable action-conditioned data engine for real-life scene simulation.
  • Cross-Embodiment Transfer: Human manipulation trajectories can be retargeted to Agibot-G1 end-effectors, producing robot videos consistent with demonstrations and physically plausible.This establishes transfer from human demonstrations to robot manipulation videos.
  • Persistent 3D States: Persistent 3D states allow EgoSim to update an out-of-view toy and regenerate it consistently when the camera returns to the object.The example concerns a green Crayon Shin-chan toy in an in-the-wild scene.

3 Interaction-aware State Updating.

The Interaction-aware State Updating pipeline reconstructs synchronized interaction metadata in 3D and incrementally fuses observations into a persistent scene state. It filters interaction candidates using semantic, spatial, and depth cues before updating geometry.

  • The pipeline outputs synchronized instance-level interaction metadata within a reconstructed 3D scene state.
  • Object masks are initialized with Grounding-DINO-guided SAM3 using interaction-object text phrases, while camera intrinsics are estimated with GeoCalib.
  • A dual-pass DROID-SLAM process aligns per-frame depths, camera poses, and intrinsic attributes across keyframe and non-keyframe observations.
  • A five-stage hierarchical filter identifies interaction states using human-centric tags, hand-object IoU, depth differences, and retrospective temporal checks.Objects require IoU greater than 0.15 with the hand box and median depth difference below 0.15m.
  • The final global 3D state fuses multiframe point clouds with TSDF integration and incremental overlap-aware updates.TSDF fusion weights and averages observations within voxels, while non-interactive objects retain their last observed geometry.

4 Data Pipeline Details

The data pipeline addresses the precision–diversity trade-off in egocentric datasets by constructing aligned, embodiment-agnostic interaction data from heterogeneous sources. It reconstructs static scenes, standardizes actions, and cleans camera trajectories for world-model training.

  • 4.1 Introduction: Existing egocentric datasets trade structural precision against data diversity, motivating a scalable pipeline and heterogeneous manipulation dataset.Synthetic data is aligned but limited and affected by sim-to-real gaps, whereas web and wearable data are diverse but noisy.
  • 4.1 Introduction: The dataset combines embodiment-agnostic sources, dexterous hand-object interactions, camera locomotion, and multimodal annotations including depth, trajectories, skeletons, and static reconstructions.
  • 4.1 Introduction: The dataset is presented as a foundation for world models generalizing to complex real-world manipulation and locomotion tasks.
  • 4.2 Data Annotation Pipeline: The four-stage pipeline standardizes clips, reconstructs static scenes, represents actions, and generates prompts from diverse human and robotic datasets.
  • 4.2 Data Annotation Pipeline: Static-scene reconstruction removes embodiment regions through masking and inpainting, then estimates spatial geometry from camera trajectories, point clouds, and depth maps.
  • 4.2 Data Annotation Pipeline: A unified skeleton maps human 21-keypoint MANO actions and robot end-effector states into a shared representation for embodiment-agnostic modeling.
  • 4.3 Data Cleaning Pipeline: Camera cleaning detects trajectory outliers from translation and rotation discrepancies, while quaternion renormalization preserves valid rotations.
  • 4.4 Dataset Statistics: 500K clips provide diverse mobile egocentric data with aligned camera trajectories and depth maps for physical grounding.

5 Detailed Implementation of EgoCap Pipeline

EgoCap reconstructs and relocalizes scenes from uncalibrated smartphone videos, producing globally consistent 3D maps and smoothed 6-DoF camera trajectories for aligned rendering.

  • EgoCap recovers precise 6-DoF camera trajectories from uncalibrated videos through reconstruct-then-relocalize mechanisms.
  • During scanning, COLMAP self-calibrates camera intrinsics while joint SLAM and 3D Gaussian Splatting produce a scale-consistent global map.
  • During interaction, dense relocalization matches frames to the reconstructed map and recalibrates intrinsics under occlusion and motion blur.
  • Temporal refinement removes anomalous jumps, interpolates translations and rotations, and applies global filtering to smooth handheld trajectories.

6 More Experiment Details

EgoSim is trained and evaluated with standardized video settings, continuous 3D state updating, and autoregressive or frame-independent baselines. Its continuous simulator renders updated states as geometric conditions for later sequences.

  • 6.1 Training and Inference: The base video diffusion model is fine-tuned for 4,000 steps at 832×480 resolution with 61 frames at 16 FPS, while text, VAE, and image encoders remain frozen.
  • 6.1 Training and Inference: Robot evaluation uses 50K training clips and 150 non-overlapping test clips, comparing models with and without hand-data pre-training.
  • 6.1 Training and Inference: EgoCap evaluation uses 30 supermarket clips for training and 20 for testing, with fine-tuning for 50 steps.
  • 6.1 Training and Inference: All evaluation settings use Flow Matching with 50 inference steps and classifier-free guidance scale 1.0.
  • 6.1 Training and Inference: Continuous simulation uses an updatable 3D interaction-aware state to preserve spatial and temporal consistency across generated sequences.
  • 6.1 Training and Inference: The accumulated 3D state is rendered with subsequent actions and camera trajectories, providing a geometric anchor that preserves modified object states.
  • 6.2 Baseline Details: Mask2IV autoregressively generates 16-frame segments from an initial frame and mask sequence, chaining four segments to reach 61 frames.
  • 6.2 Baseline Details: CosHand generates frames independently from hand masks, lacking an explicit mechanism for inter-frame consistency.

7 More Visual Results

Additional qualitative results show EgoSim handling diverse scenarios and embodiments, including dexterous manipulation, viewpoint changes, real-world object dynamics, and robotic interactions. The visualizations emphasize geometric consistency, realistic dynamics, and cross-embodiment generalization.

  • Dexterous Manipulation and Large Viewpoint Movements: EgoSim handles dexterous hand-object interactions with precise control and high visual fidelity while maintaining manipulated-object geometry.
  • Dexterous Manipulation and Large Viewpoint Movements: The method simulates interactions under large ego-motion and viewpoint shifts without severe structural distortions or blending artifacts.
  • Cross-Embodiment Manipulation: EgoSim generalizes from human hands to robotic end-effectors, generating physically plausible bimanual-gripper interactions guided by extracted 3D action keypoints.
  • Cross-Embodiment Manipulation: The simulator models complex deformable-object dynamics during robotic manipulation.
  • Real-World Object Dynamics with EgoCap: Using EgoCap data from unstructured real-world scenes, EgoSim simulates realistic object dynamics while keeping renderings aligned with complex backgrounds.
Loading 2604.01001v2…