EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

1. Introduction

• Scaling joint annotations is difficult because occlusions, rapid head motion, and blur challenge reliability, while calibrated multi-camera and motion-capture setups hinder collection across diverse real-world settings.The paper motivates a collection and annotation pipeline that reduces reliance on specialized infrastructure.Source: §1 · p. 2
• EgoXperience combines data collection, automatic annotation, unified modeling, and benchmarking to jointly reconstruct egocentric 3D activity.Existing methods often recover only subsets of coupled components, and combining their outputs can leave inconsistent scales and coordinate frames.Source: §1 · p. 2
• EgoX-Engine uses verified object tracking and SLAM-based alignment to annotate 50 hours of task-driven recordings as EgoX-Data-6M, with second-level task captions.EgoX-Bench is curated from one held-out hour and evaluates camera, scene depth, hand, and object reconstruction across capture settings; EgoX takes monocular video and first-frame object masks as input for a single-pass joint reconstruction.Source: §1 · pp. 2–3
• Against the strongest competing methods, EgoX reduces camera nATE by 68.9% on HOI4D, hand W-MPJPE and depth AbsRel by 65.6% and 42.9% on EgoX-Bench, and improves object AUC-S50 by 28.0% on HOT3D.It ranks first in depth, hand, and object metrics across four benchmarks and has the best camera accuracy on HOI4D.Source: §1 · p. 3

2.1. Scene and entity reconstruction

• EgoX extends scene and entity reconstruction to moving hands and manipulated objects, jointly estimating their motion, camera poses, and depth in a shared metric frame.Prior approaches include pairwise pointmaps, scene memory, dynamic-scene tracking, and entity-aware reconstruction.Source: §2.1 · p. 3

2.2. Hand-object reconstruction

• EgoX jointly estimates camera poses, depth, hand geometry, and rigid-object poses from RGB observations and optional first-frame object masks.Compared approaches variously require separate stages, test-time guidance, object templates, or additional object-pose inputs.Source: §2.2 · p. 3

2.3. Egocentric datasets and supervision

• EgoX-Data-6M complements existing egocentric resources with aligned camera, depth, hand, and object annotations across workspace manipulation, cross-area interaction, and outdoor locomotion.Table 1 compares dataset scale, capture settings, and annotation coverage; its object count refers to objects with 3D models.Source: §2.3 · p. 4, Table 1 · p. 4

3.1. EGOX-Engine

• EGOX-Engine tracks pre-scanned objects, verifies their poses against observed depth, and uses SLAM camera poses to align object, scene, and hand annotations in a session frame.Stereo depth provides metric geometry, and camera, scene, hand, and object estimates are transformed into the same coordinate system.Source: §3.1 · pp. 4–5

3.2. EGOX-Data-6M

• EGOX-Data-6M contains 50 hours of task-driven recordings spanning workspace manipulation, cross-room interactions, and outdoor locomotion, with aligned geometric annotations and second-level task captions.A quality-controlled one-hour subset, EGOX-Bench, is reserved for joint evaluation of camera, scene, hand, and object reconstruction.Source: §3.2 · p. 6

3.3. EGOX

• EGOX jointly predicts cameras, scene geometry, and hand-object motion in a shared frame, composing predictions through camera poses and supervising the resulting geometry.Its single-forward-pass outputs include camera intrinsics and poses, dense depth, two-hand MANO states, object poses, and positive metric scale; losses are applied where labels are available.Source: §3.3 · pp. 7, 9

4.2. Main results

• Across all four benchmarks, EgoX achieves the lowest PA-, WA-, and W-MPJPE.On HOT3D, W-MPJPE falls from 56.30 to 38.89 mm relative to HaWoR; on EgoX-Bench, it falls from 328.81 mm for WiLoR + SLAM to 113.24 mm.Source: Table 3 · p. 10, §4.2 · p. 10
• On EgoX-Bench, EgoX’s AUC-S50 is 76.59 versus 59.80 for GigaPose + MegaPose, and it leads ADD, ADD-S, and AUC-S50 across all four benchmarks.The strongest competing EgoX-Bench result uses metric CAD assets and per-frame masks; EgoX also outperforms Any6D on HOI4D, where Any6D receives estimated depth, intrinsics, and per-frame masks.Source: Table 4 · p. 10, §4.2 · pp. 10–11
• EgoX achieves the lowest AbsRel and highest δ1.25 on all four benchmarks.On EgoX-Bench, AbsRel is 0.0460 versus 0.3972 for ReViV and 0.0806 for VGGT-Ω; δ1.25 is 97.37 versus 91.57 for VGGT-Ω. EgoX uses its own predicted metric scale.Source: Table 5 · p. 10, §4.2 · p. 11
• EgoX has lower nATE and RRE than ReViV on all four benchmarks, but VGGT-Ω leads on HOT3D and EgoX-Bench, and DA3 Nested leads on H2O.On HOI4D, EgoX’s nATE is 0.0066 versus 0.0212 for VGGT-Ω, and its RRE is 0.78° versus 1.50° for VGGT.Source: Table 5 · p. 10, §4.2 · p. 11

4. Experiments

• The evaluation uses identical 150-observation clips, excludes evaluation sequences from EgoX training, and compares methods only on outputs they provide.EgoX is the only end-to-end method in the comparison that jointly reconstructs cameras, depth, hands, and object 6-DoF poses in a shared world frame.Source: §4 · p. 9, §4.1 · p. 10

4.3. Ablation studies

• With oracle scale, removing ScaleHead raises W-MPJPE from 38.89 to 40.67 mm and lowers AUC-S50 from 73.71 to 60.87.The comparison removes ScaleHead and uses oracle scale to convert the ablated model’s predictions to metric units.Source: Table 6 · p. 11, §4.3 · p. 11
• Removing DepthHead raises W-MPJPE to 67.53 mm and lowers AUC-S50 to 41.09.The remaining architecture and predicted scale are retained; the authors attribute the reconstruction change to explicit depth prediction improving hand and object reconstruction through the shared representation.Source: Table 7 · p. 11, §4.3 · p. 11

5. Conclusion

• EgoXperience combines capture, annotation, and model learning; EgoX jointly reconstructs cameras, dense depth, hands, and rigid-object 6-DoF poses in a common metric world frame.The current formulation assumes rigid objects and does not predict their meshes.Source: §5 · p. 11

A.1. Joint geometry and entity reconstruction

• Unlike methods that estimate scene geometry or dynamic content separately, EgoX directly predicts articulated hands and rigid-object poses alongside cameras and depth.The related work also describes methods for pairwise pointmaps, recurrent scene memory, moving-content reconstruction, and joint human or instance modeling.Source: §A.1 · p. 18

A.2. Hand and hand–object reconstruction

• Related hand and interaction methods use separate stages, motion priors, test-time guidance, or camera-hand estimation; EgoX additionally predicts manipulated-object motion.The discussed object and interaction methods vary in their use of templates, reference inputs, masks, and world-camera trajectories.Source: §A.2 · p. 18

A.3. Camera, depth, and object-pose pipelines

• EgoX learns a scale per window shared by depth, camera, hand, and object translations, while jointly learning camera and object estimates.The related pipelines include metric-depth estimation and object-pose systems that use CAD models, reference images, or RGB-D observations.Source: §A.3 · pp. 18–19

A.4. Datasets and geometric evaluation

• Existing hand-object datasets cover varied capture protocols, while capture volumes or clip protocols often limit evaluation of continuous camera travel between workspaces.More recent resources include indoor and outdoor interaction recordings, kitchen-video annotations, and digital-twin datasets, but their annotation purposes differ.Source: §A.4 · p. 19
• EgoX-Data-6M spans workspace manipulation, travel between indoor workspaces, and outdoor locomotion, with hand, object, and scene annotations.The data enables evaluation across camera displacement, capture setting, and sequence duration using camera, depth, and world-space interaction errors.Source: §A.4 · p. 19

B. Public Dataset Cleaning and Geometric Alignment

• Cleaning removes frames with erroneous hand annotations and spurious depth-derived points, then aligns depth geometry, MANO hand meshes, and object meshes in a common 3D coordinate system.The authors state that these steps reduce erroneous labels and conflicting geometric supervision across reconstruction tasks.Source: B. Public Dataset Cleaning and… · p. 19

• Four sequence-disjoint benchmarks cover HOT3D, EgoX-Bench, H2O, and HOI4D, with evaluation sequences excluded from EgoX training.All methods use identical sequence lists and matched 150-observation clips.Source: C. Evaluation Protocol · p. 19
• EgoX receives RGB and optional first-frame object masks, while object-pose baselines may require depth, intrinsics, per-frame masks, or metric CAD assets.Ground-truth camera poses and scale are not reconstruction inputs in the main comparisons.Source: C. Evaluation Protocol · p. 19
• Hands use PA-, W-, and WA-MPJPE; cameras use nATE and RRE; depth uses AbsRel and δ1.25; object poses use ADD, ADD-S, and AUC-S50.World-space hand metrics use each pipeline’s estimated camera trajectory, while object pose is evaluated in camera coordinates.Source: C. Evaluation Protocol · p. 20
• EgoX and native metric baselines retain metric scale, whereas VGGT and VGGT-Ω use one DA3 Nested depth-fitted scale coefficient per sequence.Alignment, aggregation, validity masks, and failure handling are held fixed across methods.Source: C. Evaluation Protocol · p. 20

• Inference time is measured per method for 150 frames on one NVIDIA A800, as reported in Table 8.Models and reusable assets are preloaded.Source: D. Runtime Analysis · p. 20, Table 8 · p. 20
• Data loading, visualization, and disk I/O are excluded from the runtime measurement.Source: D. Runtime Analysis · p. 20
• Object-pose baseline runtimes include DA3 Nested preprocessing whenever estimated depth or intrinsics are required.Source: D. Runtime Analysis · p. 20, Table 8 · p. 20

E.1. Additional hand and camera comparisons

• Figure 8 adds HOI4D and H2O qualitative comparisons for world-space hand and camera reconstruction.It supplements Figure 5.Source: §E.1 · p. 20, E. Additional Qualitative Results · p. 20, Figure 8 · p. 21

E.2. Additional object-pose comparisons

• Figure 9 adds HOI4D and H2O qualitative comparisons for camera-frame object-pose reconstruction.It supplements Figure 6.Source: E. Additional Qualitative Results · p. 20, Figure 9 · p. 21, §E.2 · p. 21

E.3. Additional depth comparisons

• Figure 10 adds HOI4D and H2O qualitative comparisons for depth predictions.It supplements Figure 7.Source: E. Additional Qualitative Results · p. 20, §E.3 · p. 21, Figure 10 · p. 22

• The approximately 280-hour training pool combines EgoX-Data-6M, six public interaction datasets, three additional real-world sources, and four synthetic sources.Training first mixes real and synthetic data, then continues with an equal mixture of retained real interaction sources.Source: F. Training and Compute · p. 22
• A 24-block reconstruction Transformer uses DINOv3 ViT-L/16 patch features, with 19 blocks exchanging all view tokens and five processing compact register and entity tokens.Camera and depth heads read fused image features; hand and object heads read view-specific entity states, without temporal encoding.Source: F. Training and Compute · p. 23
• Training normalizes metric depth and translations by z_gt, the mean distance of valid metric-depth points from the reference-camera origin after transformation into world coordinates.Rotations and intrinsics remain unchanged.Source: F. Training and Compute · p. 23
• The objective combines parameter and dense reconstruction losses, including camera, hand, object, and depth supervision with annotation-validity and in-frame masks.Object rotation losses account for object symmetry; missing annotations are not treated as negative labels.Source: F. Training and Compute · p. 23
• Training samples 1–24 observations per input set and jointly optimizes all modules with AdamW, warm-up, cosine decay, mixed precision, and module-wise gradient clipping.Batch size varies with observation count to maintain an approximately constant image budget.Source: F. Training and Compute · p. 23