PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the World and the Self from Egocentric Video

Kai Guan, Minchao Jiang, Ruichen WangLi, Wentao Zhu, Lei Zhang

arXiv:2609.01276v1cs.CV

TL;DR

Egocentric 3D perception must recover the scene and the wearer’s motion in one metric frame, but existing methods typically address only one side. RESELF couples metric geometry reconstruction with geometry-conditioned diffusion and closed-loop kinematic feedback, and outperforms task-specific state-of-the-art methods across depth, camera tracking, and full-body motion estimation.

  • Problem

    Existing methods separate scene reconstruction, which ignores the wearer, from motion estimation, which lacks explicit scene geometry and often relies on external trajectories.

  • Method

    RESELF adapts a geometry foundation model to egocentric video, conditions diffusion-based motion generation on camera trajectories and geometric features, and refines the camera through closed-loop kinematic feedback.

  • Results

    RESELF outperforms state-of-the-art methods designed for individual tasks across depth estimation, camera tracking, and full-body motion estimation.

  • Takeaways & Limitations

    Joint scene-and-self reconstruction provides a metrically consistent representation linking spatial perception with the wearer’s kinematic action.

  • Takeaways & Limitations

    The motion module avoids gravity and floor assumptions because the predicted camera trajectory has arbitrary global orientation and no reliable ground plane.

Abstract

from arXiv · show

Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Existing methods typically address scene reconstruction and motion estimation separately: scene reconstruction methods ignore the wearer, whereas motion estimation methods lack explicit scene geometry and often depend on external trajectories. Joint recovery is challenging because the two tasks exhibit asymmetric visibility and require different prediction paradigms. The largely visible scene supports deterministic geometric regression, whereas the severely occluded body requires generative motion inference. We therefore propose RESELF (REconstructing the Scene and the sELF), a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation. RESELF adapts a geometry foundation model pre-trained on large-scale exocentric data to egocentric video using frame-wise scale and relative-pose consistency objectives. The resulting camera trajectory and latent geometric features condition a diffusion model that recovers the wearer's motion. A subsequent closed-loop kinematic feedback stage further refines the camera head while preserving the reconstructed scene geometry. To support training and evaluation, we curate EE4D-JSM from EgoExo4D by aligning egocentric video, sparse metric scene geometry, camera trajectories, and full-body motion annotations. Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation. Code, models, and datasets will be available at https://ka1guan.github.io/RESELF/.

1 Introduction

RESELF frames egocentric 3D perception as joint recovery of scene geometry, camera motion, and the wearer’s full-body motion. It couples deterministic geometric reconstruction with generative motion inference in a shared metric representation.

  • Motivation: Egocentric video couples observations of the environment with camera motion induced by the wearer’s actions, requiring joint recovery of scene geometry, camera trajectory, and full-body motion.The scene provides spatial context for the action, while the wearer’s motion must be situated within that environment.
  • Motivation: Existing methods are self-blind or world-blind: scene models omit the wearer, while motion estimators lack explicit scene geometry and often rely on external camera trajectories.Scene reconstruction can also remain sensitive to rapid egomotion, whereas generative motion methods operate under severe self-occlusion.
  • Motivation: Asymmetric visibility motivates different predictors: the largely observable scene favors deterministic geometric reconstruction, whereas the severely occluded body requires generative motion inference.RESELF asks whether a shared metric representation can couple these task-specific predictors.
  • RESELF: RESELF adapts a geometry foundation model to establish a shared metric frame, then conditions motion generation on predicted camera trajectories and latent geometric features.Frame-wise metric-scale and relative-pose consistency objectives address sparse egocentric geometry supervision and adaptation challenges.
  • RESELF: A closed-loop kinematic feedback stage uses motion supervision to refine the camera head while preserving the reconstructed scene geometry.This motion-to-camera pathway imposes an additional kinematic constraint on the camera trajectory.
  • Results: RESELF jointly addresses scene-only and motion-only limitations and outperforms methods designed for either individual task across depth estimation, camera tracking, and full-body motion estimation.The framework is presented as leveraging a mutually reinforcing relationship between scene and motion.

2 Related Work

Related work separates egocentric scene reconstruction from egocentric human-motion reconstruction. Scene methods omit the wearer, while motion methods reconstruct the body under self-occlusion using head poses, diffusion, or precomputed trajectories.

  • Egocentric Scene Reconstruction: Feed-forward scene reconstruction models predict cameras, depth, point maps, and 3D tracks, but egocentric scene reconstruction remains self-blind by masking or omitting the wearer.The related methods include VGGT, Pi3, CUT3R, and EgoMono4D.
  • Egocentric Human Motion Reconstruction: Egocentric human-motion methods recover full-body motion under severe self-occlusion using estimated head poses, conditional diffusion, or precomputed trajectories.EgoEgo generates motion from head poses, UEM unifies reconstruction, forecasting, and generation, and EgoAllo conditions a motion prior on trajectories.

3 Method

RESELF jointly reconstructs metric scene geometry, camera trajectory, and full-body motion from monocular egocentric video. It adapts a geometry backbone, conditions diffusion-based motion synthesis on geometric outputs, and refines camera estimation through kinematic feedback while preserving scene geometry.

  • 3.3 Geometry-Conditioned Motion Generation: A diffusion motion head uses predicted camera trajectories and latent geometric features to synthesize the wearer’s temporally coherent full-body motion.The motion sequence represents body-joint transforms, reference trajectory, hand poses, foot contacts, and SMPL-X shape coefficients.
  • 3.1 Problem Definition: The geometry backbone establishes a shared metric frame containing per-view scene point maps, camera-to-world transforms, and the wearer’s body state.The body state includes metric root motion and full-body articulation, enabling a metrically consistent scene-and-self representation.
  • 3.2 Egocentric Scene Reconstruction: RESELF adapts Pi3X to dynamic egocentric video with metric-scale and relative-egomotion objectives, producing camera trajectories and latent geometric features.The adaptation addresses frame-wise inconsistencies, metric-scale errors, and camera-pose drift caused by rapid motion and dynamic foreground content.
  • 3.3 Geometry-Conditioned Motion Generation: A head-aligned camera trajectory provides the canonical kinematic reference, avoiding gravity and floor assumptions during motion reconstruction.The factorization separates global egomotion from local body articulation while preserving metric alignment with the reconstructed scene.
  • 3.4 Closed-Loop Kinematic Feedback: Closed-loop kinematic feedback updates only the camera head, propagating motion supervision through a differentiable trajectory condition while keeping other geometry components fixed.Geometric camera supervision remains active, and the feedback refines camera trajectories without altering reconstructed scene geometry.
  • 3.5 EE4D-JSM Dataset: EE4D-JSM aligns egocentric videos, sparse metric scene geometry, camera trajectories, and full-body motion, yielding 16,807 training and 5,212 testing sequences.The dataset spans 18 activities and more than 70 scenes after filtering sequences with nearly unusable geometry.

4 Experiments

RESELF is evaluated on camera trajectory, scene depth, and wearer motion using aligned EE4D-JSM supervision and controlled comparisons. It achieves state-of-the-art performance across these tasks, with ablations attributing gains to metric-scale objectives, geometry-conditioned motion, and kinematic feedback.

  • Egocentric Scene Reconstruction and Trajectory: ATE 0.012 and Abs-R 0.125 are RESELF’s best Similarity Alignment results across trajectory and depth metrics.Under Absolute Metric Scale, it also reduces ATE from 0.026 to 0.022 and RPE-t from 0.019 to 0.010 versus Pi3X.
  • Egocentric Human Motion Reconstruction: RESELF reduces MPJPE from 116.1 mm to 109.8 mm and H-MPJPE from 184.2 mm to 174.6 mm versus UEM.It also attains PA-MPJPE of 75.3 mm and reduces HTE from 65.9 mm to 50.1 mm, with consistent HRE improvement.
  • Qualitative Comparisons: Qualitative sequences show RESELF following basketball camera ground truth more closely and recovering more plausible bike and cooking motion than UEM.RESELF reduces cumulative camera drift, improves bike-sequence articulation and foot-ground consistency, and limits cooking-sequence global root drift.
  • Ablation Study: Frame-wise metric-scale and relative-pose objectives jointly provide the best balance between camera tracking and scene depth.Lfm notably improves trajectory estimation, Lego primarily improves relative camera motion, and combining both yields RESELF-s1.
  • Ablation Study: Geometry features improve body and hand reconstruction beyond generic appearance cues, while the camera trajectory anchors global motion.Removing the trajectory substantially increases global body error, whereas combining trajectory and geometry features achieves the best overall performance.
  • Ablation Study: Kinematic feedback, rather than camera-only fine-tuning, consistently improves all trajectory metrics while the geometry encoder and depth head remain frozen.The controlled comparison uses the same additional 10 epochs and averages results over three independent runs.

5 Conclusion

RESELF jointly reconstructs the egocentric scene and wearer’s full-body motion from a single video, linking spatial perception and kinematic action in a shared representation. Experiments report state-of-the-art performance across scene reconstruction and human motion estimation.

  • RESELF jointly reconstructs 3D scene geometry and the wearer’s full-body motion from a single egocentric video stream.
  • A conditional diffusion motion model is anchored in an ego-adapted, metric-scale geometric foundation to connect spatial perception with kinematic action.
  • Closed-loop kinematic feedback uses human kinematic priors to regularize global camera trajectories and close the optimization loop between body and environment.
  • RESELF achieves state-of-the-art performance in both scene reconstruction and human motion estimation.
  • Jointly modeling the world and the self yields mutually consistent gains and a physically grounded 3D representation for embodied intelligence.

A Supplementary Material

The supplementary material documents the dataset, objective sensitivity, motion-generation feature aggregation, and closed-loop feedback analyses.

  • The supplement details EE4D-JSM construction, alignment, camera processing, statistics, and scene-quality filtering.
  • It analyzes the frame-wise metric-scale and adjacent-frame relative-pose loss weights used for egocentric scene reconstruction.
  • It reports ablations of register-token aggregation strategies and transformer-layer selection for geometry-conditioned motion generation.
  • It includes controlled experiments analyzing closed-loop kinematic feedback.

A.1 Details of EE4D-JSM

EE4D-JSM is built from EgoExo4D to provide aligned egocentric observations, scene points, camera information, and motion-learning sequences across diverse activities and settings.

  • EgoExo4D covers seven activity domains across diverse participants and environments using synchronized egocentric and exocentric Project Aria recordings.
  • Figure 7 filters sequences by valid Project Aria scene-point density, retaining sequences averaging 1.4768% and discarding those averaging 0.0090%.
  • EE4D-JSM contains 16,807 training sequences and 5,212 testing sequences after removing 12 testing sequences.
  • The dataset uses EgoExo4D because its scale and diversity cover seven activity domains and 94 physical settings while providing the observations and annotations required for the joint task.

A.2 Sensitivity Analysis of Egocentric Scene Reconstruction Objectives

The sensitivity analysis shows that metric-scale and relative-pose objectives require balanced weighting: weaker or excessive emphasis harms different aspects of reconstruction, while the selected configuration performs best overall.

  • The selected objective-weight configuration provides the best overall balance between metric scene reconstruction and camera trajectory consistency.
  • Frame-Wise Metric-Scale Loss Weight: The frame-wise metric-scale loss weight λfm = 0.5 achieves the best performance across all reported metrics.
  • Frame-Wise Metric-Scale Loss Weight: Reducing λfm degrades scene depth accuracy and trajectory estimation, while increasing it to 1.0 also degrades both metrics.
  • Adjacent-Frame Relative Pose Loss Weight: A small λego = 0.01 causes increased rotational drift, whereas overly strong adjacent-frame regularization degrades ATE and scene depth accuracy.
  • Adjacent-Frame Relative Pose Loss Weight: The selected λego = 0.02 achieves the best ATE and Abs-R while matching the best RPE-t.

A.3 Feature Aggregation for Geometry-Conditioned Motion Generation

This analysis compares ways to aggregate geometry foundation-model register tokens for geometry-conditioned motion generation. Attention pooling over concatenated tokens provides the most informative geometric condition among the tested alternatives.

  • Aggregation strategy: Attention pooling over concatenated register tokens consistently outperforms mean pooling and an MLP aggregator across body, aligned-body, and hand reconstruction metrics.The comparison keeps the trajectory condition, motion representation, and diffusion architecture unchanged.
  • Layer selection: Using either transformer layer alone remains competitive, with the second-to-last layer performing slightly better.

A.4 Additional Analysis of Closed-Loop Kinematic Feedback

The additional analyses show that closed-loop kinematic feedback, rather than extra camera-only optimization, improves trajectory estimation. Camera supervision remains necessary, while a small motion-feedback weight gives the best overall performance.

  • Controlled feedback analysis: Closed-loop kinematic feedback yields lower mean errors across all trajectory metrics than camera-only optimization, especially for ATE and RPE-r.Both final-stage variants receive the same additional 10 epochs, and the comparison averages three independent runs.
  • Controlled feedback analysis: Camera-only optimization produces marginal changes within run-to-run variation, indicating that prolonged camera training alone provides little benefit.
  • Motion-feedback sensitivity: Removing Lcam substantially degrades all trajectory metrics, showing that motion gradients alone do not provide sufficient geometric constraints for stable metric tracking.
  • Motion-feedback sensitivity: λmot = 0.01 achieves the best overall performance when camera supervision is retained.Larger feedback weights provide limited benefit and can interfere with the established geometric representation.

A.5 Regression versus Generative Motion Modeling

The analysis tests whether deterministic regression can resolve the ambiguity of full-body reconstruction from severely partial egocentric observations. Conditional diffusion performs better because it models multiple plausible motion hypotheses rather than averaging them into one solution.

  • Comparative results: The regression variant performs consistently worse across all motion metrics, with particularly large degradation in head translation, global joint position, and hand reconstruction.Both variants use the same motion representation and conditioning inputs.
  • Interpretation: Deterministic regression recovers approximate local articulation but struggles to place the body coherently in the global scene.Its degradation is relatively smaller after pose alignment.
  • Interpretation: The diffusion head models the conditional motion distribution through iterative denoising, supporting motion consistent with the estimated camera trajectory and surrounding geometry.This addresses the one-to-many ambiguity created by partial observability.

A.6 Additional Visualization Results

The visualization shows RESELF jointly reconstructing scene geometry, camera motion, and the wearer’s full-body motion in a shared metric frame. The failure analysis highlights that camera drift can preserve local articulation while corrupting global motion placement and orientation.

  • Overall visualization: On a cooking sequence, RESELF produces unified 3D reconstructions containing the scene point cloud, camera pose, and wearer’s SMPL mesh in a shared metric frame.The recovered body remains spatially aligned with the estimated camera and surrounding scene throughout the sequence.
  • Failure analysis: In a failure case with inaccurate camera trajectory estimation, predicted motion preserves local articulation but deviates in global position and orientation.The motion overlay and camera trajectories exhibit corresponding global drift.
  • Failure analysis: Camera errors propagate to global body motion because the estimated trajectory provides the global anchor for motion generation.Severe errors are more likely under rapid head motion, motion blur, limited parallax, or weak and repetitive scene textures.
Loading 2609.01276v1…