PaperlayerView exact revision on arXiv

Source-linked AI summary

ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

Xiaozhong Lyu, Gen Li, Zhiyin Qian, Xucong Zhang, Marc Pollefeys, Siyu Tang

arXiv:2607.17790v1cs.CVcs.AI

TL;DR

Egocentric 4D reconstruction remains fragmented because prior methods separate scene perception from human ego-motion and often require auxiliary camera trajectories. ReViV jointly learns viewer and view dynamics from monocular RGB video with masked multimodal generation, achieving state-of-the-art holistic reconstruction while retaining competitive scene estimation and faster feed-forward inference.

  • Problem

    Prior egocentric methods often separate scene perception from human ego-motion and rely on auxiliary camera trajectories, leaving unified monocular viewer–view reconstruction insufficiently addressed.

  • Method

    ReViV uses a Masked Generative Egocentric Transformer to learn joint multimodal distributions and reconstruct body, hands, gaze, camera trajectory, and depth from monocular RGB video.

  • Results

    ReViV achieves state-of-the-art holistic ego-body, hand, and gaze reconstruction while maintaining competitive depth estimation and camera tracking across multiple benchmarks.

  • Takeaways & Limitations

    A single feed-forward model can reconstruct temporally coherent viewer and view dynamics across heterogeneous egocentric modalities.

  • Takeaways & Limitations

    Unified discrete tokenization discards some high-frequency spatial detail, slightly limiting depth estimation relative to task-specific continuous regression methods.

Abstract

from arXiv · show

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.

1 Introduction

ReViV addresses the fragmented and auxiliary-input-dependent state of egocentric reconstruction with a unified model that jointly reconstructs the viewer and the surrounding view from monocular RGB video. Its masked multimodal modeling yields temporally consistent predictions and strong performance across holistic reconstruction tasks.

  • Motivation: Egocentric reconstruction must jointly model the wearer’s actions and surrounding environment, but prior methods commonly treat scene dynamics and human motion independently.Existing human-pose approaches also often depend on auxiliary camera trajectories.
  • Motivation: Third-person and multi-view 4D advances leave reconstruction of the wearer’s body dynamics from monocular egocentric video largely unresolved.
  • Approach: ReViV jointly predicts body, hand pose, gaze, camera trajectory, and depth from a single monocular egocentric RGB video.The predictions are integrated into a temporally consistent viewer–view reconstruction.
  • Approach: Its Masked Generative Egocentric Transformer learns cross-modal dependencies by predicting randomly masked multimodal tokens from visible context.A unified tokenization scheme represents heterogeneous spatiotemporal signals in compact latent representations.
  • Results: ReViV achieves state-of-the-art holistic ego-body, hand, and gaze reconstruction while maintaining competitive depth estimation and camera tracking.Its feed-forward architecture also enables faster inference than optimization-based methods.

2 Related Work

Prior work advances dynamic scene reconstruction, egocentric scene understanding, and human-motion estimation along largely separate tracks. ReViV is positioned as a unified generative approach that combines body, hand, gaze, and scene geometry for egocentric 4D reconstruction.

  • Dynamic 4D reconstruction: Dynamic 4D reconstruction has progressed from object trajectories and multi-view video toward monocular inputs, while feed-forward methods improve efficiency by avoiding test-time optimization.
  • Egocentric scene reconstruction: Egocentric scene methods model modalities such as RGB, depth, gaze, camera trajectories, scenes, or point-cloud sequences, but remain focused on scene understanding or reconstruction.
  • Egocentric human motion: Egocentric human-motion methods address full-body pose, head-conditioned motion, pose forecasting, and object-aware estimation, with distortion, self-occlusion, and missing hand recovery remaining challenges.
  • Egocentric human motion: Gaze has been used for 3D forecasting and gaze-regularized egocentric behavior understanding, rather than as part of a unified viewer–view reconstruction.
  • Research gap: ReViV addresses the open challenge by jointly modeling human body, hand, and gaze dynamics together with scene geometry in egocentric 4D reconstruction.

3 Data Engine

ReViV’s data engine harmonizes heterogeneous egocentric datasets and fills missing geometric supervision to support large-scale multimodal pretraining. The resulting annotations provide temporally consistent signals for learning 4D structural priors.

  • Dataset construction: The automated multimodal data engine harmonizes spatial representations and compensates for missing modalities across diverse egocentric datasets.It extends a scene-centric corpus with dense hand and full-body motion data.
  • Geometric supervision: Video Depth Anything generates high-quality, temporally consistent video-depth pseudo-labels for datasets lacking dense 3D geometry.
  • Geometric supervision: Large-scale depth annotations expand geometric supervision to diverse unconstrained environments and provide consistent signals for learning robust 4D structural priors.Evaluations report improved video-depth prediction in complex scenes compared with existing baselines.
  • Representation standardization: Motion signals are standardized into complementary reference frames that separate local manipulation from global navigation.Camera-space hand kinematics project hand joints into the camera space of their respective frames.

4 Method

ReViV jointly reconstructs viewer- and scene-centric states from monocular egocentric RGB by modeling their multimodal dependencies with masked token prediction. Modality-specific discrete representations and a masked transformer enable conditional reconstruction of unobserved states from observed video.

  • Problem Formulation: ReViV formulates egocentric 4D reconstruction as multimodal generative modeling rather than deterministic prediction, addressing ambiguity from severe ego-body occlusion.The modeled states include hand, body, gaze, depth, and camera dynamics conditioned on RGB video.
  • Unified Discrete Representation: Modality-specific VQ-VAEs convert heterogeneous continuous signals into discrete token sequences that form a unified multimodal representation.The representation covers RGB, depth, camera, gaze, hand, and body modalities, with modality-specific codebooks and embeddings.
  • Unified Discrete Representation: A decoupled dual-stream tokenizer independently encodes body and hand motion, allowing disjoint body-only and hand-only motion-capture data to contribute.The tokenizers process 3D joint sequences using local kinematic features before producing latent representations.
  • Masked Generative Egocentric Transformer: MGET learns arbitrary conditional distributions by randomly masking tokens across modalities and predicting them from visible context.This objective captures both intra-modal temporal dynamics and cross-modal correlations while approximating the joint distribution.
  • Masked Generative Egocentric Transformer: At inference, visible RGB tokens condition MGET while human and scene modalities are masked and reconstructed through the model’s modality-specific prediction heads.The resulting conditional reconstruction targets the unobserved viewer and scene states from egocentric video.

5 Experiment

Experiments evaluate ReViV across body, hand, gaze, camera, depth, and ablation settings on egocentric benchmarks. The results show strong accuracy and substantial inference-speed advantages, while depth estimation remains constrained by discrete tokenization and the lack of depth-specific priors.

  • Egocentric Body Motion Reconstruction: ReViV establishes state-of-the-art monocular egocentric body-motion reconstruction across pose accuracy, motion realism, and semantic similarity metrics.It outperforms baselines using estimated or ground-truth camera trajectories while operating without explicit camera trajectories.
  • Egocentric Body Motion Reconstruction: 100× faster than EgoAllo and over 10× faster than UniEgoMotion, ReViV delivers comparable or stronger body-motion quality with feed-forward inference.The unified model avoids compounding camera-tracking errors by directly modeling the joint distribution p(X, Y).
  • Egocentric Hand Motion Reconstruction: ReViV achieves state-of-the-art hand-motion accuracy across four benchmarks while being 100× faster than HaMeR and over 400× faster than Dyn-HaMR.The evaluation reports GA-MPJPE, RA-MPJPE, and PA-MPJPE errors.
  • Camera, Gaze, and Depth Evaluation: ReViV evaluates camera tracking, gaze estimation, and depth estimation on ADT, with real-time camera-tracking runtime reported for ReViV and EgoM2P.Gaze evaluation reports significantly lower error than EgoM2P on unseen data.
  • Egocentric Hand Motion Reconstruction: Qualitative results show plausible, temporally consistent hand motion under severe occlusion and when the hands are entirely out of view.The model maintains biomechanically plausible motion across temporal gaps between sampled frames.
  • Camera, Gaze, and Depth Evaluation: Depth estimation remains weaker than depth-specialized alternatives because ReViV lacks depth expert initialization and discrete tokenization introduces quantization errors.Despite this gap, ReViV achieves over 20× faster inference than EgoMono4D.
  • Ablation Studies: A task-specific RGB-only body specialist suffers a massive accuracy drop compared with ReViV, indicating the value of joint cross-modal supervision.The ablation compares task-specific and unified modeling choices.

6 Conclusion

ReViV unifies egocentric scene reconstruction and unobserved human kinematics from monocular RGB video through a masked generative transformer. Its large-scale pretrained model achieves state-of-the-art holistic perception while retaining a limitation in fine-grained depth reconstruction from discrete representations.

  • Conclusion: ReViV presents a unified generative framework that reconstructs egocentric scene dynamics and unobserved human kinematics from monocular RGB video.The framework jointly models body, hand, gaze, camera, and depth signals.
  • Conclusion: MGET aligns heterogeneous viewer and view modalities in a shared, temporally coherent latent space.The approach uses large-scale pretraining and avoids specialized hardware or pre-computed SLAM.
  • Conclusion: ReViV establishes a foundational baseline for holistic egocentric perception with state-of-the-art performance across the reconstructed modalities.The conclusion frames unified viewer–view modeling as a step toward embodied AI systems with human-centric reasoning.
  • Limitations and Future Directions: Discrete multimodal representations discard high-frequency spatial details, slightly limiting depth estimation relative to task-specific continuous regression methods.The paper suggests continuous generative priors as a future direction for recovering fine-grained geometry.

–Supplementary Material–

The supplementary material details ReViV’s multimodal tokenization, masked pretraining, decoding, depth-scale alignment, and qualitative camera, gaze, and reconstruction outputs.

  • Tokenizer Design: ReViV tokenizes RGB and depth with Cosmos, while camera, gaze, body, and hand signals use modality-specific VQ-VAEs.The visual tokenizer uses a 64,000-entry codebook; body and hand tokenizers use dedicated architectures.
  • Masked Pretraining: MGET trains on masked unified token sequences spanning RGB, ViT, depth, camera, gaze, hand, and body modalities.The training algorithm samples datasets and modality masks, then optimizes masked-token cross-entropy.
  • Inference: MGET reconstructs masked human and scene tokens through three iterative parallel-decoding steps conditioned on visible video and previously predicted states.Each step predicts a uniformly sampled subset of unpredicted target tokens using conditional and unconditional passes.
  • Qualitative Results: Qualitative results show smooth camera trajectories, close ground-truth alignment, and gaze predictions that are closer to targets.The camera comparisons include projected trajectories and 3D frustum visualizations, while gaze is shown qualitatively.
  • Depth Alignment: ReViV aligns relative depth to metric ego-body space by matching scene-floor points with an estimated ego-body floor, or with VIPE depth when the floor is absent.The alignment solves for scale and translation using least squares; the scene-floor estimate uses the lowest 5% of point-cloud depths.

E Additional Experiment Results

Additional experiments show ReViV’s camera trajectories remain accurate under difficult egocentric motion, while gaze and body reconstructions preserve target-directed and semantic motion.

  • Camera Tracking: ReViV produces smooth camera trajectories that closely follow ground truth despite rapid head movements and limited camera parallax.Baselines instead show drift, trajectory deviations, or temporal jitter.
  • Camera Tracking: Across indoor sequences, predicted camera poses remain tightly aligned with ground truth across multiple timesteps.The qualitative comparison uses 3D camera frustums over Frames 05, 25, and 45.
  • Gaze Estimation: ReViV produces closer gaze predictions, indicating stronger inference of human intent from egocentric observations.The result is presented as a qualitative comparison in Fig. 3.
  • Body Motion Reconstruction: On unseen ADT sequences, ReViV reconstructs body motions more faithfully from monocular RGB than baselines conditioned on ground-truth camera trajectories.The comparison spans multiple indoor activities and timesteps.
  • Body Motion Reconstruction: ReViV generalizes more strongly to unseen ADT than UniEgoMotion and achieves semantic body-motion alignment from full-image egocentric video.The authors attribute the stronger generalization to large-scale multimodal pretraining.

F.1 Egocentric Hand Motion Reconstruction Ablation

The hand-motion ablation shows that joint multimodal learning is important: a hand-only specialist fails severely, while broader data without hand annotations causes only modest degradation.

  • Cross-Benchmark Results: Hand reconstruction trends across HoloAssist, HOT3D, ARCTIC, and TACO mirror the main body ablation.Table 2 extends the body ablation analysis to hand motion.
  • Cross-Benchmark Results: The task-specific ReViV-HandSpec. fails severely across all four hand-reconstruction benchmarks.This result underscores the weakness of isolated single-modality training for hand motion.
  • Cross-Benchmark Results: Training only on hand-annotated data produces modest degradation, showing that datasets without hand annotations still provide useful cross-modal priors.The evidence supports joint-distribution learning beyond directly annotated hand data.
  • Body Ablation Context: ReViV’s body-motion visualization reports state-of-the-art accuracy and semantic alignment from purely monocular egocentric video.The comparison notes that ReViV avoids dependence on explicit camera tracking when camera pose estimates fail.

F.2 Vision Transformer Encoding Branch

The continuous xViT branch preserves high-frequency visual context lost by quantization, improving depth, gaze, camera, and hand-reconstruction outcomes.

  • Branch Ablation: Removing xViT degrades performance across the predicted modalities, demonstrating the value of continuous RGB encoding.The ablation evaluates the effect of removing the raw-RGB Vision Transformer branch.
  • Interpretation: The xViT stream bypasses the quantization bottleneck and preserves high-frequency spatial context needed across modalities.This provides a common visual complement to the discrete multimodal token representation.
  • Gaze and Camera: Preserving pixel-level detail through xViT reduces gaze MSE and improves camera Absolute Trajectory Error.The branch supports precise target localization and global egomotion consistency.
  • Hand Reconstruction: Removing continuous xViT degrades all three hand-reconstruction metrics on TACO.The result confirms the importance of unquantized visual features for fine-grained 3D hand-pose recovery.

F.3 VQ-VAE Ablation

The VQ-VAE ablation identifies a codebook size and spatial-temporal convolution configuration that best balance reconstruction fidelity and generalizability.

  • 8.26 mm reconstruction error is achieved with a codebook size of 2048, while 4096 degrades accuracy to 9.02 mm.Smaller codebooks perform worse: 512 and 1024 yield 11.06 and 8.67 mm errors, respectively.
  • 8.26 mm reconstruction error is achieved with the spatial-temporal convolution configuration, outperforming at 15.47 mm and at 13.05 mm.The selected configuration provides the reported optimal balance between reconstruction fidelity and generalizability.
Loading 2607.17790v1…