PaperlayerView exact revision on arXiv

Source-linked AI summary

ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

Xiaozhong Lyu, Gen Li, Zhiyin Qian, Xucong Zhang, Marc Pollefeys, Siyu Tang

arXiv:2607.17790v1cs.CVcs.AI

TL;DR

A unified approach to reconstructing egocentric body, hand, and gaze dynamics alongside scene geometry remains an open challenge. ReViV uses masked generative modeling on monocular RGB video and reports state-of-the-art viewer reconstruction with competitive depth estimation and camera tracking.

  • Problem

    A unified approach that jointly reconstructs egocentric body, hand, and gaze dynamics with scene geometry remains an open challenge.

    Source: §2.3 · p. 5

  • Method

    ReViV uses masked generative modeling to predict missing viewer- and scene-centric modalities from monocular RGB observations in a single feed-forward pass.

    Source: §1 · p. 3

  • Results

    ReViV achieves state-of-the-art holistic ego-body, hand, and gaze reconstruction, competitive depth estimation and camera tracking, and faster inference than optimization-based methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper concludes that one monocular RGB video can support reconstruction of the camera wearer’s full-body, hand, and gaze behavior alongside the dynamic 4D environment.

    Source: §6 · p. 15

    The unified discrete representation discards high-frequency spatial detail, slightly limiting depth estimation relative to task-specific continuous-regression methods.

    Source: §6 · p. 15

Abstract

from arXiv · show

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.

1 Introduction

ReViV reconstructs the viewer and surrounding scene jointly from a single monocular egocentric RGB video, predicting human motion and scene-aware signals in one feed-forward framework. It targets the temporal inconsistencies and auxiliary-input requirements of prior approaches by modeling cross-modal dependencies.

1 Introduction

  • From monocular RGB, ReViV predicts body and hand motion, gaze, camera trajectory, and depth in one feed-forward pass, using affine-invariant depth and lightweight alignment when geometric anchors are available.The framework formulates the task as masked generative modeling over multimodal signals.Source: Figure 1 · p. 2, §1 · p. 3
  • Prior egocentric methods often separate environment reconstruction [60] from pose estimation, overlooking temporal coupling and sometimes requiring auxiliary tracking inputs.This separation can produce temporally inconsistent view geometry and viewer motion, while third-person and multi-view advances leave monocular egocentric body dynamics unresolved.Source: §1 · pp. 2–3
  • MGET tokenizes diverse spatiotemporal modalities into compact latents and learns their joint distribution by predicting randomly masked tokens from available context.Its unified multi-task objective is designed to learn human-motion dynamics and cross-modal correlations, inferring missing kinematics and scene geometry from visible cues.Source: §1 · p. 3
  • ReViV reports state-of-the-art holistic ego-body, hand, and gaze reconstruction, competitive depth estimation and camera tracking, and faster inference than optimization-based methods [17] [60].The reported results span multiple benchmarks; the introduction does not provide metric values.Source: §1 · p. 3
  • Training scales to 7B unique multimodal tokens and 500B training tokens to learn robust cross-modal dependencies and temporally consistent viewer–view dynamics.The introduction identifies augmented and virtual reality, assistive robotics, and human–computer interaction as settings where systems need to understand the environment and the wearer’s actions in real time.Source: §1 · pp. 2–3

2 Related Work

Prior work advances 4D reconstruction and egocentric scene or motion modeling, but leaves open a unified approach that reconstructs human dynamics and scene geometry from egocentric video. ReViV is proposed to address this gap.

2.1 4D Reconstruction

  • Prior 4D reconstruction progressed from object trajectories through multi-view to monocular video, but largely misses egocentric settings.Long-term Gaussian methods require per-scene test-time optimization, while feedforward methods [52] [55] offer fast, generalized reconstruction without inference-time optimization.Source: §2.1 · pp. 3–4

2.2 Scene Reconstruction from Egocentric Videos

  • Egocentric scene methods address limited views, camera motion, and occlusion, but focus mainly on scene or object geometry rather than full human motion and interactions.EgoM2P jointly models RGB, depth, gaze, and camera trajectories; EgoGaussian reconstructs scenes and tracks objects, while EgoMono4D [60] reconstructs point-cloud sequences.Source: §2.2 · p. 4

2.3 Human Motion Estimation from Egocentric Videos

  • Egocentric motion methods often require auxiliary trajectories, point clouds, or hand trackers, while a unified reconstruction of body, hand, gaze, and scene geometry remains open.Gaze-related work forecasts 3D gaze or uses gaze-regularized attention for egocentric behavior understanding; ReViV is proposed as the first unified generative framework for egocentric 4D reconstruction.Source: §2.3 · pp. 4–5

3 Data Engine

ReViV’s data engine harmonizes multimodal signals across heterogeneous egocentric datasets, expands pretraining data with hand and body kinematics, adds temporal depth supervision, and standardizes motion reference frames.

3 Data Engine

  • The data engine extends EgoM2P with dense hand and full-body kinematics, unifying pretraining data from 4B to 7B tokens for holistic egocentric 4D modeling.It harmonizes spatial representations and compensates for missing modalities across diverse datasets.Source: §3 · p. 5

3.1 Temporally Consistent Geometric Pseudo-Labeling

  • Video Depth Anything generates temporally consistent depth pseudo-labels, extending geometric supervision from constrained hand-object interactions to diverse real-world environments and improving video depth prediction against existing baselines.The resulting annotations provide consistent geometric signals over time for learning robust 4D structural priors.Source: §3.1 · pp. 5–6

3.2 Unified Kinematic Representation

  • Motion signals are standardized into camera-space hand kinematics for viewpoint-relative interaction priors and gravity-aligned global body kinematics that isolate navigation from initial camera tilt.The gravity-aligned frame uses the first camera pose with its up-vector projected opposite to gravity, avoiding pitch and roll artifacts.Source: §3.2 · p. 6

4 Method

ReViV treats occlusion-ambiguous egocentric reconstruction as joint multimodal generation, using discrete representations and a masked transformer to reconstruct viewer and scene states from monocular video. A lightweight alignment step places the reconstruction in a shared metric 4D coordinate system.

4.1 Problem Formulation

  • Because ego-body occlusion makes deterministic reconstruction ill-posed, ReViV models the joint distribution of RGB, hand and body motion, gaze, depth, and camera trajectories, then samples states conditioned on video.Masked multimodal token prediction captures both within-modality temporal dynamics and cross-modal correlations.Source: §4 · p. 6, §4.1 · pp. 6–7
  • A stable floor plane regularizes scene scale for metric alignment; when the floor is not visible, optional VIPE [17] metric-depth predictions provide a geometric anchor.These steps place the reconstructed viewer and scene in a shared metric-aligned 4D coordinate system.Source: §4.1 · p. 7

4.2 Unified Discrete Representation

  • Modality-specific VQ-VAEs map continuous signals to discrete token sequences; decoupled body and hand tokenizers use separate motion data with spherical quantization and EMA updates.RGB and depth use Cosmos Tokenizers, while gaze and camera tokenizers are retrained from EgoM2P.Source: §4.2 · pp. 7–8

4.3 Masked Generative Egocentric Transformer

  • MGET combines a T5-based encoder-decoder, modality and spatiotemporal embeddings, and raw-video ViT features; random multimodal masking trains within- and cross-modal conditional predictions.At inference, video tokens remain visible while human and scene tokens are masked, then iterative parallel decoding reconstructs the temporally coherent 4D state.Source: §4.3 · pp. 8–9

5 Experiment

Across unseen benchmarks, ReViV reconstructs egocentric body and hand motion, camera trajectories, gaze, and depth from monocular video, with strong accuracy and fast inference. Ablations indicate that unified multimodal training, model capacity, masking, and data volume contribute to performance.

5.1 Egocentric Body Motion Reconstruction

  • On ADT, ReViV sets a state of the art for monocular egocentric body-motion reconstruction, dominating estimated-camera baselines across all metrics, while running 100× faster than EgoAllo and over 10× faster than UniEgoMotion.Against ground-truth-camera baselines, it outperforms on PA-MPJPE, Similarity, and FID; perfect tracking is slightly better on GA-MPJPE.Source: Table 2 · p. 10, §5.1 · pp. 10–11

5.2 Egocentric Hand Motion Reconstruction

  • Across four egocentric hand benchmarks, ReViV achieves state-of-the-art local pose accuracy and global alignment, running 100× faster than HaMeR and over 400× faster than Dyn-HaMR.Its spatiotemporal priors also produce plausible, temporally consistent hand motion through severe occlusion and when hands are out of view.Source: Table 3 · p. 11, §5.2 · p. 12, Figure 3 · p. 12

5.3 Egocentric Camera Tracking

  • On unseen ADT sequences, ReViV attains the lowest camera-tracking RTE and RRE, while VIPE alone performs better on ATE; ReViV predicts poses in one feed-forward pass without geometric post-processing.The comparison reports real-time performance for ReViV and EgoM2P on camera tracking.Source: Table 4 · p. 12, §5.3 · p. 13

5.4 Egocentric Gaze Estimation

5.5 Egocentric Video Depth Estimation

  • On unseen ADT data, ReViV significantly outperforms EgoM2P on depth estimation but trails EgoMono4D on both metrics, while running over 20× faster.The authors attribute the gap to EgoMono4D’s UniDepth priors and quantization errors from ReViV’s discrete tokenizer.Source: Table 4 · p. 12, §5.5 · pp. 13–14

5.6 Ablation Studies

  • Ablations show that task-specific RGB training, reduced model capacity, alternative masking strategies, and less training data each degrade reconstruction relative to ReViV’s unified model.The authors attribute gains to cross-modal supervision, sufficient capacity, balanced mask sampling, and generalized priors learned from larger multimodal datasets, including data without explicit task annotations.Source: Table 5 · p. 14, §5.6 · p. 14

6 Conclusion · A Tokenizer Details · B Masked Generative Egocentric Transformer Details

ReViV unifies reconstruction of the egocentric viewer and environment from monocular RGB video in one feed-forward framework, using MGET to model multimodal signals in a temporally coherent latent space. Its discrete tokenization enables joint modeling but trades away some fine spatial detail for depth estimation.

6 Conclusion

  • ReViV jointly reconstructs the dynamic 4D environment and the wearer’s body, hands, and gaze from one monocular RGB video, without specialized hardware or precomputed SLAM.The authors describe it as a unified generative framework and a foundational baseline for holistic egocentric perception.Source: §6 · p. 15
  • VQ-VAE quantization discards some high-frequency spatial detail, slightly limiting depth estimation; conditional diffusion at decoding is proposed as a future direction.The authors aim to retain multimodal reasoning while recovering finer geometric detail.Source: §6 · p. 15

A Tokenizer Details

  • RGB and depth videos use the Cosmos tokenizer with a 64,000-entry codebook, while camera, gaze, body, and hand signals use modality-specific VQ-VAEs.The visual-token pipeline follows EgoM2P.Source: A Tokenizer Details · p. 20
  • Camera and gaze tokenizers use 1D temporal convolutions followed by 12-layer ViT-B encoders; masked MSE handles invalid gaze values, with codebooks of 256 and 512, respectively.Both VQ-VAEs are retrained from scratch on the newly curated large-scale database for generalization across environments and behaviors.Source: A Tokenizer Details · p. 20
  • Body and hand tokenizers use 2D convolutions to capture temporal and skeletal correlations, downsampling time by 2 and joints by 3 into a 768-dimensional latent space.They use 12-block ViT-B Transformers and standard MSE reconstruction loss; codebooks contain 2048 body tokens and 1024 hand tokens, with separate left- and right-hand tokenizers trained using mirrored motions.Source: A Tokenizer Details · pp. 20–21

B Masked Generative Egocentric Transformer Details

  • MGET training uses AdamW at a 1 × 10−4 base learning rate and global batch size 1024, consumes approximately 500B tokens, and takes about 24 hours on 256 NVIDIA H100 GPUs.Training includes a warm-up schedule over the first 10 epochs to stabilize optimization.Source: B Masked Generative Egocentric… · pp. 21–22
  • MGET pretraining samples datasets and modality token allocations, masks mutually exclusive input and target tokens, then predicts masked targets with cross-entropy.The algorithm uses unrestricted global self-attention on visible inputs and combines intra-modal self-attention with inter-modal cross-attention in the decoder.Source: B Masked Generative Egocentric… · pp. 21–22

C Inference Details · D Ego-body anchored depth scale optimization

ReViV uses iterative parallel decoding to reconstruct masked human and scene states, then estimates metric depth scale by aligning scene depth with the reconstructed ego-body or, when the floor is not visible, a VIPE metric-depth anchor.

C Inference Details

  • ReViV reconstructs all n masked human and scene state tokens over three parallel decoding steps, uniformly sampling n/3 unpredicted targets at each step.Each step estimates distributions for sampled targets using forward passes conditioned on visible video context and previously predicted state tokens.Source: C Inference Details · p. 22
  • Final tokens are sampled from combined conditional and unconditional logits using classifier-free guidance (w = 2) and nucleus sampling (top-p = 0.8).The unconditional pass masks video context but retains previously predicted states.Source: C Inference Details · p. 22

D Ego-body anchored depth scale optimization

  • The depth-scale formulation aligns scene depth with reconstructed human scale to produce a physically grounded metric reconstruction for egocentric settings.It provides a lightweight way to map the reconstructed viewer and view into a metrically consistent 4D coordinate system.Source: D Ego-body anchored depth scale… · pp. 23–24
  • The ego-body floor level is estimated by averaging the minimum 50% of predicted foot-joint z-values across the sequence.The method uses predicted ego-body motion in world space to approximate the ground plane location.Source: D Ego-body anchored depth scale… · p. 23
  • For the scene floor estimate, relative depth is lifted into an unscaled point cloud, assuming its lowest visible points lie on the floor and selecting the lowest 5% of z-values.The selected scene points serve as the floor correspondences for alignment.Source: D Ego-body anchored depth scale… · pp. 23–24
  • A least-squares fit estimates scale s and translation t by matching the lowest 5% of scene points to the estimated ego-body floor level.Source: D Ego-body anchored depth scale… · p. 24
  • When the floor is not visible, ReViV uses VIPE metric-depth predictions as an anchor and fits global scale s and translation t to minimize discrepancy with reconstructed relative depth.The body-floor and metric-depth-anchor strategies provide complementary alignment options.Source: D Ego-body anchored depth scale… · p. 24

E Additional Experiment Results … E.3 Egocentric Body Motion Reconstruction

Additional experiments qualitatively assess ReViV’s camera tracking, gaze estimation, and body-motion reconstruction, including body motion on unseen Aria Digital Twin sequences from monocular RGB input.

E.1 Egocentric Camera Tracking

  • ReViV produces smooth, accurate camera trajectories close to ground truth despite rapid head movements and limited parallax, while baselines show drift, deviations, or jitter.Source: §E.1 · p. 24
  • Across indoor sequences, ReViV’s poses remain aligned with ground truth, while EgoM2P, EgoMono4D, and VIPE develop translational drift and rotational misalignment.The qualitative comparison shows camera frustums at Frames 05, 25, and 45.Source: §E.1 · p. 24

E.2 Egocentric Gaze Estimation

  • ReViV produces closer gaze predictions, indicating stronger inference of human intent from egocentric observations.Source: §E.2 · p. 25

E.3 Egocentric Body Motion Reconstruction

  • On unseen ADT, ReViV reconstructs motion more faithfully from monocular RGB than baselines conditioned on ground-truth camera trajectories.Examples include turning right and stopping at a table, and turning left while reaching for an item.Source: §E.3 · p. 25
  • ReViV achieves state-of-the-art ego body motion reconstruction accuracy with strong semantic alignment by learning from full-image semantics and the joint distribution of motion and video.The passage attributes stronger generalization on unseen ADT to large-scale multimodal pretraining.Source: §E.3 · p. 25
  • ReViV generalizes more strongly to unseen ADT than UniEgoMotion, which the authors attribute to large-scale multimodal pretraining.The passage contrasts this with EgoAllo, which can deviate from the input video’s semantics when relying on localized hand-guidance priors.Source: §E.3 · p. 25

F Additional Ablation Studies · F.1 Egocentric Hand Motion Reconstruction Ablation

The hand-motion ablation extends the body study across four benchmarks and finds that ReViV consistently outperforms specialist, data-restricted, masking, and lightweight variants. The results support joint learning as important for hand reconstruction and indicate that datasets without hand annotations still provide useful cross-modal priors.

F.1 Egocentric Hand Motion Reconstruction Ablation

  • Tab. 2 extends the main paper’s body ablation to hand reconstruction on HoloAssist, HOT3D, ARCTIC, and TACO.Source: §F.1 · p. 25
  • ReViV consistently outperforms the masking ablations and ReViV-Tiny across all four hand-reconstruction benchmarks.The reported variants are ReViV-1to1Mask, ReViV-Uniform, and ReViV-Tiny.Source: §F.1 · p. 25
  • ReViV-HandSpec. fails severely across HoloAssist, HOT3D, ARCTIC, and TACO, underscoring the importance of joint rather than isolated single-modality training.The passage characterizes this as a finding consistent with the body ablation.Source: §F.1 · p. 25
  • Restricting training to hand-annotated data only causes modest degradation, indicating that datasets without hand annotations provide useful cross-modal priors through joint distribution learning.Source: §F.1 · p. 25
  • ReViV-1to1Mask consistently underperforms ReViV across all four hand-reconstruction benchmarks.The passage groups this variant with ReViV-Uniform among the masking ablations.Source: §F.1 · p. 25
  • ReViV-Uniform consistently underperforms ReViV across all four hand-reconstruction benchmarks.The passage groups this variant with ReViV-1to1Mask among the masking ablations.Source: §F.1 · p. 25

F.2 Vision Transformer Encoding Branch

The continuous xViT branch processes raw RGB context to avoid quantization-related loss of fine spatial detail. Removing it degrades reconstruction across depth, gaze, camera trajectories, and hands.

F.2 Vision Transformer Encoding Branch

  • The xViT branch processes raw RGB context continuously, bypassing discrete tokenization to preserve high-frequency spatial details needed across modalities.Source: §F.2 · pp. 27–28
  • Removing xViT worsens depth errors and inlier accuracy, increases gaze MSE, and raises camera Absolute Trajectory Error.Unquantized RGB context supports fine 3D geometry, precise visual-target localization, and global egomotion consistency.Source: §F.2 · p. 27
  • On TACO, removing the continuous ViT degrades all three hand reconstruction metrics, underscoring its importance for fine-grained 3D hand pose recovery.Source: §F.2 · p. 27

F.3 VQ-VAE Ablation · Supplementary References

The body-module VQ-VAE ablation shows that reconstruction accuracy depends on both codebook size and spatial-temporal convolution settings. Larger codebooks help up to a point, while the configuration provides the best balance among the tested convolution settings.

F.3 VQ-VAE Ablation

  • The spatial-temporal convolution configuration achieves 8.26 mm error, outperforming at 15.47 mm and at 13.05 mm.The authors describe as the optimal balance between reconstruction fidelity and generalizability.Source: §F.3 · p. 28
  • Increasing codebook size from 512 to 2048 reduces reconstruction error from 11.06 to 8.26 mm, but size 4096 raises it to 9.02 mm.The authors attribute the degradation at 4096 to codebook utilization inefficiency.Source: §F.3 · p. 28

Related papers

Loading 2607.17790v1…