PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Shengze Wang, Michael Stengel, Tianye Li, Seonwook Park, Amrita Mazumdar, Koki Nagano, Alex Trevithick, Shalini De Mello

arXiv:2608.20534v1cs.CV

TL;DR

Single-exo-video to ego-video generation is difficult because geometric conditioning becomes unreliable under extreme view changes and unobserved regions. Grounded-Exo2Ego combines geometric anchoring and semantic grounding with camera re-localization and synthetic data. On EgoExo4D, it outperforms recent state-of-the-art methods across all metrics, with ablations validating its contributions.

  • Problem

    Exo-to-ego generation lacks reliable geometric conditioning because monocular reconstructions misalign with real-world ego poses and leave occluded or out-of-frame regions unobserved.

  • Method

    Grounded-Exo2Ego is a dual-branch video diffusion model that combines 3D-rendering-based geometric anchoring with object-level semantic grounding, camera re-localization, and an automated synthetic data engine.

  • Results

    Grounded-Exo2Ego significantly outperforms recent state-of-the-art exo-to-ego methods across all metrics on EgoExo4D, with ablations validating its architectural and data-level improvements.

  • Takeaways & Limitations

    The framework establishes an exo-to-ego generation approach that combines spatially anchored semantics with improved real and synthetic training data.

  • Takeaways & Limitations

    Generation length is limited by VRAM constraints, while occlusion and out-of-frame content remain difficult to reconstruct and can require substantial hallucination.

Abstract

from arXiv · show

Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions. We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context. Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments. Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

1 Introduction

Exo-to-ego synthesis is harder than conventional novel-view synthesis because geometric conditioning becomes unreliable under extreme viewpoint changes and unobserved regions. Grounded-Exo2Ego addresses these challenges through semantic grounding, camera re-localization, and synthetic data, and outperforms recent methods across all metrics on EgoExo4D.

  • Motivation: Exo-to-ego synthesis is practically difficult because monocular depth errors distort ego-view geometry while large ego-view regions remain unobserved.These coupled failures undermine standard geometry-based conditioning.
  • Data contributions: A fully automated synthetic data engine produces animatable 3D humans in randomly generated environments and renders interactions from multiple participants’ perspectives.The pipeline supplies accurately annotated training data and addresses the lack of human-human egocentric interaction data.
  • Method: Grounded-Exo2Ego uses a dual-branch video diffusion model combining geometric anchoring with semantic grounding of object-level context onto the 3D reconstruction.The geometric branch provides scene structure, while the semantic branch helps identify what distorted or occluded regions should depict.
  • Data contributions: Camera re-localization reduces misalignment between geometric renderings and ground-truth videos, improving synthesis accuracy during learning.The method addresses an incompatibility between monocular reconstructions and real-world-calibrated ego poses.
  • Results: Grounded-Exo2Ego significantly outperforms recent state-of-the-art exo-to-ego methods across all metrics on the EgoExo4D benchmark.Ablations validate contributions from both the architectural and data-level improvements.

2 Related work

Related work spans single-exo-video to ego-video synthesis and camera-controlled video generation conditioned on rendered 3D scene representations. Grounded-Exo2Ego’s method overview combines geometric anchoring with semantic, object-specific conditioning.

  • Exo-to-ego video generation: Exo-to-ego video generation studies synthesizing a first-person video from a single exocentric video and a target ego camera trajectory.Camera intrinsics and extrinsics are used when available and estimated otherwise.
  • Exo-to-ego video generation: Several follow-up methods synthesize ego videos from four synchronized exo views, with Exo2EgoSyn adding pose-aware multi-view conditioning.These settings differ from the single-exo-video formulation described here.
  • Camera-controlled video and dynamic view synthesis: Camera-controlled video methods condition generation on point-cloud renderings at the target view, while Vista4D extends this design to dynamic 4D point clouds.GEN3C, Uni3C, and Lyra lift source views into 3D point clouds before target-camera rasterization.
  • Method overview: Grounded-Exo2Ego’s overview pairs geometric anchoring with semantic grounding that reprojects object masks into the ego view for object-specific cross-attention.The semantic branch routes object-specific text tokens to relevant video tokens.

3 Method

Grounded-Exo2Ego generates an ego video from an exo video and target ego trajectory using a pretrained latent video model with geometric and semantic conditioning. Camera re-localization aligns renderings with ground-truth videos, while semantic masks and object prompts guide object-level generation.

  • Problem setup: The task maps an exocentric video, camera intrinsics, and a target ego trajectory to an egocentric video consistent with that trajectory.
  • Preliminaries: A pretrained latent video model uses flow matching to predict velocity fields between Gaussian noise and encoded ground-truth ego-video latents.The generated latent is decoded into the output ego video during inference.
  • Geometric anchoring: The geometric anchoring branch reconstructs the scene in 3D, renders it from the target ego camera, and injects the rendering as a spatial scaffold into the DiT.The rendering is encoded into latent tokens, appended separately from noisy latent tokens, and near-blank regions are masked.
  • Semantic grounding: The semantic grounding branch extracts scene and object descriptions, reprojects exo-view object masks into the ego view, and routes object-specific tokens through spatially gated cross-attention.The model selects up to K=10 object candidates and pads each object-token packet to L=10 tokens.
  • Semantic grounding: Each visual token attends only to the text-token packet of the object whose reprojected region contains it, while the full scene prompt remains globally accessible.This preserves object semantics in the pretrained text latent space and was more effective than alternative feature-injection designs.
  • Real and synthetic data pipelines: Camera re-localization refines ego trajectories against monocular reconstructions to establish coordinate consistency, while retaining inaccurate reconstructions to match inference-time errors.The method cannot convert locally incorrect depth into geometrically correct reconstruction; it instead seeks a camera pose producing an aligned rendering.

4 Experiments

Experiments on the in-the-wild EgoExo4D benchmark show broad gains from Grounded-Exo2Ego, with ablations attributing improvements to training data, camera re-localization, and semantic grounding. The study also identifies longer-video generation and incomplete 3D reconstruction as remaining boundaries.

  • Quantitative comparisons: Grounded-Exo2Ego significantly outperforms recent state-of-the-art exo-to-ego methods across all reported metrics on EgoExo4D.The benchmark includes Unseen and New Action splits spanning diverse activities, environments, motion, and imperfect geometry, masks, and camera poses.
  • Quantitative comparisons: PSNR improves by +1.67 dB/+2.59 dB and T-LPIPS decreases by −37.5%/−45.9% on Unseen/New Action splits.The gains indicate improvements in reconstruction quality, alignment accuracy, and temporal dynamics.
  • Ablation studies: Camera re-localization improves all nine metrics, including IoU by +30.0%/+35.7% and T-LPIPS error by −23.4%/−24.4%.It aligns ego cameras with monocular reconstructions, improving both spatial and temporal agreement with ground-truth video.
  • Ablation studies: Full semantic grounding improves both scene-level and object-level accuracy, including IoU by +10.2%/+15.0% and FVD error by −7.7%/−26.8%.Object tokens, spatial masks, and location-conditioned routing contribute to the gains; spatial masks particularly improve alignment metrics.
  • Ablation studies: Accurate segmentation masks improve every evaluated metric, including IoU by +9.5%/+5.0% and contour error by −13.6%/−18.5%.This evaluation uses first-frame masks from the actual ego video and is therefore not deployable at test time.
  • Limitations: The method remains constrained by short video lengths and incomplete 3D reconstructions for occluded or out-of-frame content.When the ego view is largely blank in the reconstruction, the model must hallucinate much of the content; chunk-autoregressive finetuning is reported as a route to longer videos.

5 Conclusion

The framework addresses exo-to-ego video generation through architectural and data-level improvements, including spatially anchored semantic grounding beyond traditional geometry-based synthesis.

  • Grounded-Exo2Ego tackles exo-to-ego generation at both the architectural and data levels.
  • Its Semantic-Grounding mechanism guides generation with spatially anchored semantics beyond traditional geometry-based video synthesis.
  • Ablation studies demonstrate the effectiveness of the proposed contributions.

A Additional qualitative results

Figure 7 presents additional New-Activity scenes using the same column organization as Figure 4.

  • Figure 7 contains additional New-Activity scenes.
  • The figure uses the same column organization as Figure 4.
  • Figure 7 is intended for qualitative comparison across the displayed scene columns.

B Evaluation metrics

The evaluation defines Chamfer contour distance and explains the treatment of VBench metrics, including comparison with ground truth and reported variance across generations.

  • Chamfer contour distance matches generated and ground-truth objects using DINOv3 features and Hungarian matching before comparing mask boundaries.Boundary distances are normalized by the square image side length l = 448 px, making the score dimension-agnostic; lower is better.
  • VBench temporal flickering, motion smoothness, and dynamic degree are no-reference metrics that do not reliably indicate fidelity when scored higher.The evaluation therefore reports delta deviation from ground truth, |score − GT score|, while T-LPIPS is used in the main table because it compares against ground truth.
  • Table 3 reports delta VBench on the New-Activity seen and unseen splits, where lower magnitude is better and zero indicates a perfect ground-truth match.
  • Main-paper results are averaged over five independent generations, with Table 4 reporting their standard deviation.

C Robustness to imperfect geometry, masks, and ego pose

The robustness analysis evaluates corruption of depth, pose, and segmentation masks, with depth and pose perturbations applied before rendering and mask perturbations applied after reprojection.

  • Robustness setup: The conditioning uses monocular depth, object segmentation masks, and the target ego camera pose, each of which is corrupted to measure quality degradation.
  • Robustness setup: Depth and pose noise jointly affects rendering, reprojection, positional encoding, occupancy, and mask reprojection because perturbations occur before rendering.
  • Robustness setup: Mask perturbation radii are measured in gate pixels because erosion and dilation occur after reprojection.
  • Robustness setup: The robustness experiments use Table 1 values as their uncorrupted baselines and separately examine pose and mask noise.
  • Imperfect geometry: Under imperfect geometry, Grounded-Exo2Ego remains ahead in absolute object and video metrics at every tested noise level, while EgoX overtakes on PSNR and SSIM.The analysis compares against the strongest EgoX variant and reports that the proposed method degrades faster in relative terms.

D Method details

The method combines geometric renderings with structured object semantics to condition video generation, while masking unsupported regions and training semantic inputs from multi-camera information.

  • Geometric conditioning: Mesh rasterization converts re-localized ego poses and monocular depth into ego-view RGB, depth, and UV maps for downstream conditioning.Neighboring depth-grid pixels form triangles; triangles crossing 5% depth discontinuities are culled before rasterization.
  • Geometric conditioning: Render tokens are appended to noisy target latents and assigned spatial coordinates offset by the ego-video width.The tokens share RoPE spatial coordinates with noisy latent tokens while using an x-shift.
  • Geometric conditioning: Occupancy masks remove tokens whose underlying spatial-temporal cells contain less than 10% occupied pixels.The masks prevent unsupported black regions from misleading the generator.
  • Semantic grounding: Captions combine scene descriptions, up to 15 visible-object noun phrases, spatial and interaction relations, and predicted ego-view descriptions.Captioning is applied to the first frame across ego and exo cameras.
  • Semantic grounding: Object extraction merges noun phrases across cameras, prioritizes ego phrases, and ranks up to 50 deduplicated objects by viewpoint relevance and interactions.The process merges phrases referring to the same object rather than ranking solely by image area.
  • Semantic grounding: Gemma-3 embeddings are padded with learned registers to length L=10 and projected into the DiT cross-attention space.Raw noun-phrase embeddings have a maximum dataset length of 6, so padding is always applied; training replaces reprojected masks with ground-truth ego masks with probability p=0.5.

E.1 Real data pipeline

The real-data pipeline re-localizes ego cameras within inaccurate monocular reconstructions, preserving reconstruction noise during learning and supporting synthetic and in-the-wild deployment settings.

  • Camera re-localization: Ground-truth ego poses can produce misaligned or blank renderings because monocular reconstructions do not match real-world scale and geometry.EgoExo4D camera poses are calibrated to real-world geometry, whereas monocular depth reconstructions are not.
  • Camera re-localization: The pipeline aligns MoGe-2 reconstructions to a metric Depth-Anything-3 reconstruction, then computes a non-rigid alignment for ego-camera transformation.Depth-Anything-3 is used only to calculate the camera transformation, not during training or testing.
  • Camera re-localization: Adding camera re-localization alone improves all nine metrics over 14.0M EgoExo4D frames.The reported comparison evaluates the re-implemented EgoX configuration with relocation.
  • Camera re-localization: Re-localization enforces coordinate consistency but does not repair non-rigid reconstruction distortion or make renderings match ground-truth frames everywhere.The method removes camera-coordinate misalignment rather than correcting the reconstruction itself.
  • Camera re-localization: Training on original MoGe-2 reconstructions avoids train-test mismatch because ground-truth depth is unavailable at inference.The resulting correction behavior is tuned partly to the reconstruction model’s error characteristics.
  • Deployment: At test time, the model can generalize to in-the-wild data when the ego camera is localized within the MoGe-2 reconstruction.Face tracking or manual view selection can provide the required localization.
  • Synthetic data: The synthetic pipeline randomizes interior scenes, rigged characters, animations, ego cameras, and valid exo-camera placements while recording rendering metadata.It includes 640 scenes, 550 characters, and 3,000 animations, with ego cameras attached to character neck bones and temporal smoothing applied.
  • Baseline alignment: Vista4D is aligned to metric depth before ego-camera localization, yet it remains unable to generate ego views correctly despite consistent viewpoint and scene content.The comparison uses the same Depth-Anything-3 alignment strategy for Vista4D’s up-to-scale reconstruction.

G Limitations and societal impact

The deployed system is bounded by output length and resolution, camera-trajectory and perception dependencies, and societal risks including first-person-view misrepresentation and fraud.

  • Limitations: Accuracy is bounded by test-time ego-camera trajectory estimation because the method follows a supplied path rather than inferring where the wearer looks.The in-the-wild pipeline estimates the trajectory from human and face pose together with depth.
  • Limitations: First-frame object masks become stale during substantial camera or object motion, while permissive off-the-shelf segmentation masks can undermine generation.Per-frame masks would address staleness but increase preprocessing cost and storage requirements.
  • Limitations: The system depends on 3D reconstruction quality and can be constrained when exocentric depth is severely incomplete or corrupted.This limitation is part of the deployed model’s stated dependence on reconstruction and semantic segmentation.
  • Societal impact: The authors identify potential misrepresentation of first-person viewpoints and fraud as negative societal impacts of generative exo-to-ego systems.They contrast these risks with positive impacts described in the motivation.
  • Societal impact: Provenance safeguards such as invisible digital watermarking could help downstream tools flag synthesized first-person video.The watermark embeds an imperceptible signature recoverable by specialized detector models.
Loading 2608.20534v1…