PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos

Jilan Xu, Yifei Huang, Baoqi Pei, Junlin Hou, Qingqiu Li, Guo Chen, Yuejie Zhang, Rui Feng, Weidi Xie

arXiv:2504.11732v1cs.CV

TL;DR

Cross-view video prediction aims to generate future first-person frames from a synchronized third-person video, an initial first-person frame, and text, despite a substantial perspective shift. EgoExo-Gen predicts hand-object masks across views and uses them to guide video diffusion; it outperforms prior models on two benchmarks and shows zero-shot transfer to unseen actions and environments.

  • Problem

    Cross-view video prediction must align the initial ego-view scene with actor motion and environmental context from a synchronized exo-video and textual instruction despite a substantial perspective shift.

    Source: §1 · p. 1

  • Method

    EgoExo-Gen predicts future ego-view hand-object interaction masks from exo-video and the first ego-frame, then uses the masks to guide diffusion-based frame generation.

    Source: §1 · p. 2

  • Results

    EgoExo-Gen significantly outperforms prior video prediction models on Ego-Exo4D and H2O and demonstrates zero-shot transfer to unseen actions and environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports potential applications for its hand-object-aware video prediction in AR/VR and embodied AI.

    Source: §5 · p. 10

    Performance is bounded by sub-optimal mask prediction, particularly for future frames that are not visible.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Generating videos in the first-person perspective has broad application prospects in the field of augmented reality and embodied intelligence. In this work, we explore the cross-view video prediction task, where given an exo-centric video, the first frame of the corresponding ego-centric video, and textual instructions, the goal is to generate futur frames of the ego-centric video. Inspired by the notion that hand-object interactions (HOI) in ego-centric videos represent the primary intentions and actions of the current actor, we present EgoExo-Gen that explicitly models the hand-object dynamics for cross-view video prediction. EgoExo-Gen consists of two stages. First, we design a cross-view HOI mask prediction model that anticipates the HOI masks in future ego-frames by modeling the spatio-temporal ego-exo correspondence. Next, we employ a video diffusion model to predict future ego-frames using the first ego-frame and textual instructions, while incorporating the HOI masks as structural guidance to enhance prediction quality. To facilitate training, we develop an automated pipeline to generate pseudo HOI masks for both ego- and exo-videos by exploiting vision foundation models. Extensive experiments demonstrate that our proposed EgoExo-Gen achieves better prediction performance compared to previous video prediction models on the Ego-Exo4D and H2O benchmark datasets, with the HOI masks significantly improving the generation of hands and interactive objects in the ego-centric videos.

1 INTRODUCTION

EgoExo-Gen addresses the challenge of generating future first-person frames from a synchronized third-person video, the first ego-frame, and text. It predicts hand-object interaction masks across views and uses them to guide video generation.

  • The task is to predict future ego-view frames from the first ego-frame, a synchronized exo-video, and a textual instruction despite the substantial viewpoint shift.Source: §1 · pp. 1–2
  • Cross-view prediction can support egocentric world models from third-person demonstrations and perspective alignment for applications such as augmented reality and robotics.Source: §1 · p. 2
  • Earlier cross-view approaches used 3D scene reconstruction or precise human annotations, which the paper identifies as limits on scalability and generalization.Source: §1 · p. 2
  • EgoExo-Gen predicts future ego-view hand-object interaction masks from exo-video and the first ego-frame, then uses them as structural guidance for diffusion-based frame generation.Source: §1 · p. 2
  • EgoExo-Gen significantly outperforms prior video prediction models on Ego-Exo4D and H2O and shows zero-shot transfer to unseen actions and environments.Source: §1 · p. 2

2 METHODOLOGY

EgoExo-Gen predicts future ego-view frames from a synchronized exo-video, the first ego-frame, and text, without camera poses or depth. It first predicts future hand-object masks, then uses them to guide video diffusion; an automated pipeline supplies masks for training.

2.1 PROBLEM FORMULATION

  • The task predicts an ego-video from a synchronized exo-video, one starting ego-frame, and text, without camera-pose or depth information.At test time, only the first ego-frame is assumed available.Source: §2.1 · pp. 2–3

2.2 CROSS-VIEW MASK PREDICTION

  • The mask predictor generates future ego masks autoregressively, using current exo-frame features to attend to historical ego/exo features and their mask features.During training, an ego-visual-query residual is blended with the exo-query output, with its weight annealed from 1.0 to 0.0.Source: §2.2 · pp. 3–4

2.3 HOI-AWARE VIDEO DIFFUSION MODEL

  • The diffusion model conditions future-frame generation on the first ego-frame and text, while a separate mask encoder injects predicted HOI features into the denoising UNet.Mask features are fused with latent features by element-wise addition and passed to temporal attention.Source: §2.3 · pp. 4–5

2.4 DATA, TRAINING AND INFERENCE PIPELINE

  • An automated pipeline builds HOI masks from frame-wise detection or segmentation and SAM-2 tracking; training first fits the UNet, then freezes it to train mask guidance, while inference feeds predicted masks into diffusion.EgoHOS supplies ego-video masks, whereas exo-video processing uses 100DOH and Sapiens.Source: §2.4 · p. 5, Figure 3 · p. 5

3 EXPERIMENTS

The experiments test EgoExo-Gen on Ego-Exo4D and zero-shot H2O, compare HOI and conditioning designs, and examine cross-view mask prediction and qualitative limits.

3 EXPERIMENTS

  • Ego-Exo4D evaluation uses cooking clips and held-out takes and locations, with video quality measured by SSIM, PSNR, LPIPS, and FVD.The study also evaluates transfer to H2O, a tabletop-activity dataset.Source: §3 · pp. 5–6

3.1 QUANTITATIVE COMPARISON

  • EgoExo-Gen consistently outperforms prior methods on every Ego-Exo4D metric and surpasses them in zero-shot H2O transfer without retraining.Table 1 compares video prediction models on Ego-Exo4D and zero-shot transfer to H2O.Source: Table 1 · p. 6, §3.1 · pp. 6–7

3.2 ABLATION STUDIES AND DISCUSSIONS

  • Future-frame hand-object masks yield the best results across metrics; hand-only and object-only masks improve the baseline, random masks do not, and splitting hands adds no gain.These comparisons use masks extracted from future frames to preserve mask quality.Source: Table 2 · p. 7, §3.2 · p. 7
  • All three conditions yield the best performance, while first-frame-plus-HOI beats the first-frame-plus-text baseline; removing text harms instruction adherence and removing the first frame loses scene context.The tested modalities are the first frame, text instruction, and HOI masks.Source: Table 3 · p. 7, §3.2 · p. 7
  • Exo-memory attention predicts unobserved masks better than ego-only attention, while combining both improves segmentation across observed and unobserved frames and downstream video prediction.Ego-only attention struggles after the visible first frame because it has zero-image features as queries.Source: §3.2 · p. 7, Table 4 · p. 8
  • Directly injecting exo RGB or exo masks into one diffusion model is suboptimal; ego-view hand-object movement provides explicit pixel-aligned visual cues.The comparison concerns alternatives to the model's cross-view mask-prediction stage.Source: §3.2 · pp. 7–8, Table 5 · p. 8
  • SAM-2 tracking improves mask quality and generation, while cross-view masks also boost performance with both SEINE and ConsistI2V.EgoHOS provides per-frame segmentation; SAM-2 tracks prompted hands and objects across frames to improve temporal consistency.Source: Table 6 · p. 8, §3.2 · p. 8, Table 7 · p. 8

3.3 QUALITATIVE COMPARISON AND LIMITATIONS

  • Qualitative outputs show reasonable hand-object motion, but complex hand movement defeats the default and comparator; oracle future masks yield realistic near-ground-truth videos, while mask prediction bounds performance.The qualitative comparison's final row shows a complex-hand-movement failure case.Source: §3.3 · pp. 8–9, Figure 4 · p. 9

4 RELATED WORK

Prior work spans joint ego-exo video understanding, cross-view generation, video diffusion prediction, and hand-object segmentation. EgoExo-Gen focuses on predicting cross-view hand-object masks and modeling their dynamics for ego-centric video prediction.

Egocentric-Exocentric video understanding and generation

  • Cross-view research includes video-level understanding and translation or generation, while EgoExo-Gen predicts spatio-temporal hand-object masks from exo-view to ego-view.Prior examples include exo-to-ego image generation guided by hand pose (Luo et al., 2024b).Source: §4 · p. 10

Diffusion models for video prediction

  • Video prediction methods condition diffusion models on a first frame, while ego-centric approaches also decompose instructions or use specialised adapters.ConsistI2V uses spatio-temporal attention and DynamiCrafter uses dual-stream image injection; EgoExo-Gen explicitly models hands and interacting objects.Source: §4 · p. 10

Hand-Object segmentation

  • VISOR and EgoHOS segment hands and interacting objects in ego-view but generally fail in exo-view; HOISTformer addresses third-person segmentation.SAM-2 tracks prompted objects, while Sapiens segments body parts including hands and arms; the paper's annotation pipeline builds on these foundation models.Source: §4 · p. 10

5 CONCLUSION

EgoExo-Gen combines cross-view hand-object mask prediction with mask-guided video diffusion and an automated annotation pipeline. Experiments report stronger hand-object video prediction than prior models, with potential applications in AR/VR and embodied AI.

5 CONCLUSION

  • EgoExo-Gen combines cross-view mask prediction for unobserved ego-frames with HOI-aware video diffusion guided by the predicted masks.The method models spatio-temporal ego-exo correspondence and includes an automated pipeline for generating training masks.Source: §5 · p. 10
  • An automated HOI mask annotation pipeline generates masks for both ego- and exo-videos to enhance the method's scalability.Source: §5 · p. 10
  • Experiments report superiority over prior video prediction models for realistic hand-object movement, with potential applications in AR/VR and embodied AI.Source: §5 · p. 10
Loading 2504.11732v1…