PaperlayerView exact revision on arXiv

Source-linked AI summary

Exo2EgoHOI: Hand-Object-Interaction Aware Exocentric-to-Egocentric Video Generation

Hongjia Zhai, Xiyu Zhang, Haoran Zhang, Zhichao Ye, Haomin Liu, Guofeng Zhang, Ian Reid, Xingxing Zuo

arXiv:2609.38615v1cs.CVcs.AI

TL;DR

Wearable egocentric manipulation data are costly to scale, while existing viewpoint translation can alter demonstrated hand-object interactions. Exo2EgoHOI combines explicit HOI guidance with object-centric reference anchoring; evaluations report improved object and hand consistency across two datasets.

  • Problem

    Wearable egocentric manipulation data are costly to scale, and existing viewpoint translation can alter demonstrated hand-object interactions.

    Source: §1 · p. 1

  • Method

    Exo2EgoHOI combines a unified 4D HOI prior and dual-branch residual adapter with Decomposed Gated Cross-Attention for object-specific anchoring.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Across ARCTIC-HOI and Ego-Exo4D, Exo2EgoHOI improves object and hand consistency; on ARCTIC-HOI, MPJPE decreases from 2019.81 to 1318.74 mm.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The reported results support better preservation of manipulated objects and hand configurations under large viewpoint changes.

    Source: §5 · p. 9

    The framework relies on estimated depth, hand geometry, and object segmentation, whose errors may propagate into the 4D HOI prior and affect generation quality.

    Source: §5 · p. 9

Abstract

from arXiv · show

Egocentric videos of human manipulation provide valuable visual experience for embodied intelligence, yet collecting such data at scale is costly. Exocentric-to-egocentric video generation offers a scalable alternative by transforming abundant third-person manipulation videos into first-person observations. However, existing methods often struggle to faithfully preserve demonstrated hand-object interactions (HOI) across large viewpoint changes due to insufficient fine-grained interaction guidance and weak object-centric anchoring. We present Exo2EgoHOI, an HOI-aware video generative framework for interaction-preserving exocentric-to-egocentric translation. To preserve fine-grained HOI, we introduce a unified 4D HOI prior that combines scene geometry, articulated hand renderings, and dense hand-object relation fields, together with a dual-branch residual adapter for injecting structural and relational cues into the video generation backbone. To preserve object consistency, we introduce Decomposed Gated Cross-Attention, which separately encodes object and background references and adaptively integrates global semantic and local appearance features as object-centric anchors. Experiments on ARCTIC-HOI and Ego-Exo4D demonstrate substantial improvements in object consistency and HOI preservation while maintaining competitive visual fidelity. In particular, on ARCTIC-HOI, Exo2EgoHOI improves object mIoU by 32.3% and reduces MPJPE and PA-MPJPE by 34.7% and 50.0%, respectively, relative to the respective best baseline results. Project page: https://rcl-robotics.github.io/Exo2EgoHOI/.

1 INTRODUCTION

Wearable egocentric manipulation data are costly to scale, motivating translation of abundant third-person footage into first-person observations. Exo2EgoHOI targets interaction preservation and object consistency through explicit HOI guidance and object-centric anchoring.

1 INTRODUCTION

  • Wearable-camera collection of egocentric manipulation videos requires substantial effort to scale across people, environments, and tasks.Abundant third-person footage motivates translating exocentric videos into first-person observations.Source: §1 · pp. 1–2
  • Existing methods can produce plausible target views while altering hand-object interactions, and large viewpoint changes leave object appearance, shape, or state weakly constrained.Scene geometry can miss fine hand configurations and local hand-object relations, while global image conditioning can dilute object-specific cues.Source: §1 · p. 2
  • Exo2EgoHOI combines a unified 4D HOI prior and dual-branch residual adapter with Decomposed Gated Cross-Attention for object consistency.The prior combines scene geometry, articulated hand renderings, and hand-object relation fields; DGCA separately encodes object and background references and adaptively integrates semantic and appearance features.Source: §1 · p. 2
  • Experiments on ARCTIC-HOI and Ego-Exo4D show improvements in object consistency and interaction preservation while maintaining strong visual fidelity.The work also constructs an interaction-oriented benchmark that evaluates visual fidelity, object consistency, and hand-pose consistency.Source: §1 · p. 2

2 RELATED WORK

Prior work advances controllable viewpoint synthesis and HOI modeling, but the paper identifies fine-grained interaction preservation and persistent object-centric anchoring as remaining challenges.

2 RELATED WORK

  • Controllable video-generation methods use camera trajectories and rendered scene geometry to support viewpoint and camera-motion control.The paper situates these advances alongside large-scale egocentric and synchronized ego-exo datasets.Source: §2 · p. 2
  • Exocentric-to-egocentric methods variously use synchronized views, scene geometry, hand-pose conditioning, or object-level semantic grounding.The paper contrasts these approaches with its focus on explicitly modeling HOI structure throughout the video.Source: §2 · p. 3
  • HOI research spans articulated hand reconstruction, hand-object relation modeling, and synthesis, while this work aims to preserve an observed interaction across viewpoints.The cited approaches include contact-centric representations and joint hand-object motion modeling.Source: §2 · p. 3

3 METHOD

Exo2EgoHOI conditions target-view generation on source video and camera information, text, a unified 4D HOI prior, and object/background masks. The prior supplies scene, hand, and interaction cues, while DGCA anchors object identity and appearance using decomposed references and learned global-local gates.

3.1 PROBLEM FORMULATION

  • Generation is conditioned on the exocentric video, text, camera calibration, a unified 4D HOI prior, and first-frame object and background masks.The formulation maps the source interaction sequence to an egocentric sequence while aiming to preserve the observed HOI.Source: §3.1 · p. 3

3.2 UNIFIED 4D HOI PRIOR

  • The unified prior combines target-view scene layout, articulated hand geometry, and per-vertex hand-object cues for contact likelihood, distance, and relative direction.These complementary cues address the limits of scene geometry alone, which can leave fine-grained HOI underconstrained when hands and objects are small or occluded.Source: §3.2 · pp. 4–5
  • Independent spatiotemporal branches encode the hand rendering and relation field, fuse their features, and inject them as a residual into scene-conditioned tokens.The projection is zero-initialized, and the resulting tokens integrate scene context, hand structure, and hand-object relations.Source: §3.2 · p. 5

3.3 DECOMPOSED GATED CROSS-ATTENTION

  • DGCA decomposes references by object versus background and global semantics versus local appearance, then uses learned block-specific gates to fuse global and local attention responses.Its object-centric anchoring complements the geometric prior to help preserve object identity and appearance across viewpoints and throughout generation.Source: Figure 3 · p. 5, §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate Exo2EgoHOI on ARCTIC-HOI and Ego-Exo4D, where it improves object and hand consistency while retaining strong visual fidelity. Ablations examine the HOI priors, attention design, and number of exocentric training views.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation uses ARCTIC-HOI (7K training, 180 non-overlapping test, and 60 in-the-wild clips) and Ego-Exo4D, with EgoWorld, EgoX, WAN VACE, and Vista4D as baselines.WAN VACE and EgoX were fine-tuned with the same configuration; EgoWorld and Vista4D were evaluated using public inference implementations.Source: §4.1 · pp. 6–7
  • Evaluation separates visual fidelity (PSNR, SSIM, LPIPS, CLIP-I), object consistency (mIoU, Center-Err., CLIP-O), and hand consistency (MPJPE, PA-MPJPE, WA-MPJPE).Object masks and hand estimates are extracted using SAM 2 and Potamias et al. (2025), respectively.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • On ARCTIC-HOI, Exo2EgoHOI lowered LPIPS 0.4324→0.3860 and MPJPE 2019.81→1318.74 mm, while mIoU rose 0.2933→0.3881 and PA-MPJPE fell 20.36→10.18 mm.These are comparisons with the strongest baselines; the reported MPJPE and PA-MPJPE reductions are 34.7% and 50.0%, respectively.Source: Table 1 · p. 7, §4.2 · p. 7
  • On Ego-Exo4D, PSNR rose 13.14→14.11 dB and MPJPE and WA-MPJPE fell to 5208.17 and 1749.36 mm, but PA-MPJPE remained higher than WAN VACE and EgoX.Exo2EgoHOI improved most visual and object metrics over EgoX.Source: §4.2 · p. 7, Table 1 · p. 7
  • Figure 4 compares generated egocentric views and recovered hand meshes; Exo2EgoHOI better preserves object appearance and hand-object configurations under large viewpoint changes.Examples include the hinged object's raised lid and circular interior, and the scissors' target-view placement and bimanual configuration.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 ABLATION STUDIES

  • Adding scene, articulated-hand, and interaction priors progressively improves consistency; adding interaction cues raises mIoU from 0.3323 to 0.3881 and lowers MPJPE and PA-MPJPE to 1318.74 and 10.18 mm.The reported results characterize scene layout, hand articulation, and hand-object relations as complementary geometric guidance.Source: §4.3 · p. 8
  • More exocentric views generally improve performance, but not monotonically: five views peak at 14.01 dB PSNR, while six achieve the best LPIPS, mIoU, and PA-MPJPE.The reported results suggest performance largely saturates around five to six views.Source: Figure 5 · p. 9, §4.3 · p. 9
  • DGCA's global branch improves CLIP-I and its local branch improves mIoU and PA-MPJPE; learned gates and object-background decomposition further improve consistency.With learned per-block gates, mIoU rises from 0.3354 to 0.3881 and PA-MPJPE falls from 13.61 to 10.18 mm.Source: Table 2 · p. 8, §4.3 · p. 9

5 CONCLUSION

Exo2EgoHOI combines explicit interaction guidance with object-centric anchoring to preserve hand-object interactions across large viewpoint changes. Experiments show improved object and hand consistency, while the framework depends on estimated depth, hand geometry, and object segmentation.

5 CONCLUSION

  • On ARCTIC-HOI, object mIoU improves by 32.3% and PA-MPJPE decreases by 50.0% relative to the respective best baselines.Experiments on ARCTIC-HOI and Ego-Exo4D show better preservation of manipulated objects and hand configurations under large viewpoint changes.Source: §5 · p. 9
  • The unified 4D HOI prior models scene structure, articulated hands, and hand-object relations, while Decomposed Gated Cross-Attention anchors object-specific semantic and appearance cues.Together, these components target fine-grained interaction preservation and object consistency throughout egocentric video generation.Source: §5 · p. 9
  • Errors in estimated depth, hand geometry, or object segmentation may propagate into the reconstructed 4D HOI prior and affect generation quality.Future work will explore more robust interaction reconstruction and generated egocentric videos for downstream embodied learning and robot manipulation.Source: §5 · p. 9

A APPENDIX

The supplementary material provides implementation, evaluation, dataset-construction, and qualitative details for Exo2EgoHOI, including representative failure cases and an analysis of current limitations.

A APPENDIX

  • The supplement describes the model architecture, HOI adapter, global-local anchor construction, and training and inference settings.It also details ARCTIC-HOI construction, preprocessing for scene geometry, object masks, and hand reconstruction, and definitions of visual-fidelity, object-consistency, and hand-consistency metrics.Source: A APPENDIX · p. 13
  • Additional qualitative comparisons on ARCTIC-HOI and Ego-Exo4D are accompanied by representative failure cases and an analysis of current limitations.Source: A APPENDIX · p. 13
  • The supplement includes complete definitions of the visual fidelity, object consistency, and hand consistency metrics used in the main paper.Source: A APPENDIX · p. 13

A.1 ARCHITECTURE AND IMPLEMENTATION DETAILS

The implementation uses a frozen Wan2.1 video-generation backbone with separate HOI-prior branches and global-local reference features. Training updates LoRA parameters, the adapter, and gates while freezing other pretrained weights.

A.1 ARCHITECTURE AND IMPLEMENTATION DETAILS

  • The backbone is Wan2.1-I2V-14B-480P with 40 transformer blocks, width 5120, 40 attention heads, and latent patch size (1, 2, 2).Exocentric, ego-target, and rendered scene-prior videos are encoded separately before widthwise latent concatenation; only ego latents receive noise and supervision.Source: §A.1 · p. 13
  • Table 3 specifies adapter layers with C = 3 for the hand mesh prior and C = 5 for the interaction prior.Branch convolutions use padding one and SiLU; fusion and projection use zero padding.Source: Table 3 · p. 13
  • The adapter independently encodes hand and interaction priors, then adds their projected residual to the ego token grid after input patchification.The hand prior retains [-1, 1] scaling, the interaction prior retains geometric normalization, and the final projection is zero-initialized.Source: §A.1 · p. 13
  • Figure 6 depicts the reconstructed 3D scene and camera configuration alongside corresponding egocentric and exocentric observations.Colored camera frustums mark different exocentric viewpoints.Source: Figure 6 · p. 14
  • Object crops expand the tight mask box by 20% per side, while the background reference replaces object pixels with RGB value 0.5.Their CLIP [CLS] tokens form global references; a shared pretrained projector maps features to width 5120.Source: §A.1 · p. 14
  • Training jointly updates LoRA parameters, the adapter, and gates while freezing the other pretrained weights.Inference uses second-order UniPC with 50 sampling steps and guidance scale 5 to generate 49-frame videos at 30 fps.Source: §A.1 · p. 14

A.2 ARCTIC-HOI BENCHMARK CONSTRUCTION

ARCTIC-HOI pairs exocentric and egocentric clips from selected ARCTIC cameras and adds in-the-wild paired clips. The construction also describes scene rendering and per-frame object and hand reconstruction procedures.

A.2 ARCTIC-HOI BENCHMARK CONSTRUCTION

  • The benchmark contains approximately 7,000 training clips and 240 test clips, including 60 in-the-wild paired clips.Camera 0 is designated egocentric; cameras {1, 3, 4, 5, 6, 8} are exocentric, while cameras 2 and 7 are excluded for viewpoint deviation.Source: §A.2 · p. 14
  • For each source view, the pipeline back-projects its RGB-D observation into a point cloud and renders that source-view cloud into the target egocentric camera.Scene geometry is reconstructed offline, and no egocentric RGB images are used for depth estimation.Source: §A.2 · p. 14
  • Object and hand masks are detected separately per frame, and WiLoR predicts 778-vertex MANO hand meshes that are aligned to observed scene depth.Hands without sufficient depth support are discarded; rendered hand priors use fixed left/right colors and hand-only visibility without object-depth occlusion.Source: §A.2 · p. 15

A.3.1 VISUAL FIDELITY

Visual fidelity is evaluated with four image-level metrics computed per frame and then averaged, using metric-specific preprocessing.

A.3.1 VISUAL FIDELITY

  • PSNR, SSIM, LPIPS, and CLIP-I are averaged from per-frame visual-fidelity scores.PSNR and SSIM use RGB images with a 255 data range; SSIM uses uniform 7 × 7 windows.Source: §A.3.1 · p. 15
  • LPIPS uses pretrained AlexNet with inputs normalized to [−1, 1] and no additional resizing.These are the stated LPIPS preprocessing conditions.Source: §A.3.1 · p. 15
  • CLIP-I is cosine similarity between ℓ2-normalized openai/clip-vit-base-patch32 image features, after bicubic resizing and a 224 × 224 center crop.The short side is resized to 224 before cropping.Source: §A.3.1 · p. 15

A.3.2 OBJECT CONSISTENCY

The supplementary evaluation specifies object-mask and hand-pose metrics and presents qualitative comparisons across ARCTIC-HOI and Ego-Exo4D, including representative failure cases.

A.3.2 OBJECT CONSISTENCY

  • mIoU scores mask overlap on reference-present frames, assigning zero to missing predictions, while Center-Err. measures normalized centroid displacement when both masks are present.Object masks are independently extracted with Grounding DINO Swin-T OGC and SAM 2.1 Hiera-L; CLIP-O compares object-crop appearance and is also conditional on both masks being present.Source: §A.3.2 · p. 15

A.3.3 HAND CONSISTENCY

  • MPJPE measures camera-coordinate joint error without wrist centering; PA-MPJPE aligns each frame separately, while WA-MPJPE uses one alignment per video and reference side.The hand metrics use WiLoR-estimated 21-joint poses; WA-MPJPE aligns camera-coordinate sequences and does not recover a physical world coordinate system.Source: §A.3.3 · pp. 15–16

A.4 ADDITIONAL QUALITATIVE RESULTS

  • ARCTIC-HOI comparisons visualize reconstructed 3D hand meshes and per-vertex errors across varied hand configurations, object geometries, and bimanual interactions; Ego-Exo4D examples cover diverse real-world scenes and activities.The Ego-Exo4D examples complement ARCTIC-HOI with evaluation beyond its relatively controlled setting, including scene reconstruction, object consistency, and hand-object configurations.Source: §A.4 · p. 16, Figure 7 · p. 17, Figure 8 · p. 18
  • Severe occlusion can leave insufficient geometric evidence for interaction reconstruction, while complex articulated or thin objects remain difficult to recover accurately.The authors identify dependence on accurate upstream geometry and interaction reconstruction as a limitation.Source: §A.4 · p. 16, Figure 9 · p. 18
Loading 2609.38615v1…