Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation

1 INTRODUCTION

• The task requires inferring scene appearance and semantics across a large viewpoint change while maintaining the target trajectory and generating unseen egocentric content.Paired exocentric-egocentric recordings are expensive to collect.Source: §1 · p. 2
• Prior methods impose input constraints or require separate finetuning per activity, while EgoX generalizes across activities using a caption and geometry-guided attention bias.Some approaches require a ground-truth first egocentric frame or four synchronized exocentric cameras.Source: §1 · p. 2
• A human-written caption describes a view that does not yet exist, and geometry-guided attention adds cost, propagates reconstruction errors, and can suppress visual and textual context.The paper illustrates suppressed text describing a mirror reflection that differs across viewpoints.Source: §1 · p. 2
• Dynamic Captioning replaces external text with conditioning tokens derived from hidden states and adapted to diffusion timestep and network depth.No text is generated; the module supplies the conditioning role of a caption.Source: §1 · pp. 2–3
• Training on all exocentric viewpoints lets the model learn cross-view correspondence implicitly, drop geometric attention, and use only a lightweight depth prior.This removes dependence on reconstruction quality and human annotation at inference.Source: §1 · p. 3
• Ego-Forge outperforms the state of the art on standard Ego-Exo4D metrics, runs 4.3× faster end-to-end, and generalizes to in-the-wild footage.The paper highlights cases where over-reliance on geometry blocks appearance inference.Source: §1 · p. 3

2 RELATED WORK

• Camera-control and 3D-consistency methods generate along continuous camera paths, while large, discontinuous viewpoint changes from an existing video remain largely unexplored.The source and target cameras differ by a large and discontinuous pose change.Source: §2 · p. 3
• Exo-to-egocentric generation must synthesize unseen egocentric content, and existing methods add constraints; these approaches remain category-specific.Ego-Exo4D pairs each egocentric video with four exocentric views, whereas EgoX selects the best camera view and Ego-Forge uses all views.Source: §2 · p. 3
• EgoX generalizes across activities but relies on a caption of the nonexistent egocentric view and geometry-guided attention that can propagate reconstruction errors.Its attention bias is applied to every query-key pair.Source: §2 · p. 4
• Prior methods for caption-free conditioning may add learned modules at multiple backbone depths, whereas Dynamic Captioning uses one resampler shared across the network.The shared resampler is conditioned on diffusion timestep and block index to recover depth-specific behavior with fewer parameters.Source: §2 · p. 4

3 METHOD

• Ego-Forge has three training stages: text-conditioned adaptation, Dynamic Captioner training, and caption-free finetuning, followed by caption-free inference.Stage 1 learns cross-view correspondence without geometric attention bias; later stages replace the caption stream with conditioning derived from model hidden states.Source: §3 · pp. 4–5

3.1 STAGE 1: TEXT-CONDITIONED ADAPTATION

• Stage 1 trains on a stitched exocentric/egocentric canvas with a reprojected depth prior, using all exocentric viewpoints and no geometry-guided attention bias.Text conditions cross-attention, while the flow-matching objective supervises only the egocentric half of the canvas.Source: §3.1 · p. 5

3.2 STAGE 2: LEARNING THE DYNAMIC CAPTIONER

• Stage 2 trains a shared Dynamic Captioner to replace text conditioning with tokens derived from hidden states and adapted to block depth and diffusion timestep.The module uses separate query sets: exocentric queries attend only to exocentric tokens, while egocentric queries attend to the full canvas; it is supervised against the caption-conditioned cross-attention output.Source: §3.2 · pp. 5–6

3.3 STAGE 3: CAPTION-FREE FINETUNING

• Stage 3 removes captions and finetunes the Dynamic Captioner and self-attention layers under the diffusion objective, while keeping cross-attention frozen.The objective supervises only the egocentric half, with Dynamic Captioner tokens replacing the caption at every block.Source: §3.3 · p. 6

3.4 INFERENCE

• At inference, the model needs only exocentric video and a target head trajectory; it uses a reprojected depth estimate without captions or geometry-guided attention bias.The Dynamic Captioner rebuilds conditioning tokens from the current hidden state at every block.Source: §3.4 · p. 6

4 EXPERIMENTS

• The evaluation uses all available exocentric viewpoints, yielding 100K training clips and 400 evaluation clips across seen and unseen scenes.Clips are motion-curated; the dataset provides synchronized egocentric and exocentric recordings with camera poses across everyday activities.Source: §4 · pp. 6–7

4.1 QUANTITATIVE AND QUALITATIVE RESULTS

• Figure 4 compares Ego-Forge with EgoX on unseen and in-the-wild footage, illustrating arm placement, camera-motion handling, visual cue use, and unseen-content synthesis.Its examples also show the baseline generating the caption-mentioned “EMERGENCY” sign at the wrong time.Source: §4.1 · p. 7, Figure 4 · p. 7
• Ego-Forge achieves the best overall performance on seen and unseen Ego-Exo4D scenes; Table 1 presents the quantitative comparison.The evaluation scenes do not overlap with the training set.Source: §4.1 · p. 7, Table 1 · p. 8

4.2 ABLATION STUDY

• Removing the textual stream substantially degrades performance, while Stage 2 learns from caption-based teacher signals and Stage 3 surpasses that teacher.Table 2 isolates the Dynamic Captioner’s contribution by comparing cross-attention inputs, including an empty input.Source: §4.2 · p. 8, Table 2 · p. 8
• Removing block-index conditioning degrades performance most; as denoising proceeds, attention shifts toward the egocentric half, by 0.17 in the last block versus 0.03 in the first.Table 3 compares conditioning with and without block index and timestep; Figure 6 plots attention share against a uniform-attention reference.Source: §4.2 · p. 8, Table 3 · p. 9, Figure 6 · p. 9
• Table 4 compares Dynamic Captioning with a static captioner, and Table 5 reports that Stage 3 improves quantitative results and generation quality over Stage 2.Figure 8 provides examples for the Stage 2 and Stage 3 comparison.Source: §4.2 · p. 8, Table 4 · p. 9, Table 5 · p. 9
• Figure 7 compares the models’ training size and inference cost, with inference measured on 25 frames.The supplied text does not state the plotted values.Source: §4.1 · p. 7, Figure 7 · p. 9

5 CONCLUSION

• Ego-Forge outperforms the state of the art on Ego-Exo4D and runs 4.3× faster end-to-end, while generalizing to in-the-wild footage.It also handles challenging cases where geometric correspondence alone is insufficient to recover appearance.Source: §5 · p. 9
• Dynamic Captioning replaces the unavailable egocentric-view caption with tokens from model hidden states, conditioned on network depth and diffusion timestep.A staged procedure first learns from the pretrained text pathway, then jointly refines the captioner with the diffusion model without text.Source: §5 · p. 9
• Scaling training across all available exocentric viewpoints allows cross-view correspondence to be learned from data without geometric attention, while requiring only a lightweight geometric prior at inference.The paper reports using generative AI for synthetic data, drafting, writing polish, and code assistance, with code read, executed, and verified by the authors.Source: §5 · p. 9, AI USE STATEMENT · p. 9

A TEXTUAL AND VISUAL SUPPRESSION BY GGA

• Removing GGA allows EgoX to recover the person’s reflection, but the reflection remains inaccurate.The generated video still fails to produce the correct egocentric view.Source: A TEXTUAL AND VISUAL SUPPRESSION BY GGA · p. 14
• The recovered reflection is a limited appearance improvement, not evidence that EgoX generates the correct egocentric view.The passage distinguishes the reflection result from overall view correctness.Source: A TEXTUAL AND VISUAL SUPPRESSION BY GGA · p. 14
• The example concerns EgoX’s performance depending on whether GGA is present.Figure A is identified as the comparison of GGA presence.Source: A TEXTUAL AND VISUAL SUPPRESSION BY GGA · p. 14

Motivation

• Windows with little head movement are excluded because they offer little learning signal and trivial cross-view correspondence.The egocentric prior and target view change little in nearly stationary clips.Source: B TRAINING DATA CURATION · p. 14

Motion metric

• Head motion is measured between consecutive frames by adding translational distance and a rotational geodesic angle, weighted equally at λrot = 1.The poses are metric, and over a short window the two terms are of comparable magnitude.Source: B TRAINING DATA CURATION · p. 14

Selection

• Windows are retained when mean motion exceeds a threshold; med(m) + 2 MAD(m) also identifies motion concentrated in a few frames.The burstiness statistic is the fraction of frames exceeding that threshold.Source: B TRAINING DATA CURATION · p. 14

Category-balanced ordering

• Round-robin sampling across shuffled activity groups avoids long same-category runs, and retaining all exocentric viewpoints increases the training set to 100k clips.No clip is repeated or discarded; only the order changes.Source: B TRAINING DATA CURATION · pp. 14–15

Caption generation

• Qwen3-VL-32B-Instruct generates a two-view caption for every training clip because Ego-Exo4D supplies no such annotation.Using an open-weight model makes captioning this dataset feasible; Ego-Forge removes captioning at inference.Source: C CAPTION GENERATION PIPELINE · p. 15

Inputs and ordering

• The egocentric video is placed first in the joint input, so over-budget truncation drops exocentric detail; prompts allocate roughly 150 words to ego and 50 to exo.The egocentric block describes the requested output, while exocentric pixels are already provided.Source: C CAPTION GENERATION PIPELINE · p. 15

Prompt design

• Prompt directives enforce view separation, precise observable attributes, objective wording, and honest abstention when little happens or hands are out of frame.The egocentric block uses first person and excludes the wearer's body.Source: C CAPTION GENERATION PIPELINE · p. 15

Output and filtering

• Captions use two blocks, each with a scene overview and action analysis, and are filtered for empty or truncated text and near-duplicate view blocks.Clips failing either automated check are excluded from Stage 1 training.Source: C CAPTION GENERATION PIPELINE · p. 15

Training setup

• All stages run on NVIDIA H200 GPUs at batch size 1 per GPU with gradient checkpointing because the 14B backbone and stitched canvas limit memory.The hardware and batch-size setup applies across stages.Source: D TRAINING BUDGET · pp. 15–16

Preprocessing

• Preprocessing requires one-time depth estimation and reprojection for egocentric priors, plus Qwen3-VL-32B-Instruct caption generation for Stage 1.Both passes are sharded across GPUs and cached.Source: D TRAINING BUDGET · p. 15

Stage budgets

• Stage 1 uses 2 GPUs for 5 days, Stage 2 uses 1 GPU for 3 days, and Stage 3 uses 4 GPUs for 2 days.Trainable components are LoRA on S and X plus patch embedding; R only; then R and LoRA on S, respectively.Source: D TRAINING BUDGET · p. 16

Overview

• The Static Captioner translates exocentric visual tokens into the text-embedding space consumed by pretrained cross-attention, without a caption at inference.This injects visual conditioning through the pathway originally trained on text.Source: E STATIC CAPTIONING · p. 16

Architecture and inputs

• Frozen CLIP and DINOv2 features are concatenated into 2048-dimensional tokens, then projected to 4096-dimensional umT5 embeddings as the backbone's sole cross-attention input.No textual caption is used at inference.Source: E STATIC CAPTIONING · p. 16

Pretraining data

• Pretraining uses approximately 200K filtered OpenVid-1M video-caption pairs, with CLIP, DINOv2, and umT5 features precomputed and cached.The external, diverse corpus is used so the visual-to-text mapping generalizes beyond a single domain.Source: E STATIC CAPTIONING · p. 16

Distribution-alignment objective

• Mean alignment matches centroids, CORAL aligns covariances, and SWD matches sorted projected marginals; these set-level objectives preserve spatial token layout without linguistic ordering.Distribution-level alignment avoids per-token correspondence between linguistically ordered captions and spatially ordered visual tokens.Source: E STATIC CAPTIONING · p. 16

• Supplementary videos make the temporal evolution and flow of generated egocentric videos more apparent than individual frames.Source: F ADDITIONAL VISUALIZATIONS · p. 17
• The training prompt sends the egocentric video first and the exocentric video second in one call to Qwen3-VL-32B-Instruct.Source: F ADDITIONAL VISUALIZATIONS · p. 18
• The prompt establishes a static scene description before analyzing action transitions, with exhaustive object identification and detailed hand-object interactions.Source: F ADDITIONAL VISUALIZATIONS · p. 18
• The required output separates first-person ego-view and third-person exo-view blocks, allocating about 150 words to ego and 50 to exo.Source: F ADDITIONAL VISUALIZATIONS · p. 18