PaperlayerView exact revision on arXiv

Source-linked AI summary

LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

Suhwan Cho, Yonwoo Choi, Soongjin Kim, Jicheol Park, Taegyu Lim

arXiv:2610.12442v1cs.CV

TL;DR

Exocentric-to-egocentric video generation is challenging because the views share little content, and explicit reconstruction can misplace content when depth is wrong. LEGO conditions video diffusion on a directly learned egocentric render and its confidence; it outperforms the explicit pipeline and generalizes to other datasets without retraining.

  • Problem

    Exocentric and egocentric views share little content, leaving much of the target view unobserved, while depth-based explicit reconstruction can misplace rendered content.

    Source: §1 · p. 2

  • Method

    LEGO conditions video diffusion on a learned direct egocentric render, using correspondence confidence to mask uncertain regions and guide early denoising.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    LEGO outperforms the state-of-the-art explicit pipeline on seen and unseen scenes and generalizes to EgoHumans and Nymeria without retraining.

    Source: §1 · p. 2, §4 · p. 8

  • Takeaways & Limitations

    The synthesizer supplies view structure and confidence for conditioning and guidance, while the diffusion model restores detail.

    Source: §5 · p. 9

    Because the synthesizer renders each frame from a single exocentric frame, it cannot recover unseen content, and the pipeline still requires both views’ camera poses.

    Source: §C.6 · p. 20

Abstract

from arXiv · show

Generating an egocentric video from a single exocentric recording is a challenging case of novel view synthesis, as the two cameras share little overlap and much of the target view is unobserved. Current state-of-the-art methods reconstruct the scene explicitly by estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. This deterministic mapping assigns each pixel to a single reprojected location, which preserves texture but translates depth errors into misplaced content. We ask what a video diffusion model should receive as its condition and propose a lifting-free answer: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly without depth, point clouds, or reprojection, resolving cross-view correspondence internally. In contrast, its probabilistic mapping averages each region over candidate source locations according to a learned correspondence distribution, preserving structure while fine texture is averaged away. We argue that this trade-off suits a diffusion generator, whose denoising training excels at restoring detail, so an effective condition should prioritize structural alignment over sharpness. This distribution's concentration also yields a per-region confidence, used both to mask low-confidence regions and to guide the generator toward high-confidence areas during early layout-forming denoising steps. Our approach consistently outperforms the state-of-the-art explicit pipeline and generalizes to other datasets without retraining. The synthesizer thus supplies view structure, and the diffusion model its detail.

1 INTRODUCTION

LEGO replaces explicit scene reconstruction with a learned synthesizer that renders the egocentric view directly, then uses its confidence for conditioning and guidance. The paper reports stronger results than the explicit state-of-the-art pipeline and generalization without retraining.

1 INTRODUCTION

  • The task is difficult because exocentric and egocentric views share little content, while depth errors in explicit reconstruction misplace rendered content.The task is especially relevant where first-person data is scarce and third-person footage is abundant.Source: §1 · p. 2
  • An LVSM-style transformer learns to render the egocentric view directly from an exocentric frame and target camera rays, replacing depth estimation, point clouds, and reprojection.The frozen synthesizer render replaces the point-cloud render in the diffusion model’s conditioning input.Source: §1 · p. 2
  • The learned probabilistic render may blur content, but diffusion denoising can restore detail more readily than it can correct misplaced content.The correspondence distribution also provides per-region confidence, which is used to mask low-confidence regions.Source: §1 · p. 2
  • ARC reuses synthesizer confidence to guide early denoising toward high-confidence regions, without additional training.The paper motivates early guidance because layout is decided at high noise and detail emerges later.Source: §1 · p. 2
  • LEGO outperforms the state-of-the-art explicit pipeline on seen and unseen scenes and generalizes to EgoHumans and Nymeria without retraining.On Ego-Exo4D, the synthesizer render outperforms the point-cloud render as a condition, and ARC alone outperforms the prior depth-derived attention bias.Source: §1 · p. 2

2 RELATED WORK

Prior exocentric-to-egocentric systems use privileged inputs, implicit conditioning, or explicit reconstruction, while camera-controlled generation often relies on 3D lifting for large viewpoint changes. LEGO draws on learned view synthesis and training-free diffusion guidance, using learned correspondence and confidence in place of reconstructed geometry.

Exocentric-to-egocentric video generation

  • Earlier exocentric-to-egocentric systems used privileged inputs or implicit features; EgoX reconstructs a point cloud and trains with a depth-derived attention bias, while LEGO replaces both with learned synthesis and training-free guidance.LEGO retains EgoX’s single-video setting and diffusion backbone.Source: §2 · pp. 2–3

Camera-controlled video generation

  • Camera-controlled generation uses pose inputs directly or explicit 3D lifting for larger viewpoint changes; LEGO keeps the pose inputs but is lifting-free.Source: §2 · p. 3

Learned view synthesis and emergent correspondence

  • Feed-forward view synthesis models can learn geometry and correspondence from rendering or self-supervised objectives, which LEGO leverages for its exocentric-to-egocentric synthesizer.The synthesizer’s correspondence concentration supplies per-region confidence.Source: §2 · p. 3

What a diffusion model needs from its condition

  • Prior diffusion work shows that generation can recover detail from coarse or blurred conditions, while structural layout is decided at high noise; this motivates prioritizing alignment over sharpness.Source: §2 · p. 3

Training-free guidance in diffusion models

  • Existing training-free diffusion guidance assumes reliable content and its mask are given, whereas ARC derives both from the synthesizer and acts only in early steps.Source: §2 · p. 3

3 METHOD

LEGO changes the condition supplied to an otherwise unchanged video generator: a frozen synthesizer renders the target view directly, with confidence used for gating and early guidance.

3 METHOD

  • LEGO replaces depth estimation, point-cloud construction, and reprojection with a frozen synthesizer render, changing the condition and sampling process rather than the generator.The render fills the egocentric half of the conditioning latent, and confidence also drives ARC during early denoising.Source: §3 · p. 4, Figure 2 · p. 4

3.1 LEARNED VIEW SYNTHESIZER

  • A fine-tuned LVSM-style synthesizer maps exocentric frames and camera-ray maps to egocentric frames, supervised only by the target recordings.Fine-tuning adapts the synthesizer to the target fisheye camera; its probabilistic mapping trades sharp texture for content that stays closer to the correct location.Source: §3.1 · pp. 4–5

3.2 CONFIDENCE-GATED CONDITIONING

  • The synthesizer’s correspondence confidence is the probability mass of its k most likely sources; locations below a threshold are replaced with neutral gray.The gated render is encoded as the generator’s conditioning input, while the frozen synthesizer’s renders and confidence maps are precomputed once per clip.Source: §3.2 · p. 5

3.3 ADAPTIVE RENDERING CONSISTENCY

  • ARC steers early denoising toward the synthesizer render in proportion to confidence, then leaves later steps unconstrained for detail generation.The correction is deterministic, adds no noise, and is strongest in high-confidence regions; intermediate confidence is down-weighted by squaring the confidence.Source: §3.3 · p. 6
  • Table 2 organizes ablations of conditioning inputs and denoising guidance under one training recipe, with ARC applied at inference and GGA at training and inference.Figure 4 presents the corresponding qualitative ablation on one unseen clip, with enlarged crops in its second row.Source: Table 2 · p. 7, Figure 4 · p. 7

4 EXPERIMENTS

On Ego-Exo4D, LEGO outperforms EgoX on image criteria and FVD, while temporal flickering and motion smoothness are near saturation and dynamic degree is comparable. Ablations, confidence analyses, and transfer tests examine the conditioning choices and their scope.

Quantitative evaluation

  • LEGO outperforms EgoX on all image criteria and FVD on both splits, with the largest gains on PSNR and LPIPS; temporal flickering and motion smoothness are near saturation, and dynamic degree is comparable.Other baselines perform substantially worse than both methods.Source: §4 · p. 7

Ablation Study

  • ARC improves all metrics without retraining, while fine-tuning the synthesizer for the egocentric camera improves on the point-cloud render; gating lowers FVD, and restoring exocentric tokens brings further gains.GGA does not improve over full conditioning, and applying ARC to a point-cloud render with a binary hole mask performs worse than no guidance.Source: §4 · pp. 7–8

Alignment versus sharpness

  • The synthesizer render is blurriest but best aligned, whereas the point-cloud render is sharp but poorly aligned; despite blur, the synthesizer conditions diffusion better.Table 3 compares condition quality on the unseen split; generations from both renders reach the same sharpness, indicating that the output recovers condition details.Source: §4 · p. 8, Table 3 · p. 8

Confidence

  • Higher-confidence windows have lower render and output errors than discarded windows on seen and unseen scenes, and their errors remain below the all-window average at every cutoff.Figure 5 plots error for the most confident fraction of windows against the average over all windows.Source: §4 · p. 8, Figure 5 · p. 8

Generalization to other datasets

  • Without fine-tuning, LEGO outperforms EgoX† on all metrics on EgoHumans and Nymeria; the gate retains roughly 15% of renders there, as on unseen Ego-Exo4D scenes.Table 4 evaluates both datasets using weights trained on Ego-Exo4D only; each dataset has 200 test clips.Source: §4 · pp. 8–9, Table 4 · p. 9

Qualitative evaluation

  • On benchmark and in-the-wild clips, LEGO preserves layout where EgoX’s depth errors misplace hands and nearby objects; without ground truth, LEGO’s generated views remain consistent with the exocentric video.Figure 6 includes benchmark and in-the-wild comparisons; ground truth is unavailable for the in-the-wild clips.Source: §4 · p. 9, Figure 6 · p. 9

5 CONCLUSION

LEGO conditions video diffusion on a learned synthesizer’s confidence-gated render, prioritizing view alignment over sharpness. The paper reports state-of-the-art results and transfer without retraining, and identifies privacy risks for use outside consented settings.

Conclusion

  • LEGO uses a learned view synthesizer’s confidence-gated render instead of point-cloud reconstruction because diffusion restores detail more readily than it corrects misplaced content.The synthesizer supplies view structure and the diffusion model supplies detail.Source: §5 · p. 9
  • The lifting-free system uses synthesizer confidence for gating and training-free guidance, and its weights transfer to new scenes and recordings without retraining.The conclusion also notes that a stronger synthesizer could be substituted as one becomes available.Source: §5 · p. 9

Reproducibility statement

  • The authors state that protocols, hyperparameters, code, configurations, weights, precomputed renders and confidence maps, and test clip lists will be released upon publication.These materials are described as sufficient to reproduce every reported configuration.Source: REPRODUCIBILITY STATEMENT · p. 10

Limitations and ethics

  • The work uses three consented public benchmarks, collects no new data, and runs no human study; identifiable-person generation presents privacy and impersonation risks outside consented settings.The authors plan research-only model terms and will not release generated videos of identifiable individuals.Source: ETHICS STATEMENT · p. 10

Use of large language models

  • Large language models assisted with drafting and editing, while the authors say the research, experiments, and reported numbers are their own and reviewed.The authors state that they take full responsibility for the content.Source: THE USE OF LARGE LANGUAGE MODELS · p. 10

A IMPLEMENTATION DETAILS

The implementation derives target rays from calibrated camera models, fine-tunes an LVSM-style synthesizer, and trains a video generator under a matched EgoX protocol with specified differences.

A.1 CAMERA MODELS

  • The exocentric camera uses released intrinsics without radial distortion; the egocentric rays use an inverted Aria fisheye model and are expressed in the exocentric frame.Applying the supplied radial distortion coefficients raises annotated-joint reprojection error from 4 to 143 pixels at 3840×2160; rays outside the image circle are not masked.Source: §A.1 · p. 15

A.2 VIEW SYNTHESIZER

  • The LVSM-style synthesizer is fine-tuned on Ego-Exo4D at 448×448 egocentric and 448×784 exocentric resolution, using image losses and 10k training steps.Its 24-block, width-768 transformer is initialized from a published checkpoint; confidence is derived from attention in the last eight layers and pooled to the generator token grid.Source: §A.2 · p. 15

A.3 GENERATOR

  • The generator uses Wan2.1-I2V-14B with rank-256 LoRA for 20k steps; inference uses 50 denoising steps, and ARC guides the first 80% of the schedule.For EgoHumans and Nymeria, prompts are generated from eight exocentric frames; ARC hyperparameters were chosen on the unseen split because no separate validation split is provided.Source: §A.3 · p. 15

B.1 COMPARISON PROTOCOL

  • The comparison matches EgoX’s released configuration on most settings, but LEGO trains for 20k steps on four GPUs versus EgoX’s one-day run on eight GPUs.The authors use their own evaluation implementation; the released checkpoint scores 15.57 (16.05) dB PSNR on seen and 13.53 (14.38) dB on unseen scenes, with parentheses giving Kang et al.’s reported numbers.Source: §B.1 · p. 15

B.2 IMAGE AND VIDEO METRICS

The evaluation compares generated and ground-truth videos at clip level, using separate image, video, sharpness, and chroma criteria with specified regions and sampling.

Evaluation protocol

  • Generated and ground-truth videos are compared over 49 frames per clip; frame scores are averaged within clips and then across each split.The EgoX metrics are reimplemented from paper descriptions because its evaluation code is not public, with calibration against the released checkpoint.Source: §B.2 · p. 16

Image criteria

  • Image criteria are computed on the 448×448 egocentric half of the canvas: PSNR, SSIM, LPIPS, and CLIP-I.SSIM uses an 11×11 Gaussian window with σ=1.5; LPIPS uses AlexNet, and CLIP-I uses cosine similarity between CLIP ViT-B/32 embeddings.Source: §B.2 · p. 16

Video criteria

  • FVD uses I3D features from one 16-frame sample per video, while flickering, motion smoothness, and dynamic degree use VBench on the full canvas.Dynamic degree is a per-clip binary decision based on RAFT optical flow with the same flow threshold for every method.Source: §B.2 · p. 16

Sharpness and chroma

  • Condition sharpness is Laplacian-response variance relative to ground truth, and chroma is mean chroma magnitude relative to ground truth within the valid egocentric circle.Both ratios are averaged over 13 latent-aligned frames and clips; a value of 1 means the same amount as ground truth.Source: §B.2 · p. 16

B.3 OBJECT-LEVEL METRICS

The object-level evaluation fixes a protocol for computing EgoX’s metrics and finds that LEGO outperforms the released checkpoint on both splits, while the published seen row is not reproducible. The section also specifies metrics for evaluating conditioning-render quality against ground-truth egocentric frames.

B.3 OBJECT-LEVEL METRICS

  • The three object-level metrics are LocErr for matched-box center distance, IoU for box overlap, and Contour for shape agreement.The protocol settings were selected by comparing the released checkpoint with six published values; combinations retaining fewer than 30 matched pairs on either split were discarded.Source: §B.3 · p. 16
  • Contour is computed as boundary F-score after centroid-aligning masks, because this reproduces published values unlike EgoX’s literal mask-IoU definition.The protocol segments and tracks objects with SAM 2, matches crop embeddings with DINOv3, and averages LocErr and IoU over clips.Source: §B.3 · p. 16
  • LEGO outperforms the released EgoX checkpoint on all three object-level metrics on both splits under the fixed protocol.Table 5 also reports a separate sweep combination closest to the published seen row.Source: §B.3 · p. 16, Table 5 · p. 17
  • The protocol reproduces the published unseen row nearly exactly, but no searched combination reproduces the published seen row.For unseen scenes, LocErr is 146.5 against 149.9 and IoU is 0.091 against 0.092; the published seen row is quoted for reference only.Source: §B.3 · p. 16, Table 5 · p. 17

B.4 CONDITION QUALITY METRICS

  • Condition renders are evaluated on 13 generator-aligned frames from the unseen split, using alignment, texture, sharpness, and coverage measures.The measures include patchwise ZNCC, DINO co-location, Laplacian-variance sharpness ratio, coverage of the valid circle, and alignment in dB as PSNR.Source: §B.4 · p. 17
  • The valid evaluation region is the ground-truth-lit circle eroded by 4 pixels, further restricted to rendered pixels for the point-cloud render.The point-cloud region is eroded by another 3 pixels to exclude hole edges.Source: §B.4 · p. 17

B.5 CROSS-DATASET EVALUATION

The cross-dataset evaluation uses held-out EgoHumans and Nymeria clips recorded with Aria fisheye cameras and applies the EgoX evaluation pipeline with shared hyperparameters. In the conditioning-render comparison, the synthesizer is better aligned and covers more of the valid circle than the point-cloud render, while sharpness and coverage alone do not ensure alignment.

B.5 CROSS-DATASET EVALUATION

  • EgoHumans evaluation uses 200 held-out 49-frame clips across seven activity scenarios, with one exocentric camera selected per clip.The selected camera places the wearer closest to image center; egocentric frames are raw fisheye recordings.Source: §B.5 · p. 17
  • Nymeria evaluation uses 200 held-out clips from 26 sequences, retaining windows with a nearly static observer camera and a visible participant.The observer is treated as the exocentric camera, and its fisheye frames are undistorted to a pinhole view.Source: §B.5 · pp. 17–18
  • Both datasets are evaluated on the same clips with EgoX’s released checkpoint and pipeline, using the Ego-Exo4D hyperparameters.The EgoX pipeline includes depth estimation, point-cloud rendering, and geometry-guided attention.Source: §B.5 · p. 18

C.1 ALIGNMENT, SHARPNESS, AND COVERAGE

  • The synthesizer render is better aligned than the point-cloud render by 4.2 dB inside the valid circle, including when evaluated only on point-cloud-valid pixels.This indicates the alignment difference is not simply due to missing point-cloud regions.Source: §C.1 · p. 18
  • The point-cloud render covers 53.4% of the valid circle, compared with 98.1% for the synthesizer render.The untuned synthesizer is sharper than the fine-tuned version (0.50 vs. 0.28) and covers 99.6%, yet has the worst ZNCC alignment of the three.Source: §C.1 · p. 18

C.2 OUTPUT SHARPNESS AND COLOR

Generated sharpness remains similar across conditioning renders, but generated color tracks the render more closely. The paper attributes the color behavior to the different uncertainty responses of the synthesizer and generator, and links color drift to part of the unseen FVD gap.

C.2 OUTPUT SHARPNESS AND COLOR

  • On unseen scenes, output sharpness is similar with point-cloud and synthesizer conditioning: Laplacian-variance ratios are 1.10 and 1.04, respectively.The conditioning renders differ in sharpness, so output sharpness is not strongly determined by the render’s sharpness.Source: §C.2 · p. 18
  • On unseen scenes, output chroma ratios are 0.45 with synthesizer conditioning, 0.82 with point-cloud conditioning, and 1.26 without conditioning.The synthesizer render’s chroma ratio is 0.41; gating it partially recovers output chroma to 0.62.Source: §C.2 · p. 18
  • Under correspondence uncertainty, the regression synthesizer averages chroma toward gray, while the generator samples a plausible mode using its pretrained color prior.The paper states that color drift accounts for part of the unseen FVD gap because FVD is sensitive to per-frame appearance statistics.Source: §C.2 · p. 18

C.3 ADAPTING TO THE FISHEYE CAMERA

Fine-tuning the synthesizer for the fisheye target camera reduces spill and improves alignment; confidence gating uses the synthesizer’s confidence to distinguish more reliable regions. LEGO’s condition-building and guidance costs are lower than the corresponding EgoX costs, while the approach retains limits inherited from the synthesizer.

C.3 ADAPTING TO THE FISHEYE CAMERA

  • The untuned synthesizer spills scene content into 95.9% of the region outside the valid fisheye circle and trails the fine-tuned model in alignment by 5.4 dB on unseen scenes and 10.3 dB on seen scenes.Conditioning the generator on the untuned render reduces performance by 0.8 dB on unseen scenes and 2.5 dB on seen scenes.Source: §C.3 · p. 18
  • Fine-tuning leaves only 0.1% spill outside the image circle, compared with 22.4% for the point-cloud render.The paper concludes that adapting the synthesizer to the target camera is necessary.Source: §C.3 · p. 19

C.4 CONFIDENCE GATING

  • At τ=0.3, high-confidence windows have lower render and output luminance errors than discarded windows on both seen and unseen scenes.Confidence is the probability mass of the top 16 sources; the gate sets tokens below threshold to neutral gray.Source: §C.4 · p. 19

C.5 COMPUTATIONAL COST

  • Building LEGO’s condition takes 4.7 seconds per clip, versus 69.2 seconds for EgoX depth estimation and point-cloud rendering.ARC updates the clean estimate in 2.7 ms per step, while GGA adds 252 ms per forward pass; overall, LEGO is faster and uses slightly less memory.Source: Table 6 · p. 19, §C.5 · p. 20

C.6 LIMITATIONS

  • A single-frame synthesizer cannot recover content absent from the exocentric frame and blurs uncertain correspondences; colors also collapse on unfamiliar scenes.LEGO still requires camera poses for both views, but a stronger feed-forward synthesizer can replace the frozen one without changing the diffusion model or ARC.Source: §C.6 · p. 20

D ADDITIONAL QUALITATIVE RESULTS

In additional qualitative comparisons, LEGO keeps scene content aligned with the ground truth where EgoX places sharp content incorrectly. The pattern also appears across Ego-Exo4D, EgoHumans, and Nymeria, with softer detail and less saturated colors remaining on unseen scenes.

D ADDITIONAL QUALITATIVE RESULTS

Loading 2610.12442v1…