PaperlayerView exact revision on arXiv

Source-linked AI summary

Repurposing Geometric Foundation Models for Multi-view Diffusion

Wooseok Jang, Seonghu Jeon, Jisang Han, Jinhyeok Choi, Minkyung Kwon, Seungryong Kim, Saining Xie, Sainan Liu

arXiv:2603.22275v1cs.CV

TL;DR

Novel view synthesis needs a latent space that preserves geometric consistency across viewpoints, whereas existing methods typically use view-independent VAE latents. GLD repurposes geometric foundation-model features for multi-view diffusion, achieving stronger image and 3D consistency metrics with over 4.4× faster convergence than VAE while remaining competitive with methods using text-to-image pretraining.

  • Problem

    NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in view-independent VAE latent spaces.

  • Method

    GLD uses the feature space of geometric foundation models as the latent space for multi-view diffusion and synthesizes only through an optimized boundary with cascaded feature generation.

  • Results

    GLD improves 2D image quality and 3D consistency over VAE and RAE baselines, accelerates convergence by over 4.4×, and remains competitive with methods using large-scale text-to-image pretraining.

  • Takeaways & Limitations

    Geometry-informed latent representations can support view-consistent image generation while enabling zero-shot depth and 3D reconstruction as byproducts.

  • Takeaways & Limitations

    Severe occlusion, sparse spatial coverage, extreme lighting changes, and large temporal gaps can cause hallucinations, artifacts, or unreliable cross-view correspondences.

Abstract

from arXiv · show

While recent advances in generative latent spaces have driven substantial progress in single-image generation, the optimal latent space for novel view synthesis (NVS) remains largely unexplored. In particular, NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in a view-independent VAE latent space. In this paper, we propose Geometric Latent Diffusion (GLD), a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion. We show that these features not only support high-fidelity RGB reconstruction but also encode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space. Notably, GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite training its diffusion model from scratch without such generative pretraining.

1 Introduction

GLD repurposes geometrically consistent foundation-model features as the latent space for multi-view diffusion, targeting NVS’s need for coherent spatial structure across viewpoints. It supports high-fidelity reconstruction and improves image quality, 3D consistency, and training efficiency relative to standard latent spaces.

  • NVS must predict unseen viewpoints while preserving coherent spatial structure and plausibly completing occluded regions.
  • GLD uses geometric foundation-model features as the latent space for multi-view diffusion.The framework is designed to operate directly on geometry-aware representations.
  • GLD’s features support high-fidelity, view-consistent RGB reconstruction and encode cross-view geometric structure for NVS.
  • GLD’s synthesized latents can be decoded into geometric predictions, and decoded point clouds exhibit strong global consistency.
  • GLD explicitly synthesizes an optimal feature boundary while deriving deeper features through a frozen backbone and shallower features through cascading.This design reduces the cost of diffusing the full multi-level feature hierarchy while maintaining cross-level alignment.
  • 4.4× faster convergence is achieved by GLD while it improves pixel-level fidelity and cross-view 3D consistency over VAE and RAE baselines.

2 Related Work

Prior work has established latent-space effects for single-image generation and developed geometry-aware models for 3D reconstruction, but multi-view diffusion still commonly uses spaces lacking explicit cross-view structure.

  • Novel View Synthesis with Diffusion Models: Classical NVS methods produce photorealistic renderings but require dense multi-view captures and costly per-scene optimization.
  • Novel View Synthesis with Diffusion Models: Recent multi-view diffusion methods reduce these constraints by using generative priors to synthesize novel views from sparse inputs.
  • Novel View Synthesis with Diffusion Models: Pixel and VAE latent spaces lack cross-view geometric structure, burdening models with implicitly discovering geometric correspondences.
  • Latent Spaces for Diffusion Models: VAE-based latent diffusion, RAE, and SVG show that compressed or semantic representations can improve reconstruction, convergence, and generation quality for single-image synthesis.
  • Geometric Foundation Models: Geometric foundation models use feed-forward architectures to predict camera parameters and depth maps from arbitrary unposed views.

3 Preliminaries

The preliminaries describe representation autoencoders and geometric foundation models as feature-based alternatives to VAE latents, with multi-level features supporting RGB reconstruction and geometric prediction.

  • Representation Autoencoder: RAE replaces the conventional VAE latent with a pretrained frozen vision encoder’s feature space and trains a decoder for reconstruction.
  • Representation Autoencoder: The RAE decoder reconstructs RGB images from semantic features, demonstrating their sufficiency for high-fidelity reconstruction.
  • Representation Autoencoder: F = E(I) represents a single-view image as a tokenized feature sequence with token length T and channel dimension C.
  • Representation Autoencoder: Diffusion synthesizes a feature F̃, which the decoder maps to a synthesized image Ĩ = D(F̃).
  • Geometric Foundation Models: Geometric foundation models encode multi-view inputs with a ViT-based encoder, 3D attention, and multi-level feature sequences.
  • Geometric Foundation Models: A geometric decoder aggregates multi-level features to produce dense predictions such as depth or camera parameters.

4 Method

GLD uses geometric foundation-model features as a multi-view diffusion latent, synthesizing shallow features explicitly and deriving deeper features through a frozen encoder. The method validates reconstruction fidelity, selects level 1 as the synthesis boundary, and uses cascaded generation to maintain feature alignment.

  • Geometric latent space: GLD uses geometric foundation-model features as the latent space for multi-view diffusion and decodes synthesized features into target RGB views and geometry.The framework uses DA3 features across four intermediate levels and includes an RGB decoder plus a geometry decoder.
  • Pipeline overview: The three-stage pipeline synthesizes features through boundary layer k=1, propagates deeper features with the frozen DA3 encoder, and decodes the complete feature set.Feature propagation avoids explicitly diffusing every feature level.
  • Multi-view diffusion: GLD jointly generates source and target features because 3D attention couples views and downstream decoding requires features from all views.The diffusion model conditions on source-only features while generating the full joint feature map.
  • Reconstruction capability: DA3 features support high-fidelity RGB reconstruction, validating their suitability as a latent space for diffusion.A ViT-based decoder reconstructs RGB images from frozen multi-level features, with level-wise dropout improving robustness to partial inputs.
  • Boundary selection: Boundary k=1 provides the best NVS performance across metrics, while deeper boundaries degrade performance and level 0 alone causes feature-hierarchy misalignment.Deeper features are deterministically derived from the synthesized boundary feature, and level 0 is generated conditionally on level 1 through a cascaded model.

5 Experiments

GLD is evaluated across in-domain and out-of-domain NVS benchmarks using 2D fidelity, 3D consistency, qualitative comparisons, decoder reconstruction, and ablations. It generally outperforms VAE and DINO baselines, remains competitive with pretrained methods, and produces geometrically coherent auxiliary outputs.

  • Setup: GLD is evaluated on Re10K, DL3DV, and out-of-domain Mip-NeRF 360, with two source views in the main protocol.Training uses four datasets, while the main evaluation covers two in-domain benchmarks and one object-centric out-of-domain benchmark.
  • 2D Metrics: GLD consistently outperforms VAE and DINO baselines in PSNR, SSIM, and LPIPS across all evaluated benchmarks.The results support DA3 features as a more suitable latent representation for NVS than general-purpose visual encoders.
  • 2D Metrics: GLD surpasses fine-tuned state-of-the-art methods across 2D metrics on both in-domain benchmarks and achieves state-of-the-art PSNR out of domain.GLD is trained from scratch on smaller, scene-level datasets, whereas some competitors use object-centric data during fine-tuning.
  • 3D Metrics: Up to 2.8× lower ATE and 2.6× lower RPE are achieved against VAE and DINO baselines, with further improvements in reprojection error and MEt3R.GLD also shows stronger 3D consistency than most fine-tuned methods on in-domain benchmarks, while remaining slightly less accurate on some out-of-domain pose measures.
  • Qualitative Results: GLD generates structurally coherent target views under large viewpoint changes and avoids artifacts associated with failed external geometry estimation.It remains competitive with fine-tuned methods despite not using T2I pretraining.
  • Latent Representation: DA3 features support high-fidelity RGB reconstruction, while level 1 balances geometric alignment with photometric information.Deeper features cause color loss, smoothed textures, and lower PSNR, whereas the selected level preserves both geometry and appearance.
  • Ablation Studies: With N=1 source view on DL3DV, GLD achieves over 3× lower ATE and RPEr than both baselines, compared with an approximately 2× gap at N=4.The larger advantage with fewer views is consistent across both datasets.
  • Ablation Studies: Cascading shallower features from deeper synthesized features consistently improves both 2D and 3D metrics.The ablation evaluates Re10K with N=4 source views and supports cross-level conditioning for coherent multi-level representations.

6 Conclusion

The paper presents GLD as a geometry-informed latent space for novel view synthesis and identifies a feature level balancing geometric correspondence with photometric fidelity. GLD improves 2D quality and 3D consistency over standard latent spaces, remains competitive with pretrained methods despite scratch training, and enables zero-shot geometric outputs.

  • GLD repurposes a geometric foundation model’s feature space as the latent space for novel view synthesis.
  • The selected feature level balances geometric correspondence and photometric fidelity for diffusion-based NVS.
  • GLD improves both 2D image quality and 3D consistency over standard VAE and DINOv2 latent spaces.
  • GLD remains competitive with methods using large-scale text-to-image pretraining despite being trained entirely from scratch.
  • The generation process naturally enables zero-shot depth and 3D reconstruction as byproducts.

A.2 Architecture Details

GLD extracts multi-level geometric features, diffuses them with a conditioned multi-view architecture, and decodes them into RGB images. The design uses hierarchical feature processing, cross-view attention, camera conditioning, and cascaded synthesis for efficiency.

  • RGB Decoder: The RGB decoder is a ViT with 12 transformer layers, intermediate dimension 3072, dropout 0.5, and patch size 14.It reconstructs RGB images from geometric latent features.
  • Multi-Level Feature Extraction: The frozen geometric encoder extracts four normalized feature levels from intermediate DA3-Base backbone outputs for diffusion and decoding.The levels come from blocks 5, 7, 9, and 11; channel-wise normalization uses training-set statistics before diffusion and is inverted before decoding.
  • Multi-view Diffusion Model: Each level-wise diffusion model separates a condition encoder from a velocity decoder, following the DiTDH architecture.The encoder uses hidden dimension C1=768 and 28 blocks, while the velocity decoder uses C2=2048; level-wise decoders have 6 blocks and the cascaded decoder has 2.
  • Multi-view Diffusion Model: The model concatenates noisy latents with source-only conditions, zero-padded for target views, while cascaded synthesis additionally conditions on deeper-level features.Separate patch embedders distinguish source and target views in both encoder and decoder.
  • Multi-view Diffusion Model: 3D self-attention jointly processes tokens across views, with camera intrinsics, extrinsics, and source/target indicators supplied through 7D embeddings and PRoPE.Per-pixel Plücker coordinates encode camera geometry before projection to the hidden dimension.

A.3 Dataset Details

GLD is trained across four datasets with eight-view samples and evaluated on in-domain, out-of-domain, and depth benchmarks using fixed sampling protocols.

  • RGB Decoder Dataset: The RGB decoder is trained on Re10K and DL3DV with equal sampling, then evaluated on 4,000 images from 500 eight-view scenes at 504 × 504 resolution.The evaluation samples 500 scenes and all eight views per scene.
  • Diffusion Training: The multi-view diffusion model is trained on Re10K, DL3DV, HyperSim, and TartanAir with mixing ratio 4 : 4 : 1 : 1.Camera poses are normalized relative to the last view and scaled so the maximum within-batch camera distance is 1.
  • Evaluation Protocol: NVS evaluation uses 200 samples each from Re10K, DL3DV, and Mip-NeRF 360, while depth evaluation uses 50 ETH3D samples.When datasets have fewer scenes than the target count, randomized view sampling produces multiple samples per scene.

B Evaluation Details

Evaluation adapts baselines to a common output resolution and tests GLD with both DA3 and VGGT geometric backbones across 2D and 3D metrics.

  • Baseline Adaptation: Baseline outputs are resized from 512×512 to 504×504, using each method’s default classifier-free guidance scale for comparison.This standardizes output resolution with GLD.
  • Quantitative Evaluation: Table 13 compares DA3- and VGGT-based GLD against baseline latent representations on in-domain and out-of-domain datasets using 2D and 3D metrics.Bold and underlined values denote the best and second-best results, respectively.
  • Alternative Geometric Backbone: GLD is additionally evaluated with VGGT by replacing the DA3 encoder and decoder and training the diffusion models from scratch under the same configuration and objective.VGGT intermediate features are repurposed as the latent space.
  • Alternative Geometric Backbone: With VGGT, GLD consistently outperforms VAE and DINO baselines, especially on 3D metrics and pose estimation errors such as ATE.Overall performance is slightly lower than with the DA3 backbone, but the geometric consistency advantage remains.

C.2 Comparison with Method Trained from Scratch

The paper compares GLD with a from-scratch baseline to isolate the contribution of large-scale text-to-image pretraining. On out-of-domain Mip-NeRF 360, GLD substantially outperforms the from-scratch MVGenMaster baseline.

  • Controlled Comparison: The controlled comparison removes large-scale generative pretraining by training MVGenMaster from scratch under the same setup as GLD.MVGenMaster is the only included baseline with publicly available training code.
  • Results: On out-of-domain Mip-NeRF 360, from-scratch MVGenMaster performs substantially worse than both its finetuned version and GLD.The dataset lies outside GLD’s training domain and remains challenging for GLD.
  • Interpretation: The authors attribute MVGenMaster’s gap to limited robustness to depth–camera misalignment, which can induce warping errors during generation.The finetuned model benefits from stronger image priors inherited from text-to-image pretraining.

C.3 Additional Qualitative Results

Additional results show that GLD produces more 3D-consistent and photorealistic multi-view outputs than VAE and DINO baselines across varying numbers of source views. Compared with Matrix3D, GLD also yields geometrically accurate reconstructions with fewer cross-view misalignments.

  • Additional qualitative results: GLD consistently produces more 3D-consistent and photorealistic results than VAE and DINO across one, two, and four source views.The comparisons emphasize preservation of geometric structure and appearance across viewpoints.
  • Additional 3D visualizations: GLD produces consistent and geometrically accurate 3D reconstructions in comparisons against Matrix3D.Reference reconstructions are generated from ground-truth RGB images using DA3 because ground-truth depth is unavailable.
  • Additional 3D visualizations: In extrapolative settings, Matrix3D often misaligns generated unseen content with source views, producing artifacts such as duplicated objects and blurry structures.The reported examples include a duplicated clock, blurry chairs, and inconsistently generated unseen regions.
  • Additional 3D visualizations: In interpolative settings, GLD synthesizes viewpoints and RGB-D maps that preserve rigid structures and object boundaries without noticeable misalignment artifacts.These outputs accurately correspond with the source views.

D.1 Analysis of Geometric Correspondences in Diffusion Features

The analysis evaluates cross-view correspondence in diffusion attention maps across DA3, VAE, and DINO latent spaces, then relates correspondence patterns to geometric consistency and computational design. DA3 yields the strongest correspondence, which emerges primarily in the velocity decoder, while feature propagation reduces the cost of generating deeper levels.

  • Correspondence measurement: Cross-view correspondence is measured by nearest-neighbor matching between query and key descriptors using cosine distance, evaluated with PCK on ScanNet.Measurements are taken from 3D attention maps at each diffusion layer following the CAMEO protocol.
  • Latent-space comparison: The DA3 latent-space model exhibits the strongest cross-view correspondence across nearly all layers, with the largest margin in decoder blocks.This aligns with the reported relationship between stronger internal correspondence and more geometrically consistent multi-view generation.
  • Layerwise analysis: Correspondence is largely absent in conditional encoders but emerges sharply in velocity decoders and peaks at intermediate layers 31–32 across all three latent spaces.The analysis interprets the encoder as preserving per-view conditioning and the decoder as establishing cross-view geometry through 3D attention.
  • Computational cost: Generating complete GLD features requires two sampling stages, making GLD slower at inference than VAE despite propagation being more efficient than independently generating deeper feature levels.The latency analysis supports propagating level 2 and level 3 features rather than explicitly synthesizing them.
  • Limitations: Severe occlusion, sparse spatial coverage, extreme lighting changes, and large temporal gaps can cause hallucinations, artifacts, or unreliable cross-view correspondences.These conditions define the main reported failure boundary for GLD.
Loading 2603.22275v1…