Source-linked AI summary
Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations
Denis M. Akola, David F. Fouhey
TL;DR
The paper asks whether 3D foundation-model representations contain enough geometric knowledge to infer unseen or hidden scene structure from sparse views. It introduces Z3D, which applies latent diffusion in the 3DFM representation space and decodes the result into novel-view depth maps, producing realistic and geometrically consistent predictions across multiple settings while remaining sensitive to limited source-target overlap.
Problem
Sparse observations leave unseen and occluded scene geometry ambiguous, while prior rendering and 2D depth-generation methods provide limited generalizable 3D-consistent depth synthesis.
Method
Z3D conditions latent diffusion on pretrained 3DFM scene representations and target-view poses, then uses the pretrained decoder to produce novel-view depth maps.
Results
Z3D predicts realistic novel-view depth maps and produces cleaner, more coherent 3D structures while closely matching foundation-model target-view depths across evaluated settings.
Takeaways & Limitations
3DFM internal representations can support zero-shot geometric completion when paired with diffusion, extending depth prediction beyond directly visible surfaces.
Takeaways & Limitations
Z3D struggles with fine-grained novel-view depth when source and target views have limited overlap, inheriting 3DFM sensitivity to reference-view choice and insufficient coverage.
Abstract
from arXiv · showhide
3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.
1 Introduction
Novel depth synthesis seeks geometrically consistent depths for unseen viewpoints from sparse observations, including occluded structure. Z3D addresses this by combining pretrained 3D foundation-model representations with latent diffusion to generate plausible, multi-view-consistent depth.
- Motivation: Sparse-view novel depth synthesis must infer geometrically consistent structure outside visible surfaces, including regions hidden by occlusions.The capability supports physical interaction, planning, and spatial reasoning in incomplete environments.
- Limitations of Prior Work: Neural scene representations improve novel-view rendering but mainly interpolate observed views, while 2D generative depth models lack strong 3D consistency priors.These limitations make generalizable depth synthesis from sparse inputs difficult.
- Z3D: Z3D combines pretrained 3DFM intermediate representations with latent diffusion, then decodes predicted latents into geometry for unseen views.This decouples representation learning from generative completion and supports hallucination of occluded or unobserved structure.
- Z3D: The method targets consistent depth maps from sparse inputs without per-scene optimization, conditioning generation on scene representations and target-view poses.Unlike approaches focused primarily on photometric quality, it explicitly models geometry beyond visible surfaces.
- Z3D: Z3D uses pretrained 3DFM features to decouple scene representation from generation and supports plausible depths for multiple novel viewpoints without restricting target cameras to the origin.This distinguishes it from prior methods that learn representations within diffusion or use restrictive camera assumptions.
3 Method
Z3D combines pretrained 3DFM scene representations with conditional latent diffusion to synthesize depth from sparse source views at novel camera poses. The method first probes hidden-surface information in 3DFM features, then decodes diffusion-predicted target-view tokens with a pretrained depth head.
- 3 Method: Z3D targets novel-view depth from one or more input views and target camera poses, with additional views expected to improve target-depth accuracy.The base case uses one source and one target view; the framework also supports multiple target viewpoints.
- 3.1 Do 3DFMs Know about Unseen Structure?: 3DFMs’ patch tokens encode information about occluded scene structure, as linear probes predict layered depth beyond the first visible surface.The probe predicts four depth layers and confidence maps; VGGT and WorldMirror linear probes halve the error rate relative to the dataset average.
- 3.2 Network Architecture: Z3D predicts novel-view depth by applying conditional diffusion directly to 3DFM patch-token representations using source images and target relative poses.Only target-view tokens are noised during training, while clean source tokens condition the denoiser through cross-attention.
- 3.2 Network Architecture: The pipeline encodes source images with a 3DFM, conditions DiT blocks on pose and intermediate activations, and sends predicted tokens through a frozen DPT head.Pose and diffusion timestep information enter through adaptive normalization or timestep embeddings, while 3DFM activations provide geometric conditioning.
- 3.2 Network Architecture: Diffusion is restricted to layer-17 scene tokens because they contribute most to depth prediction, reducing computation while retaining the most depth-relevant representation.The backbone otherwise aggregates high-dimensional tokens from multiple intermediate layers, making direct diffusion on all concatenated outputs costly.
4 Experiments
Z3D is evaluated across indoor, outdoor, in-domain, and out-of-domain settings using depth and point-cloud metrics, qualitative comparisons, and multi-view reconstruction tests. Across these evaluations, Z3D generally provides smoother depth, cleaner geometry, and stronger out-of-domain performance, while remaining sensitive to limited view overlap.
- Qualitative Results: Z3D produces smoother novel-view depth maps than direct depth- or patch-space diffusion baselines, which show gradient spikes and less smooth predictions.This pattern is reported for both VGGT and WorldMirror variants.
- Qualitative Results: Projected point clouds from Z3D are cleaner and more geometrically coherent than those from VGGT-DD and WM-DD, which exhibit higher noise and reduced sharpness.In the two-source, four-target setting, Z3D also maintains consistent depth across novel views and yields a coherent reconstructed point cloud.
- Experimental Settings: Z3D performs well across one-source/one-target and two-source/four-target settings, often outperforming baselines on depth metrics while remaining comparable on point-cloud reconstruction metrics.The evaluation covers indoor and outdoor scenes, including out-of-domain datasets.
- Out-of-domain Evaluation: Z3D-VGGT and Z3D-WM consistently outperform depth-diffusion baselines in out-of-domain point-cloud accuracy and depth metrics, while performing similarly to each other.The authors attribute this to richer 3DFM scene representations and implicit diffusion-based modeling of novel-depth synthesis.
- Outdoor Evaluation: Z3D substantially outperforms depth-diffusion models on MegaDepth depth metrics and point-cloud accuracy, with slightly lower point-cloud completeness.The result supports applicability across both indoor and outdoor scenes.
- Generalization to Newer 3DFMs: Z3D can be adapted to the newer VGGT-Ω foundation model without algorithmic modifications while retaining performance comparable to Z3D-VGGT and Z3D-WM.The authors describe this as evidence that the framework is largely model-agnostic.
- Limitations: Z3D performance degrades when source and target views have limited overlap, making accurate novel-view depth difficult under insufficient scene coverage.The authors identify this as a typical failure mode of both Z3D-VGGT and Z3D-WM.
5 Conclusion
Z3D is a novel depth synthesis model that uses at least one view and a new view pose to predict depth for that view.
- Z3D predicts novel-view depth from at least one input view and a new view pose.
-Supplementary Material
The supplementary material identifies the authors and their affiliations at New York University.
- Denis Mbey Akola and David F. Fouhey are the paper's authors.
- Both authors are affiliated with New York University.
- The listed NYU affiliations are the Tandon School of Engineering and the Courant Institute of Mathematical Sciences.
1 Introduction
The introduction section passage identifies the supplementary material as covering dataset curation, implementation details, additional results, and limitations.
- The supplementary material includes dataset curation, diffusion implementation details, additional results, and Z3D limitations.
2 Training dataset curation
Training and evaluation sequences are curated from depth- and camera-annotated datasets using strong geometric overlap, with dataset statistics summarized in Table 1.
- Camera sequences are constructed from diverse depth- and camera-annotated datasets to provide strong geometric overlap for Z3D training and evaluation.
- Scenes with fewer than 1,000 images use all camera poses, while larger scenes are evenly subsampled to 1,000 cameras.
- Depth maps across camera pairs produce an N×N symmetric overlap matrix using bidirectional frustum overlap and its minimum to enforce mutual visibility.
- Table 1 summarizes the total scenes and images for each dataset used in the experiments.
- Habitat-Matterport scenes include 1,000 RGBD observations generated per scene with the Habitat 2.0 simulator.
- DTU, 7-Scenes, and NRGB-D follow the same evaluation strategy and use each dataset's official splits.
3 Implementation details
The implementation trains the method in two stages and uses data-dependent diffusion scheduling; additional LDI probing tests whether 3DFM features encode hidden scene structure.
- Training: Two-stage training uses target–source view pairs first, then 2 source views and 4 target views.Stage 1 runs for 98k steps with effective batch size 128; Stage 2 runs for 156k steps with effective batch size 32.
- Diffusion scheduling: The noise scheduler shifts timesteps according to latent dimensionality to reduce information corruption during diffusion training.The latent dimension is m = n_target view × P × C, with n = 4096 used as the base token dimension.
- LDI probing: Linear probing expands a 3DFM depth head to predict four depth layers and confidence maps while keeping the remaining layers fixed.The probe evaluates whether the backbone contains information extending beyond the first visible surface.
4 Additional LDI Results
Additional LDI results show that pretrained VGGT and WorldMirror backbones support layered depth prediction, reinforcing their use for hidden-surface reasoning.
- Additional LDI results: Additional LDI prediction results are provided for both pretrained VGGT and WorldMirror backbones.These results extend the layered-depth analysis beyond the primary examples.
5 Which layer features of 3DFM backbones should be used for diffusion?
The analysis identifies layer 17 as the most useful depth representation for diffusion, enabling a lower-cost target process while retaining richer source conditioning.
- Computational constraint: Diffusing all aggregated backbone tokens is computationally expensive because each token has a high-dimensional representation.VGGT and WorldMirror use 2048-dimensional token representations across multiple intermediate layers.
- Layer analysis: The layer study evaluates every combination of four geometry-informative layers against depth inferred from the full layer set.This comparison determines which subset best approximates the full representation.
- Layer analysis: Layer 17 features produce depth predictions nearly as good as all four selected layers for both VGGT and WorldMirror.Configurations containing layer 17 consistently outperform those without it across single-layer, two-layer, and three-layer combinations.
- Design choice: Z3D restricts target-view diffusion to layer 17 tokens while conditioning on all four source-view aggregation layers.This design reduces diffusion cost while preserving access to richer multi-layer source geometry.
6 More Qualitative Results
Qualitative and quantitative evaluations show that Z3D produces smoother, cleaner, and geometrically consistent novel-view depth and point-cloud reconstructions across sparse-view settings.
- Point-cloud reconstruction: Z3D-VGGT and Z3D-WM produce cleaner and more coherent point clouds than VGGT-DD and WM-DD in single-source and multi-target settings.The DD baselines show higher noise and reduced geometric sharpness, while Z3D produces high-fidelity point clouds when views overlap sufficiently.
- Novel-view depth: Z3D predictions are substantially smoother than predictions from a method operating directly in patch-space.The qualitative comparison includes source RGB, target RGB, ground-truth depth, 3DFM depth, DD prediction, and Z3D prediction.
- Quantitative evaluation: Across 1-source→1-target and 2-source→4-target settings, Z3D predictions closely match target-view depths produced by the foundation models.The evaluation uses point-cloud and depth metrics on both in-domain and out-of-domain data, with 3DFM predictions treated as reference depths.
8 Limitations of Z3D
Z3D struggles when source and target views have limited overlap, especially in predicting fine-grained novel-view depths. Its outputs preserve overall scene structure but omit details and produce inconsistent projected point clouds.
- 8 Limitations of Z3D: Limited overlap between source and target views causes Z3D-VGGT and Z3D-WM to struggle with fine-grained novel-view depth prediction.The limitation is attributed to insufficient shared scene content between the views.
- 8 Limitations of Z3D: Both variants preserve overall scene structure, but their depth maps miss finer details and yield inconsistent projected point clouds.This behavior is shown for point maps obtained from the predicted depth maps.