Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations

1 Introduction

• Novel depth synthesis targets plausible, geometrically consistent depths from sparse views, including regions hidden by occlusion and geometry outside the visible surface.Such inferred geometry can support physical interaction, planning, and spatial reasoning in incomplete environments.Source: §1 · p. 1
• NeRF, 3D Gaussian Splatting, and related methods mainly interpolate observed views, while 2D generative depth models lack strong 3D consistency priors.Sparse-view depth generalization remains limited, with geometric ambiguity and transfer across diverse scenes posing challenges.Source: §1 · pp. 1–2
• Z3D conditions latent diffusion on pretrained 3DFM representations and novel-view poses, then decodes predicted latents into depth geometry.The method separates representation learning, handled by the 3DFM, from generative completion, handled by diffusion.Source: §1 · p. 2, §2 · p. 2
• A lightweight decoder attached to pretrained 3DFM features can infer hidden surfaces, indicating that their representations encode geometry beyond visible surfaces.This supports using the representations to reason about occluded geometry and unseen viewpoints.Source: §1 · p. 2, §2 · p. 3

2 Related Work

• Unlike novel-view methods focused on photometric quality, Z3D explicitly models geometry and predicts depth beyond visible surfaces without per-scene optimization.NeRF and extensions often require many inputs and per-scene optimization; generalizable methods including PixelNeRF, SRT, RUST, LVSM, and RayZer emphasize photometric quality.Source: §2 · p. 2
• Unlike CUT3R, Z3D combines pretrained 3DFM representations with diffusion; it predicts plausible depths for multiple target viewpoints without requiring target cameras at the origin.Prior joint view-and-depth work uses learned scene representations and restrictive camera assumptions, while CUT3R queries virtual cameras to infer unobserved regions.Source: §2 · p. 3
• 3DFMs such as DUSt3R, VGGT, and WorldMirror [23] predict poses, depth, and point maps feed-forward, but are not designed for novel-view synthesis out of the box.Z3D repurposes their internal scene-level geometric features by pairing them with diffusion models.Source: §2 · pp. 2–3
• Existing diffusion depth models [51] predict only surfaces visible in inputs, whereas Z3D conditions diffusion on high-dimensional pretrained 3DFM features.Using those features requires careful tuning of noise schedulers.Source: §2 · p. 3

3.1 Do 3DFMs Know about Unseen Structure?

• Linear probes produce plausible hidden-surface layered depth images and halve error versus the dataset average, indicating 3DFMs encode occluded scene structure.The probe predicts four depth layers and confidence maps from VGGT’s DPT head while keeping the other layers fixed.Source: Figure 1 · p. 4, §3.1 · p. 5, Table 1 · p. 5

3.2 Network Architecture

• Z3D diffuses target-view 3DFM tokens conditioned on source-view features and relative poses, then decodes sampled tokens into novel-view depth maps.Diffusion runs on layer 17 target tokens; all four aggregation layers from source views condition the model through cross-attention, and target pose embeddings are added to timestep embeddings.Source: §3.2 · pp. 5, 7

3.3 Implementation Details

• Training proceeds from one source and one target view for 98k steps to two sources and four targets for 156k steps, initialized from Stage 1.Stage 1 uses effective batch size 128 and Stage 2 uses 32; Stage 2 initialization was more stable and converged faster than training that configuration from scratch.Source: §3.3 · p. 8

3.4 Training Datasets

• Z3D trains on overlapping camera-view sequences drawn from five real and synthetic datasets spanning indoor and outdoor environments.The datasets are MegaDepth, Hypersim, Taskonomy, Replica [43], and Habitat HM3D.Source: §3.4 · p. 8

4.1 Baselines

• The baselines combine LVSM novel-view synthesis with VGGT or WorldMirror depth estimation, or train Z3D’s diffusion architecture directly on depth maps.A direct comparison with MVGD was not possible because its code was unavailable.Source: §4.1 · p. 9

4.2 Evaluation

• Evaluation covers DTU, NRGBD, 7-Scenes, and in-domain test data, using aligned depth metrics and point-cloud accuracy and completion.Predicted depths are aligned to ground truth by least-squares fitting before computing AbsRel and δ < 1.25.Source: §4.2 · pp. 9–10
• Z3D predicts reasonable novel-view depths, while depth-space diffusion yields less smooth maps, gradient spikes, and noisier point clouds.With two source views, Z3D also produces consistent depths across novel views.Source: §4.2 · p. 10, Figure 3 · p. 10, Figure 5 · p. 11

Quantitative Evaluation

• In the 2-source-to-4-target setting, baselines often perform better on depth-map metrics, while their point-cloud reconstruction metrics are comparable to Z3D.Z3D’s results are qualitatively smoother than depth-diffusion methods.Source: Quantitative Evaluation · pp. 10–11
• Z3D-VGGT and Z3D-WM consistently outperform baselines on out-of-domain point-cloud accuracy and depth metrics, and perform comparably to each other.The reported results attribute this performance to rich 3D foundation model scene representations and diffusion-based implicit modeling of novel-depth synthesis.Source: Quantitative Evaluation · p. 12, Table 2 · p. 13, Table 3 · p. 13
• On MegaDepth, Z3D substantially outperforms depth-diffusion models on depth metrics and has higher point-cloud accuracy but slightly lower completeness.The results include outdoor data and show that Z3D works in both indoor and outdoor scenes.Source: Quantitative Evaluation · p. 13, Table 6 · p. 14
• Without algorithmic changes, Z3D-VGGT-Ω achieves performance comparable to Z3D-VGGT and Z3D-WM, demonstrating adaptation to a newer 3D foundation model.It is evaluated with one source view and one target view.Source: Quantitative Evaluation · p. 13, Table 7 · p. 14, Table 8 · p. 15
• When source and target views have limited overlap, Z3D struggles to predict accurate fine-grained depth, producing inconsistent point clouds.The setup uses the first source image as the reference camera, and the paper notes that insufficient scene coverage makes reliable reconstruction challenging.Source: Quantitative Evaluation · pp. 13–15, Figure 7 · p. 15

5 Conclusion

• Z3D predicts novel-view depth from an arbitrary number of posed images, conditioned on the new view’s camera pose and the scene’s latent 3DFM representation.The model requires at least one view and a pose for the new view.Source: §5 · p. 15

• The supplementary material provides dataset-curation and diffusion-model implementation details, additional results, and Z3D limitations.Source: §1 · p. 22

2 Training dataset curation

• The authors select diverse datasets with depth maps and camera annotations to construct strongly overlapping camera sequences for training and evaluation.Table 1 summarizes the number of scenes and images in each dataset.Source: §2 · p. 22, Table 1 · p. 22
• For scenes with fewer than 1,000 images, all camera poses are used; larger scenes are evenly subsampled to 1,000 cameras before overlap-based sequence extraction.Depth maps are projected across camera pairs to compute a symmetric overlap matrix; each entry is the minimum of the two directional frustum overlaps, enforcing mutual visibility.Source: §2 · p. 22
• Habitat-Matterport scenes receive 1,000 RGBD observations each from the Habitat 2.0 simulator, and training and evaluation use each dataset’s official splits.Evaluation datasets include DTU, 7-Scenes, and NRGB-D.Source: §2 · p. 23

3 Implementation details

• Training uses 98k Stage 1 steps on target-source pairs and 156k Stage 2 steps with 2 source and 4 target views, using effective batch sizes 128 and 32, respectively.Both stages use AdamW with β1 = 0.90, β2 = 0.95, and learning rate 2e-4.Source: §3 · p. 23
• Because default diffusion noise schedulers do not work out-of-the-box for high-dimensional 3DFM latents, Z3D matches DiT width to the latent token dimension, following.The architecture is summarized in Tab. 2.Source: §3 · p. 23
• Z3D makes the noise-scheduler timestep shift data-dependent using a dimension-dependent scaling rule, where m is the target-view latent-token dimensionality and n = 4096 is the base token dimension.Here m = ntarget view × P × C; for VGGT and WorldMirror, P = 1369 and C = 2048. The passage’s displayed scaling expression is incomplete.Source: §3 · pp. 23–24

• Figure 1 presents additional LDI prediction results for both pretrained VGGT and WorldMirror backbones.Source: §4 · p. 24

5 Which layer features of 3DFM backbones should be used for diffusion?

• To choose diffusion features, the analysis compares depth predictions from combinations of layers 4, 11, 17, and 23 against predictions using all four layers.These layers were identified in as informative for geometric reasoning; applying diffusion to all four together is computationally intractable because of their high dimensionality.Source: §5 · pp. 24–25
• Layer 17 alone produces depth predictions nearly as good as all four selected layers for both VGGT and WorldMirror, and combinations including it consistently outperform those without it.The trend holds across single-, two-, and three-layer configurations, with layer 17 producing sharper depth maps across datasets.Source: §5 · p. 25, Figure 2 · p. 26, Figure 3 · p. 27
• The method diffuses target-view scene tokens from layer 17 and conditions on outputs from all four aggregation layers of the source views through cross-attention.This retains multi-layer geometric information in the conditioning while reducing the computational cost of diffusion.Source: §5 · p. 25

6 More Qualitative Results

• In 1-source-to-1-target reconstructions, Z3D-VGGT and Z3D-WM produce cleaner, more coherent 3D structures than the noisier, less geometrically sharp VGGT-DD and WM-DD.The comparison is shown in Figures 5 and 6.Source: §6 · p. 28, Figure 5 · p. 29, Figure 6 · p. 30
• With sufficient view overlap, Z3D-VGGT and Z3D-WM generate geometrically consistent depth maps across 4 target views from 2 source views, forming coherent, high-fidelity point clouds.Figures 7 and 8 show predictions conditioned on the target camera poses.Source: §6 · p. 28, Figure 7 · p. 30, Figure 8 · p. 31
• Z3D produces substantially smoother novel-view depth predictions than methods operating directly in patch space.Figure 4 compares Z3D with depth-diffusion baselines WM-DD and VGGT-DD.Source: Figure 4 · p. 28, §6 · p. 28

7 More Quantitative Results

• Evaluation treats 3DFM-predicted target-view depth maps as pseudo ground truth and uses the same depth and point-cloud metrics for all methods.Results cover in-domain test splits of the training datasets and out-of-domain datasets never seen during training, assessing consistency with 3DFM geometric priors.Source: §7 · p. 32
• Across both source-to-target settings, Z3D predictions closely match target-view depths produced by the foundation models.The reported evaluations cover point-cloud metrics in Tables 3 and 5 and depth metrics in Tables 4 and 6.Source: Table 3 · p. 32, Table 4 · p. 32, §7 · p. 32

Limitations

• With limited source–target overlap, Z3D-VGGT and Z3D-WM preserve overall scene structure but miss fine details and produce inconsistent projected point clouds.Figure 9 illustrates this failure for novel views.Source: §8 · p. 34, Figure 9 · p. 34
• Z3D uses the first source image as its reference camera, and prior work [50] reports that reference-view choice significantly affects 3DFM performance.Z3D therefore inherits this 3DFM limitation.Source: §8 · p. 34