ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image

• Real-world 360-degree NVS must produce realistic, 3D-consistent views while capturing diverse explanations of unobservable regions, especially in complex backgrounds.Source: §1 · p. 1
• ZeroNVS trains a conditional diffusion model on diverse scene datasets using a new camera parametrization and normalization scheme.The mixed data include CO3D, RealEstate10K, and ACID, whose camera settings and 3D supervision differ substantially.Source: §1 · p. 1
• SDS anchoring samples diverse DDIM anchor views and uses them as randomized conditions, preserving 3D consistency while improving background diversity.The anchors serve as conditions rather than direct RGB supervision, avoiding SDS's tendency toward monotone backgrounds.Source: §1 · pp. 1, 3
• ZeroNVS achieves state-of-the-art LPIPS on DTU in the zero-shot setting, outperforming methods fine-tuned on DTU, and performs best on the proposed Mip-NeRF 360 benchmark.The Mip-NeRF 360 dataset extends evaluation to more challenging scene-level viewpoint changes than DTU.Source: §1 · p. 3

• Diffusion-based 3D methods commonly distill NeRFs from conditional models, while earlier GAN-based approaches are mainly restricted to object categories or synthetic data.Source: §2 · p. 3
• PixelNeRF, DietNeRF, and related methods infer NeRFs from sparse images but do not produce crisp natural renderings from a single image.Source: §2 · p. 3
• Recent diffusion approaches improve single-image NVS, but many support limited camera motion or specific object categories; ZeroNVS generates 360-degree motion across diverse scene types.The paper attributes this capability to new camera conditioning representations and SDS anchoring.Source: §2 · p. 3

3. Approach

• ZeroNVS first trains a diffusion model for novel view synthesis from a single real image, then uses it for 3D SDS distillation, targeting scene-specific camera/scale and inference challenges.Source: §3 · p. 3

3.1. Representing Objects for View Synthesis

• 3DoF conditioning uses elevation, azimuth, and radius, but cannot represent roll or arbitrary six-degree-of-freedom camera poses in real-world scenes.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2. Representing Scenes for View Synthesis

• 6DoF+1 represents relative six-degree-of-freedom pose together with field of view and is invariant to rigid scene transformations, addressing arbitrary pose conventions in real-world datasets.Source: §3.2 · p. 4

3. Approach

• ZeroNVS sets a new DTU LPIPS state of the art without DTU fine-tuning, while Table 2 reports better zero-shot performance than Zero-1-to-3 on the proposed Mip-NeRF 360 benchmark.Source: Table 1 · p. 6, Table 2 · p. 6

3.3. Addressing Scale Ambiguity with a New Normalization Scheme

• Viewer-centric normalization uses the input-view depth quantile to scale camera translations, avoiding scene-wide aggregation and making scale conditioning independent of additional cameras.Source: §3.3 · pp. 5–6

3.4. Improving Diversity with SDS Anchoring

• SDS anchoring samples azimuth-spaced novel views with DDIM, then guides distillation using the nearest input or sampled view to improve background diversity.Source: Figure 4 · p. 5, §3.4 · p. 6

4.1. Setup

• Models are trained on a uniformly sampled mixture of CO3D, ACID, and RealEstate10K, then evaluated on held-out 2D synthesis and zero-shot 3D synthesis using DTU and Mip-NeRF 360.Source: §4.1 · p. 6

4.2. Main Results

• The evaluation uses PSNR, SSIM, and LPIPS, with LPIPS weighted more heavily because PSNR and SSIM do not correlate well with performance in this setting.Source: §4.2 · p. 7, Figure 7 · p. 7
• ZeroNVS achieves state-of-the-art LPIPS on DTU despite being fully zero-shot, outperforming methods trained directly on DTU.Source: §4.2 · p. 7
• ZeroNVS is best on LPIPS for the proposed zero-shot Mip-NeRF 360 benchmark and exceeds Zero-1-to-3 and zero-shot PixelNeRF on all DTU metrics.The reported ZeroNVS and Zero-1-to-3 results use NeRFs predicted from SDS distillation unless otherwise noted.Source: §4.2 · p. 7
• Qualitative results show that SDS anchoring produces more diverse backgrounds than standard SDS and varies noticeably with the random seed.Source: Figure 9 · p. 8
• SDS anchoring receives user preferences of 78% for realism, 82% for creativity, and 80% overall across 21 users.The study compares scenes generated with and without SDS anchoring on Mip-NeRF 360.Source: §4.2 · p. 8
• Removing any of CO3D, ACID, or RealEstate10K reduces performance, while progressively refined camera conditioning improves held-out diffusion results.The conditioning ablation models were trained for fewer steps than the main model, causing slightly worse performance relative to Table 1.Source: §4.2 · p. 8

• ZeroNVS enables 3D-consistent novel view synthesis from a single image for generic scenes and achieves state-of-the-art performance on existing benchmarks.The paper also proposes Mip-NeRF 360 as a more challenging single-image novel view synthesis benchmark.Source: §5 · p. 8

A.1. Model

• The main model is trained for 60,000 steps with batch size 1536, although performance saturates after about 20,000 steps.This saturation pattern holds across the evaluated camera-conditioning models.Source: §A.1 · p. 11
• Five camera-conditioning variants are evaluated, including normalized, aggregated, and viewer-based 6DoF representations.The variants compare alternative ways to encode camera information for scene synthesis.Source: §A.1 · p. 11
• The M6DoF+1 representation embeds field of view as [f, sin(f), cos(f)] and concatenates it with a 16-dimensional relative pose matrix.This produces a 19-dimensional conditioning vector.Source: §A.1 · p. 11
• Depth maps are completed with DPT-SwinV2-256 after disparity alignment to sparse COLMAP or ORB-SLAM depth.Invalid regions are infilled only after optimal scale-and-shift alignment, and maps are downsampled 4× for efficient quantile evaluation.Source: §A.1 · p. 11
• At inference, the unknown Q20(¯D) value is chosen heuristically, with values between .7 and 1. working for most images.Distillation can fail when this value is incompatible with the desired SDS camera radius because cameras may intersect visible content.Source: §A.1 · p. 11

A.2. Dataloader

• The dataloader streams sharded scene archives in parallel and samples random view pairs using a tunable rate parameter.Lower rates favor fully random sampling, while higher rates bias sampling toward denser scene coverage under bandwidth constraints.Source: §A.2 · p. 11
• Individual dataset streams are combined and randomly sampled to form the mixture dataset.The implementation is built on WebDataset and supports the paper’s mixed multiview training setup.Source: §A.2 · p. 11

B.1. SDS Anchoring

• SDS anchoring samples two novel-view anchors at 120° and 240° azimuth relative to the input camera.The anchors are used to increase the diversity of synthesized scenes.Source: §B.1 · p. 11
• Anchoring randomly chooses the input view or nearest anchor for guidance, gating anchor gradients to rendered depths above 1.0.This design targets richer backgrounds while mitigating artifacts from 3D-inconsistent foreground content; inconsistent anchors can still produce unusual generations.Source: §B.1 · p. 11

B.2. Hyperparameters

• Ending with maximum noise levels as low as .025 benefits results, while anisotropic scheduling helps optimize challenging 180-degree views in Mip-NeRF 360.Progressive azimuth and elevation sampling improves training stability, with resolution staged from 128x128 to 256 × 256.Source: §B.2 · p. 11

C. Experimental setups

• ZeroNVS and Zero-1-to-3 are both trained for 60,000 steps for the main DTU and Mip-NeRF 360 results.ZeroNVS appears to saturate earlier than Zero-1-to-3, which was trained for about 100,000 steps.Source: C. Experimental setups · p. 12
• The retrained PixelNeRF baseline is trained on CO3D, ACID, and RealEstate10K for about 560,000 steps.This baseline uses the same mixture dataset used for the paper’s scene-focused training.Source: C. Experimental setups · p. 12

C.1. Main results

• All single-image NeRF distillation evaluations assume camera elevation, field of view, and content scale are given.These parameters are fixed across DTU scenes but vary across Mip-NeRF 360 scenes.Source: §C.1 · p. 12
• Mip-NeRF 360 input views are manually selected for approximate 3DoF rotational compatibility and broad visibility of scene content.The selection supports comparison with Zero-1-to-3 and compatibility with Threestudio’s SDS sampling scheme.Source: §C.1 · p. 12
• The field of view comes from COLMAP, while camera elevation is computed from the camera forward axis after PCA standardization.Elevation is measured relative to the world’s XY-plane.Source: §C.1 · p. 12
• Zero-shot methods require manual world-scale grid search because scene scales are unknown, and they generally underperform DTU-finetuned methods on SSIM and PSNR.Even after scale correction, five to nine samples within [.3, 1.5] generally suffice to find an appropriate scene scale.Source: §C.1 · p. 12

C.2. User study

• 21 respondents evaluated seven Mip-NeRF 360 scenes reconstructed with and without SDS anchoring.The videos were shown in random order, and respondents were unaware which version used SDS anchoring.Source: §C.2 · p. 12
• Respondents generally preferred scenes produced with SDS anchoring.Source: §C.2 · p. 12
• The preference for SDS anchoring was especially strong for perceived creativity.Source: §C.2 · p. 12

C.3. Ablation studies

• The ablations examine dataset selection and camera representations.Source: §C.3 · p. 12
• Novel-view metrics are computed on held-out scenes using randomly sampled input and target view pairs.Source: §C.3 · p. 12
• Ablation diffusion models are trained for 25,000 steps instead of 60,000 for computational reasons.Source: §C.3 · p. 12