Source-linked AI summary
NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors
Congyue Deng, Chiyu "Max'' Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov
TL;DR
Single-view 2D-to-3D reconstruction is ill-posed, and existing approaches may require calibrated multiview supervision or distribution-matched training cases. NeRDi instead optimizes a NeRF with diffusion-based image priors, two-section language guidance, and depth-based geometric regularization, achieving higher-quality novel views than supervised baselines on DTU MVS and demonstrating zero-shot synthesis for in-the-wild images.
Problem
Single-view NeRF synthesis is ill-posed, while existing methods may require matched multiview images with calibrated camera poses or test cases following the training distribution.
Method
NeRDi formulates single-view reconstruction as conditioned 3D generation, optimizing NeRF parameters with diffusion losses on arbitrary views, two-section language guidance, and depth-based geometric regularization.
Results
NeRDi achieves higher-quality zero-shot novel-view synthesis than supervised baselines on DTU MVS and generates high-quality novel views for in-the-wild images.
Takeaways & Limitations
Well-designed language guidance and geometric constraints allow general image priors to support specific image-to-3D reconstruction with generalizable and adaptable frameworks.
Takeaways & Limitations
The method is affected by biases in multiple large pretrained image models, is less robust to highly deformable instances, and is more suitable for object-centric images than large complex scenes.
Abstract
from arXiv · showhide
2D-to-3D reconstruction is an ill-posed problem, yet humans are good at solving this problem due to their prior knowledge of the 3D world developed over years. Driven by this observation, we propose NeRDi, a single-view NeRF synthesis framework with general image priors from 2D diffusion models. Formulating single-view reconstruction as an image-conditioned 3D generation problem, we optimize the NeRF representations by minimizing a diffusion loss on its arbitrary view renderings with a pretrained image diffusion model under the input-view constraint. We leverage off-the-shelf vision-language models and introduce a two-section language guidance as conditioning inputs to the diffusion model. This is essentially helpful for improving multiview content coherence as it narrows down the general image prior conditioned on the semantic and visual features of the single-view input image. Additionally, we introduce a geometric loss based on estimated depth maps to regularize the underlying 3D geometry of the NeRF. Experimental results on the DTU MVS dataset show that our method can synthesize novel views with higher quality even compared to existing methods trained on this dataset. We also demonstrate our generalizability in zero-shot NeRF synthesis for in-the-wild images.
1. Introduction
NeRDi addresses the ill-posedness and supervision requirements of single-view NeRF synthesis by treating reconstruction as conditioned 3D generation with diffusion-based image priors. It combines two-section language guidance, geometric regularization, and zero-shot evaluation to improve novel-view synthesis.
- Single-view NeRF synthesis is ill-posed because one 2D image does not uniquely correspond to a 3D scene.
- Existing prediction-based approaches require matched multiview images with calibrated camera poses, while unsupervised methods still assume test cases follow the training distribution.
- Two-section semantic guidance combines image captions with visual textual features to narrow the diffusion prior and enforce semantically and visually coherent novel views.
- NeRDi formulates single-view reconstruction as conditioned 3D generation and optimizes a NeRF using 2D priors from large-dataset diffusion models without 3D supervision.
- A geometric regularization term uses estimated depth maps with 3D uncertainties, and experiments report higher-quality zero-shot novel views than supervised baselines on DTU MVS and on in-the-wild images.
2. Related Work
Related work develops NeRF novel-view synthesis with less informative inputs through geometric, appearance, semantic, and adversarial regularization, while other methods learn scene priors from 3D datasets. Diffusion models provide a complementary general image-generation prior with strong text-image semantic correlation.
- Novel view synthesis with NeRF: NeRF represents static 3D scenes from posed images and renders realistic novel views using differentiable rendering and neural network scene parametrizations.
- Novel view synthesis with NeRF: Research reduces NeRF input requirements from posed images to unposed or sparse views, increasing optimization difficulty and motivating varied regularization losses.
- Novel view synthesis with NeRF: Regularization methods target geometry and appearance, depth maps, cross-view semantic consistency, or patch-based adversarial objectives.
- Novel view synthesis with NeRF: Another line of work pre-trains NeRF-based novel-view predictors on large datasets of 3D scenes containing dense views.
- Diffusion-based generative models: Language-guided diffusion models have achieved strong performance in image generation, completion, and editing, with strong semantic correlation to text prompts.
3. Method
NeRDi formulates single-view NeRF synthesis as conditioned 3D generation, optimizing arbitrary-view renderings against 2D diffusion priors while constraining the input view. Two-section language guidance narrows the prior toward the input’s semantic and visual characteristics, while depth correlation regularizes geometry.
- Conditioned 3D generation: NeRDi optimizes NeRF parameters as a conditioned 3D generation problem rather than predicting them directly from the input image.The optimization seeks a 3D scene distribution whose rendering at the input pose matches the observed image.
- NeRF representation: The method uses a Lambertian NeRF without view-direction inputs to enforce stronger multiview consistency.This design choice removes view-direction dependence from the NeRF representation.
- Diffusion-based optimization: Arbitrarily sampled NeRF views are pushed toward a pretrained latent diffusion image prior using stochastic gradients backpropagated through rendered images.The diffusion model parameters remain fixed while gradients update the NeRF parameters.
- Semantics-conditioned image priors: Joint semantic and visual guidance concatenates caption and textual-inversion embeddings to narrow the generic image prior around the input image.Captioning supplies semantics, while textual inversion captures visual cues; their joint feature balances both properties.
- Geometric regularization: A geometric regularizer maximizes Pearson correlation between estimated input-view depth and NeRF-rendered depth instead of directly back-projecting ambiguous depth values.Estimated depth has scale, shift, camera-intrinsic ambiguities, and prediction error, so correlation is used as the regularization signal.
4. Experiments
Experiments evaluate NeRDi on synthetic DTU scenes, in-the-wild objects, and complex images, including ablations of semantic guidance and geometric regularization. Results show improved perceptual quality and qualitative coherence, while depth regularization supports geometry and multiview consistency.
- Experimental setup: The evaluation covers quantitative DTU comparisons, synthetic-scene visualizations, in-the-wild objects, complex backgrounds, and component ablations.The DTU setup uses 15 test scenes and compares against PixelNeRF and DietPixelNeRF, while additional experiments use Google Scanned Objects and COCO images.
- Synthetic scenes: NeRDi's DTU PSNR and SSIM are slightly below pixelNeRF and on par with DietPixelNeRF, while LPIPS shows a significant improvement over the baselines.The paper argues that PSNR and SSIM are less indicative under single-view uncertainty because reasonable predictions can differ from ground truth, whereas LPIPS better reflects perceptual quality.
- Synthetic scenes: NeRDi removes noise and blur from pixelNeRF-initialized renderings, producing cleaner DTU novel views with realistic geometries and appearances.The method is applied directly to test scenes without additional fine-tuning on the DTU training set.
- Images in the wild: On in-the-wild objects, NeRDi balances semantic meaning and visual cues, whereas DietNeRF misses global semantics and SS3D misses fine geometric details.The comparison includes backpacks and hats from the Google Scanned Objects Dataset, where SS3D also produces incorrect hat structures.
- Semantic guidance: Combining captions with textual inversion preserves both object semantics and visual details, while removing either component causes semantic failure, blur, noise, or weaker appearance coherence.Textual inversion also enables inferred details such as a can's pull tab and coherent appearances in regions unseen from the input view.
- Geometric regularization: Removing geometric regularization can retain realistic appearances but yields erroneous 3D geometry and fails to enforce multiview consistency; depth alone cannot resolve monocular 3D ambiguities.The full model combines diffusion-based appearance synthesis with depth-based geometric regularization.
5. Conclusions
NeRDi enables zero-shot single-view NeRF synthesis without 3D supervision by combining 2D diffusion priors, semantic guidance, and geometric regularization. The method demonstrates improved novel-view synthesis while exposing limitations from pretrained-model biases, deformable objects, and complex scenes.
- 5. Conclusions: NeRDi performs zero-shot single-view NeRF synthesis for in-the-wild images without 3D supervision by applying 2D diffusion priors to NeRF optimization.The framework uses general image priors from large-scale diffusion models.
- 5. Conclusions: Ablations show that removing semantic or geometric guidance reduces synthesis quality, producing meaningless objects, blur, noise, erroneous geometry, or inconsistent novel views.The semantic-guidance ablation reports failures without captions and degraded detail without textual inversion; the geometric ablation reports plausible appearances but incorrect geometry without depth correlation.
- 5. Conclusions: The two-section semantic guidance combines image captions and textual-inversion embeddings to provide semantic and visual conditioning for novel-view synthesis.The guidance combines complementary semantic and visual features in the text embedding space.
- 5. Conclusions: Geometric regularization uses estimated depth maps with 3D uncertainties to constrain the underlying NeRF geometry.This component is evaluated through ablations of the geometric regularization.
- 5. Conclusions: The method is affected by biases in pretrained image models and is less robust to highly deformable instances whose physical states and dynamics are not described by language guidance.Independent views of deformable instances can remain visually plausible while representing different states.
- 5. Conclusions: The approach is more suitable for object-centric images because complex scenes violate its assumption of view-invariant semantics, while a single-world embedding limits representation of subtle content.View changes, occlusions, and limited embedding expressiveness constrain large, complex scenes.
A. Additional Results
Additional in-the-wild comparisons show that NeRDi can better capture complex object geometry and unseen-region appearance than DietNeRF, while diffusion priors influence ambiguous occluded content.
- A. Additional Results: NeRDi captures complex object geometry more effectively than DietNeRF, especially for chairs and baskets in the in-the-wild comparisons.DietNeRF maintains appearance consistency but fails to capture overall geometry for objects with complex structures.
- A. Additional Results: For unseen back regions, NeRDi produces skirt textures closer to the input image than DietNeRF.The comparison specifically reports improved unseen-back appearance for the skirt examples.
- A. Additional Results: NeRDi fills occluded basket regions with varied fruits, flowers, or vegetables rather than duplicating the input view, reflecting diffusion-model priors.The synthesized content in unseen regions is influenced by the 2D distribution learned by the image diffusion model.
- A. Additional Results: A pumpkin synthesized by NeRDi inherits a Jack-o’-lantern face on its unseen back because the diffusion model frequently generates that appearance for “a pumpkin.”The passage reports that half of the reference generations contain Jack-o’-lanterns.