Source-linked AI summary
SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image
Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang
TL;DR
NeRF typically depends on dense views, but training from a single image in complex scenes remains difficult because of missing geometry and appearance constraints. SinNeRF addresses this with semi-supervised geometry and semantic pseudo-label regularization, producing photo-realistic novel views and outperforming state-of-the-art NeRF baselines across the reported single-image cases. Similar to most NeRF approaches, its training efficiency remains a limitation.
Problem
NeRF requires dense views and camera poses, while single-view reconstruction of complex scenes is difficult because insufficient views lead to incorrect geometry and blurry appearance.
Method
SinNeRF trains a neural radiance field from one reference view using semi-supervised geometry and semantic pseudo labels, with progressive camera-pose sampling.
Results
SinNeRF produces photo-realistic novel-view synthesis results and significantly outperforms state-of-the-art NeRF baselines in all single-image cases.
Takeaways & Limitations
Single-view training from scratch can produce photo-realistic novel views of complex scenes without pre-training on multi-view datasets.
Takeaways & Limitations
SinNeRF has a training-efficiency issue, which the authors identify as a future direction.
Abstract
from arXiv · showhide
Despite the rapid development of Neural Radiance Field (NeRF), the necessity of dense covers largely prohibits its wider applications. While several recent works have attempted to address this issue, they either operate with sparse views (yet still, a few of them) or on simple objects/scenes. In this work, we consider a more ambitious task: training neural radiance field, over realistically complex visual scenes, by "looking only once", i.e., using only a single view. To attain this goal, we present a Single View NeRF (SinNeRF) framework consisting of thoughtfully designed semantic and geometry regularizations. Specifically, SinNeRF constructs a semi-supervised learning process, where we introduce and propagate geometry pseudo labels and semantic pseudo labels to guide the progressive training process. Extensive experiments are conducted on complex scene benchmarks, including NeRF synthetic dataset, Local Light Field Fusion dataset, and DTU dataset. We show that even without pre-training on multi-view datasets, SinNeRF can yield photo-realistic novel-view synthesis results. Under the single image setting, SinNeRF significantly outperforms the current state-of-the-art NeRF baselines in all cases. Project page: https://vita-group.github.io/SinNeRF/
1 Introduction
SinNeRF addresses the challenge of training NeRF from a single view in complex scenes by combining geometry and semantic pseudo-label regularization. Its semi-supervised strategy enables photo-realistic novel-view synthesis and outperforms state-of-the-art NeRF baselines under the single-image setting.
- Motivation: Dense views and camera poses are normally required for NeRF, while insufficient views cause incorrect geometry and blurry appearance.Collecting dense coverage can be challenging or infeasible in applications such as AR/VR and autonomous driving.
- Motivation: SinNeRF trains a neural radiance field from only one view, targeting realistically complex scenes without pre-training or auxiliary multi-plane representations.This setting extends sparse-view reconstruction to the single-view extreme, where prior methods were limited on complex scenes or relied on other techniques.
- Approach: SinNeRF introduces a semi-supervised framework that uses reference-view supervision and propagated geometry and semantic pseudo labels to guide training.Geometry labels are derived through image warping, while semantic labels provide complementary guidance for unseen views.
- Approach: Geometry pseudo labels enforce multi-view consistency, while semantic regularization preserves local texture quality and global structure.The two forms of guidance address complementary aspects of the learned radiance field.
- Results: SinNeRF yields photo-realistic novel-view synthesis results and significantly outperforms state-of-the-art NeRF baselines in all single-image cases.The reported results are obtained without bells and whistles.
2 Related Works
Related work spans NeRF-based representations, single-view 3D reconstruction, and single-image training. SinNeRF differs by training from scratch on complex 3D environments while using a single view.
- NeRF-based view synthesis: NeRF research has improved view-synthesis quality, speed, artistic effects, and generalization through diverse scene representations and network designs.Examples include anti-aliasing with conical frustums, unbounded-scene parameterization, and accelerated architectures.
- NeRF-based view synthesis: Single-view NeRF methods often require at least two or three views, low-resolution inputs, or large-scale pre-training, limiting texture detail or generalization.PixelNeRF renders from one input but performs well on simple objects while its complex-scene results remain unsatisfactory or unknown.
- Single-view 3D reconstruction: Single-view 3D reconstruction methods use image cues or learned priors, but limited 3D-annotation datasets and simple object benchmarks constrain the field.Several methods specialize in particular object categories such as humans, rather than complex scenes.
- Single-view view synthesis: SinNeRF is distinct from prior view-synthesis work because it trains a neural scene representation from scratch and evaluates complex 3D environments.The paper reports photo-realistic rendered results without relying on pre-trained feature extractors or multi-plane images.
- Single-image training: Single-image training methods such as SinGAN and InGAN generate visually pleasing content from one reference image, but their diversity is limited and they often copy image patches.SinNeRF extends single-image training toward adversarially trained radiance fields for view synthesis.
3 Method
SinNeRF trains a neural radiance field from one image by constraining unseen views with geometry and semantic pseudo labels, then progressively expanding sampled regions and viewing angles.
- Overview: Single-view NeRF training treats the reference image and available depth as labeled data while unseen views are unlabeled.The framework introduces supervision signals for unseen views to prevent collapse toward the provided view.
- Preliminary: NeRF maps spatial coordinates and viewing directions to color and density, then uses volumetric rendering to synthesize pixel colors.The rendered colors are ordinarily optimized against ground-truth colors with a mean squared error loss.
- Geometry Pseudo Label: Image warping reprojects reference-view depth into unseen views to create geometry pseudo labels, using camera intrinsics, extrinsics, and nearest-depth selection for collisions.The resulting pseudo labels are imperfect because depth misalignment and occlusions create uncertain regions.
- Geometry Pseudo Label: Geometry regularization combines bidirectional depth consistency with an edge-aware smoothness loss to constrain the reconstructed field.The smoothness term is computed on a downscaled resolution, while unseen views are also reprojected back to the reference view.
- Semantic Pseudo Label: Semantic regularization combines adversarial local texture guidance with global structure guidance from DINO-ViT [CLS] tokens.The discriminator compares augmented rendered patches with randomly cropped reference patches, while the ViT loss enforces representation consistency between reference and unseen views.
- Progressive Training Strategy: Progressive training begins with strided ray patches and nearby camera poses, then reduces stride and enlarges viewing angles as optimization proceeds.The schedule covers broader regions with limited rays and gradually focuses on local regions while stabilizing training.
4 Experiment
SinNeRF is evaluated on complex-scene benchmarks against state-of-the-art baselines, with quantitative, visual, and component-ablation studies. It achieves strong novel-view synthesis quality and shows that geometry and semantic regularizations contribute complementary benefits.
- Experimental Setup: Experiments cover NeRF synthetic, LLFF, and DTU datasets, comparing SinNeRF with DietNeRF, PixelNeRF, and DS-NeRF using PSNR, SSIM, and LPIPS.The evaluation includes complex objects, forward-facing scenes, and tabletop objects.
- Evaluation Protocol: 60 test images are generated for NeRF synthetic scenes by uniformly rotating the camera around the y-axis within [−30°, 30°].A single randomly selected view serves as the reference view, while surrounding views are unseen during training.
- NeRF Synthetic and LLFF Results: SinNeRF achieves the best results in both pixel-wise error and perceptual quality on the NeRF synthetic evaluation.The quantitative evaluation uses the Lego and Hotdog scenes.
- NeRF Synthetic and LLFF Results: SinNeRF preserves the strongest geometry and perceptual quality on NeRF synthetic scenes, avoiding baseline-specific geometry errors, ghosting, and blur.DS-NeRF misrepresents the Lego geometry, PixelNeRF produces ghost hotdogs, and DietNeRF has off-center and blurry novel views.
- NeRF Synthetic and LLFF Results: On LLFF and DTU, SinNeRF produces the most visually pleasing results and best quantitative performance, while baselines show blur, artifacts, or incorrect geometry.LLFF includes Flower and Room; DTU reports average values across scenes.
- Ablation Study: Ablations show that removing local texture guidance causes blur, removing the global structure prior causes wrong structure, and removing geometry pseudo labels causes wrong geometry.The ablation studies are conducted on the Lego scene and evaluate pseudo-label variants and global-structure-prior choices.
5 Conclusions
SinNeRF trains a neural radiance field from one view of a complex scene using synthesized geometry and semantic pseudo labels. Experiments across three complex-scene datasets report outperforming current state-of-the-art NeRF frameworks, while training efficiency remains a limitation.
- SinNeRF trains a neural radiance field on a single view from a complex scene.
- Geometry and semantic pseudo labels stabilize SinNeRF’s semi-supervised training process.
- SinNeRF outperforms current state-of-the-art NeRF frameworks on the NeRF synthetic, LLFF, and DTU datasets.
- Training efficiency remains a limitation of SinNeRF and a direction for future work.