PaperlayerView exact revision on arXiv

Source-linked AI summary

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

Ajay Jain, Matthew Tancik, Pieter Abbeel

arXiv:2104.00677v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Few-shot NeRF reconstruction is underconstrained because per-scene models lack prior knowledge and can overfit sparse observed poses. DietNeRF adds CLIP-based semantic consistency supervision from arbitrary poses, improving perceptual few-shot synthesis, including single-view fine-tuning and unseen-region completion. The approach remains subject to a detail-quality trade-off under regularization.

  • Problem

    Few-shot NeRF lacks prior knowledge and can find degenerate solutions when sparse pose supervision underdetermines scene reconstruction.

  • Method

    DietNeRF combines known-pose pixel reconstruction with CLIP feature similarity between ground-truth and rendered images from random poses.

  • Results

    DietNeRF improves perceptual few-shot view synthesis, reconstructs scenes from 8 views, completes occluded regions, and fine-tuned DietPixelNeRF outperforms baselines on perceptual metrics.

  • Takeaways & Limitations

    Pretrained single-view image representations can transfer useful semantic prior knowledge to underconstrained 3D novel-view synthesis.

  • Takeaways & Limitations

    Regularization can recover plausible geometry but sacrifices high-frequency fine details.

Abstract

from arXiv · show

We present DietNeRF, a 3D neural scene representation estimated from a few images. Neural Radiance Fields (NeRF) learn a continuous volumetric representation of a scene through multi-view consistency, and can be rendered from novel viewpoints by ray casting. While NeRF has an impressive ability to reconstruct geometry and fine details given many images, up to 100 for challenging 360° scenes, it often finds a degenerate solution to its image reconstruction objective when only a few input views are available. To improve few-shot quality, we propose DietNeRF. We introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses. DietNeRF is trained on individual scenes to (1) correctly render given input views from the same pose, and (2) match high-level semantic attributes across different, random poses. Our semantic loss allows us to supervise DietNeRF from arbitrary poses. We extract these semantics using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse single-view, 2D photographs mined from the web with natural language supervision. In experiments, DietNeRF improves the perceptual quality of few-shot view synthesis when learned from scratch, can render novel views with as few as one observed image when pre-trained on a multi-view dataset, and produces plausible completions of completely unobserved regions.

1. Introduction

Few-shot novel view synthesis is underconstrained because per-scene NeRF lacks prior knowledge and is supervised only at observed poses. DietNeRF adds semantic supervision from arbitrary poses using pretrained image encoders, improving perceptual reconstruction and completion of unseen regions.

  • Novel view synthesis rerenders sparsely observed scenes from arbitrary viewpoints while requiring both 3D reconstruction and high-frequency texture synthesis.
  • With few views, NeRF can overfit observed poses, producing artifacts and poor extrapolations because it lacks prior knowledge about scenes and objects.Regularization reduces artifacts but sacrifices fine-grained detail.
  • DietNeRF combines pixel-space reconstruction at known poses with a CLIP-based semantic consistency loss between ground-truth and arbitrarily posed renderings.The semantic loss operates on high-level feature representations rather than pixels.
  • DietNeRF learns realistic reconstructions from as few as 8 views and can produce reasonable views of completely occluded regions.
  • Fine-tuning pixelNeRF enables novel-view synthesis from a single observation, while DietPixelNeRF improves perceptual quality over baselines.The reported improvements concern perceptual view quality rather than only pixel accuracy.

2. Background on Neural Radiance Fields

NeRF represents a scene’s five-dimensional light field with a continuous neural network that predicts color and density from position and viewing direction. Volume rendering integrates these predictions along camera rays to produce images.

  • A plenoptic function describes light radiating from every point in every direction, but its high dimensionality makes explicit high-resolution estimation impractical.
  • NeRF parameterizes this function with an MLP that maps spatial position and viewing direction to RGB color and differential volume density.Density depends only on position, while color also depends on direction to model view-dependent effects.
  • For each camera ray, NeRF uses volume rendering and transmittance-weighted samples along the ray to estimate the observed pixel color.
  • NeRF uses hierarchical sampling with separate coarse and fine MLPs, training each scene from tens to hundreds of photos from varied perspectives.

3. NeRF Struggles at Few-Shot View Synthesis

Few-shot NeRF fails because sparse pose supervision leaves high-capacity scene representations underdetermined, enabling degenerate solutions and poor unseen-region extrapolation. Regularization improves geometry but removes fine detail.

  • NeRF fails at few-shot novel view synthesis because per-scene training lacks prior knowledge from similar scenes.
  • NeRF overfits to training views: NeRF trains by rendering each observed image pose and minimizing pixel-wise MSE over sampled training rays.
  • NeRF overfits to training views: With many views, MSE densely supervises the volume and supports accurate textures, occupancy, and interpolation to new views.
  • NeRF overfits to training views: With few views, NeRF’s high-frequency capacity permits near-field and view-dependent degeneracies that fit training images but fail at novel poses.
  • Regularization fixes geometry, but hurts fine-detail: Regularization can recover plausible geometry by lowering model frequency and simplifying optimization, but high-frequency fine details are lost.
  • No prior knowledge, no generalization to unseen views: Because unseen regions receive no MSE supervision, per-scene NeRF relies on its inductive bias and generalizes poorly to occluded object parts.

4. Semantically Consistent Radiance Fields

DietNeRF guides few-shot NeRF optimization with semantic consistency between observed images and renders from arbitrary poses, using pre-trained visual representations.

  • 4. Semantically Consistent Radiance Fields: DietNeRF augments NeRF with a semantic consistency loss that supervises renders at arbitrary camera poses.The loss compares high-level semantic features between observed and rendered views.
  • 4.1. Semantic consistency loss: The method compares representations of images across viewpoints because pixel-wise LMSE is useful only when rendered and observed poses are aligned.Semantic cues allow views of the same object to be compared despite viewpoint changes.
  • 4.2. Visual representations: CLIP and ImageNet models provide candidate supervision representations, while Vision Transformers can learn view-consistent features from diverse single-view image data.CLIP representations remain highly similar across views, including diametrically opposing cameras.
  • 4.3. Pose sampling distribution: Training samples an observed image and a rendered image from a random pose, using a uniform upper-hemisphere distribution for bounded 360° scenes.For unbounded or difficult-to-define pose spaces, the method interpolates between three randomly sampled known poses.
  • 4.4. Efficient semantic rendering: DietNeRF renders semantic-consistency images at low resolution, using only 15-20% of the rays required for a full-resolution training image.Strided-grid sampling keeps objects mostly visible while reducing training cost.
  • 4.5. Training schedule: LSC converges faster than LMSE, so DietNeRF applies it every k iterations and then fine-tunes with LMSE alone for 20-70k iterations.The authors hypothesize that LSC helps recover plausible geometry early but is less useful for fine-grained details.

5. Experiments

Experiments evaluate DietNeRF on synthetic and real sparse-view scenes, including training from scratch, single-view fine-tuning, and reconstruction of unobserved regions. Across these settings, semantic consistency improves perceptual quality and supports more plausible novel views.

  • 5. Experiments: CLIP representations show high cosine similarity between views of the same object, even across large camera changes, and low similarity across scenes.This experiment motivates using CLIP features as a semantic signal for multi-view synthesis.
  • 5.1. Realistic Synthetic scenes from scratch: +8.5 PSNR, -0.2 LPIPS, and -156 FID over NeRF are achieved by combining DietNeRF’s semantic consistency loss with LMSE.Semantic consistency gives the best performance, while scene-specific regularization alone improves PSNR by +5.1 over full NeRF.
  • 5.2. Single-view synthesis by fine-tuning: DietPixelNeRF improves single-view perceptual quality on DTU, outperforming baselines on LPIPS, FID, and KID.The method fine-tunes pixelNeRF using both pixel-space MSE and semantic consistency loss.
  • 5.2. Single-view synthesis by fine-tuning: DietPixelNeRF produces sharper novel views than pixelNeRF, whereas pixel-space MSE fine-tuning mainly improves the observed view.The comparison emphasizes perceptual quality rather than only local pixel-aligned metrics.
  • 5.3. Reconstructing unobserved regions: With observations from only one side of a Lego scene, DietNeRF learns more accurate colors in missing regions than NeRF and Simplified NeRF.The experiment uses 14 nearby right-side views; FID and KID are excluded because one scene provides too few samples for accurate estimation.

6. Ablations

The ablations examine semantic encoder choice and LMSE fine-tuning duration. CLIP improves PSNR over ImageNet supervision, while extended LMSE fine-tuning eventually reduces performance through likely overfitting.

  • Choosing an image encoder: Large ViT L models do not improve results over the base ViT B on the Lego scene with 8 views.The comparison uses quality metrics on the Realistic Synthetic Lego scene.
  • Choosing an image encoder: CLIP improves PSNR by +1.8 over an ImageNet model with the same ViT architecture, indicating benefits from data diversity and language supervision.Both encoder types nevertheless produce representations that transfer usefully to view synthesis.
  • Varying LMSE fine-tuning duration: Fine-tuning DietNeRF with LMSE improves quality by better reconstructing fine details, but optimization beyond 50k iterations reduces performance.The authors suggest that longer optimization may overfit the 8 input views.

7. Related work

DietNeRF relates to few-shot radiance fields, novel-view synthesis, and semantic representation learning. Its approach uses semantic priors to complement rendering-based supervision and address sparse or unobserved views.

  • Few-shot radiance fields: Few-shot radiance-field methods either condition decoders on encodings of a few observations or adapt radiance-field weights to new scenes.Related approaches include discriminator-supervised patch rendering, latent-conditioned fields, and meta-learning.
  • Novel view synthesis, image-based rendering: DietNeRF’s renderings of an occluded Lego region fill in details mostly consistent with the observed half, unlike NeRF’s blocked or poorly generalized reconstructions.The figure uses 14 images showing mostly the right side of the scene during training.
  • Novel view synthesis, image-based rendering: DietNeRF differs from Neural Volumes by using image semantics as an auxiliary prior, whereas Neural Volumes penalizes opacity using geometric intuitions.Other related systems use category-specific multi-view training, semantic segmentation, or geometric rendering strategies.
  • Semantic representation learning: Contrastive representation-learning methods, including CLIP, learn visual representations by matching related pairs such as captions and images.This places DietNeRF’s semantic supervision within a broader representation-learning literature.

8. Conclusions

The paper concludes that single-view 2D representations can transfer to underconstrained volumetric novel-view synthesis. This transferable prior knowledge improves perceptual quality in the NeRF few-shot setting and may extend to broader 3D applications.

  • 8. Conclusions: Single-view 2D representations transfer effectively to challenging, underconstrained 3D reconstruction problems such as volumetric novel view synthesis.The conclusion attributes this observation to results using pre-trained visual representations.
  • 8. Conclusions: Pre-trained representations from enormous 100M+ image datasets help address NeRF optimization issues and partial observability, improving perceptual quality.The supported scope is the NeRF family of scene representations and few-shot transfer.
  • 8. Conclusions: The authors believe diet-friendly few-shot transfer may play a greater role across a wide range of 3D applications.This is presented as a future outlook rather than an established result.

A. Experimental details

The experiments use specified view-selection protocols, tuned NeRF baselines, fixed image and optimization settings, and several paired-image and distributional quality metrics.

  • View selection: Most Realistic Synthetic experiments randomly subsample 8 of 100 training renders, while extrapolation tests manually select 14 views mostly showing Lego’s right side.DTU pixelNeRF experiments reuse the source viewpoint from prior work.
  • Simplified NeRF baseline: The Simplified NeRF baseline is tuned per Realistic Synthetic scene because published NeRF can be unstable with 8 views and sensitive to initialization and complexity settings.The tuning targets geometry convergence for fair comparison.
  • Implementation: NeRF, Simplified NeRF, and DietNeRF are trained with Adam for 200k iterations.This implementation uses a PyTorch port and 400×400 images for NeRF-family models.
  • Metrics: PSNR, SSIM, and LPIPS compare pixel-aligned image pairs, whereas FID and KID compare distributions of image features across sample sets.FID and KID require sufficiently large sample sizes because they compare distributions rather than individual images.

B. Per-scene metrics

The Realistic Synthetic evaluation reports per-scene PSNR, SSIM, and LPIPS for eight observed views, with DietNeRF variants consistently near the top.

  • Figure 8 compares CLIP ViT embedding similarity with camera-origin distance for pairs of Realistic Synthetic scenes.
  • Across all few-shot scenes, DietNeRF or its 50k-iteration LMSE fine-tuned variant is best or second-best.
  • FID and KID are excluded because the evaluation requires a larger sample size.
  • Table 7 reports PSNR, SSIM, and LPIPS separately for each Realistic Synthetic scene with 8 observed views.

C. Qualitative results and ground-truth

Qualitative experiments use randomly sampled training views and compare DietNeRF with baselines on unseen poses, including one-shot DTU reconstruction.

  • Qualitative results: DietNeRF produces the highest-quality Realistic Synthetic reconstructions without the distortions or occluders seen in the baselines.It still misses some high-frequency detail and exhibits green-color leakage on the back of a chair.
  • Qualitative results: DietPixelNeRF produces sharper one-shot DTU renderings than pixelNeRF when both are fine-tuned for 20k iterations from the same checkpoint.The semantic-loss variant performs especially well on rectangular objects, but sometimes fails to preserve accurate geometry in this under-determined setting.
  • Alternative approaches: An auxiliary adversarial loss introduced artifacts on Realistic Synthetic scenes, both alone and combined with semantic consistency loss.
  • Semantic comparison: CLIP ViT embeddings are generally more similar across views of the same scene than across different scenes.This supports using semantic embeddings to compare rendered and reference views despite differing poses.
  • Ground-truth views: Randomly sampled views create uneven observation, including redundant Lego views and sparsely observed sides of Hotdog.The sampling reflects challenges associated with real-world data capture, while carefully selected views might improve results.
Loading 2104.00677v1…