PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

Jiawei Yang, Marco Pavone, Yue Wang

arXiv:2303.07418v1cs.CV

TL;DR

Few-shot neural rendering is difficult because NeRF overfits sparse training views and can produce near-camera artifacts. FreeNeRF introduces frequency and occlusion regularization with minimal changes to plain NeRF. It outperforms existing state-of-the-art methods across multiple datasets with minimal overhead, though regularization can introduce dataset- and metric-specific trade-offs.

  • Problem

    NeRF overfits sparse training views, failing to recover multi-view-consistent geometry and synthesize reliable novel views.

  • Method

    FreeNeRF combines frequency regularization, which gradually exposes positional-encoding frequencies, with occlusion regularization that penalizes near-camera density fields.

  • Results

    FreeNeRF outperforms existing state-of-the-art methods across Blender, DTU, and LLFF with minimal overhead.

  • Takeaways & Limitations

    Frequency is a fundamental factor in NeRF training under sparse-data conditions, motivating further investigation beyond few-shot rendering.

  • Takeaways & Limitations

    Occlusion regularization can over-regularize and incompletely represent near-camera objects in DTU, while longer frequency schedules trade higher PSNR for lower LPIPS.

Abstract

from arXiv · show

Novel view synthesis with sparse inputs is a challenging problem for neural radiance fields (NeRF). Recent efforts alleviate this challenge by introducing external supervision, such as pre-trained models and extra depth signals, and by non-trivial patch-based rendering. In this paper, we present Frequency regularized NeRF (FreeNeRF), a surprisingly simple baseline that outperforms previous methods with minimal modifications to the plain NeRF. We analyze the key challenges in few-shot neural rendering and find that frequency plays an important role in NeRF's training. Based on the analysis, we propose two regularization terms. One is to regularize the frequency range of NeRF's inputs, while the other is to penalize the near-camera density fields. Both techniques are ``free lunches'' at no additional computational cost. We demonstrate that even with one line of code change, the original NeRF can achieve similar performance as other complicated methods in the few-shot setting. FreeNeRF achieves state-of-the-art performance across diverse datasets, including Blender, DTU, and LLFF. We hope this simple baseline will motivate a rethinking of the fundamental role of frequency in NeRF's training under the low-data regime and beyond.

1. Introduction

Few-shot neural rendering is difficult because NeRF overfits sparse training views, while existing solutions often add costly supervision, pre-training, or rendering complexity. FreeNeRF instead analyzes low-data failure modes and combines frequency and occlusion regularization in a minimal, efficient baseline.

  • NeRF overfits training views and struggles to synthesize novel views when only a few inputs are available.
  • Existing methods use pre-training, external depth supervision, or patch-based regularization, increasing pipeline or implementation complexity.
  • FreeNeRF stabilizes low-data training by regularizing visible input frequencies and penalizing near-camera density fields that cause floaters.
  • The method is dependency-free and overhead-free because it requires neither costly pre-training or extra supervision nor additional patch-rendering during training.
  • FreeNeRF is presented as a simple baseline that outperforms existing state-of-the-art methods across Blender, DTU, and LLFF at almost no additional computation cost.
  • The paper argues that its observations motivate reconsidering the fundamental role of frequency in NeRF positional encoding.

2. Related Work

NeRF research has struggled with sparse-view rendering, prompting methods that add external information, transferable models, or costly regularization. FreeNeRF reframes the problem around frequency and retains plain-NeRF efficiency while addressing identified failure modes.

  • NeRF commonly requires hundreds of input images for high-quality scene representations and fails with sparse inputs such as 3, 6, or 9 views.
  • Prior few-shot methods use external models, depth supervision, semantic consistency, geometry regularization, or large curated datasets.
  • These approaches can require expensive pre-training or costly training-time patch rendering, adding methodology, engineering, and budget overhead.
  • Positional encoding helps neural networks learn high-frequency functions, and FreeNeRF examines frequency regularization as a sparse-data strategy.

3. Method

FreeNeRF addresses few-shot NeRF failure by controlling input frequencies during training and penalizing near-camera density fields. These regularizers target high-frequency overfitting and novel-view floaters while retaining minimal changes to plain NeRF.

  • Few-shot NeRF overfits sparse training views, producing low-loss representations that fail to recover multi-view-consistent geometry in novel views.
  • High-frequency positional-encoding inputs can accelerate undesirable high-frequency artifacts and cause catastrophic failure in few-shot rendering.
  • With 10% of embedding bits visible, mipNeRF reaches 17.62 PSNR, versus 9.01 PSNR for plain mipNeRF at 100% visibility on three-view DTU.
  • Frequency regularization starts from raw inputs and linearly exposes higher-frequency bands during training, increasing visibility by 3 bits at each step.
  • Frequency regularization does not eliminate all artifacts because limited views leave least-overlapped regions difficult to estimate geometrically.
  • Occlusion regularization penalizes density values among the nearest sampled ray points, targeting near-camera floaters and walls directly during training.

4. Experiments

FreeNeRF is evaluated on Blender, DTU, and LLFF under few-shot settings, where it generally improves synthesis quality over competing methods while adding negligible training overhead. Qualitative and ablation results also expose trade-offs involving geometry detail, LPIPS, and near-camera representations.

  • Datasets & metrics: FreeNeRF is evaluated on Blender, DTU, and LLFF using PSNR, SSIM, LPIPS, and geometric-mean metrics under few-shot protocols.Blender uses 8 training views and 25 test images; DTU and LLFF follow RegNeRF’s protocol.
  • Comparison: FreeNeRF generally achieves the best synthesis results across Blender, DTU, and LLFF, outperforming competing methods in most reported settings.The paper reports best or near-best results across image-quality metrics and input-view settings.
  • Blender: On Blender, FreeNeRF outperforms other methods in PSNR and SSIM while achieving a comparable LPIPS score to the best method.DietNeRF’s fine-tuned variant still underperforms the reported FreeNeRF result.
  • DTU: On DTU, FreeNeRF achieves the best results among per-scene optimization methods and preserves fine-grained details that RegNeRF’s piece-wise smoothing can erase.The comparison highlights eyes, fingers, and wrinkles in the Buddha scene, while more challenging scenes also show higher-quality FreeNeRF results.
  • LLFF: On LLFF, FreeNeRF generally predicts more precise geometry, smoother occupancy fields, and fewer floaters than RegNeRF.RegNeRF can produce visually satisfactory RGB images despite inaccurate depth, whereas FreeNeRF yields less noisy geometry.
  • Training overhead: FreeNeRF adds negligible training overhead of 1.02−1.04×, compared with 1.62−2.8× for other approaches.DietNeRF and RegNeRF incur extra computation from patch rendering and additional models, whereas FreeNeRF adds no such steps.
  • Ablation study: A longer frequency curriculum improves PSNR, with the 90%-schedule performing best, but can lower LPIPS; fine-tuning can improve LPIPS.The ablation reports a trade-off between PSNR and LPIPS as regularization duration increases.
  • Limitations: FreeNeRF has two reported limitations: longer curricula may reduce LPIPS, and occlusion regularization may over-regularize near-camera DTU objects.Per-scene tuning can alleviate the DTU issue, but the paper does not use it.

5. Conclusion

FreeNeRF links few-shot neural-rendering failure to input frequency and addresses it with a simple frequency regularizer. It outperforms existing state-of-the-art methods across multiple datasets with minimal overhead.

  • FreeNeRF reveals a deep relation between input frequency and few-shot neural-rendering failure.
  • A simple frequency regularizer can drastically address the failure of few-shot neural rendering.
  • FreeNeRF outperforms existing state-of-the-art methods on multiple datasets with minimal overhead.
  • Frequency-regularized NeRF produces smoother normal estimation, which can facilitate applications involving glossy surfaces.

Supplement to FreeNeRF: Improving Few-shot Neural Rendering with Free

The supplement provides additional quantitative and qualitative results on FreeNeRF's motivation and limitations, along with experimental and implementation details.

  • The supplement adds quantitative and qualitative results discussing FreeNeRF's motivation and limitations.
  • It also documents experimental settings and implementation details.

A. Additional Results

Additional experiments show that high-frequency inputs can cause catastrophic failure in few-shot rendering, while restricting early inputs to low frequencies stabilizes scene learning before detail refinement.

  • High-frequency inputs cause catastrophic failure in few-shot neural rendering.
  • Using 10% of the total embedding bits lets mipNeRF reconstruct scenes despite over-smoothness, whereas higher-frequency inputs make reconstructions unrecognizable and collapse.
  • Restricting inputs to low-frequency components stabilizes early scene representations before high-frequency signals refine details.

A.1. Limitations

The supplement identifies trade-offs in frequency-regularization duration and failure cases from occlusion regularization, including over-regularized near-camera objects and remote floaters.

  • High-frequency details comparison: With relatively sufficient input views, shorter frequency schedules usually preserve more high-frequency details.
  • Trade-off between PSNR and LPIPS: Longer frequency curricula usually increase PSNR but decrease LPIPS, creating a quality trade-off.Under 9 input views, a 90%-schedule gives object PSNR 25.59 and LPIPS 0.117, while a 50%-schedule gives 25.38 and 0.096.
  • Limitations of Locc: Occlusion regularization can over-regularize some scenes, producing incomplete near-camera objects such as a white desk.
  • Limitations of Locc: Occlusion regularization does not solve small remote floaters, although low-frequency-only NeRF training avoids those floaters in the reported examples.

A.2. Depth Evaluation

FreeNeRF improves depth estimation over its mipNeRF baseline, while RegNeRF performs better on object-centric DTU and FreeNeRF on scene-scale LLFF. Occlusion regularization can also produce incomplete desks and does not remove remote floaters.

  • Depth Evaluation: Aggressive occlusion regularization can produce incomplete white desks, although reducing M from 10 to 5 alleviates the issue somewhat.
  • Depth Evaluation: Occlusion regularization does not solve remote floaters located far from cameras.
  • Depth Evaluation: FreeNeRF significantly improves depth estimation over mipNeRF, while RegNeRF performs better on DTU and FreeNeRF performs better on LLFF.The depth reference uses mipNeRFs trained on all views rather than ground-truth depth.

A.3. Additional Qualitative Results

Additional results show FreeNeRF reconstructing scenes closer to ground truth than DietNeRF and outperforming mipNeRF for normal estimation. On DTU, it performs best on most average metrics but shows limited LPIPS improvement due to slight blurriness.

  • DTU and LLFF: FreeNeRF achieves the best Average metrics in most DTU settings, but its LPIPS improvement is smaller because slight blurriness lowers the score.
  • Blender: FreeNeRF reconstructs Blender scenes closer to ground truth than DietNeRF, whose patches are blurry and stylistically mismatched.
  • DTU and LLFF: Figures A.5 and A.6 provide additional FreeNeRF renderings for 3 input views on DTU and LLFF.

B.1. Dataset and metrics.

The experiments use Blender, DTU, and LLFF under few-shot protocols, with dataset-specific view sampling and image resolutions. Evaluation reports PSNR, SSIM, and LPIPS, alongside qualitative and normal-estimation examples.

  • Datasets: DTU experiments optimize directly on 15 test scenes without pre-training, using 3-, 6-, or 9-view inputs at 300 × 400 resolution.
  • Datasets: LLFF contains 8 forward-facing scenes, with every eighth image reserved for evaluation and images downsampled 8× to 378 × 504.
  • Additional Results: The appendix includes qualitative Blender comparisons, DTU and LLFF renderings, and a Shiny Blender normal-estimation comparison.

B.2. Implementations.

The implementations use plain NeRF and mipNeRF codebases with Adam optimization, positional-encoding frequency settings, and occlusion regularization applied to secondary-stage outputs.

  • DietNeRF Codebase: The DietNeRF codebase uses a plain NeRF with coarse and fine MLPs, trained for 200k Adam iterations with maximum coordinate frequency L = 9.
  • RegNeRF Codebase: The RegNeRF codebase uses plain mipNeRF with maximum coordinate frequency 16, concatenated original coordinates, learning-rate decay, warm-up, and gradient clipping.
  • Occlusion Regularization: Occlusion regularization uses weight 0.01 and is computed on fine-MLP or second-query outputs from the secondary rendering stage.
Loading 2303.07418v1…