Source-linked AI summary
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, Pratul P. Srinivasan
TL;DR
NeRF-based view synthesis struggles to accurately represent glossy, view-dependent appearance. Ref-NeRF reparameterizes outgoing radiance as reflected radiance and structures it with spatially varying scene properties, while regularizing normals. The model significantly improves specular realism and normal accuracy, with interpretable representations that support scene editing.
Problem
NeRF-based methods often fail to accurately capture and reproduce glossy surfaces and produce unstable specular appearance across novel viewpoints.
Method
Ref-NeRF reparameterizes outgoing radiance using reflected viewing directions, Integrated Directional Encoding, diffuse and specular components, and a volume-density regularizer for normals.
Results
Ref-NeRF significantly improves view-dependent appearance and normal-vector accuracy, producing state-of-the-art novel-view renderings and better results on highly specular objects.
Takeaways & Limitations
The structured representation yields interpretable normals, roughness, diffuse texture, and specular tint that enable convincing scene editing.
Takeaways & Limitations
Ref-NeRF is roughly 25% slower than mip-NeRF and improves less when interreflections or non-distant illumination are present.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) is a popular view synthesis technique that represents a scene as a continuous volumetric function, parameterized by multilayer perceptrons that provide the volume density and view-dependent emitted radiance at each location. While NeRF-based techniques excel at representing fine geometric structures with smoothly varying view-dependent appearance, they often fail to accurately capture and reproduce the appearance of glossy surfaces. We address this limitation by introducing Ref-NeRF, which replaces NeRF's parameterization of view-dependent outgoing radiance with a representation of reflected radiance and structures this function using a collection of spatially-varying scene properties. We show that together with a regularizer on normal vectors, our model significantly improves the realism and accuracy of specular reflections. Furthermore, we show that our model's internal representation of outgoing radiance is interpretable and useful for scene editing.
1. Introduction
Ref-NeRF addresses NeRF’s poor interpolation and unstable modeling of glossy, view-dependent appearance by reparameterizing reflected radiance and improving normal estimation. The resulting representation improves novel-view realism and supports interpretable scene editing.
- Motivation: NeRF’s view-dependent radiance produces glossy artifacts that fade and reappear across viewpoints instead of moving smoothly across surfaces.Its volumetric explanation can also create semitransparent or “foggy” shells around objects.
- Method: Ref-NeRF replaces viewing-direction input with the reflected viewing vector about the local surface normal, simplifying radiance interpolation.This lets the directional MLP share appearance observations between nearby scene points.
- Method: Integrated Directional Encoding, together with diffuse and specular components, keeps reflected radiance smooth across varying materials and textures.The representation explicitly includes diffuse color, roughness, and specular structure.
- Method: A volume-density regularizer concentrates density around surfaces and improves normal vectors, enabling more accurate reflection directions.Accurate normals are necessary because the reflection-based parameterization depends on them.
- Results: Ref-NeRF produces state-of-the-art novel-view renderings, substantially improves glossy-object synthesis, and exposes interpretable properties for scene editing.Its interpretable components include normals, roughness, diffuse texture, and specular tint.
2. Related Work
Related work spans neural and graphics-based representations for novel-view synthesis, glossy appearance, and normal estimation. Ref-NeRF adapts reflected-radiance ideas to image-based scene recovery while modeling spatially varying materials and lighting without several strong assumptions used by prior relighting methods.
- 3D scene representations for view synthesis: View synthesis reconstructs scene geometry from captured images to render novel viewpoints, because exhaustive light-field sampling is usually impractical.Prior systems use meshes, volumetric representations, or multiplane images to support reprojection and blending.
- 3D scene representations for view synthesis: NeRF uses continuous coordinate-based MLPs to represent scene geometry and view-dependent appearance, motivating extensions across multiple application domains.Ref-NeRF focuses specifically on improving NeRF’s view-dependent appearance representation.
- Relighting and normal estimation: Prior relighting methods often require assumptions such as known lighting, no self-occlusions, or single-material scenes to recover explicit lighting and material parameters.Ref-NeRF structures components without requiring them to have precise physical meanings.
- Efficient rendering of glossy appearance: Graphics methods use prefiltered environment maps for efficient specular rendering, whereas Ref-NeRF recovers a renderable scene model from images.Its directional MLP also represents spatial variation from lighting, roughness, and texture without storing separate maps for every material.
- NeRF preliminaries: NeRF training queries spatial and directional MLPs along rays, alpha-composites predicted densities and colors, and minimizes pixelwise L2 error.Coarse and fine MLP sets are trained hierarchically.
3. Structured View-Dependent Appearance
Ref-NeRF structures outgoing radiance around reflected directions and spatially varying material properties, making specular appearance easier to interpolate across materials and viewpoints. Its directional representation combines spherical-harmonic encoding, roughness-aware integration, and diffuse–specular decomposition.
- Structured View-Dependent Appearance: Ref-NeRF represents outgoing radiance using reflected direction, diffuse color, material roughness, and specular tint.These components are explicitly used in the directional MLP and support scene editing.
- Reflection Direction Parameterization: The reflection-direction parameterization makes view-dependent radiance better suited for interpolation than direct view-direction conditioning.For rotationally symmetric BRDFs without interreflections or self-occlusions, view-dependent radiance depends only on the reflection direction.
- Integrated Directional Encoding: Integrated Directional Encoding represents reflection-direction distributions with spherical harmonics and a von Mises–Fisher distribution.The concentration parameter is inverse roughness, so larger roughness produces a wider distribution and attenuates higher-frequency directional content.
- Integrated Directional Encoding: Spherical harmonics replace NeRF’s sinusoidal direction encoding because they are stationary on the sphere.This encoding is combined with integration over a roughness-dependent distribution of reflection vectors.
- Diffuse and Specular Colors: Ref-NeRF separates diffuse color from specular color by predicting diffuse color and specular tint spatially and combining them with directional specular color.A fixed tone-mapping function converts the resulting linear color to sRGB and clips it to [0, 1].
- Structured View-Dependent Appearance: A spatial bottleneck vector lets reflected radiance vary with position under interreflections and self-occlusion.The bottleneck is output by the spatial MLP and passed into the directional MLP.
4. Accurate Normal Vectors
Ref-NeRF improves normal estimation by predicting smoother normals and regularizing their orientation and the volume-density structure. The resulting normals support accurate reflection directions and prevent specularities from being explained by hidden emitters beneath foggy surfaces.
- Limitations of NeRF Geometry: NeRF-based volume densities produce noisy normals and can explain specular highlights with emitters embedded inside foggy surfaces.This representation partially occludes emitters with semitransparent diffuse content.
- Visual Comparison: Figure 5 contrasts mip-NeRF’s blurry, view-inconsistent reflections and diffuse interior emitters with Ref-NeRF’s nearly perfect reconstruction and concentrated surface.The comparison uses accumulated normals and rendering weights along the central scanline.
- Predicted Normals: Ref-NeRF predicts a normalized normal vector at each sampled position and ties it to the corresponding density-gradient normal with a penalty.The predicted normals tend to be smoother because density gradients act as a high-pass filter on the MLP’s interpolation kernel.
- Normal Orientation Regularization: A normal-orientation regularizer penalizes backfacing normals at samples that contribute to the rendered ray color.The penalty targets visible samples where volume density decreases along the ray.
- Effects on Reflection Modeling: The orientation penalty prevents hidden emitters beneath semitransparent surfaces from explaining specularities and improves reflection-direction computation.Ref-NeRF therefore uses the improved normals to query its directional MLP.
5. Experiments
Ref-NeRF is evaluated on synthetic and real scenes, where it improves glossy appearance, normal accuracy, and view-consistent editing. Ablations support reflected-radiance parameterization and normal regularization, while the method incurs higher computation and weaker gains for non-distant illumination or interreflections.
- Evaluation: Ref-NeRF outperforms prior methods on glossy-scene rendering and normal-vector accuracy across the Shiny Blender and real captured scene evaluations.The Shiny Blender comparison covers six glossy objects, while real-scene examples include sedan, garden spheres, and toy car captures.
- Ablation Study: Replacing view directions with reflection directions significantly reduces reconstruction metrics, while removing orientation regularization severely degrades normals and renderings.Adding view directions alongside the integrated directional encoding also slightly decreases performance.
- Blender Dataset: On the standard Blender dataset, Ref-NeRF outperforms prior work across all image-quality metrics and improves normal-vector MAE by 35% relative to mip-NeRF.VolSDF has 15% lower normal MAE, but Ref-NeRF achieves 6dB higher PSNR and qualitatively sharper geometry.
- Scene Editing: Ref-NeRF’s structured radiance components enable view-consistent scene edits, including changing diffuse color without affecting glossy-paint reflections and plausibly modifying roughness.The representation exposes interpretable components such as diffuse color, roughness, and specular behavior.
- Limitations: Ref-NeRF is roughly 25% slower than mip-NeRF because normal computation requires backpropagating through the spatial MLP, and its gains diminish with interreflections or non-distant illumination.Integrated directional encoding is also slightly slower than standard positional encoding.
6. Conclusion
The paper concludes that prior neural scene representations struggle with specularities and reflections. Ref-NeRF combines structured reflected-radiance modeling with normal regularization to improve view-dependent appearance and normal accuracy.
- Prior neural representations fail to accurately represent and render scenes with specularities and reflections.
- Ref-NeRF’s parameterization, radiance structuring, and normal-vector regularization improve view-dependent appearance and normal accuracy in synthesized views.
A. Integrated Directional Encoding Proofs
The appendix derives attenuation functions for spherical harmonics under a vMF distribution and proves an efficient large-κ approximation. It also explains how normal-vector regularization supports reliable reflection directions.
- Attenuation-function derivation: The expected spherical-harmonic value under a vMF distribution is reduced by rotating the mean direction to the z-axis and integrating over azimuth.Only the zero-azimuthal-order term survives, producing an expression involving the corresponding Wigner D-matrix element.
- Attenuation-function derivation: The appendix obtains a closed-form expression for the ℓth attenuation function and proves it by recurrence followed by induction.The proof verifies the base cases ℓ = 0 and ℓ = 1, then matches coefficients for ℓ ≥ 2.
- Approximation: The exact attenuation-function computation is inefficient and numerically unstable because of catastrophic cancellation.This motivates the approximation presented in the main paper.
- Approximation: For large κ, the approximation is exact up to an O(1/κ^2) term, while Figure S1 shows close agreement for all κ > 0 and orders ℓ.The approximation’s accuracy improves as ℓ increases.
- Normal regularization: The regularization term penalizes backwards-facing normals contributing to rendered color, improving normals used to compute reflection directions.Applying the penalty to spatial-MLP normal outputs avoids optimization effects that can occur when penalizing density-gradient normals directly on fine geometric structures.
C. Optimization
The implementation uses a mip-NeRF-based JAX setup with weighted normal losses, bottleneck noise, ray-batch choices, and a modified optimization schedule.
- The implementation is based on the official JAX implementation of mip-NeRF.
- Synthetic experiments use normal orientation and prediction loss weights of 0.1 and 3·10^-4, while real captured data uses a 10^-3 prediction-loss weight.The same weights are used during coarse and fine stages.
- Training adds i.i.d. Gaussian noise with standard deviation 0.1 to the bottleneck to slightly stabilize results in some cases.The authors use this to prevent early bottleneck reliance.
- Blender experiments use single-image batching, whereas Shiny Blender experiments sample random ray batches from all images.
- Training uses 250k Adam iterations with batch size 214, learning-rate annealing from 2 × 10^-3 to 2 × 10^-5, warm-up, and gradient clipping.The schedule uses β1 = 0.9, β2 = 0.999, ε = 10^-6, a 512-iteration warm-up, and gradient clipping to norm 10^-3.
D. Dataset Details
The evaluation uses newly created Shiny Blender scenes adapted from public BlendSwap models and a real-scene dataset captured by the authors.
- The Shiny Blender dataset contains Coffee, Toaster, Car, and Helmet scenes adapted from specified BlendSwap models.The source models have CC-0 or CC-BY licenses.
- The real-scene dataset was captured by the authors.
E. Evaluation Details
Evaluation computes rendered normal vectors through volume accumulation and reports image-quality metrics and normal errors on real and Blender test sets.
- Normal evaluation: A ray’s normal vector is computed by sampling normals along the ray and applying the volume-rendering procedure.
- Normal evaluation: Normal-map visualization uses gray values for high variance along a ray, while MAE evaluation uses normalized accumulated normals.The evaluated normal is ˆN = N/∥N∥.
- Image-quality evaluation: Real captured scenes are evaluated with PSNR, SSIM, and LPIPS on their test sets.
- Image-quality evaluation: The Blender evaluation reports per-scene PSNR, SSIM, LPIPS, and normal MAE.
- Baselines: Comparisons use official mip-NeRF and PhySG implementations, tuned PhySG hyperparameters for Blender, and author-run VolSDF results.
F. Additional Results
Additional results report quantitative metrics across three real captured scenes, the original synthetic Blender dataset, and the Shiny Blender dataset. The Shiny Blender results are organized by standard image-quality metrics and per-scene test sets.
- Real captured scenes: Table S1 reports quantitative test-set metrics for three real captured scenes.Every eighth image is held out for testing, following the NeRF evaluation setup.
- Synthetic datasets: Tables S2–S5 contain quantitative metrics for the original synthetic Blender dataset.
- Shiny Blender dataset: Tables S6–S9 contain per-scene test-set metrics for the Shiny Blender dataset.
- Shiny Blender metrics: The Shiny Blender evaluation reports PSNR, SSIM, LPIPS, and MAE.These metrics are presented in Tables S6, S7, S8, and S9, respectively.