PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects

Zhiwen Yan, Chen Li, Gim Hee Lee

arXiv:2303.14435v1cs.CVcs.GR

TL;DR

Dynamic NeRFs struggle to reconstruct moving specular objects because canonical-space warping does not preserve reflection changes. NeRF-DS conditions color on observation-space surface position and orientation and uses moving-object masks to guide deformation, achieving better novel-view synthesis quality on dynamic specular scenes than prior dynamic and reflective NeRFs. Its main limitation is reliance on accurate surface-normal predictions, which reflected textures can distort.

  • Problem

    Existing dynamic NeRFs do not explicitly model surface changes during warping, causing failures on challenging moving specular objects.

  • Method

    NeRF-DS conditions the color branch on observation-space position and surface normal and adds moving-object masks as deformation guidance.

  • Results

    NeRF-DS achieves better novel-view synthesis quality than prior dynamic and reflective NeRFs on dynamic specular scenes.

  • Takeaways & Limitations

    Surface-aware color modeling and mask-guided deformation address reflected-color changes and correspondence learning during canonical-space warping.

  • Takeaways & Limitations

    The method relies on accurate surface-normal predictions, but reflected textures can mislead NeRF geometry and produce incorrect normals, especially in dynamic specular scenes.

Abstract

from arXiv · show

Dynamic Neural Radiance Field (NeRF) is a powerful algorithm capable of rendering photo-realistic novel view images from a monocular RGB video of a dynamic scene. Although it warps moving points across frames from the observation spaces to a common canonical space for rendering, dynamic NeRF does not model the change of the reflected color during the warping. As a result, this approach often fails drastically on challenging specular objects in motion. We address this limitation by reformulating the neural radiance field function to be conditioned on surface position and orientation in the observation space. This allows the specular surface at different poses to keep the different reflected colors when mapped to the common canonical space. Additionally, we add the mask of moving objects to guide the deformation field. As the specular surface changes color during motion, the mask mitigates the problem of failure to find temporal correspondences with only RGB supervision. We evaluate our model based on the novel view synthesis quality with a self-collected dataset of different moving specular objects in realistic environments. The experimental results demonstrate that our method significantly improves the reconstruction quality of moving specular objects from monocular RGB videos compared to the existing NeRF models. Our code and data are available at the project website https://github.com/JokerYan/NeRF-DS.

1. Introduction

Dynamic NeRFs reconstruct moving scenes by warping observations into a common canonical space, but existing methods struggle with changing reflections on moving specular objects. NeRF-DS addresses this with surface-aware color modeling, mask-guided deformation, and a dynamic specular dataset.

  • Motivation: Existing dynamic NeRFs often fail on moving specular objects because they do not model reflected-color changes during warping.Specular surfaces can reflect different colors as their position and orientation change.
  • Method: NeRF-DS conditions color on observation-space position and surface normal while retaining canonical-space geometry.Corresponding points can share geometry while preserving colors determined by their original surface position and orientation.
  • Method: A moving-object mask guides deformation learning and provides more consistent correspondence cues than changing RGB values.The mask also distinguishes moving and static regions for the deformation field.
  • Results: Experiments report significantly higher novel-view reconstruction quality and fewer serious artifacts than existing NeRF models on dynamic specular scenes.The comparison includes prior dynamic and reflective NeRFs and visualizes results against HyperNeRF.
  • Dataset: The authors collect a dataset of moving and deforming specular objects in realistic environments for training and testing monocular videos.The dataset addresses the limited representation of dynamic specular objects in existing datasets.

3. Dynamic NeRF Preliminaries

Dynamic NeRFs use monocular RGB video by mapping observations at each time to a shared canonical space. A deformation field predicts canonical coordinates, after which a canonical NeRF renders density and color for novel views.

  • NeRF: NeRF represents a scene by mapping spatial position and viewing direction to volume density and color.An MLP predicts density and a bottleneck, while a second head uses viewing direction to predict color.
  • Volumetric Rendering: Novel-view rendering samples points along each camera ray and combines their colors using transmittance and local volume density.Sample weights depend on accumulated transmittance, step size, and density.
  • Dynamic NeRF: Dynamic NeRF transforms observation-space points at time t into a common canonical space so multiple times and views can train a static representation.This shared space compensates for the lack of strict multiview observations at each moment.
  • Dynamic NeRF: A deformation MLP predicts canonical coordinates from the sampled observation coordinate and time embedding, while HyperNeRF additionally predicts a hyper-coordinate.The canonical coordinates and hyper-coordinate are then supplied to the canonical NeRF to predict density and color.

4. Our Method: NeRF-DS

NeRF-DS makes dynamic NeRF surface-aware by conditioning color on observation-space position and normal, while using masks to guide deformation for moving specular objects. It preserves shared canonical geometry while addressing reflected-color changes and correspondence failures during motion.

  • Surface-Aware Dynamic NeRF: Existing dynamic NeRFs can map different observation-space points or surface orientations to one canonical point and render them with the same color.This under-parameterizes specular appearance because reflected color depends on observation-space position and surface normal.
  • Surface-Aware Dynamic NeRF: NeRF-DS adds observation-space position and surface normal to the color branch while leaving canonical-space volume-density prediction unchanged.Its modified function predicts density from the canonical coordinate and color from canonical coordinate, viewing direction, observation-space position, and normal.
  • Surface-Aware Dynamic NeRF: Annealed positional encoding initially cuts observation-space coordinates off from the model and gradually introduces them during training.The encoding parameter τ starts at 0 and is slowly increased.
  • Surface-Aware Dynamic NeRF: The method estimates canonical surface normals from volume-density gradients, predicts smoother normals with the NeRF MLP, and warps them back to observation space using the deformation rotation.This preserves surface-normal consistency across corresponding points over time.
  • Mask Guided Deformation Field: A mask-guided deformation field uses moving-object masks because specular color changes hinder RGB-only temporal correspondence learning.The mask remains consistent during motion and provides guidance for deforming regions.
  • Mask Guided Deformation Field: Mask input produces a more meaningful deformation field that differentiates moving foreground from static background, unlike deformation learned without masks.The unmasked field fails to capture the foreground motion completely.

5. Experiments

Experiments evaluate NeRF-DS on a newly collected dynamic specular dataset against representative dynamic and reflective NeRF baselines. NeRF-DS reduces artifacts and improves quantitative novel-view synthesis quality, while both surface information and mask guidance contribute to performance.

  • 5.1. Dynamic Specular Dataset: The dataset contains 8 everyday scenes with moving or deforming specular objects, two rigidly mounted forward-facing cameras, and approximately 500 frames per video.One camera provides training frames and the other provides testing views; masks generated from MiVOS support registration and mask supervision.
  • 5.2. Experimental Setups: NeRF-DS is compared with HyperNeRF, Nerfies, and Ref-NeRF, representing state-of-the-art dynamic and static-specular NeRF reconstruction.The implementation follows official configurations for 250k iterations at 480 × 270 resolution.
  • 5.3. Evaluation Results: NeRF-DS produces fewer geometric artifacts and more accurate reflected colors than the baselines on dynamic specular scenes.HyperNeRF and Nerfies often produce blurry or torn objects, while Ref-NeRF assumes a static scene and can predict incorrect geometry for moving objects.
  • 5.3. Evaluation Results: NeRF-DS significantly outperforms all baselines on LPIPS and achieves better MS-SSIM and PSNR in most scenes and in the overall average.The authors identify LPIPS as a better reconstruction-quality measure than MS-SSIM and PSNR because blurry predictions may not substantially affect the latter metrics.
  • 5.3. Evaluation Results: Removing either the surface-aware dynamic NeRF or the mask-guided deformation field reduces performance.The ablations are named NeRF-DS w/o Surface and NeRF-DS w/o Mask.

6. Limitations

NeRF-DS improves reconstruction quality but depends on accurate surface-normal predictions, which can be distorted by reflected textures.

  • 6. Limitations: NeRF-DS relies on accurate surface-normal predictions, but reflected textures can mislead NeRF geometry and make predicted normals follow reflections instead of surface shape.The problem is more severe in dynamic than static specular scenes because dynamic scenes provide weaker geometry constraints.

7. Conclusion

NeRF-DS extends dynamic NeRF for more accurate reconstruction and rendering of dynamic specular scenes. Its surface-aware representation and mask-guided deformation address reflected-color modeling and correspondence under motion.

  • 7. Conclusion: NeRF-DS models reflected colors during canonical-space warping with a surface-aware dynamic NeRF and a mask-guided deformation field.Both components are described as essential for modeling reflected colors during warping.
  • 7. Conclusion: NeRF-DS achieves better novel-view synthesis quality than prior dynamic and reflective NeRFs on dynamic specular scenes.

1. Qualitative Result Videos

Supplementary videos provide a visual comparison of rendered sequences from NeRF-DS and baseline models.

  • 1. Qualitative Result Videos: Rendered videos of NeRF-DS and baseline models are included in the supplementary zip file.The videos are intended to demonstrate qualitative performance differences.

2. Implementation Details

NeRF-DS uses positional encoding and dedicated prediction modules for masks, deformation, hyper coordinates, and canonical radiance. The implementation adds modest parameter and training-time overhead, while qualitative comparisons isolate the two proposed components.

  • Model modules: The implementation includes dedicated mask, deformation, hyper-coordinate, and canonical NeRF prediction modules.The respective architectures are shown in Figures 8–11.
  • Input encoding: Positional encoding is applied to spatial coordinates, viewing direction, and surface normals, with input-specific widths and annealing schedules.The encoding details are summarized in Table 2.
  • Computational cost: The full model contains 1.45M parameters and requires 6 hours to train, versus 1.30M parameters and 5 hours for the baseline.
  • Ablation comparisons: Qualitative ablations compare the full NeRF-DS model separately against versions without surface-aware dynamic NeRF and without mask-guided deformation.These comparisons are shown in Figures 12 and 13.

4. Additional Experiment Results

Additional experiments tune positional encoding, mask sharpening, and surface-normal selection, while also testing NeRF-DS on non-specular HyperNeRF scenes. The selected settings delay coordinate use, sharpen masks gradually, and use temporally consistent warped normals.

  • Positional encoding: NeRF-DS delays observation-space coordinate x in the color branch for 50k iterations, then increases its bandwidth to 4 over the next 50k iterations.This schedule was selected from quantitative results on the Sheet scene.
  • Mask sharpening: The mask-weight standard deviation is gradually decreased from 1 to 0.1 during the first 30k training iterations.The schedule was chosen based on quantitative results in Table 4.
  • Surface normals: The main results use surface normals warped from canonical space, n = R^Tn′, to provide better consistency over time.Canonical-space and directly observation-space normals are evaluated on the Sheet scene.
  • Non-specular scenes: On the non-specular HyperNeRF dataset, the reproduced comparison uses one-tenth the original batch size and ten times the number of iterations because of limited hardware.The authors state that this setting disadvantages their reproduced comparison because their reproduced HyperNeRF models perform worse than reported models.

5. Additional Qualitative Analysis

Qualitative and ablation analyses connect observation-space surface normals and mask guidance to reflected-color rendering and geometric quality. The case study shows that surface normals alone provide a substantial LPIPS improvement, while masks further suppress artifacts.

  • Surface-normal analysis: Different surface normals across frames allow NeRF-DS to render different reflected colors from the same surface.The case study uses an early NeRF-DS result without the mask.
  • Ablation comparisons: The qualitative comparisons include full NeRF-DS versus versions without surface-aware dynamic NeRF and without mask-guided deformation.The relevant comparisons are presented in Figures 12 and 13.
  • Artifact comparison: HyperNeRF fails to recognize corresponding surfaces across frames and produces severe geometric artifacts, whereas additional masks can further suppress those artifacts.
  • Dataset visualization: The dynamic specular dataset includes paired training- and test-camera videos across eight scenes, as illustrated in Figure 14.The dataset contains moving specular objects in everyday environments.
  • Ablation result: 20.3% LPIPS improvement from baseline is attributed to the surface normal alone in the ablation study.

6. Dynamic Specular Dataset Details

The dynamic specular dataset contains eight scenes of varied moving specular objects in everyday environments, captured by two rigidly connected cameras. Figure 14 shows paired training and test videos, while Table 7 details each scene.

  • Dataset composition: The dataset consists of 8 scenes containing various dynamic specular objects in everyday environments.Different object and surface types are listed in Table 7.
Loading 2303.14435v1…