Source-linked AI summary
HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, Steven M. Seitz
TL;DR
Deformation-based dynamic NeRFs struggle with topology changes because their continuous deformation fields cannot represent the required discontinuities. HyperNeRF lifts NeRF into higher-dimensional space and represents each image’s radiance field as a slice, outperforming existing methods on interpolation and novel-view synthesis while remaining limited by camera registration and unobserved motion.
Problem
Continuous deformation fields in dynamic NeRFs cannot represent topological changes that require discontinuities.
Method
HyperNeRF extends NeRF into higher dimensions, representing each observation as a slice through a hyper-space while retaining deformation fields.
Results
HyperNeRF produces sharper, higher-quality results with fewer artifacts on smooth moment interpolation and fixed-moment novel-view synthesis.
Takeaways & Limitations
Combining deformation fields with higher-dimensional slices enables reconstruction of scenes containing both large motions and topological variations.
Takeaways & Limitations
Reconstruction quality depends on camera registration, and poorly observed moments such as rapid motion cannot be reconstructed from color-only input without domain-specific priors.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at https://hypernerf.github.io.
1 INTRODUCTION
HyperNeRF addresses topological changes in dynamic scenes by combining deformation-based reconstruction with higher-dimensional NeRF representations inspired by level sets. It targets smooth moment interpolation and fixed-moment novel-view synthesis from monocular video.
- Topological changes in real-world motion create discontinuities or singularities that challenge moving-scene reconstruction algorithms.
- Level-set methods model changing scenes as static objects in higher-dimensional ambient spaces, allowing topological changes as smooth transformations.
- HyperNeRF adapts level-set ideas to deformable neural radiance fields, adding multiple ambient dimensions and allowing curved slicing manifolds represented by MLPs.
- HyperNeRF produces sharper, higher-quality results with fewer artifacts for smooth moment interpolation and fixed-moment novel-view synthesis.
- Deformation-based NeRFs map observations to a canonical template with continuous fields, preventing them from representing topological variations such as mouth openings.
- HyperNeRF combines deformation-based motion modeling with higher-dimensional coordinates and deformations, enabling topology changes that strictly deformation-based approaches cannot model.
3 MODELING TIME-VARYING SHAPES
The paper explains topology-varying shapes as slices through higher-dimensional surfaces, then introduces deformable slicing surfaces to represent shape mixtures compactly. This perspective motivates HyperNeRF’s higher-dimensional scene representation.
- 3.1 Level Set Methods: Level-set methods represent surfaces as zero-level sets of auxiliary functions and encode topology variation by adding ambient dimensions.
- 3.1 Level Set Methods: A collection of shapes can be encoded as individual slices of one learned higher-dimensional implicit surface using an MLP and learned ambient coordinates.
- 3.2 Deformable Slicing Surfaces: Axis-aligned slicing requires every desired shape to exist as a cross-section, which can inefficiently duplicate spatially separated shape components.
- 3.2 Deformable Slicing Surfaces: Deformable slicing surfaces let different spatial locations reference different ambient coordinates, producing more compact representations of arbitrary shape mixtures.
4 METHOD
HyperNeRF extends deformable NeRFs into higher-dimensional space so topologically varying scenes can be represented as slices of a hyper-space template. Deformable slicing surfaces allocate different ambient coordinates across spatial positions, improving reconstruction quality and interpolation over axis-aligned planes.
- NeRF foundations: NeRF maps 3D positions and viewing directions to color and density through an MLP, using sinusoidal positional encoding for high-frequency signals.The encoding bandwidth is controlled by the number of frequencies m.
- Deformable NeRFs: Deformable NeRFs map observation-space coordinates to a canonical template using a latent deformation code and an MLP-parameterized transformation.Nerfies also uses elastic regularization to encourage near-rigid deformations and windowed positional encoding for coarse-to-fine optimization.
- Motivation: Continuous deformation fields cannot easily represent topology changes because such changes require deformation-field discontinuities.The continuity follows from representing the fields with smooth MLP interpolators.
- Hyper-space representation: HyperNeRF extends the template NeRF into higher dimensions and represents each observation as a slice through a high-dimensional ambient space.The approach adapts the level-set idea of representing changing shapes as cross-sections while still rendering 3D scenes.
- Slicing surfaces: Deformable slicing surfaces map each sample point to ambient coordinates, allowing different spatial positions to reference different parts of the hyper-space template.This uses the template more efficiently than axis-aligned planes, which force all ray samples in an observation to share one ambient coordinate and can cause fading artifacts.
- Evaluation: HyperNeRF outperforms Nerfies, Neural Volumes, and NSFF in most interpolation cases, while qualitative comparisons show sharper reconstructions and better geometry than Nerfies.The authors note that some image-quality metrics can prefer blurry results.
6 LIMITATIONS AND CONCLUSION
HyperNeRF reconstructs topologically varying scenes by combining deformation-based NeRFs with higher-dimensional slicing. The authors note camera registration and insufficiently observed moments as limitations.
- 6 LIMITATIONS AND CONCLUSION: Camera registration affects reconstruction quality, as with all NeRF-like methods.
- 6 LIMITATIONS AND CONCLUSION: Moments poorly captured in training data, such as during rapid motion, cannot be reconstructed by the method.The model uses only color images and imposes no domain-specific priors.
- 6 LIMITATIONS AND CONCLUSION: HyperNeRF reconstructs topologically varying scenes with discontinuous deformations by representing them as slices through a higher-dimensional space.The method combines level-set ideas with deformation-based dynamic NeRFs.
A NETWORK ARCHITECTURE
The supplementary material provides detailed architecture diagrams for HyperNeRF’s template, deformation, and ambient slicing-surface networks.
- A NETWORK ARCHITECTURE: Architecture diagrams detail the template MLP, deformation MLP, and ambient slicing surface MLP.
B EVALUATION B.1 Additional Qualitative Results
Additional qualitative results from the interpolation experiments are presented in Figures 11 and 12.
- B.1 Additional Qualitative Results: Figures 11 and 12 show qualitative results from the interpolation experiments.
B.2 Additional Quantitative Metrics
The supplementary material reports per-sequence PSNR, MS-SSIM, and LPIPS metrics for each interpolation sequence and method.
- B.2 Additional Quantitative Metrics: Per-sequence interpolation evaluations report PSNR, MS-SSIM, and LPIPS scores for each sequence and method.Tables 3, 4, and 5 report the respective metrics.
B.3 Details of the NSFF Experiments
The NSFF experiments use an updated implementation with tuned hyperparameters and a dataset-length-dependent decay schedule for supervised losses.
- The experiments use an updated NSFF implementation containing bug fixes and improvements.
- NSFF hyperparameters are tuned because the defaults perform poorly on the authors’ sequences.
- Supervised depth and flow losses decay over 1000N iterations, where N is the number of dataset frames, instead of a fixed 25,000 iterations.The fixed schedule caused poor performance on longer sequences.
B.4 Details of the Neural Volumes Experiments
The Neural Volumes experiments follow the procedure described in Section E.2 of Nerfies.
- The experiments closely follow Nerfies’ procedure from Section E.2.
C 2D LEVEL SET EXPERIMENTS
The 2D level set experiments learn topologically varying signed distance functions with template and ambient slicing-surface MLPs, then interpolate shapes through latent codes.
- Setup: The experiments learn a family of topologically varying 2D signed distance functions using the method that generates Fig. 3.
- Model: The template MLP takes normalized 2D coordinates and an ambient coordinate, producing a signed distance that defines a 3D surface.The surface is sliced by either an axis-aligned plane or a deformable slicing surface represented by an ambient slicing-surface MLP.
- Model: Per-shape latent codes are used alongside spatial coordinates, while detailed architectures are provided for the template and deformable slicing-surface MLPs.
- Training: Training samples points from continuous truncated SDFs and optimizes a Pseudo-Huber loss comparing predicted and ground-truth SDF values.The steepness parameter is set to δ = 0.005.
- Implementation: Positional encoding uses minimum and maximum frequency degrees, with the template and ambient slicing-surface MLPs assigned separate degree settings.For the ambient slicing-surface MLP, the reported settings are m− = −2 and m+ = 2; training uses Adam for 2000 iterations with batch size 512.
- Interpolation: Interpolated shapes are generated by linearly interpolating the shape code ωi between instances.