HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

• Topological changes in real-world motion create discontinuities or singularities that challenge moving-scene reconstruction algorithms.Source: §1 · p. 1
• Level-set methods model changing scenes as static objects in higher-dimensional ambient spaces, allowing topological changes as smooth transformations.Source: §1 · p. 1
• HyperNeRF adapts level-set ideas to deformable neural radiance fields, adding multiple ambient dimensions and allowing curved slicing manifolds represented by MLPs.Source: §1 · p. 2
• HyperNeRF produces sharper, higher-quality results with fewer artifacts for smooth moment interpolation and fixed-moment novel-view synthesis.Source: §1 · p. 2
• Deformation-based NeRFs map observations to a canonical template with continuous fields, preventing them from representing topological variations such as mouth openings.Source: §2.2 · p. 3
• HyperNeRF combines deformation-based motion modeling with higher-dimensional coordinates and deformations, enabling topology changes that strictly deformation-based approaches cannot model.Source: §2.2 · p. 3

3.1 Level Set Methods

• Level-set methods represent surfaces as zero-level sets of auxiliary functions and encode topology variation by adding ambient dimensions.Source: §3.1 · p. 4
• A collection of shapes can be encoded as individual slices of one learned higher-dimensional implicit surface using an MLP and learned ambient coordinates.Source: §3.1 · p. 4

3.2 Deformable Slicing Surfaces

• Axis-aligned slicing requires every desired shape to exist as a cross-section, which can inefficiently duplicate spatially separated shape components.Source: §3.2 · p. 4
• Deformable slicing surfaces let different spatial locations reference different ambient coordinates, producing more compact representations of arbitrary shape mixtures.Source: §3.2 · p. 4

NeRF foundations

• NeRF maps 3D positions and viewing directions to color and density through an MLP, using sinusoidal positional encoding for high-frequency signals.The encoding bandwidth is controlled by the number of frequencies m.Source: §4.1 · p. 5

Deformable NeRFs

• Deformable NeRFs map observation-space coordinates to a canonical template using a latent deformation code and an MLP-parameterized transformation.Nerfies also uses elastic regularization to encourage near-rigid deformations and windowed positional encoding for coarse-to-fine optimization.Source: §4.1 · p. 5

Motivation

• Continuous deformation fields cannot easily represent topology changes because such changes require deformation-field discontinuities.The continuity follows from representing the fields with smooth MLP interpolators.Source: §4.2 · p. 6

Hyper-space representation

• HyperNeRF extends the template NeRF into higher dimensions and represents each observation as a slice through a high-dimensional ambient space.The approach adapts the level-set idea of representing changing shapes as cross-sections while still rendering 3D scenes.Source: §4.2 · p. 6

Slicing surfaces

• Deformable slicing surfaces map each sample point to ambient coordinates, allowing different spatial positions to reference different parts of the hyper-space template.This uses the template more efficiently than axis-aligned planes, which force all ray samples in an observation to share one ambient coordinate and can cause fading artifacts.Source: §4.2 · p. 6, Figure 6 · p. 6

Evaluation

• HyperNeRF outperforms Nerfies, Neural Volumes, and NSFF in most interpolation cases, while qualitative comparisons show sharper reconstructions and better geometry than Nerfies.The authors note that some image-quality metrics can prefer blurry results.Source: §5.2.1 · p. 9, §5.2.2 · p. 9

6 LIMITATIONS AND CONCLUSION

• Camera registration affects reconstruction quality, as with all NeRF-like methods.Source: §6 · p. 9
• Moments poorly captured in training data, such as during rapid motion, cannot be reconstructed by the method.The model uses only color images and imposes no domain-specific priors.Source: §6 · p. 9
• HyperNeRF reconstructs topologically varying scenes with discontinuous deformations by representing them as slices through a higher-dimensional space.The method combines level-set ideas with deformation-based dynamic NeRFs.Source: §6 · p. 9

A NETWORK ARCHITECTURE

• Architecture diagrams detail the template MLP, deformation MLP, and ambient slicing surface MLP.Source: A NETWORK ARCHITECTURE · p. 13

B.1 Additional Qualitative Results

• Figures 11 and 12 show qualitative results from the interpolation experiments.Source: §B.1 · p. 13

B.2 Additional Quantitative Metrics

• Per-sequence interpolation evaluations report PSNR, MS-SSIM, and LPIPS scores for each sequence and method.Tables 3, 4, and 5 report the respective metrics.Source: §B.2 · p. 13

• The experiments use an updated NSFF implementation containing bug fixes and improvements.Source: §B.3 · p. 13
• NSFF hyperparameters are tuned because the defaults perform poorly on the authors’ sequences.Source: §B.3 · p. 13
• Supervised depth and flow losses decay over 1000N iterations, where N is the number of dataset frames, instead of a fixed 25,000 iterations.The fixed schedule caused poor performance on longer sequences.Source: §B.3 · p. 13

• The experiments closely follow Nerfies’ procedure from Section E.2.Source: §B.4 · p. 13

Setup

• The experiments learn a family of topologically varying 2D signed distance functions using the method that generates Fig. 3.Source: C 2D LEVEL SET EXPERIMENTS · p. 13

Model

• The template MLP takes normalized 2D coordinates and an ambient coordinate, producing a signed distance that defines a 3D surface.The surface is sliced by either an axis-aligned plane or a deformable slicing surface represented by an ambient slicing-surface MLP.Source: C 2D LEVEL SET EXPERIMENTS · p. 13
• Per-shape latent codes are used alongside spatial coordinates, while detailed architectures are provided for the template and deformable slicing-surface MLPs.Source: C 2D LEVEL SET EXPERIMENTS · p. 13, Figure 16 · p. 14, Figure 17 · p. 14

Training

• Training samples points from continuous truncated SDFs and optimizes a Pseudo-Huber loss comparing predicted and ground-truth SDF values.The steepness parameter is set to δ = 0.005.Source: C 2D LEVEL SET EXPERIMENTS · p. 13

Implementation

• Positional encoding uses minimum and maximum frequency degrees, with the template and ambient slicing-surface MLPs assigned separate degree settings.For the ambient slicing-surface MLP, the reported settings are m− = −2 and m+ = 2; training uses Adam for 2000 iterations with batch size 512.Source: C 2D LEVEL SET EXPERIMENTS · pp. 13–14

Interpolation

• Interpolated shapes are generated by linearly interpolating the shape code ωi between instances.Source: C 2D LEVEL SET EXPERIMENTS · p. 14