PaperlayerView exact revision on arXiv

Source-linked AI summary

Nerfies: Deformable Neural Radiance Fields

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla

arXiv:2011.12948v5cs.CVcs.GR

TL;DR

Photorealistic deformable 3D capture traditionally requires specialized camera setups, limiting accessibility. This paper extends NeRF with deformation priors and coarse-to-fine optimization to reconstruct high-fidelity human models from casual cellphone captures.

  • Problem

    Photorealistic 3D scanning typically requires specialized labs, motivating accessible reconstruction of deformable subjects from casual mobile-phone captures.

  • Method

    The method extends NeRF with continuous deformation fields, elastic rigidity priors, and coarse-to-fine regularization for reconstructing deformable scenes.

  • Results

    The approach produces high-fidelity human reconstructions from cellphone captures and evaluates cross-view prediction using synchronized cameras.

  • Takeaways & Limitations

    The system enables free-viewpoint selfie reconstructions of non-rigidly deforming human subjects from casual mobile-phone imagery.

  • Takeaways & Limitations

    Sharp motion discontinuities, such as at a closing mouth, are difficult for the continuous deformation MLP and can produce incorrect but valid deformations.

Abstract

from arXiv · show

We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.

1. Introduction

The paper aims to make photorealistic 3D human reconstruction accessible from casual mobile-phone captures, despite nonrigidity and difficult materials. It generalizes NeRF [39] with per-observation deformation fields, elastic regularization, and coarse-to-fine optimization, enabling free-viewpoint selfie reconstruction.

  • Motivation: The method targets accessible photorealistic 3D human modeling from casual mobile-phone captures, replacing specialized labs with many synchronized lights and cameras.The motivation is to increase the accessibility and applications of 3D modeling technology.
  • Problem: It addresses hand-held reconstruction challenges from nonrigid motion and materials such as hair, glasses, and earrings by generalizing Neural Radiance Fields (NeRF) [39] to shape deformations.The supplied passage states that the approach recovers high fidelity 3D reconstructions from short videos, although the sentence is truncated before further qualification.
  • Method: A canonical NeRF template is paired with a per-observation MLP deformation field that warps points into the canonical frame for modeling non-rigid scenes.The deformation field maps 3D points from each observation’s frame of reference into the canonical model’s frame.
  • Method: Elastic regularization constrains deformation-field distortions and over-fitting by directly regularizing Jacobian singular values computed through automatic differentiation.The formulation draws on elastic energy methods for mesh fitting.
  • Method: Coarse-to-fine optimization initially suppresses high-frequency positional-encoding components to learn smooth deformations before progressively introducing finer frequencies.This frequency-based capacity modulation is intended to make deformation-field optimization more robust.
  • Evaluation: Evaluation uses two synchronized, rigidly attached, calibrated cameras, reconstructing from one camera and predicting views from the other.The system is designed to reconstruct free-viewpoint selfies from casual mobile-phone captures.

2. Related Work

Related work spans non-rigid reconstruction, domain-specific models, and coordinate-based scene representations. The paper distinguishes its framework from concurrent deformable-NeRF methods through its conditioning, regularization, optimization, and deformation parameterization choices.

  • Non-Rigid Reconstruction: Non-rigid reconstruction separates a geometric model from an observation-dependent deformation model, progressing from ambiguous 2D keypoints to multi-view scan fusion.DynamicFusion instead jointly reconstructs scenes using a single moving RGBD camera.
  • Non-Rigid Reconstruction: Neural Volumes [31] learns deformable scenes with a voxel grid and 3D-CNN warp field but requires dozens of synchronized cameras and fails to extend to single-camera sequences.Yoon et al. handle moving-camera trajectories using monocular-depth semantic priors.
  • Domain-Specific Modeling: Domain-specific reconstruction models target faces, bodies, or animals, but can omit fine details or category attributes such as hair, eyewear, and garments.Some monocular face methods use color and RGBD cameras.
  • Coordinate-based Models: Coordinate-based models encode spatial fields in MLP weights with lower memory than discrete representations, supporting shape [16, 38, 42], scene [39], and NeRF [39] representations.NeRFs use periodic positional encoding [51] to increase representational resolution.
  • Concurrent Work: Compared with concurrent translation-field methods [44], this framework uses per-example latent conditioning [5], elastic regularization, coarse-to-fine optimization, and an improved SE(3) deformation parameterization.Other concurrent methods [29] [60] recover time-varying NeRFs with monocular-depth and flow-estimation supervision.

3. Deformable Neural Radiance Fields

The method represents a deformable scene with a canonical NeRF template and per-frame observation-to-canonical deformation fields, enabling rendering of moving subjects from casually captured images. It uses latent-conditioned SE(3) transformations, elastic regularization, and positional-encoding bandwidth annealing to improve deformation modeling and optimization.

  • Scene representation: A canonical NeRF template plus per-frame deformation fields represents non-rigid scenes by mapping observation-space points into a shared canonical space.Each frame has a learned deformation code encoding its scene state, and rendering maps sampled observation-frame points to the template.
  • Deformation field: The deformation field uses latent-conditioned dense SE(3) transforms, allowing groups of points to undergo shared rotations and translations more naturally than independent displacement vectors.The field is parameterized by an MLP that predicts screw-motion parameters conditioned on each frame’s latent code.
  • Elastic regularization: Elastic regularization addresses ambiguities that make deformation optimization under-constrained and can otherwise produce implausible results and artifacts.The regularizer controls local deformation through the Jacobian of the continuous observation-to-canonical mapping and uses a robust error to reduce outlier influence.
  • Elastic regularization: In an under-constrained capture with 20 images and one-sided camera views, elastic regularization reduces distortion and resolves deformation ambiguity.The example concerns a face capture whose cameras are biased toward one side, making the reconstruction under-constrained.
  • Optimization: The method smoothly anneals positional-encoding bandwidth because few frequencies can underfit while many frequencies can overfit the data.The bandwidth interpretation is tied to the Neural Tangent Kernel of the NeRF MLP.

4. Nerfies: Casual Free-Viewpoint Selfies

This section applies the generic non-rigid scene reconstruction method to high-quality human-subject models from casually captured selfie photos or videos, called “nerfies.”

  • The system reconstructs high-quality human-subject models from casually captured selfies, which the authors call “nerfies”.
  • Inputs are selfie-photo sequences or selfie videos captured while the user stands mostly still.
  • Users wave the camera around their face to cover viewpoints within a 45◦ cone.

5. Experiments

Experiments evaluate Nerfies with a dedicated synchronized two-camera validation rig, comparing unseen-view reconstruction against NeRF-based baselines and ablations. Nerfies achieves the best LPIPS across all sequences, reconstructs fine scene details and smooth deformations, but remains sensitive to camera registration and under-constrained geometry.

  • Evaluation Setup: The evaluation uses a rigid two-Pixel-3 rig to capture time-synchronized views, enabling unseen-view validation despite non-rigid scene changes.Front-camera captures use sub-millisecond synchronization, while back-camera videos are manually synchronized and subsampled to 5 fps; images are registered with COLMAP.
  • Ablation Study: Ablations show that stronger elastic regularization improves dynamic scenes, while removing elastic loss harms quasi-static scenes; the study also tests SE(3), background, and coarse-to-fine components.The reported ablations remove each contribution individually and all together, with λ = 0.01 compared against λ = 0.001.
  • Qualitative Results: Nerfies reconstructs details including hair, shirt wrinkles, and glasses, generalizes beyond humans, and supports smooth animations by interpolating deformation latent codes.Qualitative comparisons are provided for dynamic and quasi-static captures in Figures 10 and 11, with additional full-body results in Figure 8.
  • Limitations and Robustness: Elastic regularization reduces distortion in under-constrained captures, while reconstruction quality depends on camera registration and fails when structure-from-motion fails.The under-constrained example uses only 20 images captured mostly from one side while the head tracked the camera.

6. Conclusion … D. Additional Illustrations

The method reconstructs non-rigid scenes from casual cellphone captures using as-rigid-as-possible deformation priors and coarse-to-fine regularization, while additional illustrations examine its motion and subject generality. Its formulation uses SE(3) deformation fields, annealed positional encoding, and elastic regularization to improve deformation modeling.

  • 6. Conclusion: The method enables high-fidelity reconstructions of human subjects from casual cellphone selfie captures, with as-rigid-as-possible deformation and coarse-to-fine regularization identified as key components.The conclusion also notes future challenges involving larger or faster motion, topological variation, and training or inference speed.
  • A. Details of SE(3) Field Formulation: The deformation formulation encodes a screw axis S = (r; v) ∈ R6, recovers translation as p = Gv, and maps points through x′ = eSx.The exponential eS is expressed in homogeneous matrix form in SE(3).
  • A. Details of SE(3) Field Formulation: An SE(3) rigid-transformation field represents rotations compactly, whereas a translation field requires different parameters across points to encode the same rotation.In the star example, a translation field needs position-dependent magnitudes, while a rotation field uses a single parameter.
  • B. Details of Coarse-to-Fine Optimization: Coarse-to-fine optimization eases positional-encoding frequencies in through α, narrowing the deformation MLP’s NTK bandwidth and permitting progressively higher-frequency deformations.The normalized NTK visualization uses an 8-layer MLP of width 256 and shows narrower interpolation-kernel bandwidth as α increases.
  • C. Details of Elastic Regularization: The elastic regularizer penalizes deviations of Jacobian singular values from 1 and directly penalizes their logarithms because expansions should receive stronger penalties than equal-factor contractions.The closest rotation is obtained from the Jacobian’s SVD, with the formulation accounting for reflections through negative singular values.
  • C. Details of Elastic Regularization: The annealed positional-encoding window ranges from identity-only at α = 0 to all frequency bands active at α = 4, effectively sliding across bands as α increases.For maximum band count m = 4, each frequency-band weight changes as the window advances.
  • D. Additional Illustrations: The illustrations show that people move unintentionally while sitting still, and that the method works agnostic of subject type.Unintentional movement is visualized through predicted-depth differences and deformation magnitude at the predicted depth point.

E. Additional Implementation Details

The method uses a deformation network with coarse-to-fine positional encoding and a canonical NeRF augmented with appearance modulation, trained with Adam and scene-independent background regularization.

  • Training: Training uses Adam [27] with a learning rate exponentially decayed by 0.1 until the maximum iteration count, with configuration-specific hyperparameters in Tab. 3.The exact hyperparameters are provided for each configuration.
  • Architecture Details: The deformation network maps an encoded position, deformation code, and annealing parameter to a deformed position using one architecture across experiments.It takes γα(x), deformation code ω, and coarse-to-fine parameter α, then outputs x′.
  • Architecture Details: The canonical NeRF retains the original NeRF MLP while adding appearance code ψ with view direction to modulate appearance as in NeRF-A [36].Its width W is configuration-dependent and specified in Tab. 3.
  • Background Regularization: Background regularization samples 16384 points per iteration, jitters inputs with ε ∼N(0, 0.001), and uses a robust Geman-McClure loss with α = −2 and c = 0.001.Sampling a fixed number of background points avoids memory issues across scenes; the loss is implemented as described by Barron.
  • Implementation: The implementation extends the JAX implementation of NeRF.

F. Experiment Details · F.1. Dataset Processing

The dataset-processing pipeline filters blurry video frames and registers cameras using foreground-aware feature computation. It retains around 600 frames per capture after Laplacian-variance filtering.

  • F.1. Dataset Processing: Blurry video frames are filtered using the variance of the Laplacian.The Laplace operator uses kernel size 3 before computing the variance of the resulting image.
  • F.1. Dataset Processing: The processing pipeline combines blur-based frame selection with semantic segmentation and masked structure-from-motion registration.
  • F.1. Dataset Processing: Around 600 frames are retained for each video capture after filtering images by blur score.The blur score is computed from the variance of the Laplacian response.
  • F.1. Dataset Processing: Each image’s blur score is computed by applying a kernel-size-3 Laplace operator and measuring the variance of the resulting image.
  • F.1. Dataset Processing: Camera registration begins by computing a foreground mask with a semantic segmentation network such as DeepLabV3.
  • F.1. Dataset Processing: COLMAP computes camera registration while the foreground mask excludes foreground pixels from feature computation.

F.2. Baselines

The baselines section adapts Neural Volumes [31] and compares against NSFF [29] for single-camera deformable-scene reconstruction. Evaluations use matched camera parameters and spatial resolution, with quantitative and qualitative comparisons reported across the paper’s tables and figures.

  • Comparison to Neural Volumes: Neural Volumes [31] is adapted from synchronized multi-camera input to single-image input and trained to reconstruct the complete face-centered scene.The background-estimation branch is disabled, and the scene is scaled to a unit cube.
  • Comparison to Neural Volumes: The Neural Volumes [31] comparison uses a 128^3 voxel grid, a 32^3 warp field, and 100k training iterations for each of five sequences.All results use identical camera parameters and spatial resolution.
  • Evaluation: Baseline comparisons report SSIM, LPIPS, MS-SSIM, and PSNR using quantitative tables alongside qualitative renderings from the corresponding figures.The supplied passages identify the reported metrics and visuals but do not provide metric values or establish a winning method.
  • Comparison to NSFF: NSFF [29] models dynamic scenes by conditioning NeRF outputs and scene-flow prediction on a position-encoded time variable, whereas this method uses a deformation field.The comparison uses official NSFF code with hyperparameters tuned for the evaluated datasets.

G. Additional Results

Additional results provide qualitative reconstructions across quasi-static and dynamic sequences, while also documenting configurations and a gaze-related hollow-face illusion in reconstructed nerfies.

  • Qualitative Results: Qualitative results cover every sequence from the quantitative evaluation, including quasi-static scenes (Fig. 11) and dynamic scenes (Fig. 10).The results are presented in the paper’s Tab. 1 and Tab. 2 contexts.
  • Configurations: The FULL configuration uses full resolution for qualitative results, whereas HALF uses half resolution for quantitative evaluation and ablation studies.Table 3 provides the hyper-parameters for each configuration.
  • Gaze Effects: When gaze consistently follows the camera, the reconstructed nerfie encodes gaze as geometry, producing a hollow-face illusion that appears camera-directed despite fixed geometry.The effect is visible in the depth map and resembles the Hollow-Face illusion [22].

H. Limitations

The method struggles with topological changes, rapid motion, orientation flips, and hollow-face ambiguity. These limitations can produce discontinuous deformations, under-constrained poses, local-minimum misalignments, or incorrect geometric modeling.

  • Topological Variation: Topological Variation: Continuous MLP deformation fields struggle with topology-changing motion, such as opening a mouth, which requires a sharp deformation discontinuity.Adjacent points on opposing sides of a lip seam must map to appropriate positions despite the topology change.
  • Rapid Motion: Rapid Motion: States visible only briefly provide sparse observations, making their volumetric density under-constrained and harder to reconstruct.This limitation occurs because NeRF relies on multiple observations to constrain where density lies in the scene volume.
  • Orientation flips: Orientation flips: Non-convex rotation optimization can trap deformation fields in local minima, causing sub-optimal alignment for differently oriented views.In the Toby example, frames showing the dog’s left side are reconstructed better than frames showing its right side.
  • Hollow Face Illusion: Hollow Face Illusion: The method may model eye concavities as geometry instead of deformation when a user fixes their gaze toward the camera.This failure arises from the illusion’s ambiguity, in which concave surfaces appear convex and seem to follow the viewer’s eye.

I. 2D Deformation Experiment

The 2D toy experiment evaluates deformation fields for reconstructing randomly transformed and centrally distorted images from a single template, showing that field parameterization and positional-encoding frequency affect reconstruction quality and convergence.

  • 2D Deformation Experiment: The toy scene consists of a single image randomly translated, rotated, and nonlinearly distorted near its center, reconstructed using a 2D deformation field referencing one MLP template.The template maps normalized image coordinates x, y ∈[−1, 1] to RGB values.
  • Deformation Formulation: An SE(2) rigid transformation field faithfully reconstructs the images with a reasonable template and smooth deformation, whereas a translation field produces artifacts and messy flow.The translation field cannot recover a reasonable template, while SE(2) captures the transformations more effectively.
  • Positional Encoding Frequencies: Increasing positional-encoding frequencies improves modeling of the central nonlinear swirl but introduces artifacts from early template overfitting.With m = 1, the template reaches the correct orientation but cannot fully represent the swirl; m = 2 . . . 6 reconstructs the swirl better while causing artifacts.
Loading 2011.12948v5…