Nerfies: Deformable Neural Radiance Fields

Motivation

• The method targets accessible photorealistic 3D human modeling from casual mobile-phone captures, replacing specialized labs with many synchronized lights and cameras.The motivation is to increase the accessibility and applications of 3D modeling technology.Source: §1 · p. 1

Problem

• It addresses hand-held reconstruction challenges from nonrigid motion and materials such as hair, glasses, and earrings by generalizing Neural Radiance Fields (NeRF) [39] to shape deformations.The supplied passage states that the approach recovers high fidelity 3D reconstructions from short videos, although the sentence is truncated before further qualification.Source: §1 · p. 1

Method

• A canonical NeRF template is paired with a per-observation MLP deformation field that warps points into the canonical frame for modeling non-rigid scenes.The deformation field maps 3D points from each observation’s frame of reference into the canonical model’s frame.Source: §1 · p. 2
• Elastic regularization constrains deformation-field distortions and over-fitting by directly regularizing Jacobian singular values computed through automatic differentiation.The formulation draws on elastic energy methods for mesh fitting.Source: §1 · p. 2
• Coarse-to-fine optimization initially suppresses high-frequency positional-encoding components to learn smooth deformations before progressively introducing finer frequencies.This frequency-based capacity modulation is intended to make deformation-field optimization more robust.Source: §1 · p. 2

Evaluation

• Evaluation uses two synchronized, rigidly attached, calibrated cameras, reconstructing from one camera and predicting views from the other.The system is designed to reconstruct free-viewpoint selfies from casual mobile-phone captures.Source: §1 · p. 2

Non-Rigid Reconstruction

• Non-rigid reconstruction separates a geometric model from an observation-dependent deformation model, progressing from ambiguous 2D keypoints to multi-view scan fusion.DynamicFusion instead jointly reconstructs scenes using a single moving RGBD camera.Source: §2 · p. 2
• Neural Volumes [31] learns deformable scenes with a voxel grid and 3D-CNN warp field but requires dozens of synchronized cameras and fails to extend to single-camera sequences.Yoon et al. handle moving-camera trajectories using monocular-depth semantic priors.Source: §2 · p. 2

Domain-Specific Modeling

• Domain-specific reconstruction models target faces, bodies, or animals, but can omit fine details or category attributes such as hair, eyewear, and garments.Some monocular face methods use color and RGBD cameras.Source: §2 · p. 2

Coordinate-based Models

• Coordinate-based models encode spatial fields in MLP weights with lower memory than discrete representations, supporting shape [16, 38, 42], scene [39], and NeRF [39] representations.NeRFs use periodic positional encoding [51] to increase representational resolution.Source: §2 · p. 2

Concurrent Work

• Compared with concurrent translation-field methods [44], this framework uses per-example latent conditioning [5], elastic regularization, coarse-to-fine optimization, and an improved SE(3) deformation parameterization.Other concurrent methods [29] [60] recover time-varying NeRFs with monocular-depth and flow-estimation supervision.Source: §2 · p. 2

Scene representation

• A canonical NeRF template plus per-frame deformation fields represents non-rigid scenes by mapping observation-space points into a shared canonical space.Each frame has a learned deformation code encoding its scene state, and rendering maps sampled observation-frame points to the template.Source: §3 · p. 3, §3.2 · p. 3

Deformation field

• The deformation field uses latent-conditioned dense SE(3) transforms, allowing groups of points to undergo shared rotations and translations more naturally than independent displacement vectors.The field is parameterized by an MLP that predicts screw-motion parameters conditioned on each frame’s latent code.Source: §3.2 · pp. 3–4

Elastic regularization

• Elastic regularization addresses ambiguities that make deformation optimization under-constrained and can otherwise produce implausible results and artifacts.The regularizer controls local deformation through the Jacobian of the continuous observation-to-canonical mapping and uses a robust error to reduce outlier influence.Source: §3.3 · pp. 4–5
• In an under-constrained capture with 20 images and one-sided camera views, elastic regularization reduces distortion and resolves deformation ambiguity.The example concerns a face capture whose cameras are biased toward one side, making the reconstruction under-constrained.Source: §3.3 · p. 4

Optimization

• The method smoothly anneals positional-encoding bandwidth because few frequencies can underfit while many frequencies can overfit the data.The bandwidth interpretation is tied to the Neural Tangent Kernel of the NeRF MLP.Source: §3.5 · p. 5

• The system reconstructs high-quality human-subject models from casually captured selfies, which the authors call “nerfies”.Source: §4 · p. 6
• Inputs are selfie-photo sequences or selfie videos captured while the user stands mostly still.Source: §4 · p. 6
• Users wave the camera around their face to cover viewpoints within a 45◦ cone.Source: §4 · p. 6

Evaluation Setup

• The evaluation uses a rigid two-Pixel-3 rig to capture time-synchronized views, enabling unseen-view validation despite non-rigid scene changes.Front-camera captures use sub-millisecond synchronization, while back-camera videos are manually synchronized and subsampled to 5 fps; images are registered with COLMAP.Source: §5.2 · pp. 6–7, Figure 9 · p. 6

Ablation Study

• Ablations show that stronger elastic regularization improves dynamic scenes, while removing elastic loss harms quasi-static scenes; the study also tests SE(3), background, and coarse-to-fine components.The reported ablations remove each contribution individually and all together, with λ = 0.01 compared against λ = 0.001.Source: §5.3 · p. 7, Table 1 · p. 8

Qualitative Results

• Nerfies reconstructs details including hair, shirt wrinkles, and glasses, generalizes beyond humans, and supports smooth animations by interpolating deformation latent codes.Qualitative comparisons are provided for dynamic and quasi-static captures in Figures 10 and 11, with additional full-body results in Figure 8.Source: Figure 8 · p. 6, §5.3 · p. 7, Figure 10 · p. 8

Limitations and Robustness

• Elastic regularization reduces distortion in under-constrained captures, while reconstruction quality depends on camera registration and fails when structure-from-motion fails.The under-constrained example uses only 20 images captured mostly from one side while the head tracked the camera.Source: §5.3 · pp. 7, 10

6. Conclusion

• The method enables high-fidelity reconstructions of human subjects from casual cellphone selfie captures, with as-rigid-as-possible deformation and coarse-to-fine regularization identified as key components.The conclusion also notes future challenges involving larger or faster motion, topological variation, and training or inference speed.Source: §6 · p. 10

A. Details of SE(3) Field Formulation

• The deformation formulation encodes a screw axis S = (r; v) ∈ R6, recovers translation as p = Gv, and maps points through x′ = eSx.The exponential eS is expressed in homogeneous matrix form in SE(3).Source: A. Details of SE(3) Field Formulation · p. 10
• An SE(3) rigid-transformation field represents rotations compactly, whereas a translation field requires different parameters across points to encode the same rotation.In the star example, a translation field needs position-dependent magnitudes, while a rotation field uses a single parameter.Source: A. Details of SE(3) Field Formulation · p. 10, Figure 13 · p. 10

B. Details of Coarse-to-Fine Optimization

• Coarse-to-fine optimization eases positional-encoding frequencies in through α, narrowing the deformation MLP’s NTK bandwidth and permitting progressively higher-frequency deformations.The normalized NTK visualization uses an 8-layer MLP of width 256 and shows narrower interpolation-kernel bandwidth as α increases.Source: Figure 14 · p. 11, B. Details of Coarse-to-Fine… · p. 11

C. Details of Elastic Regularization

• The elastic regularizer penalizes deviations of Jacobian singular values from 1 and directly penalizes their logarithms because expansions should receive stronger penalties than equal-factor contractions.The closest rotation is obtained from the Jacobian’s SVD, with the formulation accounting for reflections through negative singular values.Source: C. Details of Elastic Regularization · p. 11
• The annealed positional-encoding window ranges from identity-only at α = 0 to all frequency bands active at α = 4, effectively sliding across bands as α increases.For maximum band count m = 4, each frequency-band weight changes as the window advances.Source: B. Details of Coarse-to-Fine… · p. 10, Figure 15 · p. 11

D. Additional Illustrations

• The illustrations show that people move unintentionally while sitting still, and that the method works agnostic of subject type.Unintentional movement is visualized through predicted-depth differences and deformation magnitude at the predicted depth point.Source: D. Additional Illustrations · p. 11, Figure 16 · p. 11

Training

• Training uses Adam [27] with a learning rate exponentially decayed by 0.1 until the maximum iteration count, with configuration-specific hyperparameters in Tab. 3.The exact hyperparameters are provided for each configuration.Source: E. Additional Implementation Details · p. 11

Architecture Details

• The deformation network maps an encoded position, deformation code, and annealing parameter to a deformed position using one architecture across experiments.It takes γα(x), deformation code ω, and coarse-to-fine parameter α, then outputs x′.Source: Figure 18 · p. 12
• The canonical NeRF retains the original NeRF MLP while adding appearance code ψ with view direction to modulate appearance as in NeRF-A [36].Its width W is configuration-dependent and specified in Tab. 3.Source: Figure 19 · p. 12

Background Regularization

• Background regularization samples 16384 points per iteration, jitters inputs with ε ∼N(0, 0.001), and uses a robust Geman-McClure loss with α = −2 and c = 0.001.Sampling a fixed number of background points avoids memory issues across scenes; the loss is implemented as described by Barron.Source: E. Additional Implementation Details · p. 12

Implementation

• The implementation extends the JAX implementation of NeRF.Source: E. Additional Implementation Details · p. 12

F.1. Dataset Processing

• Blurry video frames are filtered using the variance of the Laplacian.The Laplace operator uses kernel size 3 before computing the variance of the resulting image.Source: §F.1 · p. 12
• The processing pipeline combines blur-based frame selection with semantic segmentation and masked structure-from-motion registration.Source: §F.1 · p. 12
• Around 600 frames are retained for each video capture after filtering images by blur score.The blur score is computed from the variance of the Laplacian response.Source: §F.1 · p. 12
• Each image’s blur score is computed by applying a kernel-size-3 Laplace operator and measuring the variance of the resulting image.Source: §F.1 · p. 12
• Camera registration begins by computing a foreground mask with a semantic segmentation network such as DeepLabV3.Source: §F.1 · p. 12
• COLMAP computes camera registration while the foreground mask excludes foreground pixels from feature computation.Source: §F.1 · p. 12

Comparison to Neural Volumes

• Neural Volumes [31] is adapted from synchronized multi-camera input to single-image input and trained to reconstruct the complete face-centered scene.The background-estimation branch is disabled, and the scene is scaled to a unit cube.Source: §F.2 · p. 12
• The Neural Volumes [31] comparison uses a 128^3 voxel grid, a 32^3 warp field, and 100k training iterations for each of five sequences.All results use identical camera parameters and spatial resolution.Source: §F.2 · p. 12

Evaluation

• Baseline comparisons report SSIM, LPIPS, MS-SSIM, and PSNR using quantitative tables alongside qualitative renderings from the corresponding figures.The supplied passages identify the reported metrics and visuals but do not provide metric values or establish a winning method.Source: Table 2 · p. 13, §F.2 · p. 13

Comparison to NSFF

• NSFF [29] models dynamic scenes by conditioning NeRF outputs and scene-flow prediction on a position-encoded time variable, whereas this method uses a deformation field.The comparison uses official NSFF code with hyperparameters tuned for the evaluated datasets.Source: §F.2 · p. 13

Qualitative Results

• Qualitative results cover every sequence from the quantitative evaluation, including quasi-static scenes (Fig. 11) and dynamic scenes (Fig. 10).The results are presented in the paper’s Tab. 1 and Tab. 2 contexts.Source: G. Additional Results · p. 13

Configurations

• The FULL configuration uses full resolution for qualitative results, whereas HALF uses half resolution for quantitative evaluation and ablation studies.Table 3 provides the hyper-parameters for each configuration.Source: Table 3 · p. 13

Gaze Effects

• When gaze consistently follows the camera, the reconstructed nerfie encodes gaze as geometry, producing a hollow-face illusion that appears camera-directed despite fixed geometry.The effect is visible in the depth map and resembles the Hollow-Face illusion [22].Source: Figure 21 · p. 13

Topological Variation

• Topological Variation: Continuous MLP deformation fields struggle with topology-changing motion, such as opening a mouth, which requires a sharp deformation discontinuity.Adjacent points on opposing sides of a lip seam must map to appropriate positions despite the topology change.Source: H. Limitations · pp. 13–14

Rapid Motion

• Rapid Motion: States visible only briefly provide sparse observations, making their volumetric density under-constrained and harder to reconstruct.This limitation occurs because NeRF relies on multiple observations to constrain where density lies in the scene volume.Source: H. Limitations · p. 14, Figure 22 · p. 14

Orientation flips

• Orientation flips: Non-convex rotation optimization can trap deformation fields in local minima, causing sub-optimal alignment for differently oriented views.In the Toby example, frames showing the dog’s left side are reconstructed better than frames showing its right side.Source: H. Limitations · p. 14

Hollow Face Illusion

• Hollow Face Illusion: The method may model eye concavities as geometry instead of deformation when a user fixes their gaze toward the camera.This failure arises from the illusion’s ambiguity, in which concave surfaces appear convex and seem to follow the viewer’s eye.Source: H. Limitations · pp. 14–15

2D Deformation Experiment

• The toy scene consists of a single image randomly translated, rotated, and nonlinearly distorted near its center, reconstructed using a 2D deformation field referencing one MLP template.The template maps normalized image coordinates x, y ∈[−1, 1] to RGB values.Source: §I · p. 15

Deformation Formulation

• An SE(2) rigid transformation field faithfully reconstructs the images with a reasonable template and smooth deformation, whereas a translation field produces artifacts and messy flow.The translation field cannot recover a reasonable template, while SE(2) captures the transformations more effectively.Source: §I · p. 15

Positional Encoding Frequencies

• Increasing positional-encoding frequencies improves modeling of the central nonlinear swirl but introduces artifacts from early template overfitting.With m = 1, the template reaches the correct orientation but cannot fully represent the swirl; m = 2 . . . 6 reconstructs the swirl better while causing artifacts.Source: §I · p. 15