Source-linked AI summary
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video
Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman
TL;DR
Free-viewpoint rendering from a monocular video must synthesize unseen camera views of a moving person while preserving complex motion and fine appearance details. HumanNeRF learns a canonical volumetric T-pose together with skeletal and non-rigid backward motion fields, and reports higher numerical performance and visual quality than prior work across varied captures. The method nevertheless has artifacts for unseen body regions and relies on assumptions about pose estimates, motion, lighting, temporal smoothness, and segmentation.
Problem
Free-viewpoint rendering from in-the-wild monocular video must synthesize unseen views of moving people while accounting for complex poses, cloth folds, hair movement, and other fine details.
Method
HumanNeRF jointly optimizes a canonical volumetric T-pose and a backward motion field combining skeletal rigid and non-rigid motion, with pose refinement.
Results
HumanNeRF outperforms the state of the art numerically and produces significantly higher visual quality across lab, self-captured, and permitted YouTube videos.
Takeaways & Limitations
The results point toward modeling humans in motion and eventually achieving photorealistic free-viewpoint rendering from casual captures.
Takeaways & Limitations
The method has artifacts when body regions are unseen and assumes pose-dependent non-rigid motion and fairly diffuse lighting, while in-the-wild segmentation requires manual correction.
Abstract
from arXiv · showhide
We introduce a free-viewpoint rendering method -- HumanNeRF -- that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose. This task is particularly challenging, as it requires synthesizing photorealistic details of the body, as seen from various camera angles that may not exist in the input video, as well as synthesizing fine details such as cloth folds and facial appearance. Our method optimizes for a volumetric representation of the person in a canonical T-pose, in concert with a motion field that maps the estimated canonical representation to every frame of the video via backward warps. The motion field is decomposed into skeletal rigid and non-rigid motions, produced by deep networks. We show significant performance improvements over prior work, and compelling examples of free-viewpoint renderings from monocular video of moving humans in challenging uncontrolled capture scenarios.
1. Introduction
HumanNeRF targets free-viewpoint rendering of moving people from a single in-the-wild monocular video, where unseen views must preserve complex motion and appearance details. It addresses limitations of multi-view, template-based, and small-deformation methods with a data-driven approach designed for large human motions.
- Motivation: Monocular free-viewpoint rendering aims to rotate around a performer at any paused frame while synthesizing unseen views, cloth folds, hair movement, and complex poses.The paper focuses specifically on single-camera videos captured outside controlled studio settings.
- Limitations of prior work: Prior methods commonly require multi-view or studio capture, rely on SMPL templates that can artifact on clothing and complex motions, or handle only small deformations.These assumptions limit their suitability for large full-body motions such as dancing.
- HumanNeRF: HumanNeRF optimizes a canonical volumetric T-pose and a backward motion field mapping it to each video frame, combining skeletal rigid and non-rigid volumetric motion.The approach is data-driven, trained from the video itself, and includes pose refinement without template models.
- Results: HumanNeRF outperforms the state of the art numerically and produces significantly higher visual quality across lab, self-captured, and permitted YouTube videos.The evaluation spans existing datasets, outside-lab recordings, and uncontrolled online videos.
3. Representing a Human as a Neural Field
HumanNeRF represents a person with a canonical continuous appearance field and a pose-conditioned backward motion field. The motion field combines coarse skeletal deformation, learned non-rigid offsets, and pose correction to align observed frames with canonical space.
- Canonical volume: The canonical volume Fc is a continuous field that maps a canonical-space point to color and density through an MLP.The appearance field is queried at canonical coordinates after motion is inverted.
- Canonical volume: The observation-space appearance field is obtained by evaluating the canonical field at points mapped through the pose-conditioned motion field.This backward mapping supports rendering observed poses from a shared canonical representation.
- Motion field: The motion field is decomposed into skeletal motion followed by a non-rigid offset, with skeletal deformation providing coarse skinning and non-rigid motion modeling clothing-like effects.The non-rigid component is conditioned on the skeleton-driven deformation and body pose.
- Skeletal motion: Skeletal deformation uses inverse volumetric linear-blend skinning, with canonical blend-weight volumes transformed by bone rotations and translations.A single canonical set of weight volumes is used instead of separate observation-space sets, helping avoid overfitting and improve generalization.
- Skeletal motion: The blend-weight volumes are represented explicitly and generated by a CNN, with K motion channels plus a background channel constrained by channel-wise softmax.Explicit volumes avoid repeated MLP evaluations and provide trilinear-interpolation smoothness for optimization regularization.
- Pose correction: Pose correction learns relative joint-angle updates while holding joint locations fixed, then applies the corrected pose to skeleton-driven deformation.The correction is produced by an MLP conditioned on the estimated joint angles and is intended to improve alignment when pose estimates are inaccurate.
4. Optimizing a HumanNeRF
HumanNeRF jointly optimizes canonical appearance and motion parameters by rendering warped canonical fields and matching the input frames. Delayed coarse-to-fine non-rigid optimization addresses overfitting that otherwise harms unseen-view quality.
- Objective: The objective minimizes summed image-rendering losses over input frames, poses, and cameras with respect to canonical, skeletal, non-rigid, and pose-correction network parameters.The rendered field is formed by warping the canonical volume with the pose-conditioned motion field.
- Volume rendering: Volume rendering accumulates sampled colors using transmittance-weighted opacities, with opacity determined from predicted density and sample intervals.The renderer integrates the neural field along each camera ray.
- Volume rendering: Foreground likelihood scales sample opacity so points unlikely to belong to the subject contribute less to rendering.This validity weighting uses the approximate foreground probability derived from the motion representation.
- Delayed non-rigid optimization: Jointly optimizing all parameters can overfit non-rigid motion to input images, entangle it with skeletal motion, and degrade unseen-view quality.The failure occurs because part of the subject’s skeletal motion is modeled by the non-rigid field.
- Delayed non-rigid optimization: HumanNeRF disables non-rigid motion initially and gradually enables its positional-encoding frequencies using a truncated Hann window.The schedule transitions from no non-rigid optimization to full frequency bands between iteration hyperparameters Ts and Te.
- Loss and sampling: The loss combines LPIPS perceptual loss with weighted MSE, while patch-based ray sampling supplies convolution-compatible rendered patches for LPIPS.The reported settings use λ = 0.2, six 32 × 32 patches per batch, and corresponding rendered rays.
5. Results
HumanNeRF is evaluated on lab, self-captured, and YouTube videos against Neural Body and HyperNeRF, with ablations examining motion components and optimization. It achieves stronger quantitative and visual novel-view results, while pose correction, non-rigid deformation, and delayed optimization improve reconstruction and view generalization.
- 5.1. Evaluation dataset: HumanNeRF is evaluated on ZJU-MoCap, self-captured videos, and consented YouTube videos, using static-camera inputs and unseen views for evaluation.ZJU-MoCap uses one camera as input and the other 22 cameras for evaluation; in-the-wild videos use estimated poses and manually corrected segmentations.
- 5.4. Comparisons: HyperNeRF does not produce meaningful novel-view outputs in these experiments, likely because it relies on multiple moving-camera views and handles less extreme motion.The comparison concerns monocular videos with moving subjects and large body motions.
- 5.4. Comparisons: HumanNeRF outperforms Neural Body across subjects and metrics except subject 393 on PSNR, with nearly 40% average improvement in LPIPS.The method also produces sharper details on unobserved views, whereas Neural Body tends to produce blurrier results.
- 5.5. Ablation studies: Skeletal deformation alone significantly improves over Neural Body on ZJU-MoCap, while adding non-rigid deformation provides further gains.This ablation averages results over six sequences; accurate poses were provided, so pose optimization was not ablated there.
- 5.5. Ablation studies: For in-the-wild data, pose correction improves arm alignment and details, while non-rigid deformation improves clothing alignment and shape in unseen views.The visual ablation compares successive model variants with pose correction and non-rigid motion enabled.
- 5.5. Ablation studies: Delayed optimization improves skeletal/non-rigid motion decoupling; without it, the non-rigid field can absorb skeletal motion and harm unseen-view generalization.The resulting decoupling is associated with better generalization to unseen views in the ablation figures.
6. Discussion
HumanNeRF produces state-of-the-art free-viewpoint renderings of moving people from monocular video, while retaining several practical limitations and assumptions.
- The method has artifacts when part of the body is absent from the video.
- Pose correction can fail with poor initial poses or strong artifacts such as motion blur.
- Frame-by-frame poses remain temporally unsmooth even after pose correction.
- The method assumes non-rigid motion is pose-dependent and lighting is fairly diffuse.
- For in-the-wild videos, segmentation errors require manual intervention.
- The authors report state-of-the-art monocular-video free-viewpoint renderings and identify a direction toward photorealistic casual capture.
A. Derivation of Motion Bases
The motion-basis derivation defines body pose and computes observation-to-canonical transformations for each body part from skeletal joint rotations and locations.
- Body pose p=(J, Ω) comprises K joint locations J and local axis-angle joint rotations Ω.
- Given canonical and observed poses, transformation M maps each body part from observation space to canonical space.
- The transformation composes canonical-pose joint transforms with the inverse of observed-pose joint transforms along the kinematic-tree parent set.
- The exponential map converts each axis-angle rotation vector into a 3×3 rotation matrix using Rodrigues’ rotation formula.
- The body-part rotation R_k and translation t_k are extracted from the homogeneous transformation M_k.
B. Network Architecture
HumanNeRF uses separate neural components for canonical appearance, non-rigid motion, pose correction, and motion-weight-volume generation.
- Figures 9–12 specify the architectures for canonical appearance, non-rigid motion, pose correction, and motion-weight-volume generation.
- The canonical MLP maps positional encoding γ(x) to color c and density σ using an 8-layer, width-256 network.
- The non-rigid motion MLP takes joint rotations and positional encoding to predict positional offset ∆x.
- The pose correction MLP refines initial poses from joint angles using a 4-layer network with width 256.
- The motion-weight-volume network transforms a 256-dimensional latent code through transposed convolutions into a 32×32×32×25 volume.
C. Motion Field Decomposition
The motion field is decomposed into skeletal rigid and non-rigid components, with experiments selecting the design that applies skeletal warping before non-rigid conditioning.
- The method decomposes motion into skeletal rigid motion and non-rigid motion and evaluates three formulations.
- Design 1 conditions both skeletal and non-rigid motion on the observed point position x and sums their outputs.
- Design 2 conditions skeletal motion on the position adjusted by non-rigid motion, x+T_NR(x).
- Design 3 conditions non-rigid motion on the point after skeletal warping, producing T_skel(x)+T_NR(T_skel(x)).
- Design 3 produces the best novel-view-synthesis quality on PeopleSnapshot and is adopted in the final method.
D. Additional Implementation Details
The method uses several implementation choices to improve foreground separation, motion representation, pose handling, and ray allocation during optimization.
- Skeletal rigid motion is generated from a deep-network offset relative to the global motion representation and canonical Gaussian bone weights.The canonical motion weights are computed with a softmax combining learned offsets and logarithmic Gaussian weights.
- Global body orientation is represented by rotating the camera, preserving the estimated bounding box as subject orientation changes.Axis-aligned bounding boxes are used for ease of implementation, although they vary with rotated global orientation.
- Randomly assigned solid backgrounds help separate the foreground subject from the background during optimization.The same random solid background color is applied to both the rendering and input image.
- The final layer of the non-rigid-motion and pose-correction MLPs is initialized with U(−10−5, 10−5), keeping offsets near zero and pose rotations near identity.
- Foreground pixels receive probability 0.8 during random ray sampling, while background pixels receive probability 0.2.The sampling strategy emphasizes rays associated with the segmented subject.
E.1. Additional Results
Additional experiments show strong performance on unseen ZJU-MoCap subjects, diminishing returns beyond 112 training frames, and limitations from pose correction and non-rigid clothing dynamics.
- E.1. Additional Results: HumanNeRF substantially captures unseen-region appearance details on subjects 313, 315, and 390, while Neural Body produces blurry results.The method consistently outperforms NeuralBody, particularly under LPIPS.
- E.2. Ablation Study on Sequence Length: Using more frames improves quality, but gains become unclear beyond 112 frames in the sequence-length study.The experiment uses subject 392 with training sequences ranging from 19 to 556 frames.
- E.3. Canonical Appearance: The recovered canonical T-pose appearance is shown for ZJU-MoCap, self-captured, and YouTube videos.
- E.4. Limitations: Pose correction may fail under artifacts such as strong motion blur.
- E.4. Limitations: Non-rigid clothing motion may remain unrecovered when jacket movement depends on the temporal dynamics of the subject.
- F. Societal Impact: The method is intended to reproduce observed motion sequences with unseen viewpoints, not to re-animate subjects in novel poses.Public deployment should identify viewpoint-adjusted imagery as computer-generated.