PaperlayerView exact revision on arXiv

Source-linked AI summary

A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

Shih-Yang Su, Frank Yu, Michael Zollhoefer, Helge Rhodin

arXiv:2102.06199v3cs.CVcs.GR

TL;DR

Learning detailed 3D human representations from monocular video is challenging because bodies vary in shape and appearance and move non-rigidly. A-NeRF learns a skeleton-relative neural body model from unlabelled videos and, with rough 3D pose initialization, reaches detail previously attained with parametric surface models or multi-view approaches.

  • Problem

    Learning 3D human representations from 2D observations is challenging because bodies vary in shape and appearance and move non-rigidly, while most 3D models require 3D labels.

    Source: §1 · p. 1

  • Method

    A-NeRF extends NeRF to learn user-specific neural body models and skeleton poses from unlabelled videos, using a skeleton-relative latent representation to address ill-posed inverse mapping.

    Source: §1 · pp. 1–2

  • Results

    With rough 3D pose estimates for initialization, A-NeRF learns from monocular video a detailed neural body model reaching detail previously attained with parametric surface models or multi-view approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A-NeRF supports motion capture, character animation, and appearance and motion transfer, and its pose refinement improves existing monocular skeleton reconstruction.

    Source: §1 · p. 2

    A-NeRF is computationally costly for long sequences and multiple actors, and learning pose dependencies requires seeing the person from all sides in varying poses.

    Source: §4 · p. 10

Abstract

from arXiv · show

While deep learning reshaped the classical motion capture pipeline with feed-forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets. Project website: https://lemonatsu.github.io/anerf/ .

1 Introduction

A-NeRF learns a user-specific neural 3D body model and skeleton pose from unlabelled videos, extending NeRF to articulated motion. It represents points in coordinates relative to an articulated skeleton, using one embedding per bone to address the ill-posed inverse mapping from world coordinates.

Introduction

  • Learning 3D human representations from 2D observations is challenging because humans vary in shape and appearance and move non-rigidly, while most 3D models require labels unavailable from ordinary images.Existing generative motion-capture approaches refine feed-forward estimates using personalized scans [20] or laser-scan-trained parametric models [5], but require a restrictive shape model or personal scan.Source: §1 · p. 1
  • A-NeRF learns a user-specific neural body model from unlabelled videos and refines an initial articulated skeleton pose without requiring a template body model.The method extends NeRF to single videos and articulated motion; Figure 1 depicts refinement from single or multiple views without tedious camera calibration.Source: §1 · p. 1, Figure 1 · p. 2
  • Because world-space points cannot be uniquely assigned to body parts, A-NeRF uses one skeleton-relative embedding per bone to overparameterize and resolve the ill-posed inverse mapping.This reparameterization provides local coordinates relative to the articulated skeleton and lets the network combine body-shape and appearance constraints across a sequence.Source: §1 · p. 2
  • From monocular video with rough 3D pose initialization, the combined method learns a neural body model with detail previously attained only by parametric surface models or multi-view approaches.Source: §1 · p. 2
  • A-NeRF targets a known training video without ground truth and supports motion capture, character animation, appearance and motion transfer, and dynamic novel-view synthesis.Its pose refinement improves on existing monocular skeleton reconstruction; its synthesized illumination is plausible but non-physical, and relighting requires additional steps.Source: §1 · p. 2
  • Self-supervised personalized body modeling could include people and activities underrepresented in supervised datasets, but risks creating 3D models without consent.The authors urge use only of datasets collected to develop and validate motion-capture algorithms.Source: §1 · p. 2

2 Related Work

The related work spans human pose and shape estimation, human modeling, and neural scene representations [65].

Discriminative Human Pose Estimation

Surface-based Generative Body Models

  • Surface-based models constrain plausible shape and motion with deformable meshes [20] or scan-learned parameters [11], whereas A-NeRF is surface-free and volumetrically rendered.Closest methods texture and geometrically refine a pre-existing rigged quadruped mesh for zebra images or use optical flow to refine human pose.Source: §2 · p. 3

Implicit Body Models

  • Implicit body models use level-sets [61] or Gaussian densities [21] for differentiable pose, shape, and appearance refinement, but their primitives provide only rough approximations.Source: §2 · p. 3

Neural Scene Representations

  • Neural scene representations span meshes [31], point clouds [2] [41], sphere sets [27], and dense grids [32], with limits in expressiveness or memory.Unconstrained positional-encoded MLPs [42] represent arbitrary 3D points; A-NeRF learns an articulated body from estimated poses and uncalibrated cameras, while learned rendering can be inconsistent with scarce data [43].Source: §2 · p. 3

Related articulated body models

  • Unlike NASA [13] and NeuralBody, A-NeRF learns a volumetric model with appearance and rendering, needs no surface supervision or initialization, conditions pose differently, and refines pose from unconstrained single videos.Source: §2 · p. 3

3 Formulation

A-NeRF represents a moving human with a neural radiance field conditioned on skeleton-relative coordinates, enabling joint refinement of pose and body appearance from image reconstruction. Its formulation combines ray-marched rendering with a pose-and-temporal regularized objective.

3 Formulation

  • The model optimizes pose θ and body parameters φ against image reconstruction, with an L1 data term, an initial-pose regularizer, and temporal smoothness between consecutive poses.The pose prior tolerates small shifts up to ϵ = 0.01; hyperparameters balance the three terms.Source: §3 · p. 4

3.1 NeRF and A-NeRF Image Formation Model

  • A-NeRF defines the human as a neural function over 3D points and view directions, then renders pixels by ray marching and accumulating contributions from sampled points.For static-camera videos, the background color is estimated as the median pixel color across the video.Source: §3.1 · pp. 4–5

3.2 Articulated Skeleton Pose Model

  • The skeleton encodes connectivity and fixed bone lengths in a rest pose, while each frame’s pose defines a bone-specific affine transform between local and world coordinates.It can be initialized using any skeleton pose estimator and does not include a parametric surface model.Source: §3.2 · p. 5

3.3 A-NeRF Skeleton-Relative Encoding

  • A-NeRF transforms each query location and viewing direction into bone-relative coordinates before NeRF predicts color and opacity, avoiding the need to invert surface deformation.The embedding represents a query relative to every bone, overparameterizing its position to handle complex interactions while encoding articulated motion.Source: §3.3 · p. 5
  • The final embedding combines cutoff-weighted positional encodings of per-bone distances and ray directions with relative direction vectors, limiting the influence of distant bones.Relative direction is passed directly without positional encoding; the cutoff makes the embedding local to nearby bones.Source: §3.3 · pp. 5–6
  • Reference-pose encoding assigns a query to its closest bone but cannot capture pose-dependent effects such as muscle bulging and is ambiguous at equal distances; encoding it relative to every bone avoids that single-part association.The final model favors per-bone distances over full relative positions because they are lower-dimensional, suit spherically shaped limbs, and improve reconstruction accuracy.Source: §3.3 · pp. 5–6
  • Transforming viewing rays relative to each bone and optimizing a per-image appearance code lets A-NeRF plausibly model dynamic lighting effects in the input images.The ray-direction encoding uses the rotational part of each bone’s inverse transform.Source: §3.3 · p. 6
  • The chosen embedding is invariant to global person shifts and rotations, preserves piece-wise rigidity, and allows pose-dependent deformation; Figure 3 reports improved consistency under rotation with the full model working best.The figure contrasts the skeleton-relative encoding with original NeRF and direct pose conditioning, which show artifacts or degraded geometry across varied poses.Source: Figure 3 · p. 4, §3.3 · p. 6

4 Evaluation

A-NeRF evaluation shows that pose refinement improves pose estimates and held-out visual quality, while the generative model supports novel views, animation, and volumetric reconstruction from monocular video. The experiments also report benefits from multi-view refinement and identify limits in computation time and training-pose coverage.

Novel-View-Synthesis and Character Animation

  • Trained on single or multiple uncalibrated videos, A-NeRF enables novel-view synthesis and character animation through pose changes or motion transfer, without 3D surface or multi-view supervision.Figure 4 shows novel camera views, while Figure 5 shows pose transfer and animation with body parts fixed.Source: §4 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

Human Pose Estimation

  • On Human3.6M, A-NeRF improves PA-MPJPE over the SPIN [26] initialization from 42.7 to 39.3 (8.0%) on Protocol I and 41.9 to 39.6 (5.5%) on Protocol II; Protocol II PA-Wrist improves 14% (9.2 mm).On MPI-INF-3DHP, A-NeRF provides moderate improvements over baseline estimates despite limited training frames and poses; unlike SMPLify [8], it does not degrade the 3D pose while fitting 2D joint estimates.Source: Table 1 · p. 8, §4 · pp. 8–9

Visual Quality Comparison

  • A-NeRF significantly improves held-out visual reconstruction over NeuralBody [49] and its own unrefined variant on Human3.6M and MonoPerfCap.Without refinement, noisy joint estimates lead models to predict mean pixel values; refined A-NeRF renders limbs and faces more plausibly with fewer artifacts.Source: §4 · pp. 8–9, Table 2 · p. 9, Figure 7 · p. 10

Video-based volumetric reconstruction

  • From monocular video, A-NeRF reconstructs detailed volumetric bodies without a predefined surface template or geometric smoothness term.The density model includes details such as headphones and a baseball cap that offsets to a parametric surface model could not capture.Source: §4 · p. 9, Figure 6 · p. 9

Multi-view extension

  • A-NeRF can leverage multi-view pose refinement even without ground-truth camera calibration.The evaluation reports this extension for Human3.6M Protocol II.Source: Table 1 · p. 8, §4 · p. 9

Ablation study

  • Relative-position embedding yields only half as good pose refinements as the proposed radial embedding, while embedding choices keep dimensionality moderate and match or improve PSNR and SSIM.With accurate poses and a fixed image count, a long diverse-pose video achieves visual quality comparable to multiple shorter clips.Source: §4 · p. 10

Limitations and Failure Cases

  • A-NeRF’s main bottleneck is computation time, and learning pose dependencies requires seeing the person from all sides in varying poses.Retargeting an extreme, unseen breakdance pose to a model trained on ordinary walking motions produces artifacts because the pose is far from the training distribution.Source: §4 · p. 10

5 Conclusion · Appendix

The appendix collects visualizations, additional qualitative and quantitative results, ablations, and details on view encoding, datasets, and implementation.

Appendix

  • It visualizes learned A-NeRF body geometry and adds qualitative results for novel-view synthesis and pose refinement.The supplemental video also shows results in motion.Source: Appendix · p. 16
  • It reports comprehensive ablation studies of A-NeRF.Source: Appendix · p. 16
  • It compares A-NeRF quantitatively with related work on a dataset with perfect training poses.Source: Appendix · p. 16
  • It justifies the view encoding choice for modeling view-dependent effects of the human body.Source: Appendix · p. 16
  • It provides additional dataset information and implementation details.Source: Appendix · p. 16
  • Real faces and their reconstructions are blurred for anonymity.Source: Appendix · p. 16

A Geometry Visualization

A-NeRF learns convincing body geometry without predefined template meshes, including accessories and geometry consistent across front and back when the person is seen from behind. Fine structures remain challenging, and the surface is not regularized to be smooth.

A Geometry Visualization

  • A-NeRF learns convincing body geometry without relying on predefined template meshes.The geometry is learned from monocular video.Source: A Geometry Visualization · p. 16
  • The surface-free model captures accessories such as headphones, caps, quivers, and arrows, while maintaining consistent geometry across viewing directions.Front-to-back consistency is reported when the person is seen from the back; the examples include accessories often absent from human surface models.Source: A Geometry Visualization · p. 16, Figure A2 · p. 17
  • Thin structures such as arrow shafts remain challenging, and the learned surface is not regularized to be smooth.Real faces and their reconstructions are blurred in the figures for anonymity.Source: A Geometry Visualization · p. 17, Figure A2 · p. 17

B Additional Qualitative Results for A-NeRF

Additional results show plausible novel views from different angles, including appearance details absent from human surface models, and report pose-accuracy improvements and pose-refinement effects.

B Additional Qualitative Results for A-NeRF

  • A-NeRF produces plausible novel views from different angles and reconstructs small structures such as arrows, quivers, and caps that human surface models do not model.These structures are among the most difficult to reconstruct; the examples appear in the first two rows of Figure A2.Source: B Additional Qualitative Results for… · p. 17
  • On Human 3.6M Protocol I, A-NeRF improves PA-MPJPE for joints with high initial errors, including elbows, wrists, knees, and ankles.Table A1 states that higher is better.Source: Table A1 · p. 18
  • On Human 3.6M S9 Protocol II2, Rel. Dist. encoding helps refine estimated poses, and Cutoff can further improve refinement.Table A2 evaluates the influence of pose refinement.Source: Table A2 · p. 18

C Additional Pose Refinement Results for A-NeRF · D Ablation studies

Additional pose-refinement results show A-NeRF estimates poses that align better with training images; the ablation section frames quantitative evaluation of pose-estimation accuracy and image-generation quality.

C Additional Pose Refinement Results for A-NeRF

D Ablation studies

  • The ablation studies quantify how the method’s contributions affect pose-estimation accuracy and image-generation quality, supporting summary points i), iii), and iv).Source: D Ablation studies · p. 18

D.1 Pose Estimation Accuracy Evaluation

On Human3.6M, pose refinement is evaluated with PA-MPJPE; the findings emphasize end-effector improvements and the benefits of radial-distance encoding.

Joint-wise PA-MPJPE Improvements

  • A-NeRF most improved joints with high initial error, particularly wrists, elbows, knees, and ankles, underscoring the value of pose refinement for precise end-effector positioning.Source: §D.1 · p. 18

Impact of Radial Distance Encoding

  • Rel. Dist. encoding improved wrist PA-MPJPE by 57% over Rel. Pos. and overall PA-MPJPE by 0.56, as radial information outperformed position-only encoding.Rel. Pos. encoding did not improve overall PA-MPJPE and yielded only half as good pose refinements as Rel. Dist.Source: §D.1 · p. 18

Impact of Cutoff on Refinement

  • Adding Cutoff improved wrist pose accuracy by 15% over the model without Cutoff.Source: §D.1 · p. 18

Impact of Pose Regularization

  • Pose regularization moderately improved pose refinement but was not essential to the learning framework.Source: §D.1 · p. 18

Directional Encoding Impact

  • Directional encoding had a small but noticeable impact and worked best when ray direction was represented relative to bone coordinates.Source: Table A3 · p. 20

Query Encoding Trade-off

  • World-coordinate query encoding failed on motions and consumed substantial memory, while skeleton-relative positions worked at high dimensionality; combining distance and direction performed well on both aspects.These encoding models were evaluated without Cutoff.Source: Table A4 · p. 20

D.2 Image Generation Quality Evaluation · E Visual Comparison to NeuralBody. · F Discussion on View-dependent Effects

Across these evaluations, A-NeRF’s image quality benefits from skeleton-relative directional encoding and pose-diverse training. Its view-dependent-effects design combines per-image illumination codes with Rel. Ray., though the resulting lighting is plausible rather than physically correct.

D.2 Image Generation Quality Evaluation

  • The proposed Rel. Ray. encoding offers better image quality than the alternatives evaluated in Table A3.The supplied passage does not give numerical scores for this comparison.Source: §D.2 · p. 20
  • Bone-relative position encoding significantly outperforms direct world- and joint-position conditioning, but offers no gain over Rel. Dist. + Rel. Dir. despite its much greater dimensionality.The direct-conditioning baselines reach 0.7 SSIM by rendering background colors; the proposed embeddings keep dimensionality moderate while matching or improving PSNR against higher-dimensional variants.Source: §D.2 · p. 20
  • Rel. Dir. encoding outperforms direct conditioning on θ; distance-based encoding is compact (360 dimensions) but yields lower PSNR and SSIM, and is insufficient for skeleton-relative query locations unless paired with direction information (full model: 72 dimensions, without positional encoding).Source: §D.2 · p. 20, Table A5 · p. 21
  • With accurate poses, A-NeRF learns as well from a single view as from multiple views, while additional views help only when pose diversity is insufficiently provided.For a fixed image count, one long video with diverse poses gives visual quality comparable to multiple shorter multi-view recordings.Source: Table A6 · p. 21, §D.2 · p. 21

E Visual Comparison to NeuralBody.

  • For long sequences and retargeting to diverse poses, A-NeRF works better than NeuralBody, which produces slightly deformed or expanded body features on poses unlike its training data.Both methods were trained on SURREAL with accurate poses and ground-truth cameras.Source: E Visual Comparison to NeuralBody · p. 21, Table A7 · p. 22

F Discussion on View-dependent Effects

  • Because estimated cameras make lighting inconsistent across frames, A-NeRF uses per-image codes for illumination and Rel. Ray. to encode outgoing ray direction relative to each body part.The per-image code represents illumination effects as a black-box function, while Rel. Ray. helps explain illumination on the human body.Source: F Discussion on View-dependent Effects · p. 21
  • World Ray reduces performance and causes novel-view flickering, No Ray performs worst, and per-image codes combined with Rel. Ray. perform best, although the resulting illumination is not physically correct.The authors identify explicit illumination modeling and relighting as future work.Source: F Discussion on View-dependent Effects · p. 22, Table A8 · p. 22

G Dataset Details

The dataset details specify Human 3.6M evaluation protocols and synthetic SURREAL and Mixamo data used in the experiments.

Human 3.6M

  • Human 3.6M uses PA-MPJPE on every 5th frontal-camera frame for Protocol I and every 64th frame across all cameras for Protocol II.Protocol I follows [23] [25] [26]; Protocol II follows [44].Source: G Dataset Details · p. 22

SURREAL

  • SURREAL uses 10,800 512 × 512 training images and 1,500 test images from 1,500 poses in a 4-1 train-test split.Training poses are rendered from 9 cameras and testing poses from 5 cameras; the synthetic data is used to examine factors affecting visual quality, with poses from.Source: G Dataset Details · p. 22

Mixamo

  • Mixamo supplies two synthetic subjects, James and Archer, each rendered across three motions and varying camera angles, with 1,130 training images per subject.Pose and camera parameters are extracted with [26].Source: G Dataset Details · p. 22

H Implementation Details

A-NeRF is trained from unlabelled videos by initializing per-frame poses, jointly optimizing pose and neural shape/appearance, then freezing pose for a final appearance-focused phase. The implementation also specifies skeleton, multi-view, sampling, architecture, and tuning choices.

Training procedure

  • Poses initialized with [26] are optimized with the neural parameters for 500k iterations, then frozen while the data term trains the parameters for another 200k iterations.Training uses 3,072 randomly sampled rays per batch; gradient updates are accumulated so each frame is expected to be sampled at least 30 times. Training takes approximately 60 hours on two Nvidia Tesla V100 32GB GPUs.Source: H Implementation Details · pp. 22–23

Skeleton Representation

  • The skeleton encodes connectivity and static bone lengths in a 3D-joint rest pose, with per-frame relative joint rotations and global position defining motion.The authors test 6D overparameterized rotations and axis-angle rotations; the skeleton is equivalent to SMPL without its parametric surface model and can be initialized by any skeleton pose estimator.Source: H Implementation Details · pp. 23–24

Multi-view extension

  • For multi-view pose refinement, joint rotations are initialized by averaging estimates across views without camera calibration, while global position and orientation are estimated independently per camera.Relative joint rotations are shared across views, but camera-specific global position and orientation account for unknown relative camera pose.Source: H Implementation Details · p. 24

Efficient Sampling

  • Sampling is restricted to ray segments inside a skeleton-centered cylinder and a foreground mask dilated by 5 pixels; rendering also uses coarse-to-fine importance sampling [42].The cylinder radius is the distance from the root to the farthest joint plus a 250 mm buffer; silhouette-based restriction is omitted for monochrome backgrounds.Source: H Implementation Details · pp. 23–24

Hyperparameter tuning

  • Hyperparameters are tuned on Human3.6M Subject 1, with the learning rate changed for 3DHP and gradient accumulation set relative to frame count.Table A9 lists hyperparameters tuned on Human3.6M Subject 1.Source: Table A9 · p. 23, H Implementation Details · p. 24

Network architecture

  • A-NeRF follows the NeRF architecture [42] and adds a per-frame 16-dimensional appearance code at the second-last layer to model illumination [36].The code’s late placement and low dimensionality help model global effects without affecting body geometry; NeuralBody baselines use 128-dimensional codes.Source: Figure A4 · p. 23, H Implementation Details · p. 24

Related papers

Loading 2102.06199v3…