Source-linked AI summary
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
Shih-Yang Su, Frank Yu, Michael Zollhoefer, Helge Rhodin
TL;DR
Existing 3D human models often require 3D supervision, restrictive priors, or controlled capture, making articulated human modeling from ordinary 2D video difficult. A-NeRF extends NeRF with skeleton-relative, overparameterized embeddings to learn personalized volumetric shape and appearance while refining pose from unlabelled videos. It achieves detailed monocular models and improves pose accuracy on Human3.6M, while remaining limited by computation, coverage requirements, illumination, and out-of-distribution motion.
Problem
Existing 3D models often require 3D labels or restrictive body models, whereas learning expressive human shape, appearance, and non-rigid motion from 2D observations remains challenging.
Method
A-NeRF extends NeRF with skeleton-relative local coordinates and one embedding per bone, learning a personalized volumetric body model and pose from unlabelled video.
Results
A-NeRF reaches previously reported detail levels from monocular video and improves Human3.6M PA-MPJPE by 8.0% over its initialization baseline on Protocol I.
Takeaways & Limitations
The model supports articulated neural body modeling, novel-view synthesis, and pose refinement from a single video, with a natural multi-view extension that does not require camera calibration.
Takeaways & Limitations
Long sequences and multiple actors are bottlenecked by computation, and unseen poses far outside the training distribution can produce artifacts.
Abstract
from arXiv · showhide
While deep learning reshaped the classical motion capture pipeline with feed-forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets. Project website: https://lemonatsu.github.io/anerf/ .
1 Introduction
A-NeRF learns a user-specific neural 3D body model and skeleton pose from unlabelled video, addressing the challenge of recovering articulated human shape and appearance from 2D observations. Its skeleton-relative overparameterization supports implicit modeling, while pose refinement improves motion-capture accuracy.
- 3D human representations are difficult to learn from 2D observations because natural images lack crowd-sourced 3D labels and humans vary in shape, appearance, and non-rigid motion.
- A-NeRF learns a user-specific neural 3D body model and underlying skeleton pose from unlabelled videos, extending NeRF to articulated motion and single-video training.
- Its core representation uses skeleton-relative local coordinates and one embedding per bone to resolve the ill-posed mapping from world points to body parts.The overparameterization provides a common frame for combining body-shape and appearance constraints across the captured sequence.
- A-NeRF learns detailed neural body models from monocular video using only rough 3D pose estimates for initialization.The reported detail level was previously associated with parametric surface models or multi-view approaches.
- The approach operates transductively on a specific target video, produces plausible but non-physical illumination, and requires additional steps for relighting.The model demonstrates novel-view synthesis of dynamic motions within this setting.
2 Related Work
Prior human modeling methods use discriminative initialization, constrained surface models, or implicit primitives, each with domain or representation limitations. A-NeRF differs by learning an articulated volumetric model with appearance and pose refinement from unconstrained single videos.
- Discriminative Human Pose Estimation: Discriminative pose estimators provide accurate initial skeleton estimates but can misalign with images, while direct 2D-based refinement can increase 3D errors.
- Surface-based Generative Body Models: Surface-based generative models constrain human shape and motion through template meshes or parametric models learned from laser scans.These priors support single-image reconstruction, detailed texturing, displacement mapping, manual-rigging relief, and optimization within the learned space.
- Implicit Body Models: Implicit body models based on level sets or sums of Gaussians refine pose, shape, and appearance through differentiable ray tracing, but primitives provide only rough approximations.
- Neural Scene Representations: Neural scene representations offer flexible nonlinear encodings but can remain limited by fixed mesh connectivity or the memory demands of discretized volumes.
- Neural Scene Representations: Unlike NASA and NeuralBody, A-NeRF learns a volumetric model with appearance and rendering without surface supervision or initialization, while refining pose from single unconstrained videos.
3 Formulation
A-NeRF jointly learns a detailed, time-varying neural body model and articulated poses from monocular images by rendering an implicit representation and optimizing reconstruction with pose and smoothness regularization. Its skeleton-relative encodings reparameterize query locations and viewing directions around all bones, improving consistency across articulated motion while preserving pose-dependent effects.
- 3 Formulation: A-NeRF estimates time-varying skeleton poses and learns a detailed body model from image sequences of one person.The model parameters define volumetric shape and color, while skeleton poses capture motion over time.
- 3 Formulation: The generative model initializes pose with an off-the-shelf estimator, refines it through skeleton-relative embeddings, and renders the implicit body by ray-marching.Rendering and optimization use a photometric image-reconstruction objective.
- 3 Formulation: The objective combines an L1 image data term with pose regularization and temporal smoothness, balanced by hyperparameters λt and λθ.The pose regularizer keeps estimates near the initial predictor output while allowing small shifts up to ϵ = 0.01.
- 3.1 NeRF and A-NeRF Image Formation Model: NeRF represents humans as a neural function over 3D points and view directions, then computes pixels by ray-marching sampled points along each camera ray.Each sample contributes according to opacity, emitted or reflected color, and accumulated transmittance.
- 3.2 Articulated Skeleton Pose Model: The skeleton uses a rest pose with fixed connectivity and bone lengths, while each frame’s pose defines an affine transformation between local bone and world coordinates.The inverse transformation maps world-space query locations back to local bone coordinates.
- 3.3 A-NeRF Skeleton-Relative Encoding: A-NeRF transforms query locations and view directions into skeleton-relative coordinates before NeRF predicts color and opacity, reversing the usual explicit-model deformation direction.The overcomplete representation expresses one query relative to every bone and incorporates an embedding cutoff for irrelevant bones.
- 3.3 A-NeRF Skeleton-Relative Encoding: Reference-pose encoding is ambiguous near equally distant bones and cannot capture non-rigid pose-dependent effects such as muscle bulging.Bone-relative position removes single-part association ambiguities by concatenating local coordinates for all bones, at the cost of much higher dimensionality.
- 3.3 A-NeRF Skeleton-Relative Encoding: The final embedding favors relative distances for their lower dimensionality and ability to model spherically shaped limbs, adds relative directions for orientation, and uses relative ray directions for dynamic appearance.The combined embedding is invariant to global translation and rotation while preserving piece-wise rigidity and allowing pose-dependent deformation.
4 Evaluation
A-NeRF is evaluated for pose refinement, visual quality, novel-view synthesis, animation, and volumetric reconstruction from monocular video. Across these tasks, it improves pose estimates and visual detail while learning without explicit surface templates, though long sequences, multiple actors, and unseen motions remain challenging.
- Experimental setup: The evaluation uses rough supervised pose initialization, while the body model and subsequent pose optimization are learned from one or more videos without supervision.Experiments cover Human3.6M, MPI-INF-3DHP, MonoPerfCap, and synthetic SURREAL and Mixamo data.
- Novel-View-Synthesis and Character Animation: A-NeRF supports novel-view synthesis and character animation from uncalibrated videos without 3D surface or multi-view supervision.The model changes viewpoint and pose, transfers poses between characters, and animates while selectively fixing body parts.
- Human Pose Estimation: A-NeRF improves pose refinement over SPIN on Human3.6M and provides moderate improvements over baseline estimates on MPI-INF-3DHP.On Human3.6M, it improves PA-MPJPE by 8.0% on Protocol I and 5.5% on Protocol II; the largest reported gain is 14% for PA-Wrist on Protocol II.
- Visual Quality Comparison: A-NeRF's full model significantly improves held-out visual quality over NeuralBody and gains additional detail from pose refinement.Without refinement, models produce ghosting, blurry limbs, and distorted faces; refinement reduces these artifacts and improves overall visual quality.
- Video-based volumetric reconstruction: A-NeRF reconstructs detailed volumetric bodies from monocular videos without stereo, depth, multi-view constraints, predefined templates, or geometric smoothness terms.The learned density captures details such as headphones and a baseball cap that parametric-surface offsets could not capture.
- Multi-view extension: A-NeRF can use multi-view refinement even without ground-truth camera calibration.This evaluation is reported for Protocol II on Human3.6M.
- Ablations: A-NeRF's embedding choices maintain moderate dimensionality while matching or improving PSNR and SSIM relative to higher-dimensional variants.For a fixed number of accurately posed images, long videos with diverse poses achieve visual quality comparable to multiple shorter clips.
- Limitations and Failure Cases: A-NeRF requires seeing the person from all sides in varying poses and is computationally bottlenecked for long sequences and multiple actors.Retargeting an unseen extreme breakdance pose to a model trained on normal walking motions produces artifacts.
5 Conclusion
A-NeRF integrates articulated skeletons with implicit neural fields through overcomplete re-parameterization, learning personalized volumetric body models from monocular video without predefined templates. The approach supports articulated motion and advances practical motion capture, while future work targets generalization across subjects.
- A-NeRF integrates articulated skeleton models and implicit functions through an overcomplete re-parameterization.
- The method learns interpretable, personalized volumetric density fields with texture detail and time-varying poses from a single video.
- A-NeRF does not require a parametric surface model or camera calibration and naturally extends from single-view to multi-view settings.
- The authors identify learning a general human model from a database of subjects as future work.
A Geometry Visualization
A-NeRF reconstructs convincing, view-consistent body geometry from monocular video without predefined template meshes. Its surface-free representation captures accessories, although very fine thin structures remain difficult to reconstruct.
- A-NeRF learns convincing body geometry without relying on pre-defined template meshes.
- The reconstructed geometry remains consistent between front and back when the person is observed from the back.
- The surface-free model represents accessories such as headphones, caps, quivers, and arrows that human template meshes often omit.
- Very fine details such as thin arrow shafts remain uncaptured, and the surface is not regularized to be smooth.
B Additional Qualitative Results for A-NeRF
Additional qualitative results show plausible novel-view synthesis and reconstruction of small accessories. Quantitative tables report pose-refinement improvements, especially for joints with high initial errors, with encoding and cutoff choices affecting refinement.
- A-NeRF produces plausible novel-view synthesis from different angles and reconstructs small structures such as arrows, quivers, and caps.
- A-NeRF improves pose accuracy on joints with high initial errors, including elbows, wrists, knees, and ankles.
- Relative-distance encoding helps refine estimated poses, while Cutoff can further improve the refinement.
C Additional Pose Refinement Results for A-NeRF
Additional pose-refinement results show that A-NeRF estimates human poses that align better with the training images. The accompanying experiments quantify improvements in pose-estimation accuracy and image-generation quality.
- A-NeRF estimates human poses that align better with the training images.
- The experiments quantify improvements in pose-estimation accuracy and image-generation quality.
D.1 Pose Estimation Accuracy Evaluation
A-NeRF refines initialized skeleton poses using skeleton-relative encodings, with radial distance and cutoff improving pose accuracy, especially for difficult joints such as wrists.
- A-NeRF evaluation on Human3.6M uses PA-MPJPE to measure pose-refinement accuracy.The metric is reported for the benchmark experiments.
- A-NeRF most strongly improves joints with high initial errors, including elbows, wrists, knees, and ankles.The wrist shows the most significant improvement, emphasizing refinement for precise end-effector localization.
- 57% improvement on the wrist and 0.56 overall PA-MPJPE improvement result from Rel. Dist. compared with Rel. Pos. encoding.Rel. Dist. captures radial information that relative position alone does not provide.
- 15% improvement on wrists results from adding Cutoff to the pose-refinement model without Cutoff.
- Pose regularization provides moderate improvement but is not strictly essential to the learning framework.
- The full model combines distance and direction encodings, while direction-relative transfer is the best directional strategy.
D.2 Image Generation Quality Evaluation
A-NeRF produces high-quality articulated renderings under controlled visual evaluation and benefits more from diverse poses than from additional views of insufficiently varied poses.
- A-NeRF visual quality is evaluated on SURREAL using PSNR and SSIM inside character bounding boxes.The test set contains 1,500 images rendered in novel camera views and poses, with ground-truth poses used during training.
- A-NeRF's Rel. Ray. encoding provides better image quality than the evaluated alternative directional encodings.
- Distance-only positional encoding is compact but insufficient, whereas the full distance-and-direction model performs well in both compactness and visual quality.The table reports 360 dimensions for distance-based encoding and 72 dimensions for the full model without positional encoding.
- A-NeRF can learn from a single view, while additional views do not help when the poses lack diversity.
- For a fixed number of accurately posed images, one long video with diverse poses has visual quality comparable to multiple shorter multi-view recordings.This makes the simpler monocular capture setup preferable within the reported setting.
- World Ray reduces performance and causes flickering novel-view renderings, while per-image codes combined with Rel. Ray. perform best.The learned illumination is plausible but not physically correct, limiting direct relighting applications.
G Dataset Details
The supplementary experiments use Human3.6M, SURREAL, and Mixamo, combining real-world evaluation protocols with synthetic subjects, poses, motions, and camera views.
- Two synthetic datasets supplement the real-world datasets introduced in the main paper.
- Human3.6M evaluation follows established protocols using PA-MPJPE on frontal-camera or multi-camera frames.Protocol I evaluates every fifth frontal-camera frame; Protocol II evaluates every 64th frame from all cameras.
- SURREAL contains 10,800 training images from 1,500 poses and a 1,500-image test set rendered across multiple cameras.The poses use a 4-1 train-test split, with nine training cameras and five testing cameras.
- Mixamo experiments render two synthetic subjects, James and Archer, across three provided motions and varying camera angles.Each subject has 1,130 training images, with pose and camera parameters extracted using an external method.
H Implementation Details
A-NeRF uses a skeleton-based representation, NeRF-style architecture, pose initialization and optimization, and optional multi-view constraints for implementation across datasets.
- A-NeRF initializes camera intrinsics, bone lengths, and per-frame poses, then jointly optimizes poses and neural shape and appearance parameters without supervision.
- The model uses the original NeRF architecture with a 16-dimensional per-frame appearance code and skeleton-relative inputs.
- The skeleton encodes 24-joint connectivity, static bone lengths, relative joint rotations, and global position per frame.
- Multi-view refinement averages relative joint rotations across cameras while retaining view-specific global position and orientation.This does not require calibrated cameras for the relative rotations.
- Efficient sampling restricts ray samples to a cylinder around the initial skeleton pose and an approximate foreground mask.The cylinder radius uses the farthest joint distance from the root plus a 250mm buffer.
- Novel-view visualizations use either a 0–360° bullet-time trajectory or a bounded circular trajectory with elevation and zoom changes.