PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Articulated Radiance Field

Atsuhiro Noguchi, Xiao Sun, Stephen Lin, Tatsuya Harada

arXiv:2104.03110v2cs.CV

TL;DR

Learning pose-controllable articulated 3D representations from images is challenging because existing approaches require 3D shape supervision and may learn dependencies on irrelevant articulated parts. NARF extends NeRF with a disentangled, differentiable representation that supports rendering novel views and poses with little increase in computational complexity, and experiments show strong generalization to novel poses and views.

  • Problem

    Pose-controllable articulated 3D representations are difficult to learn because the radiance field depends on relevant parts and their kinematic parents, while models may incorrectly use irrelevant pose parameters.

  • Method

    NARF extends NeRF by conditioning each location on rigid transformations from forward kinematics and learning disentangled control of viewpoint, pose, bone parameters, and appearance.

  • Results

    NARF renders novel views and poses from pose-annotated 2D images with little increase in computational complexity and generalizes to novel poses and views with high fidelity.

  • Takeaways & Limitations

    The representation provides individually controllable rendering of viewpoint, pose, bone parameters, and appearance for articulated objects.

  • Takeaways & Limitations

    Training requires pose-annotated images, although the differentiable model may be extended to reduce the required supervisory information.

Abstract

from arXiv · show

We present Neural Articulated Radiance Field (NARF), a novel deformable 3D representation for articulated objects learned from images. While recent advances in 3D implicit representation have made it possible to learn models of complex objects, learning pose-controllable representations of articulated objects remains a challenge, as current methods require 3D shape supervision and are unable to render appearance. In formulating an implicit representation of 3D articulated objects, our method considers only the rigid transformation of the most relevant object part in solving for the radiance field at each 3D location. In this way, the proposed method represents pose-dependent changes without significantly increasing the computational complexity. NARF is fully differentiable and can be trained from images with pose annotations. Moreover, through the use of an autoencoder, it can learn appearance variations over multiple instances of an object class. Experiments show that the proposed method is efficient and can generalize well to novel poses. The code is available for research purposes at https://github.com/nogu-atsu/NARF

1. Introduction

NARF extends implicit radiance fields to articulated objects by modeling pose-dependent changes through relevant object parts. It learns controllable, part-based representations from pose-annotated images and renders novel views and poses with little added computational complexity.

  • Motivation: Articulated-object rendering remains difficult because joint variation, self-occlusion, and nonlinear kinematics complicate neural deformation modeling.Existing approaches may be limited to 2D or mesh representations, restricting view-dependent appearance or rendering quality.
  • Method: NARF predicts each 3D location’s radiance field using only its most relevant articulated part and explicitly computed kinematic transformations.Part probabilities are estimated from the location and geometric configuration, while density and view-dependent radiance are conditioned on the selected part.
  • Learned representation: NARF learns disentangled controls for camera viewpoint, bone parameters, bone pose, and appearance from pose-annotated images.Its part segmentation and dense 3D representation require no additional supervision.
  • Outcome: NARF renders novel views and poses of articulated 3D objects with little increase in computational complexity.The representation is trained from pose-annotated 2D images and changes rendered poses through input pose configurations.

2. Related Work

Prior work provides implicit, differentiable representations for shapes, scenes, and deformations, but articulated representations have lacked a combination of image-based learning and appearance modeling. NARF addresses this gap within the implicit 3D representation framework.

  • Articulated 3D Shape Representations: Traditional skinning models deform articulated objects through bone transformations, but their capacity is limited by discrete template-mesh resolution and shading requirements.These models have been developed for human bodies and animals with pose- and identity-dependent deformation modeling.
  • Articulated 3D Shape Representations: NASA provides continuous, differentiable articulated shape representations but requires ground-truth occupancy and does not learn appearance.The missing appearance modeling is important for rendering.
  • Articulated Pose Conditioned Image Generation: Pose-conditioned image-generation methods target articulated objects such as people and support applications including movie making, photo editing, virtual clothing, and motion transfer.These approaches form a related line of work on generating images conditioned on target poses.
  • Implicit 3D representation: Implicit 3D representations are memory efficient, continuous, and topology-free, supporting applications from shape and texture learning to reconstruction and image synthesis.They have also been applied to articulated objects and deformation.
  • Implicit 3D representation: NeRF learns view-dependent radiance from sparse posed images of static scenes, while later extensions address deformations and dynamics.NeRF itself cannot handle deformable objects because it addresses static scenes.

3. Method

NARF extends NeRF to articulated 3D objects by conditioning radiance on pose while using explicit kinematics and part-aware transformations. Its disentangled design selects relevant parts to combine efficient computation with articulated-part dependency.

  • Neural Radiance Field Revisited: NeRF maps 3D location and viewing direction to density and view-dependent color, then uses differentiable volume rendering for novel-view synthesis.Density depends on location, while color depends on both location and viewing direction.
  • Pose-Conditioned NeRF: A Baseline: Pose-conditioned NeRF extends this representation so changing kinematic pose configurations enables rendering novel poses as well as novel views.The method assumes pre-cleaned backgrounds when modeling articulated objects.
  • Kinematic Model: The articulated-object formulation uses explicit forward kinematics to compute rigid transformations for parts arranged in a tree of fixed-length bones.Each joint has a parent, and global transformations are obtained by multiplying transformations along the root-to-joint path.
  • Pose-Conditioned NeRF: A Baseline: P-NeRF directly conditions on all pose transformations, but articulated modeling remains difficult because transformations are nonlinear and radiance depends on only relevant parts and their parents.The proposed representation addresses these issues by decomposing the object into rigid parts with explicit local coordinate systems.
  • Rigidly Transformed NeRF: RT-NeRF transforms global locations and viewing directions into each part’s local coordinates, while conditioning on transformation and bone parameters to model pose and shape variation.The local location is x_l = R^-1(x − t), and transformed viewing direction is d_l = R^-1d.
  • Neural Articulated Radiance Field: NARF compares part-wise, holistic, and disentangled designs; Holistic NARF uses one model with nearly part-count-independent cost, while Disentangled NARF introduces a selector to combine advantages.Part-wise training can be dominated by zero-density samples, whereas Holistic NARF avoids that issue but loses part dependency and cannot generate segmentation masks from its equation.

4. Results of Training on a Single Object

NARFD learns an articulated radiance field that renders RGB, depth, and segmentation maps while generalizing to novel poses and views. It outperforms the compared methods across evaluation settings with computational cost close to a single NeRF.

  • Dataset and Metrics: The evaluation uses synthetic human bodies with same or novel poses and viewpoints, assessed using PSNR, SSIM, and mask distance.Higher PSNR and SSIM indicate better image reconstruction, while lower Mask indicates closer rendered 3D shape.
  • Quantitative Results: NARFD outperforms all compared methods across every evaluation metric and test-data setting.The comparison includes NARFP, NARFH, P-NeRF, CNN, and D-NARF.
  • Quantitative Results: NARFD maintains high performance for novel poses and views, with an SSIM drop within 4%.The method generalizes to novel poses and views with high fidelity.
  • Comparison with Baselines: NARFP has good generalization but requires 10× NARFD's FLOPS and 17× its memory, while NARFH drops 8% in SSIM on novel poses.The CNN baseline drops 10% in SSIM under novel views, and P-NeRF and D-NARF fail in almost all settings.
  • Qualitative Results: NARFD produces higher-quality RGB, depth, and segmentation maps than the other methods for novel views and poses.The qualitative comparison includes RGB images, depth maps, and part segmentation maps.
  • Disentangled Representations: NARFD learns disentangled camera viewpoint, bone parameters, and pose representations that can be individually controlled during rendering.Figure 4 illustrates this control using bone-parameter interpolation.

5. Appearance Variation with Autoencoder

A NARF-based autoencoder models shape and appearance variation across multiple articulated objects from single images. Its disentangled latent and pose representations support reconstruction and individually controlled rendering properties.

  • Autoencoder Architecture: A 2D CNN encoder produces a latent vector that the NARF decoder combines with camera viewpoint and human pose to reconstruct the input image.The encoder and decoder are trained jointly with the reconstruction loss.
  • Latent Disentanglement: The latent vector is decomposed into shape and appearance vectors, which condition density and color inputs separately.The shape vector conditions location and bone parameters, while the appearance vector conditions view direction and local transformation.
  • Dataset: The autoencoder is trained on 35,450 images for training and 3,940 for evaluation, rendered from 112 male identities and varied poses and viewpoints.All rendered images have resolution 128 × 128.
  • Results: A single autoencoder reconstructs RGB images and additionally generates depth images and segmentation maps for multiple objects.The result indicates that appearance variation is modeled across the objects.
  • Results: The NARF-based autoencoder disentangles camera viewpoint, bone parameters, human pose, and color appearance for individual rendering control.Color appearance can be changed by replacing the appearance latent vector with another person's vector.

6. Conclusion and Future work

The proposed representation learns implicit articulated-object models from pose-annotated images while providing explicit control over viewpoint, pose, bone parameters, and appearance. Pose annotation remains required for training, but differentiability may support reduced supervision in future extensions.

  • NARF learns implicit representations of articulated objects from images with pose annotations.
  • The representation explicitly controls viewpoint, pose, bone parameters, and appearance.
  • Pose annotation is required during training, although differentiability may enable joint pose-estimation and segmentation training.
  • Explicit 3D shape and part segmentation may support unsupervised depth-estimation and segmentation learning.

Radiance Field

The autoencoder evaluation compares reconstruction and generation across pose and view settings using multiple image-quality and mask metrics. NARFD outperforms the alternatives across all reported settings and metrics, while also generating depth and segmentation images.

  • The evaluation uses PSNR, SSIM, and Mask across four combinations of pose and viewpoint novelty.
  • NARFD outperforms competing methods under all evaluation metrics and test-data settings.The settings include same pose/same view, novel pose/same view, same pose/novel view, and novel pose/novel view.
  • NARFD jointly generates high-quality depth and segmentation images in the autoencoder setting.
  • CNN-based models perform poorly in novel-view settings and cannot generate depth or segmentation images.
  • P-NeRF and D-NARF fail in almost all settings, while NARFH performs poorly on novel poses.

B. Ablation Studies in RT-NeRF

The RT-NeRF ablation isolates explicit coordinate transformation and transformation-conditioned color estimation. Results show that explicit transformation supports novel-pose and novel-view generalization, while omitting transformation input reduces mask performance and produces blurrier images.

  • Design elements: RT-NeRF explicitly transforms global 3D locations into local coordinates before estimating density.The ablation compares this design with RP-NeRF, which directly conditions on global locations and rigid transformations.
  • Design elements: RT-NeRF conditions color estimation on the 6D se(3) transformation representation ξ.
  • Results: RP-NeRF fails to learn a good 3D representation and does not generalize to novel poses and views.
  • Results: Removing ξ from RT-NeRF color estimation significantly reduces Mask performance and produces blurrier rendered images.

C. Additional Ablation Studies in NARF

Additional ablations examine mask supervision, NARFP temperature, and NARFD activation choices, alongside implementation and comparison details. Mask loss and softmax improve performance, while τ = 100 is empirically best for NARFP.

  • Mask loss: Removing mask loss significantly degrades all three NARF variants, especially for novel pose/novel view.Without mask loss, NARFH fails to converge and NARFP and NARFD produce blurry backgrounds.
  • Temperature parameter: τ = 100 is the empirically best-performing temperature setting for NARFP.Values near zero make selection hard and hinder training convergence, whereas very large values reintroduce part dependency issues.
  • Activation function: The sigmoid ablation replaces the softmax output with p_i = 1 / (1 + exp(−o_i)).
  • Activation function: Softmax outperforms sigmoid in NARFD, especially under the novel pose/novel view setting.
  • Implementation and comparison: The NARFD implementation retains coordinates for all parts and is more effective than D-NARF's transformation of input coordinates.
  • Implementation and comparison: Training uses Adam with batch size 16 and typically converges at about 100,000 iterations.

G. Cross Dataset Evaluation on SURREAL Dataset

The cross-dataset protocol trains NARF on THUman and evaluates it on SURREAL, whose differences in clothing and body characteristics test generalization. Novel-view and novel-pose renderings remain reasonable, though some body types reconstruct less effectively.

  • Training on THUman and testing on SURREAL evaluates the NARF autoencoder across datasets.The datasets differ in clothing and body characteristics, including SURREAL's lack of clothing on its meshes.
  • Figure 12 reports qualitative results under the cross-dataset evaluation protocol.
  • Short pants and fat people are reconstructed less effectively because THUman lacks sufficient diversity in clothing and body size.
  • Novel-view and novel-pose rendering results on SURREAL look quite reasonable.

H. Experiment on Real Human Images

NARF is evaluated on real human images using ZJU-MOCAP, including novel-view and novel-pose tests. Rendering quality is lower than on synthetic datasets, but the representation provides explicit control over viewpoint, pose, bone parameters, and appearance.

  • NARF is trained on 1,969 of 2,185 ZJU-MOCAP frames and tested on 216 frames with novel poses.The images have resolution 512 × 512.
  • Figure 13 compares training poses rendered from novel viewpoints with novel-pose and novel-view testing results.
  • Rendered-image quality on real human images is lower than on the synthetic datasets.The authors attribute this potentially to the assumption that articulated parts are rigid, which loose clothing may violate.
  • The representation explicitly controls viewpoint, pose, bone parameters, and appearance for articulated objects.
Loading 2104.03110v2…