PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, Christian Theobalt

arXiv:2106.02019v2cs.CVcs.GRcs.LG

TL;DR

Realistic neural rendering of humans in general clothing and user-controlled novel poses remains difficult. Neural Actor uses an SMPL-guided canonical-space radiance field with texture-map latents, and the authors report improved rendering quality and generalization to poses unlike those in training.

  • Problem

    Neural implicit methods have yet to make high-quality human rendering in general clothing and user-controlled novel poses straightforward.

    Source: §1 · p. 2

  • Method

    Neural Actor uses SMPL to unwarp posed space into a canonical space, where radiance fields learn residual deformation and appearance from texture-map latent features.

    Source: §3.1 · p. 4, §3.2 · p. 5, §6 · p. 12

  • Results

    Neural Actor outperforms state-of-the-art methods in rendering quality and generalizes to novel poses that starkly differ from training poses.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The method produces pose- and view-dependent appearance changes and wrinkle patterns in novel poses.

    Source: §6 · p. 12

    Because unwarping relies on SMPL topology, the method cannot handle loose clothing such as skirts.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is built upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high fidelity dynamic geometry and appearance, we leverage 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports body shape control of the synthesized results.

1 INTRODUCTION

Neural Actor addresses the difficulty of rendering realistic humans in novel, user-controlled poses by combining a coarse body model with a canonical-space radiance field. Texture-map latents help model dynamic details, while the method’s clothing scope follows the SMPL topology.

1 INTRODUCTION

  • Neural implicit methods still struggle to render humans realistically in general clothing and user-controlled novel poses, while earlier explicit-model workflows are complex and time-consuming.Traditional methods also require sophisticated reconstruction and rigging of animatable person-specific templates.Source: §1 · pp. 1–2
  • Neural Actor uses SMPL-guided inverse skinning to map posed space to a canonical pose, then learns residual deformations and pose-conditioned radiance fields.The design is intended to handle large movements efficiently.Source: §1 · p. 2
  • SMPL texture maps serve as latent variables for pose-dependent shape and appearance, with adversarial learning mitigating the tendency to predict mean appearance.The approach splits the mapping from skeletal pose to dynamic effects into pose-to-texture and texture-to-effects stages.Source: §1 · p. 2
  • The contributions include a four-sequence multi-view human-performance dataset, alongside a method for playback and challenging user-controlled poses.The authors state that they will make the dataset publicly available.Source: §1 · p. 2
  • Because unwarping uses SMPL topology, the method handles clothing that roughly follows that structure but not loose clothing such as skirts.The authors identify explicit cloth models on top of SMPL as future work.Source: §1 · p. 2

2 RELATED WORK

Prior neural scene methods span static and dynamic representations, but many dynamic systems focus on playback rather than full-body novel-pose synthesis. Related image-based human generators also lack multi-view geometry, while template-based approaches can require person-specific reconstruction and rigging.

Neural scene representation and rendering

  • Static neural scene methods learn representations for novel views, while dynamic methods use latent interpolation, deformation fields, scene flow, or space-time fields to model changing scenes.The cited dynamic-scene approaches commonly focus on playback; Neural Actor additionally targets novel poses under novel views.Source: §2 · pp. 2–3
  • Several dynamic human methods synthesize novel poses, but concurrent geometry-guided deformable NeRF methods [Chen et al. 2021; Peng et al. 2021a] do not synthesize pose-dependent dynamic appearance.Many other dynamic-scene approaches discussed here replay the same sequence under new views rather than generalizing to new poses.Source: §2 · p. 3

Generative models for humans

  • Full-body image-to-image human generators condition on pose or body-model renderings, but their 2D convolutions do not learn scene geometry or ensure multi-view consistency.They can also exhibit “shower curtain effects,” limiting their suitability for free-viewpoint rendering.Source: §2 · p. 3

Geometry-aware human rendering

  • A prior method learns motion-dependent geometry and dynamic textures from multi-view video, but relies on a person-specific template requiring a 3D scanner and manual rigging and skinning.Neural Actor instead uses coarse SMPL and supports reshaping body proportions for rendering.Source: §2 · p. 3

3 NEURAL ACTOR

Neural Actor uses an SMPL-guided deformable radiance field and texture-map latents to synthesize animatable humans from novel viewpoints and poses. Its design addresses the difficulty of learning dynamic geometry and appearance from pose alone.

3 NEURAL ACTOR

  • Pose alone underdetermines dynamic geometry and appearance, while pose-estimation errors make the association harder to learn and can produce blurry outputs.The mapping from skeletal pose to dynamic effects is not bijective, challenging deterministic regression.Source: §3 · pp. 3–4

3.2 Texture Map as Latent Variables

  • At training time, texture maps are generated from multi-view images, while test-time maps are predicted from posed-mesh normal maps by an image-translation network.The adversarial loss in vid2vid helps prevent convergence to the mean value and mitigates uncertainty.Source: Figure 2 · p. 4, §3.2 · p. 5

3 NEURAL ACTOR

  • Neural Actor addresses these challenges with an SMPL proxy for deforming implicit fields and texture maps as latent variables for dynamic effects.The texture-map prior is trained adversarially to mitigate uncertainty in pose-dependent effects.Source: §3 · p. 4, §3.2 · p. 5

3.1 Geometry-guided Deformable NeRF

  • SMPL guidance combines inverse skinning into canonical space with a learned residual deformation for non-rigid changes that standard skinning cannot capture.The residual also compensates for motion-capture tracking errors.Source: §3.1 · pp. 4–5

3.2 Texture Map as Latent Variables

  • Texture maps provide high-resolution local pose cues for inferring local geometry and appearance changes, supporting generalization to novel poses.A feature extractor uses texture features at the nearest surface point, together with the sample's local coordinate, to predict residual deformation and dynamic appearance.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments span eight sequences from three datasets and evaluate playback, novel views and poses, body-shape control, and design ablations. Neural Actor produces high-quality results on challenging poses and generalizes to poses distinct from training.

4.1 Qualitative Results

  • On DeepCap and AIST driving poses, Neural Actor renders fine-scale details and plausible challenging poses, outperforming baselines on difficult motions.Examples include crunching and bending forward; Neural Volumes cannot reenact one person driving a different person because it requires captured images at test time.Source: §4.1 · p. 6, Figure 4 · p. 7
  • Neural Actor supports body reshaping by varying SMPL shape parameters and rendering the same actor in configurations that differ substantially from the original.At inference, the method maps the reshaped posed template through canonical space before inferring color and density.Source: §4.1 · p. 6, Figure 5 · p. 8

4.2 Comparisons

  • Across eight sequences, Neural Actor generalizes to new poses and produces sharper, higher-quality results than baseline methods, which show artifacts or blur.The qualitative comparison highlights pose-dependent wrinkles and appearance details that the baselines cannot recover.Source: Figure 7 · p. 9, §4.2 · p. 10
  • For long-sequence playback, Neural Actor produces sharp dynamic appearance by combining inverse-kinematic and residual deformation with a texture-map prior.The authors report that providing multi-view images to generate ground-truth texture maps at test time can further improve results.Source: §4.2 · p. 9
  • Neural Actor achieves quality similar to original DDC without a personalized template, while DDC using coarse SMPL produces geometric artifacts.Original DDC requires a person-specific 3D-scanned template and manual rigging and skinning.Source: Figure 8 · p. 10, §4.2 · p. 10

4.3 Ablation Study

  • Texture-map features improve quality over pose-vector, raw-texture, and normal-map conditioning by encoding spatial detail and local and global information.The authors report that a single normal-map pixel does not provide pose information, although the whole map can represent it.Source: Figure 9 · p. 11, §4.3 · p. 11
  • Geometry-guided deformation performs best when inverse-kinematic motion is separated from residual deformation; directly learning full motion causes severe artifacts.With the coarse body model, transferring skinning weights and texture features from the nearest surface point improves on transferring from the nearest vertex.Source: Figure 10 · p. 11, §4.3 · p. 11
  • With five training cameras or 195 sampled frames from a 19,500-frame sequence, Neural Actor shows no significant performance drop.The camera views are uniformly distributed on an upper sphere, and the frames are uniformly sampled.Source: Figure 11 · p. 11, §4.3 · pp. 11–12

5 LIMITATIONS

Neural Actor has limits in clothing topology, hand synthesis, pose generalization, and person scope.

5 LIMITATIONS

  • SMPL-based unwarping handles clothing that roughly follows SMPL topology, but not looser garments such as skirts.The authors identify explicit cloth models as future work for unwarping loose clothing.Source: §5 · p. 12
  • Untracked hands can produce noisy hand texture features, making faithful finger and hand synthesis difficult.The tracked SMPL hand may remain open when the ground-truth hand is a fist; low-resolution hand images also make gesture tracking difficult.Source: §5 · p. 12
  • Generalization may fail for poses unlike the training types or with excessive joint bending, and the model is person-specific.Extending the approach to multiple people is left for future work.Source: §5 · p. 12

6 CONCLUSION

Neural Actor synthesizes human characters from arbitrary viewpoints and controllable poses using canonical-space modeling and texture-map features. The authors report improved rendering quality, generalization to markedly different poses, and support for controllable shapes.

6 CONCLUSION

  • Neural Actor uses a coarse parametric body model to unwarp surrounding 3D space into a canonical pose, where a neural radiance field models pose-induced geometry and pose- and view-induced appearance.Body-model texture maps act as latent variables for predicting residual deformations and dynamic appearance.Source: §6 · p. 12
  • The method outperforms state-of-the-art methods in rendering quality and produces pose- and view-dependent appearance changes and wrinkle patterns.Source: §6 · p. 12
  • Neural Actor generalizes to novel poses that differ markedly from training poses and supports synthesis with controllable new shapes.Source: §6 · p. 12

A.1 Architecture

The architecture combines image translation, texture features, deformation prediction, and NeRF rendering.

A.1 Architecture

  • The method comprises an image translation network, texture feature extractor, deformation network, and NeRF model.Source: §A.1 · p. 14
  • The image translation network uses vid2vid with default settings, producing 512 × 512 normal and texture maps.Source: §A.1 · p. 14
  • A pretrained ImageNet ResNet34 extracts texture-map features, which are jointly trained with the deformation network and NeRF model.Feature maps from before the first four pooling layers are upsampled and concatenated into 512 channels.Source: §A.1 · p. 14
  • The deformation and NeRF networks use fully connected layers with ReLU, except for unactivated outputs predicting deformation, density, and RGB color.Figure 14 shows the network architecture; “+” marks vector concatenation.Source: §A.1 · p. 14

A.2 Algorithms

The algorithms use geometry-guided ray marching for volume rendering and locate nearby SMPL surface points for sampled ray points.

A.2 Algorithms

  • Geometry-guided ray marching speeds volume rendering; the implementation uses N = 64 and γ = 0.06 or 0.08 for all sequences.Source: §A.2 · p. 14
  • The ray-marching algorithm bounds sampling depths using mesh vertices and aborts volume rendering when the ray misses the geometry.Source: §A.2 · p. 14
  • For each ray point, the nearest SMPL surface point and its (u, v) coordinate are found by testing mesh faces and projecting onto planes or edges.Source: §A.2 · pp. 14–15
  • Specialized CUDA kernels implement both algorithms for better efficiency.Source: §A.2 · p. 15

B ADDITIONAL BASELINE SETTINGS

The appendix specifies training configurations for the comparison baselines. Settings range from a shared NeRF+pose setup to separate multi-stage training for DDC.

NeRF+pose

  • NeRF+pose concatenates a 72-dimensional pose vector with positional encoding and is trained for 300K iterations on 8 GPUs.It uses the same batch size and sampling strategy as Neural Actor.Source: B ADDITIONAL BASELINE SETTINGS · p. 15

Neural Volumes

  • Neural Volumes uses batch size 4 per GPU, 128×128 rays per image, and 300K iterations on 4 GPUs.Three uniformly distributed camera images provide scene-content latent vectors during training and testing.Source: B ADDITIONAL BASELINE SETTINGS · p. 15

Neural Body and Multi-View Neural Human Rendering

Real-time Deep Dynamic Characters

  • Real-time Deep Dynamic Characters trains four networks in sequence, from EGNet and lighting through DeltaNet and TexNet.The stages run for 360,000, 30,000, 360,000, and 720,000 iterations, respectively, on four NVIDIA Quadro RTX 8000 GPUs.Source: B ADDITIONAL BASELINE SETTINGS · p. 15

C ADDITIONAL INFORMATION OF TRAINING DATA

Figure 15 contains detailed information about the eight experimental sequences. It identifies sequences from DynaCap and DeepCap alongside four sequences captured by the authors.

C ADDITIONAL INFORMATION OF TRAINING DATA

  • Figure 15 gives detailed training-data information; its images have background removed and camera parameters provided.Source: C ADDITIONAL INFORMATION OF TRAINING… · p. 15, Figure 15 · p. 16
  • The eight sequences comprise two from DynaCap [Habermann et al. 2021], two from DeepCap [Habermann et al. 2020], and four captured by the authors.The author-captured sequences are labeled N1–4; the DynaCap sequences are D1 and D2, and the DeepCap sequences are S1 and S2.Source: Figure 15 · p. 16

D DEFINITION OF EVALUATION METRICS

The appendix defines four evaluation metrics. They cover pixel error, structural similarity, perceptual similarity, and similarity between generated and real images.

D DEFINITION OF EVALUATION METRICS

Loading 2106.02019v2…