PaperlayerView exact revision on arXiv

Source-linked AI summary

AvatarReX: Real-time Expressive Full-body Avatars

Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, Yebin Liu

arXiv:2305.04789v1cs.CVcs.GR

TL;DR

AvatarReX addresses the difficulty of learning expressive full-body avatars from accessible video while supporting real-time animation and rendering. It combines part-specific representations and disentangled geometry and appearance, with a two-pass training strategy; the resulting avatars support novel poses and expressions with dynamic details.

  • Problem

    Many avatar methods omit joint control of the body, hands, and face, while dense-view capture requirements and slow NeRF rendering limit access and interactive use.

    Source: §1 · pp. 1–2

  • Method

    AvatarReX learns body, hand, and face representations separately using parametric-template priors, and disentangles geometry and appearance for rendering and training.

    Source: §1 · p. 2, §8 · p. 15

  • Results

    AvatarReX supports expressive control of the body, hands, and face together, with real-time animation and rendering and photo-realistic dynamic details for novel motions and expressions.

    Source: §1 · p. 2, §6.1 · p. 11, §8 · p. 15

  • Takeaways & Limitations

    Multi-view video of approximately 2,000 frames captured by 22 cameras can yield an avatar automatically, without pre-scanning or manual intervention, that can be driven by new poses, gestures, and expressions.

    Source: §1 · p. 2

    Occasional artifacts arise from errors and oversmoothing in the coarse SDF volume, including stitching textures and boundary artifacts during ray casting.

    Source: §7 · p. 15

Abstract

from arXiv · show

We present AvatarReX, a new method for learning NeRF-based full-body avatars from video data. The learnt avatar not only provides expressive control of the body, hands and the face together, but also supports real-time animation and rendering. To this end, we propose a compositional avatar representation, where the body, hands and the face are separately modeled in a way that the structural prior from parametric mesh templates is properly utilized without compromising representation flexibility. Furthermore, we disentangle the geometry and appearance for each part. With these technical designs, we propose a dedicated deferred rendering pipeline, which can be executed in real-time framerate to synthesize high-quality free-view images. The disentanglement of geometry and appearance also allows us to design a two-pass training strategy that combines volume rendering and surface rendering for network training. In this way, patch-level supervision can be applied to force the network to learn sharp appearance details on the basis of geometry estimation. Overall, our method enables automatic construction of expressive full-body avatars with real-time rendering capability, and can generate photo-realistic images with dynamic details for novel body motions and facial expressions.

1 INTRODUCTION

Traditional avatar creation can require expert workflows and dense camera capture, while NeRF-based methods struggle to render dynamic avatars in real time. AvatarReX combines expressive full-body control with a compositional representation, deferred rendering, and a two-pass training strategy.

1 INTRODUCTION

  • Traditional avatar creation involves tedious scanning, meshing, and rigging, and requires expert knowledge and sophisticated capture systems.These requirements limit access and increase cost.Source: §1 · p. 1
  • Prior systems often omit fine-grained face and hand control, while dense-view capture rigs limit access to the few existing full-body approaches.NeRF-based avatar methods also struggle to render dynamic avatars at real-time framerates, limiting interactive use.Source: §1 · p. 2
  • AvatarReX models the body, hands, and face with independent implicit fields, tailoring their use of parametric templates to part-specific shape and texture variation.The approach uses structured local radiance fields for clothed bodies and builds hand fields on MANO geometry.Source: §1 · p. 2
  • Disentangled geometry and appearance, SDF surfaces, and deferred shading enable rendering acceleration by two orders of magnitude without compromising quality.An explicit SDF volume supports fast surface location before color evaluation.Source: §1 · p. 2
  • A two-pass strategy combines volume rendering for geometry learning with surface rendering and patch-level perceptual supervision for sharp appearance details.Volume rendering uses sparse pixel supervision; surface rendering minimizes network queries once surface reconstruction is available.Source: §1 · p. 2
  • Approximately 2,000 frames from 22 cameras support automatic avatar learning without pre-scanning or manual intervention, with real-time driving and rendering.The camera setup uses 16 cameras for the body and hands and 6 for the face.Source: §1 · p. 2

2 RELATED WORK

Prior avatar work spans mesh-template, implicit-field, and neural-rendering approaches, with separate advances in face modeling and NeRF acceleration. AvatarReX’s related-work discussion situates its pose-dependent deferred surface pipeline among methods whose acceleration often targets static scenes or playback.

Body Avatars

  • Body-avatar work moved from statistical mesh templates for minimally clothed bodies toward more flexible implicit fields for clothed humans.Related methods include learned forward and backward skinning fields and differentiable forward skinning for pose generalization.Source: §2 · pp. 2–3
  • Avatar methods combine textured meshes, learned geometry and appearance, neural texture maps, or volumetric representations, with different capture and animation constraints.Neural Body plays back training sequences, while Remelli et al. require driving views; AvatarReX instead takes pose parameters and expression coefficients as input.Source: §2 · p. 3

Face Avatars

  • Face-avatar work ranges from morphable models and expression controls to implicit head representations, personalized NeRFs, and alternative driving signals.Many methods still require dense images or videos, while some use audio or gaze as driving signals.Source: §2 · p. 3

NeRF Acceleration

  • NeRF acceleration uses explicit grids, hashing, and tensor decompositions, but many accelerated methods target static scenes or dynamic-sequence playback rather than character animation.The body representation also uses structured local implicit fields with dynamic feature patches for added detail capacity.Source: §2 · pp. 3–4, Figure 2 · p. 4
  • AvatarReX uses deferred surface rendering and pose-dependent SDF geometry rather than a fixed mesh, which cannot represent its pose-varying topology.It caches SDF values in an explicit volumetric grid to accelerate sphere tracing.Source: §2 · p. 4

3 AVATAR REPRESENTATION

AvatarReX combines separate body, hand, and face representations that use parametric templates and SDF geometry. Each part is modeled to suit its shape and appearance, then assembled into a controllable avatar.

3 AVATAR REPRESENTATION

  • The body, hands, and face use distinct implicit representations based on their corresponding parametric templates, with geometry and color modeled separately.The shared geometry representation is an SDF whose zero-level set defines the surface.Source: §3 · pp. 4–5

3.1 Body Representation

  • Body geometry uses skinning-driven SMPL-X nodes and local MLPs, while dynamic feature patches augment local color modeling.The authors report that these patches allow color networks to learn more appearance details for the major body.Source: §3.1 · p. 5

3.2 Hand Representation

  • The hand uses the MANO template surface for its SDF geometry, while an MLP models pose-dependent color without learning complex finger motion.The color network receives the canonical surface position, posed normal, SDF value, and hand pose.Source: §3.2 · pp. 5–6

3.3 Face Representation

  • The face representation conditions NeRF on expression-derived front and side views encoded as feature triplanes, with separate geometry and color networks.Spatially varying triplane features reduce the need for a large MLP decoder and provide more detailed conditioning than a global expression condition.Source: §3.3 · p. 6

3.4 Composition

  • The parts are assembled using template correspondences and precomputed blending weights, with SDF values and colors blended near part boundaries.For the hand example, hand and body values are blended within the stated SDF interval; otherwise, the body values are used.Source: §3.4 · pp. 6–7

4 REAL-TIME ANIMATION AND RENDERING

AvatarReX uses deferred surface rendering to accelerate animation and rendering of its implicit avatar. The pipeline reconstructs geometry, creates a point map, and shades its pixels.

4 REAL-TIME ANIMATION AND RENDERING

  • The pipeline reconstructs an SDF volume, raycasts its zero-crossing surface into a point map, then queries color fields to shade the pixels.Geometry reconstruction is agnostic to the viewing camera; the point map and shading correspond to the requested viewpoint.Source: §4 · p. 7
  • Geometry reconstruction uses a camera-agnostic coarse-to-fine scheme: it first infers body geometry in a low-resolution SDF volume, then upsamples the volume fourfold and refines hands and face.The initial grid is 128×128×128 with 2 cm voxels; the passage says this resolution is not fine enough to represent thin structures such as fingers.Source: §4 · p. 7
  • The deferred surface pipeline uses an explicit geometry grid to cull color queries, achieving real-time performance with custom CUDA kernels and TensorRT.The approach depends on reliable surface prediction.Source: §4 · pp. 7–8

5 MODEL TRAINING

Section 5 describes a two-pass training strategy, multi-camera capture and processing, and a full training procedure for constructing and animating avatars. The strategy first estimates geometry with volume rendering, then uses surface rendering for patch-level appearance supervision.

5.1 Network Training Strategy

  • Topology-free volume rendering learns geometry from sparse pixels; topology-based finetuning uses the learned geometry for more pixel evaluations and patch-level perceptual supervision.The finetuning stage evaluates colors at ray-surface intersections and closes the gap between training and real-time rendering.Source: §5.1 · pp. 8–9, Figure 6 · p. 8
  • Topology-based surface rendering supports 128×128 patches, compared with roughly 20×20 patches for volume rendering, which the authors found too small to contain structural information.The volume-rendering pass is limited to 400 pixels per forward pass by network complexity and GPU memory.Source: §5.1 · p. 9

5.2 Data Capture and Processing

  • Training uses separate 16-camera body-and-hands and 6-camera face capture systems, with sequences of about 2000 frames and no pre-scanned templates or non-rigid mesh tracking.The face sequences capture expressions and natural reading; body sequences capture casual motions with neutral expressions.Source: §5.2.1 · p. 9

5.2.1 Data Capture.

  • Inverse-rendering refinement improves SMPL-X alignment by optimizing shape and pose against a foreground silhouette after off-the-shelf fitting.The figure compares initial fitting with the refined alignment of the arms to image observations.Source: §5.2.2 · p. 9, Figure 7 · p. 9

5.2.2 Data Pre-processing.

  • The pipeline trains the face networks first, then fixes them while jointly training body and hand networks; a tone-mapping network addresses skin-tone differences between the captures.The tone-mapping network maps viewing directions from the two capture systems to a color-mapping matrix.Source: §5.2.3 · pp. 9, 11, Figure 8 · p. 9

5.2.3 Full training procedure.

  • Full training takes about three days on one 24 GB NVIDIA GeForce RTX 3090 GPU, and the paper illustrates novel-pose synthesis and interactive avatar manipulation.The novel-pose results cover four identities; Table 1 reports iteration, elapsed-time, and updated-parameter statistics.Source: Figure 9 · p. 10, §5.2.3 · p. 11, Figure 10 · p. 11

6 EXPERIMENTS

The experiments evaluate AvatarReX’s real-time animation and rendering, compare its body, face, and hand results with prior methods, and test its technical components.

6.1 Results

  • AvatarReX supports simultaneous body, hand, and facial control, rendering at 25 FPS at 1024×1024 on an NVIDIA GeForce RTX 3090 GPU.The results include varied garments, cloth materials, motions, and expressions.Source: §6.1 · p. 11

6.2 Comparison

  • Novel-pose body synthesis is on par with state-of-the-art baselines, with detail comparable to Neural Actor and rendering faster than other NeRF-based methods by two orders of magnitude.Figure 11 reports that AvatarReX outperforms other baselines in appearance detail.Source: Table 2 · p. 11, §6.2 · pp. 12–13, Figure 11 · p. 12
  • Facial comparisons report sharper details such as teeth than the baselines, and the numeric evaluation finds AvatarReX superior.The comparison uses a monocular video sequence.Source: Figure 12 · p. 12, Table 3 · p. 12, §6.2 · p. 13
  • LISA recovers more hand-skin appearance detail than AvatarReX, but at higher computational load.The comparison uses the InterHand2.6M dataset.Source: §6.2 · p. 13, Figure 13 · p. 13

6.3 Ablation Study

  • Dynamic feature patches improve high-frequency detail over the MLP-only baseline and learn garment wrinkles that change with motion.Figure 14 compares the baseline, feature patches, and the further improvement after two-pass training.Source: §6.3 · p. 13, Figure 14 · p. 14, Figure 15 · p. 14
  • Topology-based finetuning improves photorealistic appearance details over one-pass training, with further evidence from the facial-area experiment.The first pass uses the vanilla NeRF-based training process; the second pass is topology-based finetuning.Source: §6.3 · pp. 13, 15, Figure 16 · p. 14
  • Patch-level perceptual supervision recovers sharp details such as wrinkles and buttons, whereas MSE-only finetuning yields blurry appearance.The first-pass pixel losses penalize pixel-wise error but ignore structural quality; the perceptual loss is more sensitive to structure errors.Source: §6.3 · pp. 14–15, Figure 17 · p. 15

7 LIMITATION AND FUTURE WORK

AvatarReX has limitations in reconstructing geometry and clothing layers, modeling self-shadowing, and representing body components beyond the body, hands, and face; the authors identify possible future directions and social risks.

7 LIMITATION AND FUTURE WORK

  • Coarse SDF errors can cause stitching textures and boundary artifacts, while assembling local fields rather than using the full SMPL-X surface can produce occasional joint artifacts.The authors describe these artifacts as occasional, although results are mostly photorealistic.Source: §7 · p. 15
  • A single model for all clothing layers may cause ghosting at clothing-skin or garment boundaries, especially with loose, dynamically deforming cloth.The authors suggest separately modeling cloth layers as future work.Source: §7 · p. 15
  • Self-shadowing from limbs interacting with or occluding the torso is learned as pose-dependent appearance rather than explicitly modeled, adversely impacting generalization.The authors suggest relighting and geometry-based shadow approximations as possible directions.Source: §7 · p. 15
  • The method models only the major body, hands, and face, leaving components such as hair, eyes, teeth, soft tissues, and accessories unmodeled.Complex dynamics of loose garments, hair, and soft tissues are beyond this paper's scope.Source: §7 · p. 15
  • Creating digital twins could enable misuse by retargeting people with poses or actions they never performed.The authors mention active watermarking and forgery detection as possible safeguards.Source: §7 · p. 15

8 CONCLUSION

AvatarReX learns full-body avatars from multi-view video, combining expressive control with real-time animation and rendering. Its compositional representation, geometry-appearance disentanglement, and two-pass training support high-quality appearance for novel poses and expressions.

8 CONCLUSION

  • AvatarReX learns full-body avatars from multi-view video and supports expressive control of the body, hands, and face together with real-time animation and rendering.The authors demonstrate synthesis for novel poses and expressions and note potential in interactive applications.Source: §8 · p. 15
  • A compositional representation with geometry-appearance disentanglement underlies the avatar, while two-pass training combines surface rendering with patch-level perceptual supervision to improve appearance quality.Source: §8 · p. 15
  • The experiments demonstrate synthesis results for novel poses and expressions, with potential for interactive applications.Source: §8 · p. 15

A NETWORK ARCHITECTURE

The appendix describes separate network architectures for body, hands, and face, along with positional encoding used before network inputs. The body uses local and blending MLPs, hands use an analytic geometry field, and the face uses triplane features and an MLP.

A NETWORK ARCHITECTURE

  • The body geometry field uses 128 local MLPs and a blending MLP, while its color field adds 128 tiny convolutional networks for dynamic feature patches.The body architecture uses MLPs except for the tiny convolutional network; its feature maps are 32×32 with 3×3 kernels.Source: A NETWORK ARCHITECTURE · pp. 17–18, Figure 18 · p. 18
  • Hand geometry is derived analytically, and the hand color field uses a simple MLP.The hand color network's fully connected layers use ReLU activation.Source: A NETWORK ARCHITECTURE · p. 18, Figure 19 · p. 18
  • The face network extracts feature triplanes from orthogonal Faceverse renderings with UNets, then uses an MLP to predict SDF or color values.The orthogonal renderings have 256×256 resolution.Source: A NETWORK ARCHITECTURE · p. 18, Figure 20 · p. 18
  • Coordinates and view directions are augmented with sinusoidal positional encoding, using m=6 for coordinates and m=4 for view directions.Source: A NETWORK ARCHITECTURE · p. 18

B ADDITIONAL IMPLANTATION DETAILS

The implementation uses PyTorch and Adam, with training and evaluation hyperparameters reported in Table 6. Real-time rendering runs fully on the GPU, using TensorRT for inference and custom CUDA kernels for other pipeline steps.

B ADDITIONAL IMPLANTATION DETAILS

C ADDITIONAL DETAILS OF DATA

The appendix describes a four-subject, two-clothing-set capture dataset and the SMPL-X fitting and optimization used to prepare training data.

C ADDITIONAL DETAILS OF DATA

  • Training data covers four subjects—two males and one female—in two clothing sets, with example frames shown in Figure 21.The figure presents example video frames from the experiments.Source: C ADDITIONAL DETAILS OF DATA · p. 19, Figure 21 · p. 19
  • Initial SMPL-X fits are refined with differentiable rendering, using foreground segmentation and rendered silhouettes.Foreground segmentation is extracted with Background Matting v2, and silhouettes are rendered with a differentiable rasterizer.Source: C ADDITIONAL DETAILS OF DATA · p. 19
  • The fitting objective penalizes keypoint reprojection errors with L2 loss and regularizes parameters toward their initial values.The experiments set λ1 = 1 × 10−6 and λ2 = 1 × 10−3.Source: C ADDITIONAL DETAILS OF DATA · p. 19
  • Table 7 compares PSNR and LPIPS on synthesized images under unseen poses from the ZJU-MoCap test set.The caption says best and second-best scores are highlighted in orange and light orange, respectively.Source: Table 7 · p. 19
  • For efficiency, shape and pose are first optimized on 20 uniformly sampled frames; shape is then fixed while pose is optimized per frame.Source: C ADDITIONAL DETAILS OF DATA · p. 19

D RESULTS ON ZJU-MOCAP

AvatarReX achieves pose-generalization performance comparable to state-of-the-art methods on ZJU-MoCap Sequence 387 with the system’s four cameras and 300-frame sequence.

D RESULTS ON ZJU-MOCAP

  • With four cameras and a 300-frame sequence, AvatarReX achieves pose-generalization performance comparable to state-of-the-art methods on ZJU-MoCap Sequence 387.The paper recommends training videos of about 2,000 frames for its everyday-action focus, while noting that the method also works with shorter videos and fewer cameras.Source: D RESULTS ON ZJU-MOCAP · p. 19

Related papers

Loading 2305.04789v1…