PaperlayerView exact revision on arXiv

Source-linked AI summary

Textured Neural Avatars

Aliaksandra Shysheya, Egor Zakharov, Kara-Ali Aliev, Renat Bashirov, Egor Burkov, Karim Iskakov, Aleksei Ivakhnenko, Yury Malkov, Igor Pasechnik, Dmitry Ulyanov, Alexander Vakhitov, Victor Lempitsky

arXiv:1905.08776v1cs.CVcs.AIcs.LG

TL;DR

Direct pose-to-image systems can struggle to generalize to new camera views, motivating neural avatars that work across poses and viewpoints. This paper combines explicit 2D textures with a network that predicts pixel texture coordinates, and reports better generalization than direct mapping approaches. Its scope remains limited by scale changes, pose-estimation errors, and the assumption of constant surface color without lighting effects.

  • Problem

    Direct pose-to-image translation networks generalize poorly to new camera views, while neural avatars seek rendering across varying poses and camera positions.

    Source: §1 · p. 2

  • Method

    The system estimates explicit body-part textures and uses a convolutional network to predict output-pixel texture coordinates from pose and camera parameters.

    Source: §1 · p. 2

  • Results

    The authors report that retaining explicit shape and texture separation achieves better generalization than direct mapping approaches.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Explicit texture representation supports more realistic renderings and better generalization for new views and/or limited training data than direct video-to-video translation [67].

    Source: §1 · p. 2

    Generalization remains limited at scales considerably different from training, hand or face pose-estimation errors cause strong artifacts, and the method assumes constant surface color while ignoring lighting effects.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present a system for learning full-body neural avatars, i.e. deep networks that produce full-body renderings of a person for varying body pose and camera position. Our system takes the middle path between the classical graphics pipeline and the recent deep learning approaches that generate images of humans using image-to-image translation. In particular, our system estimates an explicit two-dimensional texture map of the model surface. At the same time, it abstains from explicit shape modeling in 3D. Instead, at test time, the system uses a fully-convolutional network to directly map the configuration of body feature points w.r.t. the camera to the 2D texture coordinates of individual pixels in the image frame. We show that such a system is capable of learning to generate realistic renderings while being trained on videos annotated with 3D poses and foreground masks. We also demonstrate that maintaining an explicit texture representation helps our system to achieve better generalization compared to systems that use direct image-to-image translation.

1. Introduction

Neural avatars aim to render individuals across changing body poses and camera positions, where direct pose-to-image networks can generalize poorly to new views. The proposed system combines explicit 2D body-part textures with a convolutional network that predicts pixel texture coordinates from pose and camera parameters.

1. Introduction

  • Direct pose-to-image translation networks generalize poorly to new camera views, motivating alternative neural-avatar architectures.Neural avatars target rendering individual people across body poses defined by 3D joint positions and varying camera positions.Source: §1 · p. 2
  • The system combines classical graphics’ geometry–texture separation with deep convolutional networks, explicitly estimating 2D body-part textures.The network predicts output-pixel texture coordinates from body pose and camera parameters.Source: §1 · p. 2
  • Explicit texture estimation improves realism and generalization for new views and/or limited training data compared with direct video-to-video translation [67].The network also predicts a body foreground/background mask.Source: §1 · p. 2

2. Related work

Prior full-body avatar methods span explicit graphics models, direct neural synthesis, and image warping, each with different data and generalization constraints. This work sits among video-based, texture-oriented approaches seeking to handle new views and poses.

2. Related work

  • Classical avatars combine a neutral-pose mesh, texture, and skinning, but realistic modeling can require considerable designer effort.The classical approach explicitly models skin, hair, clothing, reflectivity, and motion.Source: §2 · p. 2
  • Deep ConvNets avoid explicit surface modeling, but their lack of geometric or photometric models can make generalization to new poses and especially camera views problematic.Prior neural work includes personalized faces, hair, and hands; one face-rendering system decomposes view-dependent texture and 3D shape.Source: §2 · p. 2
  • Recent neural full-body methods produce impressive results but require substantial training data and commonly assume fixed test camera views.The paper targets the harder setting of rendering from arbitrary viewpoints.Source: §2 · pp. 2–3
  • Single-photo warping methods modify gaze, expression, or body pose, but are limited in realism and the amount of change they can model.The paper distinguishes its video-trained approach as addressing full-body multi-view rendering.Source: §2 · p. 3
  • DensePose and earlier methods predict image-pixel UV coordinates, and DensePose results are used to pretrain this system.The system uses a DensePose body-surface parameterization.Source: §2 · p. 3
  • Related texture-extraction and free-viewpoint systems cover image or video collections, while this approach is restricted to single-human scenes and also targets unseen poses.A related frame-warping method requires close pose matches in its training set, limiting the poses it can handle.Source: §2 · p. 3

3. Methods

The system maps pose and camera information to a rendered avatar through predicted body-part assignments and texture coordinates, rather than directly translating pose maps into images. Its textures and network are jointly updated during training, with initialization strategies for abundant or scarce training data.

3. Methods

  • Pose maps encode projected body bones and interpolated joint depth; the system predicts an avatar image and foreground mask from these inputs.At test time, the predicted image and mask can be blended with a real or synthetic background.Source: §3 · p. 3
  • Unlike direct translation from pose maps to images and masks, the textured approach predicts body-part assignments and coordinates used to sample explicit textures.The 24 body parts each have a 2D parameterization and a color texture map.Source: §3 · pp. 3–4
  • A fully convolutional network uses separate branches to predict assignment maps and coordinate maps, which determine texture-sampling weights and locations.The final assignment channel represents background probability.Source: §3 · p. 4
  • Image and background-mask losses are backpropagated through sampling to update both the network parameters and texture maps.The image loss compares the rendered image with ground truth; the mask loss compares predicted and ground-truth background masks.Source: §3 · p. 5
  • For multiple-video training, DensePose outputs initialize the pose-to-mapping network; scarce-data training can instead transfer a network from another avatar.Texture pixels are initialized from the mean of assigned training-image pixels, with unassigned pixels set to black.Source: §3 · p. 5
  • After end-to-end learning, the texture maps and coordinate predictions differ considerably from their initialized state.The initialized texture and network typically produce images that only coarsely resemble the person.Source: Figure 3 · p. 5, §3 · p. 5

4. Experiments

The experiments evaluate textured neural avatars on multi-view and monocular video, including held-out cameras and poses. The method outperforms the baselines on SSIM and is preferred by a majority of crowd-study users, while results also expose lighting and pose-related limitations.

Datasets

  • The evaluation includes CMU sequences trained on 16 or six cameras, with test cameras and body motions held out, plus captured multi-view and external monocular videos.The captured rig spans approximately 30°; a separate setting trains from one camera.Source: §4 · p. 6

Multi-video comparison

  • On held-out sequences, the method outperforms Direct and V2V on SSIM but underperforms V2V on FID.Figure 5 compares the methods on CMU1-6 and CMU2-6 using cameras held out from training.Source: Figure 5 · p. 7, §4 · p. 7

Baselines

  • For CMU1-6, adding target-frame DensePose results to keypoints did not improve the considered metrics over V2V using body joints alone.Direct is a separate ablation that predicts RGB and mask directly instead of body-part assignments and coordinates.Source: §4 · pp. 6–7

Multi-video comparison

  • In the multi-video crowd study, users preferred the method over either competing system in side-by-side comparisons.The comparison placed each system's result beside the ground truth; the authors note that averaging lighting across viewpoints is a disadvantage in metrics and user comparison.Source: §4 · p. 7

Single video comparisons

  • With single-camera training, the method showed a strong advantage on held-out camera and motion; users preferred it to V2V in 55% and 65% of cases for two subjects.Figure 6 compares both systems at the same hold-out viewpoint after training from a single viewpoint.Source: §4 · p. 7, Figure 6 · p. 8
  • Fine-tuning from a short monocular video produced avatars that generalized to new poses, though hand parts showed considerable artefacts.The [4] sequences included a 300-frame video; Figure 7 also shows a YouTube-derived avatar.Source: §4 · p. 7, Figure 7 · p. 8

5. Summary and Discussion

The paper presents a full-body neural avatar that predicts texture coordinates while retaining an explicit texture representation. It reports better generalization than direct mapping approaches, but identifies scale, pose-estimation, and lighting limitations.

Approach

  • The system predicts image-pixel texture coordinates and texture within an end-to-end ConvNet, separating texture from shape without explicit 3D shape modeling.This combines a classical graphics idea with neural rendering for new camera views and body poses.Source: §5 · p. 9

Conclusion

  • Retaining explicit shape and texture separation helps the system generalize better than direct mapping approaches.The paper presents this as a benefit of retaining an explicit texture representation.Source: §5 · p. 9

Limitations

  • Generalization remains limited at scales considerably different from training, with hand and face pose-estimation errors, and because the method assumes constant surface color and ignores lighting.The authors suggest rescaling followed by cropping or padding for scale differences, and view- and lighting-dependent textures as a potential way to address lighting.Source: §5 · p. 9

Related papers

Loading 1905.08776v1…