Source-linked AI summary
Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars
Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, Victor Lempitsky
TL;DR
The paper addresses the challenge of generating realistic one-shot head avatars fast enough for practical deployment. It decomposes appearance into a pose-dependent coarse image and an offline-generated, pose-independent texture, then evaluates visual quality and speed against prior systems. The experiments report up to an order-of-magnitude inference speedup while maintaining state-of-the-art quality, including a real-time smartphone implementation.
Problem
Existing neural head-avatar systems are too slow for mobile real-time use, and reducing their architectures creates an unfavorable speed-realism tradeoff.
Method
The system combines a pose-dependent coarse image from a small rendering network with a warped, pose-independent high-frequency texture generated offline.
Results
Up to an order of magnitude inference speedup is reported over previous neural head-avatar models while achieving state-of-the-art quality.
Takeaways & Limitations
The bi-layer representation supports high effective resolution and fast neural rendering, including real-time smartphone inference at 42 ms per frame.
Takeaways & Limitations
The texture-enhancement updater network is trained in two stages because joint training with the base generator exceeds available memory constraints.
Abstract
from arXiv · showhide
We propose a neural rendering-based system that creates head avatars from a single photograph. Our approach models a person's appearance by decomposing it into two layers. The first layer is a pose-dependent coarse image that is synthesized by a small neural network. The second layer is defined by a pose-independent texture image that contains high-frequency details. The texture image is generated offline, warped and added to the coarse image to ensure a high effective resolution of synthesized head views. We compare our system to analogous state-of-the-art systems in terms of visual quality and speed. The experiments show significant inference speedup over previous neural head avatar models for a given visual quality. We also report on a real-time smartphone-based implementation of our system.
1 Introduction
Neural head avatars seek both realistic appearance and easy acquisition, but existing one-shot systems face severe speed limitations for mobile deployment. This work uses a bi-layer representation to improve effective resolution and rendering speed, including a reported 42 ms smartphone implementation.
- Motivation: Head avatars support applications including telepresence, gaming, AR/VR, and special effects, but realistic modeling must handle complex geometry and photometry.Earlier systems either achieved high realism with substantial person-specific acquisition and design or used a single photograph without targeting photorealism.
- Problem: Recent neural head-avatar methods reduce acquisition requirements to a few or single photographs, but their speed remains insufficient for real-time mobile deployment.Scaling down existing architectures produces an unfavorable speed-realism tradeoff.
- Approach: The proposed system generates each new pose by summing a pose-dependent coarse image with a warped, pose-independent texture estimated during avatar creation.The rendering network predicts the coarse image and texture warping, while the texture remains static at runtime.
- Approach: Separating coarse structure from high-frequency texture improves effective resolution and speed because runtime prediction can use a relatively small network without high-frequency details.High-resolution texture synthesis is performed offline, while the coarse image and texture warp are predicted at test time.
- Evaluation: The system is evaluated against analogous neural avatar methods for visual quality and speed and is reported to support real-time smartphone implementation.The figure caption reports 42 milliseconds per rendering on an Adreno 640 GPU in FP16 mode.
2 Related work
Related neural head-avatar methods differ in acquisition regime and generator architecture. They include direct image synthesis and differentiable warping, with warping applied at different stages of the representation.
- Acquisition regime: Neural head-avatar methods divide into many-shot approaches requiring videos or multiple videos and few-shot or one-shot approaches using photographs.The proposed method belongs to the one-shot category and models a person from a single photograph.
- Generator architectures: Direct-synthesis methods generate images through sequences of convolutional operators, nonlinearities, and normalization layers.Identity information can be injected through lengthy learning in many-shot settings or adaptive normalization mechanisms.
- Generator architectures: Differentiable-warping methods transform source imagery or intermediate representations, including standardized textures, mesh-related textures, and feature representations.The surveyed systems apply warping at different points in the generation pipeline.
3 Methods
The method generates each avatar view by combining a pose-dependent low-frequency image with a warped, pose-independent high-frequency texture. It uses learned embeddings, adaptive generators, specialized losses, and texture enhancement to preserve identity and detail while avoiding expensive full-image synthesis at runtime.
- Architecture: The inference generator maps target keypoints to a low-frequency image layer and a warping field, while the warped texture supplies the high-frequency layer.The final output is the sum of these two components.
- Architecture: Source images and poses are encoded into embeddings that initialize adaptive parameters for the texture and inference generators.The texture generator then synthesizes a high-frequency source texture, while the inference generator handles target-pose rendering.
- Architecture: The texture generator is encouraged to produce high-frequency content, but texture inpainting for occluded regions emerges from using different poses for initialization and loss calculation.The model does not explicitly constrain the generator to perform occluded-part inpainting.
- Training objectives: Training combines adversarial, pixelwise, perceptual, feature-matching, and texture-mapping regularization losses.Pixelwise loss is used for the low-frequency component, perceptual loss for sharper high-frequency content, and mapping regularization improves stability and texture usage.
- Training objectives: Texture mapping regularization keeps the predicted warp near an identity mapping early in training to prevent unused texture regions and excessive warp distortion.The regularizer is motivated by two compensating degrees of freedom: texture position and warping.
- Texture enhancement: A learned gradient descent updater optimizes the synthesized texture instead of fine-tuning the full generator, reducing overfitting risk in scarce one-shot data.This texture-only optimization addresses the computational cost and overfitting associated with person-specific fine-tuning.
4 Experiments
The experiments compare one-shot avatar systems across quality, complexity, extrapolation, ablation, and mobile-inference settings. The proposed method achieves stronger identity and pose results at comparable complexity, supports high-quality synthesis, and runs in real time on a smartphone.
- Evaluation setup: The evaluation uses VoxCeleb2 and VoxCeleb2-HQ, with LPIPS, CSIM, NME, and MACs measuring image similarity, identity mismatch, pose error, and computational complexity.The test protocol uses held-out identities and videos, with source frames and sequential target frames sampled at 1 FPS.
- Comparison with state-of-the-art methods: Across model families with varying parameter counts, the proposed method outperforms competing methods in pose error and identity preservation while being up to an order of magnitude faster on average.The comparison excludes selected overheads for temporal consistency and shared keypoint extraction when measuring complexity.
- Comparison with state-of-the-art methods: In a user study of 361 users and 1600 image pairs, the medium proposed model was preferred to medium FOMM in 59.6% of comparisons.Each user saw 21 pairs on average.
- Comparison with state-of-the-art methods: For different source and target poses, the proposed method has a clear advantage, while competing methods introduce more artifacts or blurriness.The qualitative comparison uses medium-sized models with similar parameter counts and emphasizes pose extrapolation.
- High-quality images: On the high-quality dataset, the method achieves a smaller identity gap and renders person-specific features such as wrinkles and moles on out-of-domain examples.The high-quality synthesis architecture uses the same parameter count as the medium architecture in the preceding comparison.
- Smartphone implementation: The medium model runs at 42 ms per frame on a Snapdragon 855, enabling real-time smartphone performance when keypoint tracking runs in parallel.SNPE was measured as about 1.5 times faster than PyTorch Mobile and up to two times faster than TensorFlow Lite.
5 Conclusion
The system synthesizes one-shot head avatars using a two-layer representation and achieves high visual quality with substantially faster inference. Ablation examples and qualitative results illustrate its ability to preserve facial details.
- Qualitative results show that the method preserves fine facial details such as Marilyn’s mole.
- The method creates head avatars from a single photograph using a pose-dependent coarse image and an offline-generated, pose-independent texture.The texture is warped and added to the coarse image during testing.
- The experiments show up to an order of magnitude inference speedup over previous neural head avatar models while achieving state-of-the-art quality.
A Methods
The methods section organizes the paper’s technical description around training, architecture choices, and the mobile inference pipeline.
- The paper first explains the training process in detail.
- It then describes the architecture and how different design choices affect final performance.
- Finally, it explains the adopted mobile inference pipeline.
A.1 Training details
The networks are optimized with Adam using a specified learning rate and momentum parameters, followed by batch-normalization statistics estimation before testing.
- The networks are optimized with Adam using a learning rate of 2 · 10^-4.
- Adam uses β1 = 0.5 and β2 = 0.999.
- Before testing, standing statistics for all batch-normalization layers are calculated using 500 mini-batches.
Texture mapping regularization.
Texture-mapping regularization prevents poor warping initialization from reducing effective resolution by constraining the warping field toward identity early in training. On VoxCeleb2, this supports texture alignment with the training images and better generalization.
- Random initialization can produce textures where the source face occupies too little area, reducing the output’s effective resolution.
- The warping field is represented as ωi(t) = I + ∆ωi(t), with an L1 penalty applied to the deviation from identity.
- The penalty weight is reduced multiplicatively to zero during training, so it does not affect final model performance.
- VoxCeleb2’s weak face alignment provides a consistent spatial region that the regularization can exploit.
- The regularization lets gradients initially reach the texture generator unperturbed, encouraging a high-frequency mean-face component that fills the training-image face area.
- PatchGAN produces patch-level realism scores, and feature matching minimizes distances between discriminator features of target and generated images.
A.2 Architecture description
The system uses residual-block networks with adaptive parameter prediction and efficiency-focused inference design. The embedder omits normalization to reduce pose-dependent overfitting, while discriminator normalization improves output quality.
- Network design: All networks use pre-activation residual blocks, with channel counts ranging from 64 to 512 and inference-generator capacity varied across model sizes.The inference generator generally uses half the feature count of the other networks, with full- and quarter-capacity variants also evaluated.
- Inference efficiency: The inference generator upsamples after its first convolution, halving the number of MACs per inference compared with the alternative ordering.This design is described as an efficiency improvement specific to the inference generator.
- Embedder: The embedder reduces channels across same-resolution blocks to bottleneck feature maps that match the corresponding generator inputs.This design supports prediction of adaptive parameters for generator blocks at higher resolutions.
- Normalization: Batch normalization is omitted from the embedder but used in the discriminator, where concatenating real and fake batches preserves label-dependent feature statistics and improves output quality.Instance normalization was more sensitive to hyperparameters and did not transfer reliably between dataset qualities.
- Adaptive parameter prediction: Embedding tensors are resized by nearest upsampling and passed through three linear layers to predict weights and biases for adaptive convolutions and batch-normalization layers.The resulting matrices match the parameter shapes of each adaptive layer and are trained separately per adaptive convolution and normalization layer.
- Adaptive parameter prediction: Adaptive parameters predicted from each embedding tensor initialize all adaptive parameters in the corresponding texture- and inference-generator block.The paper does not ablate this prediction scheme because it follows an already published method.
- Deployment: The smartphone deployment removes spectral normalization and merges fixed adaptive parameters into base layers before running FP16 inference on an Adreno 640 GPU.The model is converted from PyTorch through ONNX into Snapdragon Neural Processing Engine containers.
B.1 Training details for the state-of-the-art methods.
The competing systems are trained from official configurations and scaled into smaller model families by reducing generator channel counts. Complexity is estimated with an off-the-shelf module-level MAC counter, with known omissions for functional operations.
- First Order Motion Model: FOMM model variants reduce generator channels from 64–512 by default to 32–256 for medium and 16–128 for small models.These channel changes create a family of models for comparison at different capacities.
- Few-shot Vid-to-Vid: Few-shot Vid-to-Vid removes random crop and scale augmentation to avoid black borders and domain mismatch on already-cropped training videos.Its medium and small variants use reduced generator channel ranges of 32–256 and 16–128, respectively.
- Complexity measurement: MAC counts for competitors are computed with an off-the-shelf tool that evaluates internal PyTorch modules but may omit operations in PyTorch functional calls.The authors report accurate complexity estimates for their own model, while more complete competitor evaluation would require substantial code refactoring.
B.2 Extended evaluations.
The extended evaluation adds quantitative, qualitative, ablation, and cross-system comparisons across model families and datasets. It examines architecture variants, real-time avatar systems, texture updating, and cross-person reenactment.
- Extended comparisons: Figures 15–17 provide extended comparisons across method families, model sizes, and qualitative results for the VoxCeleb2 and VoxCeleb2-HQ settings.The figures cover medium-sized models, detailed medium-model outputs, and small, medium, and large variants.
- Quantitative comparisons: Table 2 extends model comparisons with SSIM, initialization time, and inference time for medium-sized models.The table also reports metrics for Few-shot Vid-to-Vid, FOMM, and Neural Talking Heads.
- Architectural variants: The texture generator begins with a 4 × 4 input and uses nearest upsampling, while the inference generator reshapes pose features into a 4 × 4 representation before upsampling.The embedder and discriminator operate through progressively downsampled feature maps, including final 8 × 8 stages.
- Enhancer: The enhancer uses an encoder–decoder architecture with residual blocks replacing sampling layers and no batch normalization inside the enhancer.This architecture is described as similar to a prior image-to-image translation design.
- Mesh-based comparison: The comparison with a closed-source mesh-based avatar product reports better face-shape and hair modeling for the proposed method.The first row evaluates reenactment because both methods are initialized with the frontal image.
- Ablation study: Figure 19 evaluates the texture updater by comparing results with and without the updater on VoxCeleb2-HQ.The ablation isolates the contribution of texture updating.
- MarioNETte comparison: Figure 20 compares the method with MarioNETte on one-shot self-reenactment, while noting that MarioNETte complexity was not independently evaluated.The comparison uses results from the MarioNETte paper because source code was unavailable.
- Cross-person reenactment: Cross-person reenactment preserves the source texture but retains noticeable leakage from the driving identity.This result identifies a remaining qualitative limitation in cross-person transfer.