Source-linked AI summary
MegaPortraits: One-shot Megapixel Neural Head Avatars
Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, Egor Zakharov
TL;DR
Existing one-shot neural avatar systems are limited by the resolution of large, diverse training datasets. MegaPortraits combines new avatar architectures and training methods with medium-resolution videos and high-resolution still images, achieving megapixel avatars and distilling them into a real-time, identity-locked student model.
Problem
Large, diverse datasets needed for neural avatars are limited in resolution, restricting existing one-shot systems to up to 512 × 512.
Method
The method combines latent 3D appearance and motion representations with contrastive, gaze, and cycle-consistency losses, high-resolution still-image training, and teacher-to-student distillation.
Results
The approach achieves the first megapixel neural head avatars, while its distilled student runs at 130 FPS and achieves similar render quality to the full model.
Takeaways & Limitations
Identity-locked student models provide real-time rendering for predefined avatars and can help prevent misuse while maintaining low latency.
Takeaways & Limitations
Near-frontal training views degrade rendering for strongly non-frontal poses, high-resolution static views introduce temporal flicker, and shoulder motion is not modeled.
Abstract
from arXiv · showhide
In this work, we advance the neural head avatar technology to the megapixel resolution while focusing on the particularly challenging task of cross-driving synthesis, i.e., when the appearance of the driving image is substantially different from the animated source image. We propose a set of new neural architectures and training methods that can leverage both medium-resolution video data and high-resolution image data to achieve the desired levels of rendered image quality and generalization to novel views and motion. We demonstrate that suggested architectures and methods produce convincing high-resolution neural avatars, outperforming the competitors in the cross-driving scenario. Lastly, we show how a trained high-resolution neural avatar model can be distilled into a lightweight student model which runs in real-time and locks the identities of neural avatars to several dozens of pre-defined source images. Real-time operation and identity lock are essential for many practical applications head avatar systems.
1 INTRODUCTION
MegaPortraits targets the resolution ceiling imposed by limited video datasets while improving cross-reenactment and practical deployment. It combines disentangled motion and appearance modeling, megapixel upgrading from still images, and identity-specific distillation.
- Motivation: Public video datasets limit one-shot neural avatars to resolutions up to 512 × 512 because large, diverse, high-resolution collections are difficult to obtain.The required datasets must include thousands of people with varied demographics, lighting, backgrounds, expressions, and poses.
- Core model: The proposed one-shot model improves cross-reenactment by combining latent 3D appearance volumes with latent motion representations, contrastive disentanglement, and gaze loss.The gaze loss targets more realistic and accurate eye animation.
- Megapixel upgrading: A medium-resolution video-trained model is upgraded to 1024 × 1024 using an additional high-resolution still-image dataset.The method uses the same training video dataset while adding high-resolution still images for megapixel synthesis.
- Megapixel upgrading: The megapixel method outperforms the baseline super-resolution approach for cross-reenactment and demonstrates neural head avatars at megapixel resolution.The paper presents this as the first demonstration of neural head avatars in proper megapixel resolution.
- Practical deployment: Distillation produces a student model that is ten times faster, runs at 130 FPS on a modern GPU, and targets predefined identities.Identity-specific training enables the speedup relative to the main model, which can create avatars for unseen people.
2 RELATED WORK
Related work spans implicit 4D reconstruction and convolutional talking-head generation. MegaPortraits differs by supporting one-shot appearance transfer from arbitrary driving videos while using latent motion disentanglement.
- 4D head avatars: 4D head-avatar methods treat appearance and motion as non-rigid reconstruction, but typically require multi-shot, avatar-specific training data.These methods may also fail to represent motions unseen during training.
- Talking-head generation: Convolutional talking-head systems evolved from video-trained avatars toward few-shot and one-shot generation conditioned on appearance and motion descriptors.Most methods use explicit motion representations such as keypoints or blendshapes, while some use latent motion parameterization.
- Talking-head generation: Latent motion parameterization offers more expressive motion when motion and appearance are disentangled during training.MegaPortraits adopts this approach and proposes a new disentanglement method to improve result quality.
- Resolution limitations: Talking-head resolution is bounded by video datasets of at most 512×512, motivating the use of high-resolution still-image data for enhancement.The related-work discussion frames this alternative as single-image super-resolution.
3 METHOD
MegaPortraits combines a one-shot 3D volumetric avatar with latent motion warping, perceptual and adversarial training, and cycle consistency to separate appearance from motion. A second stage enhances 512×512 outputs to 1024×1024 using high-resolution photographs, while optional distillation produces a faster identity-conditioned student.
- 3.1 Base model: The base model encodes source appearance as volumetric features and a global descriptor, while separately extracting explicit pose and latent expression motion from source and driver images.The source appearance is transformed into a canonical coordinate space before driver motion is imposed.
- 3.1 Base model: Separate warping generators remove source motion, process the canonical volume with G3D, impose driver motion, and project the resulting volume into 2D for image synthesis.The projected features are decoded by G2D into the cross-reenacted output image.
- 3.1 Base model: Latent expression descriptors are decoded into explicit 3D warping fields instead of using keypoints, providing a more person-independent representation of facial motion.The authors add cycle consistency because higher-capacity models otherwise suffer severe appearance leakage from the driving image.
- 3.1 Base model: Training combines perceptual and GAN objectives with cycle consistency to match the driver while disentangling motion from canonical-space appearance.Perceptual losses include general-content, facial-appearance, and gaze terms; adversarial training uses multi-scale patch discrimination and feature matching.
- 3.2 High-resolution enhancement: The enhancement stage freezes Gbase and trains Genh to translate 512×512 outputs into 1024×1024 images using a high-resolution photograph dataset.Because the photographs have different identities, this stage cannot form source-driver pairs differing only in motion.
- 3.3 Student model: An optional conditional student network distills the teacher formed by Gbase and Genh, using pseudo-ground truth and an appearance index for a limited set of identities.The student is trained specifically for cross-driving and is intended to accelerate inference for predefined appearances.
4 EXPERIMENTS
The experiments evaluate MegaPortraits against prior systems for cross- and self-reenactment, high-resolution enhancement, and distilled inference. MegaPortraits outperforms competitors in cross-reenactment, improves 1024 × 1024 image quality while preserving identity, and distills to a fast student model, with weaker raw-image self-reenactment performance.
- Training details: The experiments use VoxCeleb2, VoxCeleb2HQ, and FFHQ, with separate training setups for 256 × 256, 512 × 512, high-resolution enhancement, and student models.The student model is trained using a proprietary dataset of 20,000 selfie videos and 100,000 selfie pictures.
- Evaluation setup: MegaPortraits is compared with Face-V2V, FOMM, and HeadGAN using FID, CSIM, and user studies for cross-reenactment, plus standard metrics for self-reenactment.UMTN measures motion preservation and UAPP measures appearance preservation; evaluation samples were shown to approximately twenty users on average.
- Cross-reenactment evaluation: MegaPortraits outperforms competitors across all cross-reenactment metrics, including both user studies, with better preservation of source appearance and driver motion.The qualitative comparison particularly highlights gaze direction, shape, and appearance preservation relative to FOMM and HeadGAN, while implicit pose representation reduces appearance leakage compared with Face-V2V.
- Self-reenactment evaluation: In self-reenactment, MegaPortraits performs comparably to baselines with face masking but worse on unmasked raw images.The reported difference may relate to absent shoulder-motion modeling and the method’s strong motion-appearance disentanglement.
- High-resolution evaluation: At 1024 × 1024 resolution, MegaPortraits improves image quality over bicubic upsampling and HiFaceGAN while preserving source identity.The method generates more high-frequency details and achieves the same CSIM as the base model.
- Distillation: The distilled student model runs at 130 frames per second and closely matches its teacher, achieving PSNR 23.14 and LPIPS 0.208 relative to the teacher.The student contains 100 avatars and has a total model size of 800 megabytes.
5 CONCLUSION
The paper presents megapixel neural head avatars and a distilled student model that preserves teacher quality while enabling fast, identity-locked rendering. It also identifies limitations in large-pose and shoulder-motion modeling.
- The approach is presented as the first neural head-avatar system to achieve megapixel resolution.
- The distilled student model achieves up to 130 FPS while closely matching the teacher model’s predictions.
- The student network locks avatar identities to predefined appearances while increasing rendering speed.
- Limitations: The method cannot model large head rotations because training data is dominated by near-frontal views.
- Limitations: The system lacks shoulders-motion modeling, which causes misalignment in corresponding image regions.
A NETWORK ARCHITECTURES
The architecture combines appearance and motion encoders with learned 3D warping, volumetric processing, image generation, and a separate high-resolution encoder-decoder. A student network uses avatar-specific SPADE parameters to switch among predefined identities.
- Appearance encoder: The appearance encoder produces volumetric features and a global descriptor from the source image.
- Motion encoder: Two ResNet-18 motion encoders separately predict head pose and expression vectors.
- Warping generators: Warping generators transform source and driver representations using rigid motion and emotion-changing components.
- Generators: A 3D U-Net-like network canonicalizes the warped volumetric representation, followed by projection and a 2D convolutional generator.
- High-resolution model: The high-resolution model adds an encoder-decoder that transforms base-model features into 1024 × 1024 outputs.
- Student model: The student model stores identity-specific information in SPADE normalization tensors selected for each predefined avatar.
B.1 Training details
Training details specify image augmentations, adversarial objectives, optimization settings, and student-model losses, with architectures documented for the base and high-resolution generators.
- Augmentation: Source and target images receive color jitter and random flips, while driving images undergo face-centered cropping, thin-plate-spline warping, and severe color jitter.
- Adversarial training: The base and high-resolution generators use multi-resolution patchGAN losses with spectral normalization.
- Optimization: AdamW training uses betas=(0.5, 0.999), eps=1e-8, and weight decay=1e-2 for generators and discriminators.
- Student training: The student model is trained with adversarial, VGG19, gaze, and VGG Face perceptual losses against teacher predictions.
B.2 Two stage training
The authors evaluated end-to-end high-resolution training but found it impractical on available GPUs and observed lower quality when fine-tuning a decoder around a frozen encoder.
- Memory constraints: End-to-end training with the full enhancer or even one 1024x1024 decoder layer exceeded the available GPU memory.
- Alternative training: Freezing a pretrained encoder and fine-tuning a decoder with an added upsampling block produced a significant quality decrease.
- Alternative training: The observed quality reduction occurred across three different weighting coefficients compared with separately trained networks.
B.3 Datasets preprocessing
The dataset preprocessing filters VoxCeleb2 videos for sufficient crop resolution and image quality, producing a 15,000-video training set. The student model uses compact SPADE tensors with spatial dimensions capped at 64 × 64.
- VoxCeleb2 videos are cropped around detected heads and filtered to exclude crops smaller than 512×512.
- Each SPADE tensor in the student model has spatial dimensions of at most 64 × 64.
- The remaining videos are ranked by mean image-quality scores, and the bottom 50% are removed, yielding 15,000 training videos.
B.4 Evaluation of the architectural choices
The architectural ablations evaluate gaze and blink modeling alongside the core components of the proposed avatar system. Gaze supervision is distilled from pretrained estimators using a shared two-head network.
- The additional ablation removes the source encoder, source-image warping, driver augmentation, direct background prediction, or the complete base model separately.
- A specialized gaze loss distills predictions from RT-GENE and RT-BENE to improve natural facial appearance.
- The gaze-and-blink distillation model uses a common backbone with separate heads for gaze direction and blink prediction.
- The distilled estimator is trained on 60k random dataset frames using MAE and MSE losses, weighted by w_MAE = 15 and w_MSE = 10.
B.6 Explicit control of the pose
The system provides explicit control over source-image scale and translation and demonstrates additional control over frontalization and moderate head rotation. The accompanying figures evaluate these pose manipulations qualitatively.
- The system can preserve the source-image scale or apply scale and translation from the driver to mimic the driver’s framing.
- The system demonstrates frontalization and applies head rotation from a frontal pose for angles of at least 15°.
- Figure 13 presents results with different scales and translations, while Figure 14 presents frontalization results.
C ADDITIONAL RESULTS
Additional results compare self- and cross-reenactment across 256 × 256, 512 × 512, and 1024 × 1024 settings, alongside student-model and super-resolution demonstrations. The reported comparisons emphasize high-resolution detail fidelity and qualitative differences between training variants and systems.
- Resolution comparisons: The paper compares cross- and self-reenactment at 256 × 256 and 512 × 512, and cross-reenactment at 1024 × 1024 resolution.
- Video demonstrations: The paper also provides demonstration videos for megapixel output and for cross- and self-reenactment scenarios.
- Volumetric representation: The volume tensor is reported to encode source geometry and has shape 96 × 16 × 64 × 64.
- Training comparison: The two-stage training comparison focuses on differences around the eyes, glasses, teeth, hair, and skin.
- Student model: The student model is demonstrated in self-reenactment mode.
- Super-resolution: The proposed super-resolution approach preserves high-frequency detail better than bicubic upsampling and HiFaceGAN in the qualitative comparison.