PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Adversarial Learning of Realistic Neural Talking Head Models

Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, Victor Lempitsky

arXiv:1905.08233v2cs.CVcs.GRcs.LG

TL;DR

Personalized neural talking heads traditionally require substantial person-specific data and training, limiting their use when only a few images are available. This paper meta-learns adversarial generators and discriminators for rapid few- and one-shot adaptation, achieving highly realistic, personalized models, including perfect user-study realism and personalization with 32 images.

  • Problem

    Existing neural talking-head methods require several minutes of video or many photographs plus hours of GPU training for each personalized model.

  • Method

    The system meta-learns person-specific embeddings and initializes adversarial generators and discriminators for rapid adaptation from a handful of photographs, including one-shot learning.

  • Results

    With 32 images, the model achieves perfect realism and personalization in a user study, while user evaluations show higher realism and personalization than alternatives.

  • Takeaways & Limitations

    The framework enables highly realistic personalized talking heads from as little as one photograph and supports direct synthesis across a large variety of poses.

  • Takeaways & Limitations

    The method does not represent gaze and lacks landmark adaptation, causing noticeable personality mismatch when puppeteering from another person’s landmarks.

Abstract

from arXiv · show

Several recent works have shown how highly realistic human head images can be obtained by training convolutional neural networks to generate them. In order to create a personalized talking head model, these works require training on a large dataset of images of a single person. However, in many practical scenarios, such personalized talking head models need to be learned from a few image views of a person, potentially even a single image. Here, we present a system with such few-shot capability. It performs lengthy meta-learning on a large dataset of videos, and after that is able to frame few- and one-shot learning of neural talking head models of previously unseen people as adversarial training problems with high capacity generators and discriminators. Crucially, the system is able to initialize the parameters of both the generator and the discriminator in a person-specific way, so that training can be based on just a few images and done quickly, despite the need to tune tens of millions of parameters. We show that such an approach is able to learn highly realistic and personalized talking head models of new people and even portrait paintings.

1. Introduction

The paper presents a system for synthesizing personalized photorealistic talking-head video frames from face landmarks using direct adversarial ConvNets. Extensive meta-learning enables rapid few-shot—and even one-shot—adaptation to new people from only a handful of photographs.

  • Task and motivation: The task is to synthesize plausible speech expressions and mimics of a particular individual from face landmarks driving personalized head animation.The proposed capability has applications in telepresence, videoconferencing, multiplayer games, and special effects.
  • Task and motivation: Warping-based methods can operate from one image, but their artifact-free handling of motion, head rotation, and disocclusion is limited.The limitation applies to both classical warping and learned warping fields.
  • Task and motivation: Direct adversarial ConvNets offer realistic talking heads but require generator and discriminator networks with tens of millions of parameters and several minutes of video per person.These requirements make conventional personalized model training costly in data and time.
  • Proposed approach: The proposed system learns personalized talking-head models from a handful of photographs and limited training time, producing a reasonable result from a single photograph.Adding a few more photographs increases personalization fidelity.
  • Proposed approach: Extensive meta-learning on diverse talking-head videos simulates few-shot tasks and prepares adaptation from landmark positions and a small image set of a new person.The approach frames subsequent personalization as adversarial learning while initializing the model in a person-specific way.
  • Evaluation and applications: Experiments compare the system with alternative neural talking-head models using quantitative measurements and a user study, and demonstrate landmark-track video synthesis and puppeteering.The user study reports sufficient realism and personalization fidelity to deceive participants.

2. Related work

Related work spans statistical and deep face modeling, generative image techniques, adversarial meta-learning, and few-shot generative models for text-to-speech. The paper adapts these ideas to talking heads while differing in application domain, adversarial learning, meta-learning integration, and implementation.

  • Face and talking-head modeling: Face modeling has achieved strong results with classical and deep-learning methods, but talking heads additionally require modeling hair, neck, mouth cavity, and often shoulders or upper garments.The paper emphasizes that these non-face regions distinguish talking-head modeling from face modeling.
  • Generative modeling foundations: The system draws on adversarial training, conditional and projection discriminators, adaptive instance normalization, and content-style decomposition from generative modeling.Adaptive instance normalization is used in the meta-learning stage, while the other cited techniques motivate the broader system design.
  • Meta-learning: The method follows MAML’s goal of rapidly adapting from few samples but uses a substantially different implementation, alongside prior combinations of adversarial training and meta-learning.The related methods include data-augmentation GAN, MetaGAN, and adversarial meta-learning.
  • Few-shot generative models: Few-shot text-to-speech models share the setting and components of a standalone embedder network and generator fine-tuning, whereas this work targets talking heads and adds adversarial-learning and meta-learning adaptations.The paper also identifies implementation details as a difference from these text-to-speech works.

3. Methods

The method meta-learns an adversarial talking-head system from video episodes, learning person-invariant embeddings and person-specific generator initialization from landmark-conditioned frames. For unseen people, few-shot fine-tuning adapts the generator and discriminator to close identity gaps while retaining a realistic talking-head prior.

  • Meta-learning setup: The system assumes landmark locations for every video frame and rasterizes them into three-channel landmark images used to condition synthesis.An off-the-shelf face-alignment method provides the landmark locations.
  • Meta-learning setup: Meta-learning trains an embedder, generator, and discriminator, with the embedder extracting N-dimensional video-specific information invariant to pose and facial mimic.The generator combines an unseen frame’s landmarks with a predicted embedding and separates person-generic from person-specific parameters.
  • Adversarial training: Each episode uses K = 8 additional frames to average an estimated video embedding, reconstruct a held-out frame, and optimize content, adversarial, and embedding-match objectives.The content loss uses weighted L1 distances between VGG19 and VGGFace features, while the adversarial term includes realism and discriminator feature matching.
  • Adversarial training: The discriminator combines video-specific projection embeddings with general realism and landmark compatibility, while the match term aligns its embeddings with those computed by the embedder.Alternating updates train the embedder and generator against discriminator updates using a hinge loss that separates real and synthesized-image realism scores.
  • Few-shot adaptation: For a new person, the meta-learned embedder, generator, and projection matrix provide an initial landmark-conditioned model, but direct generation can leave an unacceptable identity gap.The number of new-person training frames T need not equal K.
  • Few-shot adaptation: Fine-tuning directly optimizes person-specific and person-generic generator parameters alongside discriminator parameters, using meta-learned initialization to preserve realistic extrapolation across poses and expressions.The fine-tuned generator generally fits the training sequence better, and meta-learning supplies a strong realistic talking-head prior.

4. Experiments

Experiments evaluate few-shot talking-head models on VoxCeleb1 and VoxCeleb2 using objective metrics and user studies. The method achieves higher perceived realism and personalization, with larger-scale training further improving results and enabling photograph and painting puppeteering.

  • Datasets: VoxCeleb1 supports baseline comparisons and ablations, while VoxCeleb2 provides approximately ten times more videos for demonstrating the approach’s full potential.VoxCeleb1 contains 256p videos at 1 fps; VoxCeleb2 contains 224p videos at 25 fps.
  • Evaluation protocol: Evaluation fine-tunes models on T frames from unseen people and tests them on 32 non-overlapping hold-out frames from each of 50 sampled test videos.The protocol evaluates the selfreenactment scenario after few-shot learning.
  • VoxCeleb1 comparison: 4800-triplet user studies clearly show that the proposed method achieves much higher realism and personalization than the baselines, despite inconclusive objective-metric comparisons.Each triplet was shown to five users; user accuracy has a random-guess lower bound of one third.
  • Puppeteering results: One-shot models puppeteer photographs and paintings using poses from VoxCeleb2 test videos selected by landmark-geometry similarity.Videos are ranked using CSIM between the original image and generated image to find geometrically similar people.

5. Conclusion

The framework meta-learns adversarial generative models that can create highly realistic, personalized virtual talking heads from as few as one photograph. Its main limitations are the gaze-insensitive landmark representation and lack of landmark adaptation, which can cause personality mismatch when puppeteering another person.

  • Conclusion: The framework creates highly realistic virtual talking heads from a handful of photographs, including as little as one image.It uses meta-learning of adversarial generative models with deep generator networks.
  • Conclusion: 32 images yielded perfect realism and personalization scores in a user study of 224p static images.The reported result concerns the model trained on 32 images.
  • Conclusion: The current landmark set does not represent gaze, and using another person’s landmarks causes noticeable personality mismatch.Avoiding this mismatch in fake puppeteering videos requires landmark adaptation.

A.1. Time comparison results.

The method’s feed-forward variant can substantially reduce few-shot training time when enough data is available, while fine-tuning provides the best output quality at a higher time cost. Inference is comparatively slower because of the model’s large parameter count, but reaches 13ms per frame on an NVIDIA 2080 Ti.

  • Experimental setup: The timing comparison evaluated three methods plus the method’s feed-forward variant on a single NVIDIA P40 GPU.Pix2pixHD and the proposed method used 40-epoch fine-tuning, with batches of 8 images when T exceeded 1; measurements averaged 100 iterations.
  • Training time: Given enough training data, the feed-forward variant outpaces the other methods in few-shot training time while maintaining high personalization fidelity and output realism.The comparison refers to Figure 4 for personalization fidelity and realism.
  • Training time: Approximately four and a half minutes are required to fine-tune the method on 32 training images for the best output quality.This timing was measured on the P40 GPU.
  • Inference speed: Inference is comparable to or slower than other methods because the model uses many parameters to encode prior knowledge about talking heads.The passage identifies the large parameter count as the cause of the slower inference speed.
  • Inference speed: 13ms per frame is achievable on an NVIDIA 2080 Ti, making the method sufficient for most real-time applications.The passage states that newer GPUs can drastically improve inference speed.

A.2. Ablation study

The ablation study examines person-specific initialization and adversarial fine-tuning losses. Generator initialization is necessary for plausible convergence, while adversarial fine-tuning improves image crispness and alignment with ground truth.

  • Study scope: The study focuses on training losses and fine-tuning details rather than re-evaluating fine-tuning’s overall effect on result quality.That overall effect had already been shown in Figure 4.
  • Person-specific initialization: Removing the discriminator’s LMCH term required stable one-shot meta-learning because multiple training frames caused optimization instabilities.After meta-learning, the discriminator’s person-specific vector W_i was randomly initialized for the ablation.
  • Adversarial fine-tuning: Removing the adversarial fine-tuning term produced subtly less crisp images that matched ground truth less closely in pose and image details.The comparison was made against the best model, with close-ups selected to emphasize the differences.

A.3. Additional qualitative results

The section presents additional qualitative comparisons, puppeteering results, and talking-head models learned from selfies. It also compares the model with Face2Face, highlighting lower fine-tuning data use and viewpoint synthesis for fixed poses.

  • Additional comparisons: Additional comparisons with other methods are provided in Figures 9, 10, 6, and the remaining figures.These comparisons extend the qualitative evaluation across multiple figures.
  • Puppeteering results: More puppeteering results are shown for one-shot learned portraits and photographs.The results concern both portrait paintings and photographic subjects learned from a single image.
  • Selfie-based models: Talking heads learned from selfies are also evaluated qualitatively.The section includes dedicated selfie-based results in Figure 13.
  • Comparison with Face2Face: 32 frames were used for fine-tuning this model, versus 1100 frames to train the Face2Face model.The comparison notes that the model’s output resolution is constrained by its training dataset.
  • Comparison with Face2Face: For a fixed pose specified by 3D face landmarks, the model synthesizes natural-looking frames from different viewpoints, unlike Face2Face.The figure identifies viewpoint synthesis as a limitation of the Face2Face system.

A.4. Training and architecture details

The system uses shared residual convolutional designs for embedding and discrimination, while the generator combines downsampling, bottleneck, upsampling, and self-attention components. Training runs distributed adversarial optimization with stored image embeddings, averaged generator conditioning, and two discriminator updates per generator update.

  • Embedder and discriminator: The embedder and discriminator share six residual downsampling blocks, accepting six-channel RGB-plus-landmark inputs and expanding channels from 64 to a maximum of 512.Each block downsamples by a factor of 2, and channel count doubles across blocks until reaching the maximum.
  • Embedding and conditioning: Stored vectorized image embeddings support element-wise LMCH, while their averages condition the generator and discriminator vectors determine realism scores.The embedder stores one resulting embedding per training image before averaging them for generator input.
  • Generator: The generator has four residual downsampling blocks, four bottleneck blocks, and four upsampling blocks, with self-attention before the last downsampling block and after two upsampling blocks.Bottleneck layers use 512 channels; downsampling uses instance normalization, whereas bottleneck and upsampling use adaptive instance normalization.
  • Distributed training: Training used 8 NVIDIA P40 GPUs with batch size 48 and two discriminator updates per generator update.Gradients were reduced across GPUs only for the generator and embedder using PyTorch distributed training.
  • Ablation study: The ablation study evaluates removing LMCH, randomly initializing the discriminator embedding, and removing L′ADV during fine-tuning.The comparison includes adversarial fine-tuning against regular fine-tuning without L′ADV.
Loading 1905.08233v2…