Source-linked AI summary
HeadGAN: One-shot Neural Head Synthesis and Editing
Michail Christos Doukas, Stefanos Zafeiriou, Viktoriia Sharmanska
TL;DR
Single-reference head reenactment often fails to preserve identity, realism, or driving motion. HeadGAN conditions a GAN on identity-adapted 3D face representations and optional audio features, and reports superior performance across reconstruction, reenactment, and frontalisation while supporting editing and compression-related uses.
Problem
Single-reference head reenactment methods can have poor photo-realism, identity preservation, or pose and expression transfer.
Method
HeadGAN uses identity-adapted 3D face representations to condition one-shot synthesis, with optional audio features for mouth movements.
Results
HeadGAN reports superior image quality and standard GAN-metric performance on reconstruction, reenactment, and frontalisation, including against a model trained on VoxCeleb2.
Takeaways & Limitations
The 3D face representation supports reenactment, reconstruction, pose and expression editing, frontalisation, and efficient video compression.
Abstract
from arXiv · showhide
Recent attempts to solve the problem of head reenactment using a single reference image have shown promising results. However, most of them either perform poorly in terms of photo-realism, or fail to meet the identity preservation problem, or do not fully transfer the driving pose and expression. We propose HeadGAN, a novel system that conditions synthesis on 3D face representations, which can be extracted from any driving video and adapted to the facial geometry of any reference image, disentangling identity from expression. We further improve mouth movements, by utilising audio features as a complementary input. The 3D face representation enables HeadGAN to be further used as an efficient method for compression and reconstruction and a tool for expression and pose editing.
1. Introduction
Head reenactment from a single reference image remains difficult because existing methods can compromise photo-realism, identity preservation, or pose and expression transfer. HeadGAN addresses these challenges with a one-shot 3D-face-conditioned GAN that optionally uses audio features and supports several related tasks.
- Single-image head reenactment methods often struggle with photo-realism, identity preservation, or complete transfer of driving pose and expression.
- Key-point guidance can harm identity preservation because key-points encode appearance, especially when source and driving head geometries differ.
- HeadGAN conditions one-shot GAN-based synthesis on 3D face representations derived from 3D Morphable Models to disentangle identity from expression.
- Optional speech features improve mouth synthesis, while the system also supports real-time reenactment, compression and reconstruction, expression editing, and novel-view synthesis.The paper reports operation at approximately 20 fps.
- HeadGAN reports superior image quality and standard GAN-metric performance on reconstruction, reenactment, and frontalisation, including against a model trained on VoxCeleb2.
2. Related Work
Related work spans model-free, landmark-based, 3D-face-assisted, and audio-driven approaches to head synthesis. These methods expose trade-offs involving pose transfer, identity preservation, source-pose requirements, and mouth realism.
- Model-free methods: Model-free methods animate heads without facial priors, but warping can produce unnatural deformations and poor photo-realism.
- Model-free methods: FOMM uses relative key-points to preserve source identity, but requires matching source and driving poses and may not follow the driver’s head pose.
- Landmark-based methods: Landmark-based methods can work with limited references but often require favorable source poses, small deformations, frontal neutral images, or identity-specific fine-tuning.
- Head animation assisted by 3D faces: 3DMM-based methods recover pose, expression, and identity parameters, then use rendered 3D faces to condition networks that complete missing appearance information.
- Audio-driven head synthesis: Audio-driven synthesis provides a separate research direction, while HeadGAN optionally uses audio to improve mouth realism and uses target video for pose and expression.
3. Methodology
HeadGAN preprocesses driving and reference images into identity-adapted 3D face representations, then generates source-identity frames using dense flow, visual features, and optional audio. Its 3DMM conditioning separates identity from expression and supports accurate correspondence, temporal synthesis, and efficient transmission.
- 3.1. 3D Face Representation: 3DMM fitting extracts driving camera, identity, and expression parameters, while reference fitting provides source identity and camera parameters.
- 3.1. 3D Face Representation: The method combines source identity geometry with driving expression parameters to preserve facial structure while transferring motion.
- 3.1. 3D Face Representation: The preprocessing pipeline renders identity-adapted 3D faces for the driving sequence and reference image, which condition HeadGAN’s Generator.
- 3.2. Audio features: Audio features are extracted from temporally aligned windows around each driving frame and provide complementary information for facial and mouth movements.
- 3.3. HeadGAN Framework: The Generator combines recent 3D face representations, the reference image and 3D face, visual feature maps, and optional audio features to synthesize frames.
- 3.3. HeadGAN Framework: Dense flow warps reference appearance according to the driving 3D face, while the rendering network uses SPADE visual conditioning and AdaIN audio conditioning to produce photo-realistic images.
- 3.4. Advantages of 3D face modeling: The 3D representation supplies dense facial correspondences, supports flow in regions such as hair and upper body, and helps address identity preservation.
4. Experiments
HeadGAN is evaluated across reconstruction, reenactment, frontalisation, editing, and ablation settings. It achieves strong image quality, identity preservation, pose and expression transfer, while its components and audio input are separately assessed.
- Method: HeadGAN represents 3D facial geometry as a per-pixel coordinate representation derived from a rasterised 3D face mesh.The representation is produced using camera parameters, a 3D facial shape, and the mean shape of the 3D morphable model.
- Reconstruction: In reconstruction, HeadGAN outperforms four baselines by a noteworthy margin across L1, PSNR, LPIPS, FID, FVD, and CSIM.The evaluation compares generated frames with ground truth and measures realism and identity preservation alongside reconstruction quality.
- Reenactment: HeadGAN performs well simultaneously on pose transfer, expression transfer, and identity preservation, unlike the cited FOMM variants.FOMM-Rel increases CSIM but requires matching initial poses, while FOMM-Abs transfers pose more accurately but reduces CSIM to 0.587.
- Frontalisation: For frontalisation, HeadGAN matches RaR on CSIM and surpasses the baselines on image quality and frontalisation accuracy.Frontalisation is performed by resetting the camera parameters to a frontal pose without driving frames.
- Editing: HeadGAN edits expression and pose by manually changing expression or camera parameters in a rendered 3D representation before generation.The demonstrated edits manipulate the first three principal components of expression and the camera angles.
- Ablation Study: The full model outperforms ablated variants, with the flow network essential and the 3D representation alleviating identity-preservation problems relative to landmarks.The ablation removes flow decoding layers, replaces 3D representations with landmark sketches, and evaluates the contribution of audio modality.
- Ablation Study: Audio input contributes substantially to more plausible lip movements in the lipreading evaluation.The study reconstructs BBC test videos from a random reference frame and evaluates synthetic videos with an external lipreading classifier.
5. Conclusion
HeadGAN is a one-shot head-animation method driven by 3D facial data and audio features. It achieves superior reenactment performance and photo-realism, while also supporting reconstruction, editing, and frontalisation.
- HeadGAN animates heads using 3D facial data and audio features.
- Compared to state-of-the-art methods, HeadGAN exhibits superior reenactment performance and higher photo-realism.
- The method also supports reconstruction, pose and facial expression editing, and frontalisation.
Appendix A. 3DMM fitting
The 3DMM fitting stage recovers facial shape and camera parameters from an input image by detecting, aligning, and processing dense 3D face points.
- RetinaFace-R501 regresses dense 3D facial points from the input image y.The network was pre-trained on WIDER FACE.
- Procrustes analysis registers the regressed points with the mean shape of the LSFM 3DMM.
- The fitted 3DMM yields shape parameters p and camera parameters c, with c decomposed into scale, rotation, and translation.An affine camera matrix is estimated by least squares before conversion to the scale-rotation-translation form.
Appendix B. Objective functions - Training
HeadGAN trains its generator and discriminators with adversarial, reconstruction, perceptual, feature-matching, and flow-supervision objectives for realistic frame synthesis and accurate pose transfer.
- HeadGAN trains generator G and image and mouth discriminators D and Dm using GAN Hinge loss.The mouth discriminator operates on cropped 64 × 64 mouth regions.
- The generator uses pixel-space and VGG feature-space reconstruction losses between generated and ground-truth frames.Ground-truth frames are available because training performs self-reenactment.
- Feature matching loss uses discriminator features from real and synthetic frames to improve the photo-realism of generated samples.
- L1 and VGG losses on the warped reference image supervise the dense flow network to learn flow toward the desired head pose.
- The overall generator objective sets λL1 = 50 and λVGG = λFM = 10.
C.1. Generator G
The dense flow generator combines convolutional encoding with SPADE-conditioned decoding, temporal 3D face representations, and learned up-sampling to synthesize RGB images.
- The dense flow network F uses a three-convolution encoder with instance normalization and ReLU activations, down-sampling the input twice.
- SPADE blocks inject the temporal 3D face representation xt−k:t into the decoder as its modulation input.The representation is down-sampled to match the decoder feature-map dimensions.
- Pixel Shuffle layers up-sample decoder features, followed by a convolution that computes the synthetic RGB image.The authors report better performance than simple nearest-neighbor, linear, or bilinear up-sampling.
- The image and mouth discriminators use an architecture similar to the cited discriminator and apply spectral normalization to their normalization layers.
Appendix D. Additional results
The appendix presents additional generated samples for reconstruction and reenactment, including intermediate flow and warped images, and illustrates the SPADE/AdaIN block design.
- Block architecture: The SPADE block applies feature normalization followed by modulation with gamma and beta.The displayed block labels show normalization and affine modulation before the output activation and convolution.
- Block architecture: The SPADE and AdaIN blocks retain the same number of input and output channels F and omit a residual component.The blocks are based on SPADE Resnet blocks but are adapted to preserve channel dimensionality.
Appendix E. Evaluation metrics
The evaluation uses image, video, perceptual, identity, and pose metrics to assess reconstruction quality, realism, temporal coherence, identity preservation, and pose transfer.
- Image and video metrics: L1 distance measures the mean pixelwise difference between synthesized and ground-truth frames across channels, locations, and test frames.RGB channel values are treated in the range [0, 255].
- Perceptual and realism metrics: LPIPS compares images using neural-network visual features that correlate with perceptual judgments, while FID compares real and generated image distributions.FID provides insight into the photo-realism of synthetic frames.
- Perceptual and realism metrics: FVD evaluates generated video sequences using temporal coherence and correlates with human judgments of visual quality.Unlike frame-level measures, FVD accounts for temporal relationships between frames.
- Pose metrics: Average Rotation Error measures the mean absolute Euler-angle distance from the frontal pose across generated images.ARD instead measures Euler-angle distance between driving and generated head poses across frames.