Source-linked AI summary
FaR-GAN for One-Shot Face Reenactment
Hanxiang Hao, Sriram Baireddy, Amy R. Reibman, Edward J. Delp
TL;DR
Face reenactment must handle complex facial geometry and movement, while prior methods commonly require many images of the same identity. FaR-GAN uses one source face image and target facial landmarks to generate the same identity with the target expression, achieving better visual quality than compared methods on VoxCeleb1. Its scope is limited by identity gaps for large appearance differences and by the omission of pupil movement from landmark representations.
Problem
Face reenactment requires modeling complex facial geometry and movement, and prior methods usually need many same-identity images for appearance modeling.
Source: §1 · p. 2Method
FaR-GAN is a GAN-based one-shot model that combines source appearance information with target expression landmarks to generate the reenacted face.
Source: §1 · p. 2, §3.1 · pp. 3–4Results
Takeaways & Limitations
The model can reenact faces from one image without assumptions about source identity, facial pose, expression, or background.
Source: §1 · p. 2, §5 · p. 8Large appearance differences between source and landmark-providing identities can leave a visible identity gap, and pupil movement is not represented.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Animating a static face image with target facial expressions and movements is important in the area of image editing and movie production. This face reenactment process is challenging due to the complex geometry and movement of human faces. Previous work usually requires a large set of images from the same person to model the appearance. In this paper, we present a one-shot face reenactment model, FaR-GAN, that takes only one face image of any given source identity and a target expression as input, and then produces a face image of the same source identity but with the target expression. The proposed method makes no assumptions about the source identity, facial expression, head pose, or even image background. We evaluate our method on the VoxCeleb1 dataset and show that our method is able to generate a higher quality face image than the compared methods.
1. Introduction
One-shot face reenactment seeks to generate a realistic target-expression face from one source image while preserving identity, appearance, and background. FaR-GAN addresses this challenge with a GAN-based model that composes appearance and expression information and achieves higher image quality than compared methods.
Problem
- One-shot face reenactment uses one source image and target expression landmarks to produce a photo-realistic face while preserving identity, appearance, and background.The task requires transforming source facial shape to a target shape while preserving the source face's appearance and identity.Source: §1 · p. 2
Proposed approach
- FaR-GAN transforms an input face to target expressions while retaining the source identity, background, and clothes without assumptions about identity, expression, pose, or background.The target expression is specified by facial landmarks.Source: §1 · p. 2
Contribution
Result
- FaR-GAN produces reenacted images with higher image quality than the compared methods.Source: §1 · p. 2
2. Related Work
Prior face reenactment methods use 3D modeling, GAN-based generation, or few-/one-shot learning to transfer expressions while preserving identity. These approaches differ in their handling of geometry, rendering, unknown identities, and intermediate representations.
Face Reenactment by 3D Modeling
- 3D-based methods fit a 3D morphable model to source and target faces, transfer expression components, and use neural textures with a U-Net renderer for end-to-end synthesis.The neural rendering pipeline is designed to synthesize images from imperfect coarse 3D model information.Source: §2 · p. 2
Face Reenactment by GANs
- GAN-based methods generate high-quality face images, but ReenactGAN maps target expressions into a boundary latent space and requires an identity-specific decoder, preventing use with unknown identities.The boundary latent space is intended to retain facial-expression information without identity-related information.Source: §2 · p. 2
- Few-shot and one-shot methods address unknown identities through frontalization and interpolation maps, direct expression transfer, or auto-encoder features injected with SPADE.X2Face uses frontalization before target-expression transfer, while Zakharov et al. directly transfer expressions and Zhang et al. use multi-scale landmark masks with SPADE [22].Source: §2 · p. 2
3. Proposed Method
FaR-GAN separates facial-expression encoding from source-identity transformation, using spatially adaptive conditioning and adversarial training to reenact faces from target landmarks. Its training objective combines realism, pixel, perceptual, and identity constraints, with implementation choices for stochastic detail and long-range information.
3.1. Model Architecture
- FaR-GAN uses an embedder to encode target landmark information and a U-Net-based transformer to reenact the source identity with those landmarks.The embedder preserves spatial information through multiresolution features rather than a single embedding vector.Source: §3.1 · p. 3, Figure 2 · p. 3
- SPADE injects landmark information through learned spatial scale and bias parameters, avoiding AdaIN because instance normalization can wash out sparse landmark semantics.Using multiscale landmark masks instead produced visible landmark-contour artifacts, so the method conditions SPADE on embedder features.Source: §3.1 · p. 4, Figure 4 · p. 4, Figure 3 · p. 4
- The discriminator receives either a reenacted or ground-truth face concatenated with its landmark mask, guiding both realistic appearance and target-landmark correctness.The generator also injects independent Gaussian noise at each decoder resolution to model stochastic portrait details such as hair and skin texture.Source: §3.1 · p. 4
3.2. Loss Function
- Training samples two frames from one video identity, using one frame as source and the other frame's landmark mask as the reenactment target.The generator is trained end-to-end with adversarial, pixel-wise L1, perceptual, and identity losses.Source: §3.2 · p. 5
- The adversarial term uses LSGAN, while VGG-19 perceptual loss and VGGFace identity loss constrain semantic similarity and source-identifying information.The authors selected LSGAN over vanilla GAN and WGAN-GP based on visual reenactment quality.Source: §3.2 · p. 5
3.3. Implementation Details
- Implementation details include residual and spectrally normalized convolutions, SPADE noise blocks, self-attention in generator and discriminator, Adam optimization, and linearly decayed learning rate.The weights for L1, perceptual, and identity losses are 20, 2, and 0.2, respectively; training uses an initial learning rate of 5e−5 for 100 epochs.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate FaR-GAN on VoxCeleb1 against prior methods and examine landmark representations, the discriminator, self-attention, and noise injection. FaR-GAN achieves stronger perceptual quality, while ablations identify components that improve realism and detail.
4.1. Dataset
- The VoxCeleb1 evaluation uses 24,997 videos from 1,251 identities, with identities split 8:2 between training and testing.Images are cropped, resized to 256 × 256, and paired with 68-point facial landmarks extracted using Dlib.Source: §4.1 · p. 5
4.2. Experimental Results
- FaR-GAN outperforms X2Face and Few-Shot in FID while achieving similar SSIM, with accurate expression transfer and background preservation.The proposed method also produces more high-frequency information than Few-Shot and avoids the wrinkle artifacts observed in X2Face.Source: §4.2 · p. 6, Table 1 · p. 6
4.3. Ablation Study
- Adding the discriminator produces more details such as hair, teeth, and background, indicating that it guides the generator toward better synthetic images.The discriminator receives the reenacted or ground-truth face together with its landmark mask.Source: §4.3 · p. 6, Figure 6 · p. 7
- Contour-based landmark masks achieve much lower FID than binary masks despite similar SSIM, because distinct facial-part colors provide richer pose and expression information.The contour-based representation helps the embedder treat facial components separately.Source: §4.3 · p. 6, Table 2 · p. 6
- Self-attention reduces FID from 63.9 to 35.3, and noise injection further lowers it to 27.1, making their combination the best-performing configuration.Without these components, visual results contain blob-like artifacts; noise injection additionally restores high-frequency hair details.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Figure 9 · p. 8
5. Conclusion and Future Work
FaR-GAN performs one-shot face reenactment from a single source image and target landmarks while preserving source identity and scene attributes. The authors report better visual quality than compared methods but identify identity gaps and missing gaze modeling as remaining limitations.
5. Conclusion and Future Work
- FaR-GAN transforms one image from any identity into a high-quality face with a target expression, without assumptions about input identity, pose, or expression.The model is evaluated on VoxCeleb1 and reported to produce better visual quality than compared methods.Source: §5 · p. 8
- A visible identity gap remains when the target-landmark identity differs substantially from the source, including differences in gender or face size.The authors propose future finetuning to reduce these identity changes.Source: §5 · p. 8