Source-linked AI summary
FSGAN: Subject Agnostic Face Swapping and Reenactment
Yuval Nirkin, Yosi Keller, Tal Hassner
TL;DR
Face swapping and reenactment require realistic transfer of identity, pose, and expression without costly subject-specific training. FSGAN combines subject-agnostic recurrent reenactment with interpolation, inpainting, and blending, and reports superior qualitative and quantitative results over existing systems.
Problem
Subject-specific face manipulation methods require training for each subject or pair, typically with thousands of images, while occluded regions remain difficult to handle.
Method
FSGAN uses subject-agnostic recurrent reenactment for pose and expression transfer, continuous face-view interpolation, face completion for missing regions, and blending into the target image.
Results
FSGAN's qualitative and quantitative results surpass existing work even without training on subject-specific images.
Takeaways & Limitations
FSGAN eliminates laborious subject-specific data collection and model training for face swapping and reenactment.
Takeaways & Limitations
Identity and texture quality degrade as reenactment yaw differences increase, while excessive iterations blur texture; performance is also limited by training-data resolution and sparse landmark tracking.
Abstract
from arXiv · showhide
We present Face Swapping GAN (FSGAN) for face swapping and reenactment. Unlike previous work, FSGAN is subject agnostic and can be applied to pairs of faces without requiring training on those faces. To this end, we describe a number of technical contributions. We derive a novel recurrent neural network (RNN)-based approach for face reenactment which adjusts for both pose and expression variations and can be applied to a single image or a video sequence. For video sequences, we introduce continuous interpolation of the face views based on reenactment, Delaunay Triangulation, and barycentric coordinates. Occluded face regions are handled by a face completion network. Finally, we use a face blending network for seamless blending of the two faces while preserving target skin color and lighting conditions. This network uses a novel Poisson blending loss which combines Poisson optimization with perceptual loss. We compare our approach to existing state-of-the-art systems and show our results to be both qualitatively and quantitatively superior.
1. Introduction
FSGAN addresses limitations in face swapping and reenactment by offering a subject-agnostic approach that manipulates pose, expression, and identity without person-specific training. It combines new interpolation and loss-function contributions and reports results surpassing existing work.
- Earlier methods typically handled swapping or reenactment separately and often relied on 3D face representations.
- Subject-specific methods require training for each subject or pair, often using thousands of face images, while occlusions remain difficult to handle.
- FSGAN simultaneously manipulates pose, expression, and identity without person-specific or pair-specific training, while producing high-quality, temporally coherent results.
- The method introduces continuous interpolation between multiple face views using reenactment, Delaunay Triangulation, and barycentric coordinates.
- FSGAN proposes stepwise consistency and Poisson blending losses for progressive reenactment and seamless integration of the source face into its new context.
- Qualitative and quantitative comparisons report that FSGAN surpasses existing work even without training on subject-specific images.
2. Related work
Related work spans 3D-based, GAN-based, hybrid, and feature-disentanglement approaches to facial manipulation. These methods established realistic synthesis and reenactment but differ in their handling of representations, training requirements, and occlusions.
- 3D based methods: Early swapping and reenactment methods used manual procedures or fitted 3D morphable face models to transfer expressions and facial appearance.
- 3D based methods: Nirkin et al. showed realistic face swapping without 3D face-shape estimation, but their method was not end-to-end trainable and required special attention to occlusions.
- GAN-based methods: GAN-based approaches improved realistic image generation and supported unsupervised transformations, although early GAN training could be unstable and limited to low-resolution images.
- Facial manipulation using GANs: Facial-manipulation GANs applied multi-scale translation, perceptual loss, attention maps, or landmarks for high-resolution synthesis and reenactment.
- Hybrid methods: Hybrid 3D/deep methods rendered subject-specific 3D morphable models and mapped synthetic views to photorealistic images with trained generators.
- Feature disentanglement: Feature-disentanglement methods separated face and hair or identity and geometric components in latent representations for face manipulation.
3. Face swapping GAN
FSGAN combines reenactment, segmentation, view interpolation, completion, and blending into an end-to-end face-swapping pipeline. Its recurrent reenactment handles pose changes incrementally, while interpolation and Poisson-based blending support subject-agnostic, realistic results.
- FSGAN replaces the target face with the source face while retaining the target’s pose and expression.
- The pipeline reenacts the source using target landmark heatmaps, predicts face and hair segmentations, completes missing regions, and blends the completed face into the target.
- Face reenactment and segmentation: Large pose changes are handled through recurrent small-step reenactment, with stepwise consistency training and interleaved optimization of reenactment and segmentation networks.
- Training losses: Training combines domain-specific perceptual features, pixelwise reconstruction, and multi-scale adversarial objectives to preserve facial detail, colors, and realism.
- Face view interpolation: Continuous face-view interpolation uses an appearance map, triangulated pose space, and barycentric coordinates to combine nearby source views for a target pose.
- Face blending: A novel Poisson blending loss combines Poisson optimization with perceptual loss to integrate completed faces under differing skin tones and lighting conditions.
4. Datasets and training
FSGAN is trained using curated face-video frames and complementary datasets for perceptual, segmentation, and attribute objectives. Training uses subject-balanced frame selection and progressively reduced learning rates.
- Training data: Frames are pruned for redundant views and motion blur, then filtered using face-pixel coverage and landmark variance.At most 100 frames are retained per subject, and each iteration samples frames from two randomly chosen subjects.
- Auxiliary datasets: VGG-19 perceptual-loss networks are trained on VGGFace2 for recognition and CelebA for facial-attribute classification.VGGFace2 contains 3.3M images of 9,131 identities; CelebA contains 202,599 images with 40 binary attributes.
- Auxiliary datasets: The segmentation CNN uses approximately 10k segmented face images, approximately 3k LFW Parts images, and 1k Figaro images with hair segmentations.FaceForensics++ additionally provides 1,000 real videos and 1,000 synthetic videos generated with DeepFakes and Face2Face.
- Optimization: Generators are trained from scratch with Adam at learning rate 0.0002, halved every ten epochs, using fixed loss weights and a scheduled reenactment weight.All networks were trained on eight NVIDIA GPUs.
5. Experimental results
Experiments compare FSGAN with prior swapping and reenactment methods on FaceForensics++ videos, including qualitative results, quantitative metrics, and ablations. The reported results show strong pose and expression retention, while iterative reenactment helps with large pose differences.
- Experimental setup: Experiments compare FSGAN with DeepFakes, Nirkin et al., and Face2Face on FaceForensics++ videos, including component ablations.The same FaceForensics++ source-target pairs are used for comparisons, and ablations evaluate pipeline components.
- Qualitative reenactment: Multiple reenactment iterations better preserve identity and texture for large angle differences than one iteration.Figure 5 compares one and three iterations using the same subject for clarity.
- Qualitative swapping: FSGAN is evaluated on unseen FaceForensics++ video subjects without training on those videos, with pose-matched source frames used for the image-to-image baseline.Face-view interpolation enables a video-to-video comparison with FSGAN.
- Quantitative evaluation: Quantitative evaluation measures source identity, image quality, target pose, and target expression using verification, SSIM, Euler-angle, and landmark metrics.Pose error is measured in degrees and expression error in pixels across the first 100 frames of 500 videos.
- Quantitative results: FSGAN preserves identity and image quality similarly to prior methods while retaining target pose and expression much better than its baselines.The authors caution that SSIM may underrepresent small face artifacts because they occupy only a small part of the image.
- Ablation study: Across ablations, source identities remain preserved, while the full pipeline retains target poses and expressions best.Adding inpainting and blending causes a slight SSIM drop, while their pose and expression errors remain similar.
6. Conclusion
The method removes subject-specific data collection and training, but its quality is constrained by view differences, repeated reenactment iterations, training resolution, and sparse landmark tracking.
- Larger facial yaw differences degrade identity and texture quality.
- Too many face reenactment generator iterations blur texture.
- The method is limited to the resolution of its training data.
- Sparse landmark tracking does not fully capture the complexity of facial expressions.
- The method eliminates laborious subject-specific data collection and model training, making face swapping and reenactment accessible to non-experts.
A. Additional qualitative results
Additional results evaluate challenging face-swapping examples and compare the method qualitatively with Nirkin et al. and DeepFakes.
- Additional quantitative face-swapping results use challenging pairs with partial occlusions, different ethnicities, and different skin colors.
- Figure 10 provides an additional qualitative face-swapping comparison with Nirkin et al. and DeepFakes on FaceForensics++.
B. The architecture of the generator CNNs
The generator architecture uses global and enhancer modules based on bottleneck layers, while the segmentation network uses U-Net with bilinear upsampling; additional figures show the architecture and reenactment comparison.
- The generators Gr, Gc, and Gb use a pix2pixHD-based architecture with global and enhancer modules.
- The face segmentation network Gs is based on U-Net and replaces deconvolution with bilinear interpolation upsampling.
- Figure 11 presents an additional face-reenactment comparison with Face2Face on FaceForensics++.
- Figure 12 depicts the global generator and enhancer architectures, including bottleneck blocks and bilinear upsampling followed by convolution.