Source-linked AI summary
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping
Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen
TL;DR
FaceShifter addresses the limited use of target information and difficult facial occlusions in face swapping. Its AEI-Net integrates multi-level target attributes with source identity, while HEAR-Net refines occluded regions self-supervised. The framework produces realistic, high-fidelity, identity-faithful swaps without subject-specific training, according to extensive experiments.
Problem
Existing methods often use limited target information, making realistic, high-fidelity synthesis and facial-occlusion handling challenging.
Method
FaceShifter uses AEI-Net with a multi-level attributes encoder and AAD generator, followed by HEAR-Net for self-supervised occlusion refinement.
Results
FaceShifter generates considerably more realistic and faithful face swaps than other state-of-the-art methods, including for arbitrary new face pairs without subject-specific training.
Takeaways & Limitations
The two-stage framework combines adaptive target-attribute integration with annotation-free recovery of facial anomaly regions.
Abstract
from arXiv · showhide
In this work, we propose a novel two-stage framework, called FaceShifter, for high fidelity and occlusion aware face swapping. Unlike many existing face swapping works that leverage only limited information from the target image when synthesizing the swapped face, our framework, in its first stage, generates the swapped face in high-fidelity by exploiting and integrating the target attributes thoroughly and adaptively. We propose a novel attributes encoder for extracting multi-level target face attributes, and a new generator with carefully designed Adaptive Attentional Denormalization (AAD) layers to adaptively integrate the identity and the attributes for face synthesis. To address the challenging facial occlusions, we append a second stage consisting of a novel Heuristic Error Acknowledging Refinement Network (HEAR-Net). It is trained to recover anomaly regions in a self-supervised way without any manual annotations. Extensive experiments on wild faces demonstrate that our face swapping results are not only considerably more perceptually appealing, but also better identity preserving in comparison to other state-of-the-art methods.
1. Introduction
FaceShifter targets high-fidelity, occlusion-aware face swapping by integrating target attributes thoroughly during synthesis and refining anomaly regions without manual annotations. Its two-stage design addresses inconsistencies in lighting, face shape, and resolution while producing realistic, identity-faithful results.
- Face swapping must replace source identity while preserving target pose, expression, lighting, background, and other attributes.
- Previous methods struggle to synthesize realistic, high-fidelity swaps because they extract and recombine identity and attributes inadequately.Replacement-based methods are sensitive to posture and perspective, while 3D reconstruction remains inaccurate and GAN-based methods still face realism challenges.
- Thorough target-attribute integration improves consistency with scene lighting, face shape, and target image resolution.The framework is designed to preserve target pose and expression while fitting the synthesized face seamlessly into the target image.
- FaceShifter’s first stage, AEI-Net, extracts multi-level target attributes and adaptively integrates them with identity embeddings for synthesis.Its AAD layers learn where to integrate attributes or identity embeddings rather than relying on single-level integration.
- HEAR-Net addresses facial occlusions by recovering anomaly regions self-supervised, without manual annotations.It uses reconstruction deviations from feeding the same image as source and target to guide refinement, including glasses, shadows, and reflections.
- Experiments show FaceShifter produces more realistic and faithful results than other state-of-the-art methods, while remaining subject agnostic.Once trained, it can be applied to new face pairs without subject-specific training.
2. Related Works
Face-swapping research progressed from pose-limited pixel replacement toward 3D-based and GAN-based approaches. Subject-agnostic methods reduce identity-specific training, while occlusion handling has used supervised segmentation or the self-supervised strategy described for FaceShifter.
- 2. Related Works: Early face-swapping methods handled mainly similar poses by directly replacing face pixels.
- 3D-Based Approaches: 3D-based approaches model pose or perspective differences, but may require user interaction, omit expression handling, or depend on unsatisfactory reconstruction accuracy and robustness.
- GAN-Based Approaches: GAN-based approaches include identity-specific models and later subject-agnostic methods that recombine source identity with target information.RSGAN, FSNet, and IPGAN represent identity, face, hair, or attributes as embeddings for synthesis.
- Occlusion Handling: Occlusion-aware face swapping has used supervised segmentation masks, whereas FaceShifter identifies occlusions self-supervised without manual annotations.
3. Methods
FaceShifter uses a two-stage pipeline: AEI-Net integrates source identity with multi-level target attributes to generate a high-fidelity swap, while HEAR-Net refines facial occlusions using heuristic reconstruction errors.
- FaceShifter takes source and target images, using the source for identity and the target for pose, expression, lighting, and background attributes.
- First stage: AEI-Net: AEI-Net combines an identity encoder, a multi-level attributes encoder, and an AAD generator to produce the first-stage swapped face.The attributes encoder extracts target features at multiple spatial resolutions rather than compressing them into one vector.
- First stage: AEI-Net: AAD layers adaptively combine identity and attribute activations through feature-level modulation and attention masks.The mask weights the element-wise combination of identity and attribute activations, allowing different embeddings to influence different face regions.
- First stage: AEI-Net: AEI-Net is trained with adversarial, identity-preservation, attribute-preservation, and same-image reconstruction losses.The reconstruction loss applies when source and target images are identical, while attribute preservation compares multi-level embeddings.
- Second stage: HEAR-Net: HEAR-Net uses the difference between an AEI-Net reconstruction and the target image to locate occlusions, then refines the first-stage swap with a U-Net.It is trained fully self-supervised, without manual annotations, and includes identity, consistency, and reconstruction objectives.
4. Experiments
Experiments compare FaceShifter with prior methods, evaluate identity, pose, expression, and user preferences, and analyze its adaptive integration and occlusion refinement. Results show improved fidelity to source identity and target attributes, while HEAR-Net handles diverse errors and occlusions.
- Qualitative Comparison: FaceShifter better preserves source face shapes and target lighting and resolution than FaceSwap, Nirkin et al., DeepFakes, and IPGAN.The comparisons attribute competing methods’ inconsistencies to synthesizing an inner face region before blending it into the target.
- Quantitative Comparison: FaceShifter achieves a higher ID retrieval score with a large margin on FaceForensics++ videos.ID retrieval averages whether the nearest original video frame belongs to the correct source video.
- Quantitative Comparison: FaceShifter is advantageous in expression preservation while remaining comparable with other methods in pose preservation.Pose and expression errors are measured as L-2 distances between swapped and target vectors.
- Human Evaluation: Human evaluators preferred FaceShifter over the other three methods by large margins across identity, target attributes, and realism studies.The study collected responses from 100 evaluators using randomly drawn FaceForensics++ face pairs.
- Analysis of the Framework: Adaptive attentional masks produce sharper, less ghosted faces than additive or concatenative embedding integration baselines.The masks control how identity and attribute embeddings are fused in AAD layers.
- Analysis of the Framework: Identity embeddings affect low-level features broadly but activate selectively around eyes, mouth, and contours at middle levels, while compressed attributes cause blurriness.Nearest-neighbor queries show attributes embeddings encode pose, hair color, expression, and sunglasses, helping explain occasional occlusion preservation.
- Second-Stage Refinement: HEAR-Net handles occlusions from medals, hands, hair, face painting, masks, translucent objects, eyeglasses, headscarves, and floating text.It also corrects occasional color shifts and can rectify face shape under very large target poses.
5. Conclusions
FaceShifter combines two stages for high-fidelity, occlusion-aware face swapping: AEI-Net synthesizes the face, while HEAR-Net recovers anomalous regions without manual annotations.
- AEI-Net adaptively integrates identity and target attributes to synthesize high-fidelity swapped faces.
- HEAR-Net recovers anomaly regions in a self-supervised way without manual annotations.
- The framework generates realistic face images from arbitrary face pairs without subject-specific training.
- Extensive experiments show significantly better performance than previous face swapping methods.
A. Network Structures
The appendix provides detailed network structures for AEI-Net and HEAR-Net, including convolutional-layer notation, activation settings, and AAD residual-block channel definitions.
- Figure 12 presents the detailed structures of both AEI-Net and HEAR-Net.
- Conv k,s,p denotes a convolutional layer with kernel size k, stride s, and padding p.
- ConvTranspose k,s,p denotes a transposed convolutional layer using the same k, s, and p notation.
- All LeakyReLU activations use α = 0.1.
- AAD ResBlk(cin, cout) represents an AAD residual block with cin input channels and cout output channels.
B. Training Strategies
Training uses specified sampling ratios and optimization settings, while occlusion augmentation combines hand images with rendered ShapeNet objects and synthetic occlusions.
- AEI-Net training uses 80% samples with Xt != Xs, whereas HEAR-Net training uses 50%.
- All networks are trained with ADAM using β1 = 0, β2 = 0.999, and lr = 0.0004.
- AEI-Net is trained for 500K steps and HEAR-Net for 50K steps using four P40 GPUs with eight images per GPU.
- Occlusion augmentation uses hand images from EgoHands and GTEA Hand2K.
- Rendered ShapeNet objects are also used to create synthetic occlusions.