Source-linked AI summary
HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping
Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, Rongrong Ji
TL;DR
Face swapping must preserve source identity and shape while retaining target attributes and realistic appearance, but prior approaches have geometric, lighting, and occlusion limitations. HifiFace addresses these issues with 3D shape-aware identity extraction and Semantic Facial Fusion, and the authors report higher-fidelity, more photo-realistic results than previous state-of-the-art methods.
Problem
Face swapping requires preserving source identity, including face shape, while matching target expression, posture, lighting, background, and occlusion; prior methods struggle with geometric shape and appearance conditions.
Method
HifiFace combines 3DMM and 3D face reconstruction for shape-aware identity features with Semantic Facial Fusion for adaptive feature integration and blending.
Results
HifiFace generates higher-fidelity results than previous state-of-the-art face-swapping methods quantitatively and qualitatively, with improved identity preservation especially for face shape and more photo-realistic images.
Takeaways & Limitations
The framework preserves source face shape while handling image quality, occlusion, and lighting within one face-swapping model.
Takeaways & Limitations
Earlier approaches faced ghosting, complex appearance conditions, or cumbersome separate stages for lighting and occlusion, motivating HifiFace’s unified design.
Abstract
from arXiv · showhide
In this work, we propose a high fidelity face swapping method, called HifiFace, which can well preserve the face shape of the source face and generate photo-realistic results. Unlike other existing face swapping works that only use face recognition model to keep the identity similarity, we propose 3D shape-aware identity to control the face shape with the geometric supervision from 3DMM and 3D face reconstruction method. Meanwhile, we introduce the Semantic Facial Fusion module to optimize the combination of encoder and decoder features and make adaptive blending, which makes the results more photo-realistic. Extensive experiments on faces in the wild demonstrate that our method can preserve better identity, especially on the face shape, and can generate more photo-realistic results than previous state-of-the-art methods.
1 Introduction
HifiFace targets face swapping that preserves the source identity and shape while retaining target attributes and producing photo-realistic images. It combines 3D shape-aware identity extraction with semantic facial fusion to address geometric, lighting, occlusion, and blending limitations.
- Face swapping transfers source identity into a target image while retaining target pose, expression, lighting, and background attributes.
- Prior methods struggle to preserve exact face shape because recognition networks emphasize texture, while blending restricts geometric changes.
- Poisson blending can cause ghosting, and separate learning-based stages for lighting or occlusion are cumbersome and cannot solve all problems in one model.
- HifiFace combines 3DMM-derived source and target coefficients with recognition features to enforce precise source face-shape transfer.
- Its Semantic Facial Fusion module adaptively combines encoder and decoder features and learns face masks for realistic lighting, background, occlusion, and edge blending.
- HifiFace is presented as an end-to-end framework that preserves source face shape and generates high-fidelity swapped faces.
2 Related Work
Related work includes 3D-based, source-oriented, and target-oriented face-swapping pipelines. These approaches improve expression transfer or identity similarity but retain limitations in shape preservation, blending, or lighting.
- 3D-based Methods: 3D-based methods fit 3D morphable models to transfer expression and posture, then blend generated face regions into target images.
- Source-oriented GAN pipelines use face landmarks for reenactment but may weaken identity similarity and restrict face-shape changes during blending.
- Target-oriented GAN pipelines use face recognition networks to obtain identity information for face swapping.
- FaceShifter achieved state-of-the-art identity performance with two stages but could not perfectly preserve lighting despite an additional fixing stage.
3 Approach
HifiFace combines 3D shape-aware identity extraction with semantic facial fusion to preserve source identity and target attributes while improving realism. Its pipeline uses encoder features, decoder fusion, adaptive masks, and multiple supervision losses.
- Overall Pipeline: HifiFace has four parts: an encoder, decoder, 3D shape-aware identity extractor, and Semantic Facial Fusion module.The encoder extracts target attributes; the decoder fuses them with 3D shape-aware identity features.
- 3D Shape-Aware Identity: The shape-aware identity extractor combines 3D face reconstruction and recognition features to represent source identity with explicit geometric information.It regresses source and target 3DMM coefficients, combines source identity with target expression and posture, and concatenates the result with recognition identity features.
- Semantic Facial Fusion: The SFF module adaptively blends low-level encoder and decoder features using learned masks to retain target spatial details while protecting source identity.It replaces non-facial decoder regions with clean encoder features while preserving identity-rich decoder features in facial regions.
- Image Quality: Dilated masks and contour inpainting help preserve lighting, occlusion, and background despite changes in face shape.The method also uses an upsample module to support higher-resolution outputs such as 512 × 512.
- Loss Function: HifiFace trains with identity, segmentation, reconstruction, cycle, perceptual, and adversarial objectives to support identity preservation and realism.The full realism loss combines adversarial, segmentation, reconstruction, cycle, and LPIPS terms.
4 Experiments
Experiments compare HifiFace with prior methods qualitatively, quantitatively, and through ablations. The results emphasize improved source-face shape preservation, realistic attribute recovery, and the contributions of 3D shape-aware identity and Semantic Facial Fusion.
- Comparisons: HifiFace is compared with FSGAN, SimSwap, FaceShifter, AOT, and Deeper-Forensics using qualitative and quantitative evaluations.The quantitative evaluation covers ID retrieval, pose error, face shape error, and face forgery detection performance.
- Qualitative Comparisons: HifiFace preserves target attributes while recovering the source face shape and producing high-quality swapped faces.Compared with prior methods, it preserves lighting, face style, face color, expression, and occlusion while capturing source shape.
- Quantitative Comparisons: HifiFace achieves the best ID retrieval score and the lowest reported face shape error in the FF++ evaluation.Its pose preservation is comparable with other methods.
- Quantitative Comparisons: HifiFace achieves the best score on realism evaluation using FF++ and DFDC detection models.The evaluation uses AUC and AP on 10K swapped and 10K real faces for each method.
- Ablation Studies: Ablations show that 3D shape-aware identity improves face-shape similarity, while SFF improves background, occlusion, legibility, and identity similarity.Removing or replacing these components produces shape changes, artifacts, weak attribute preservation, or weaker identity similarity.
- Face Shape Preservation: Direct mask blending struggles with changed face shapes, causing limited shape changes, occlusion errors, edge artifacts, or double faces.HifiFace avoids these issues because SFF can inpaint the edge of the predicted mask.
- Face Shape Preservation: SFF difference maps show that feature recombination focuses on the predicted-mask contour, benefiting inpainting around changed face shapes.The maps compare swapped outputs with target-self outputs and mark changed or artifact-containing regions as bright.
5 Conclusions
The conclusion presents HifiFace as a high-fidelity face-swapping method that preserves source face shape and generates photo-realistic results. It attributes these outcomes to 3D shape-aware identity extraction and Semantic Facial Fusion, supported by quantitative and qualitative experiments.
- Contributions: HifiFace preserves source face shape and generates photo-realistic face-swapping results.The paper reports higher fidelity than previous state-of-the-art methods quantitatively and qualitatively.
- Contributions: The 3D shape-aware identity extractor preserves identity, including face shape.The SFF module combines feature-level and image-level information for realistic image generation.
- Implications: The authors state that HifiFace may contribute to face forgery detection research.
Network Structures
HifiFace uses residual encoder and decoder blocks with different normalization schemes, together with resampling operations to change feature-map resolution.
- Network Structures: Encoder Res-Blocks use Instance Normalization, while decoder Res-Blocks use Adaptive Instance Normalization.
- Network Structures: Resample operations use average pooling or upsampling to change feature-map size.
- Network Structures: Leaky ReLU is used as the activation function for all residual units.
More Results
Additional analyses show how interpolated shape and identity information affect generated faces, while high-resolution examples demonstrate the method on wild and real-world images.
- More Results: Interpolated 3D identity coefficients and recognition-based identity vectors are used to analyze shape and identity control.The interpolation combines source and target information for both representations.
- More Results: Shape information controls basic face shape and identity, while the recognition-based identity vector contributes identity texture.With only target identity-vector information, shape can still change but identity detail is lacking; adding source identity improves identity similarity.
- More Results: Additional results on wild face images are presented in Figures 13 and 14.The real-world examples are intended to demonstrate natural integration into high-resolution scenes.
- More Results: Figure 12 provides detailed architectural structures for HifiFace.