Source-linked AI summary
Fast Face-swap Using Convolutional Neural Networks
Iryna Korshunova, Wenzhe Shi, Joni Dambre, Lucas Theis
TL;DR
The paper addresses face swapping that changes identity while preserving pose, expression, and lighting. It trains a convolutional feed-forward network using style-transfer objectives over multiple target-identity photographs, with alignment and segmentation before and after transformation. The resulting system produces photorealistic face swaps in or near real time, though performance is weaker for profile views and can amplify sidelighting.
Problem
Face swapping requires changing identity while keeping the input pose, facial expression, gaze direction, hairstyle, and lighting intact.
Method
A convolutional neural network transforms aligned input faces using a multi-image target-identity style loss with explicit lighting matching, followed by realignment and segmentation-based compositing.
Results
The feed-forward system achieves high photorealism and performs face swapping in or near real time.
Takeaways & Limitations
The work provides a proof of concept for fully automatic, nearly real-time face swapping that preserves original pose, expression, and lighting.
Takeaways & Limitations
The network works better for frontal than profile views and can amplify sidelights under harsh illumination conditions.
Abstract
from arXiv · showhide
We consider the problem of face swapping in images, where an input identity is transformed into a target identity while preserving pose, facial expression, and lighting. To perform this mapping, we use convolutional neural networks trained to capture the appearance of the target identity from an unstructured collection of his/her photographs.This approach is enabled by framing the face swapping problem in terms of style transfer, where the goal is to render an image in the style of another one. Building on recent advances in this area, we devise a new loss function that enables the network to produce highly photorealistic results. By combining neural networks with simple pre- and post-processing steps, we aim at making face swap work in real-time with no input from the user.
1. Introduction and related work
The paper frames face swapping as changing identity while preserving pose, expression, gaze, hairstyle, and lighting, using feed-forward neural style transfer for photorealistic, near-real-time results. It addresses limitations of prior methods that alter expression, lack output-identity control, require extensive data, or depend on expensive optimization and user guidance.
- Prior database-based replacement methods lack control over output identity and alter the input facial expression.
- Video face replacement introduces temporal alignment, facial-performance tracking, and temporal-consistency challenges, while remaining complex and user-guided.
- Face swapping replaces person A’s identity with person B’s while preserving pose, facial expression, gaze direction, hairstyle, and lighting.
- The method casts pose and expression as content and the target identity as style, drawing on neural style transfer.
- Unlike single-reference style transfer, the proposed loss uses multiple target-identity images to approximate a style manifold and explicitly match lighting conditions.
- A feed-forward network produces highly photorealistic face swaps and enables operation in or near real time after training.
2. Method
The method frames identity replacement as style transfer, using aligned images and a feed-forward convolutional network trained with content, multi-image style, lighting, and smoothness losses. Pre- and post-processing restore image regions not preserved by the transformation network.
- Input pose and expression provide content, while the target person’s identity provides style; lighting is handled separately.
- A convolutional network transforms input image x into output image ˆx using a training set of target-identity style images.The style images are used only during training.
- 2. Method: Face alignment and segmentation support the pipeline by aligning 68 facial keypoints and restoring the input background and hair through seamless cloning.The method assumes content and style images are aligned to a frontal-view reference face.
- 2.1. Transformation network: The multiscale transformation network combines branches processing downsampled inputs and produces a three-channel output from its highest-resolution branch.The 128×128 architecture has 1M parameters; larger-input branches can be added straightforwardly.
- 2.2. Loss functions: The objective jointly minimizes VGG feature-space content and style losses, with style represented by matching neural patches across multiple target-identity images at corresponding locations.The multi-image search differs from prior single-style-image matching and uses only the best Nbest style images for each input.
- 2.2. Loss functions: A siamese lighting network supplies an illumination penalty, while total-variation regularization encourages spatial smoothness in generated images.The lighting network distinguishes equal from different illumination conditions using aligned, equal-pose image pairs.
3. Experiments
Experiments show that the trained networks can swap identities while largely preserving pose, expression, gaze, and lighting, with similar quality across tested target identities. Results also expose trade-offs involving style strength, lighting, frontal views, occlusions, and residual identity features.
- Training setup: Approximately 60 Nicolas Cage photos, horizontally flipped for augmentation, trained the target-identity model alongside CelebA’s over 200,000 content images.A Taylor Swift model trained with approximately 60 images achieved similar quality using the same hyperparameters.
- Training setup: The 256 × 256 transformation network has about 2M parameters, and its additional branch required 12 hours of training.The higher-resolution branch was optimized while keeping the rest of the network fixed.
- Failure cases and comparisons: Increasing the style weight from α = 80 to α = 120 alters facial expression, illustrating a trade-off between identity strength and expression preservation.Residual input-identity features can be reduced by increasing style loss, but the resulting expression change is undesirable.
- Face-swapping results: The network noticeably changes noses, eyes, eyebrows, lips, and wrinkles while preserving head pose, expression, lighting, and gaze direction.Overall head shape is mostly unchanged, and differing input and target skin colors can produce an average skin tone.
- Face-swapping results: Omitting the lighting loss produces flat lighting and removes shadows in CageNet outputs.The lighting-loss ablation directly compares CageNet trained with and without that objective term.
- Failure cases and comparisons: The method works better for frontal than profile views, while occlusions such as glasses can be removed and cause artefacts.The frontal-view bias may reflect the greater prevalence of frontal images in both training and style sets.
4. Discussion and future work
The method achieves photorealistic face swapping but remains sensitive to style-image coverage, viewpoint, lighting, expression, occlusions, and alignment accuracy. The authors identify several loss, data, segmentation, and feature-space improvements.
- Evaluation: Subjective evaluation was used because style-transfer loss values do not provide a feasible quantitative assessment, and fair comparison with prior face-swapping methods is difficult.Image-manipulation methods produce crisp images but may transfer pose and expression inaccurately, while more complex systems require many target-identity images.
- Failure cases: Frontal views generally produce better results than profile views, likely because online photo collections contain more frontal portraits.Quality also degrades with uncommon expressions and harsh input lighting; larger, more carefully selected style collections may reduce these problems.
- Future improvements: Occluding objects such as glasses are not preserved reliably, motivating a loss function that explicitly protects them.The current system instead uses segmentation in post-processing to preserve the input background.
- Future improvements: Extreme viewing angles can make dlib’s facial keypoint estimates inaccurate, especially along the jawline, producing artifacts in swapped images.The detector approximates invisible keypoints with an average frontal-face shape when head rotation is extreme.
- Future improvements: Replacing the generic VGG network with VGG-Face could improve content and style losses because it was trained to recognize approximately 3K individuals.The authors suggest its identity-oriented feature space may better suit face swapping.
5. Conclusion
The paper presents a proof of concept for fully automatic, nearly real-time face swapping with deep neural networks. Its specific contribution is changing identity while preserving pose, expression, and lighting, while acknowledging remaining issues.
- Conclusion: The authors provide a proof of concept for fully automatic, nearly real-time face swapping with deep neural networks.The conclusion characterizes the method as a specific type of face replacement.
- Conclusion: The proposed method changes identity without altering the original pose, facial expression, and lighting.The authors state that this particular face-replacement problem had not previously been addressed to their knowledge.
- Conclusion: Feed-forward neural networks offer ease of implementation, easier addition of identities, control over effect strength, and potential for more natural-looking results.The authors note that issues still remain in neural-network-based face swapping.
- Conclusion: The authors identify neural-network-based face swapping as a challenging problem with issues that still need to be overcome.
Photo credits
The paper identifies the rights holder of the photograph used in Figure 2 and lists the licensing or public-domain status of the other photographs.
- Photo credits: The photograph used in Figure 2 is owned by Peter Matthews.
- Photo credits: Other photographs were public-domain images or were made available under a Creative Commons license by the listed rights holders.The named rights holders include Angela George, Manfred Werner, David Shankbone, Alan Light, Gordon Correll, AngMoKio, Aleph, Diane Krauss, and Georges Biard.