Source-linked AI summary

SimSwap: An Efficient Framework For High Fidelity Face Swapping

Renwang Chen, Xuanhong Chen, Bingbing Ni, Yanhao Ge

arXiv:2106.06340v1cs.CV

TL;DR

Face swapping seeks to transfer a source identity into a target face while preserving target attributes, but existing approaches can lack arbitrary-identity generalization or attribute preservation. SimSwap uses feature-level identity injection and weak feature matching, achieving competitive identity performance with better attribute preservation than previous methods.

  • Problem

    Face swapping methods may lack arbitrary-identity generalization or fail to preserve target attributes such as expression and gaze direction.

  • Method

    SimSwap injects source identity into target features with an ID Injection Module and uses Weak Feature Matching Loss to preserve target attributes.

  • Results

    SimSwap generates visually appealing face swaps and preserves target attributes better than previous methods while maintaining competitive identity performance.

  • Takeaways & Limitations

    Feature-level identity injection extends identity-specific face swapping to arbitrary identities, while weak feature matching supports preservation of target attributes.

  • Takeaways & Limitations

    The original DeepFakes decoder integrates one source identity into its weights, so it can be applied only to that specific identity.

Abstract

from arXiv · show

We propose an efficient framework, called Simple Swap (SimSwap), aiming for generalized and high fidelity face swapping. In contrast to previous approaches that either lack the ability to generalize to arbitrary identity or fail to preserve attributes like facial expression and gaze direction, our framework is capable of transferring the identity of an arbitrary source face into an arbitrary target face while preserving the attributes of the target face. We overcome the above defects in the following two ways. First, we present the ID Injection Module (IIM) which transfers the identity information of the source face into the target face at feature level. By using this module, we extend the architecture of an identity-specific face swapping algorithm to a framework for arbitrary face swapping. Second, we propose the Weak Feature Matching Loss which efficiently helps our framework to preserve the facial attributes in an implicit way. Extensive experiments on wild faces demonstrate that our SimSwap is able to achieve competitive identity performance while preserving attributes better than previous state-of-the-art methods. The code is already available on github: https://github.com/neuralchen/SimSwap.

1 INTRODUCTION

Face swapping must generalize across arbitrary faces, transfer source identity, and preserve the target’s expression, posture, and lighting. SimSwap addresses generalization with feature-level identity injection and attribute preservation with a weak feature-matching loss.

  • Face swapping frameworks should adapt to arbitrary faces while matching the source identity and preserving the target’s attributes.The target attributes include expression, posture, and lighting.
  • Source-oriented methods transfer target attributes to the source image but remain sensitive to source posture and lighting and may reproduce expression inaccurately.
  • SimSwap introduces the ID Injection Module to embed source identity into target features, removing decoder dependence on one identity.This extends an identity-specific architecture to arbitrary identities.
  • SimSwap uses Weak Feature Matching Loss because directly modifying target features can affect attributes, while explicitly constraining every attribute is impractical.

2 RELATED WORK

Face-swapping methods are primarily divided into source-oriented and target-oriented approaches, distinguished by whether they operate on the source image or target features.

  • Face-swapping methods are mainly divided into source-oriented and target-oriented approaches.
  • Source-oriented Methods: Source-oriented methods transfer target attributes to the source face before blending it into the target image.Early methods used 3D models for posture and lighting transfer but required manual intervention.

3 METHOD

SimSwap separates source-identity transfer from target-feature encoding and decoding, enabling arbitrary-identity face swapping while preserving target attributes. Its ID Injection Module and Weak Feature Matching Loss implement these goals through feature-level identity embedding and implicit attribute constraints.

  • 3 METHOD: SimSwap encodes target features, injects source identity, and decodes the modified features into the result image.The generator comprises an Encoder, ID Injection Module, and Decoder.
  • 3.1 Limitation of the DeepFakes: The original identity-specific architecture limits generalization because source identity is integrated into decoder weights.Its decoder can therefore be applied only to one specific identity.
  • 3.2 Generalization to Arbitrary Identity: The ID Injection Module extracts a source identity vector and embeds it into target features using ID-Blocks with adaptive instance normalization.The module has identity extraction and embedding parts, with a face-recognition network producing the source identity vector.
  • 3.2 Generalization to Arbitrary Identity: After injection, the decoder restores the image from modified features, leaving identity modification to the module and allowing arbitrary identities.Training uses source images from different identities, so decoder weights should not depend on one specific identity.
  • 3.3 Preserving the Attributes of the Target: Weak Feature Matching Loss compares deep discriminator features of the result with the target to implicitly preserve high-semantic attributes.It replaces the unavailable ground-truth image with the input target and uses only the last discriminator layers.
  • 3.4 Overall Loss Function: The training objective combines identity, reconstruction, adversarial, gradient-penalty, and weak feature-matching losses.Identity Loss constrains generated and source identity vectors, while Reconstruction Loss regularizes same-identity source-target pairs.

4 EXPERIMENTS

Experiments evaluate SimSwap on excluded wild-face images, comparisons with prior methods, and ablations of feature matching and identity weighting. SimSwap transfers source identity while preserving target attributes, with experiments showing a balance between identity modification and attribute preservation.

  • Qualitative Face Swapping Results: SimSwap transfers source identity while preserving target expression, gaze direction, posture, and lighting, including exaggerated expressions, face stripes, and large rotations.The face matrix uses movie-scene targets and Internet sources excluded from training; source posture and expression need not match the target.
  • Comparison with Other Methods: Compared with DeepFakes and FaceShifter on FaceForensics++, SimSwap achieves competitive identity performance while preserving attributes better.DeepFakes show severe lighting and posture mismatch, while FaceShifter does not fully preserve expression and gaze direction.
  • Comparison with Other Methods: SimSwap reproduces subtle expressions and slightly better lighting than FaceShifter, whose stronger identity modification often sacrifices expression and gaze preservation.For example, SimSwap reproduces narrowed eyes in a target face, whereas FaceShifter fails to do so.
  • Comparison with Other Methods: Compared with FSGAN, SimSwap better preserves expression, gaze direction, and lighting and is less affected by the source image’s attributes.The comparison describes SimSwap as more robust to the input source image and producing more convincing results.
  • Analysis of SimSwap: Removing Feature Matching maximizes identity performance but causes gaze deviations, whereas SimSwap retains a relatively high ID retrieval with medium Reconstruction Loss.The authors select SimSwap as a balance between identity and attribute performance; identity weighting and Feature Matching selection adjust this trade-off.
  • Analysis of SimSwap: The wFM-id+ variant has a smaller Identity Loss but is overfitted to that loss and more likely to introduce source-face hair, so SimSwap provides more stable results.SimSwap and wFM-id+ usually produce similar visual outputs, but identity retrieval and visual inspection favor nFM over wFM-id+ despite its lower Identity Loss.

5 CONCLUSION

SimSwap is an efficient framework for generalized, high-fidelity face swapping that transfers source identity while preserving target attributes better than previous methods.

  • SimSwap targets generalized and high-fidelity face swapping.
  • The ID Injection Module transfers source identity at feature level and extends identity-specific swapping to arbitrary identities.
  • Weak Feature Matching Loss supports preservation of the target face’s attributes.
  • Extensive results report visually appealing outputs and better attribute preservation than previous methods.

A ADDITIONAL COMPARISON WITH DEEPFAKES

The supplementary comparison evaluates SimSwap against DeepFakes on FaceForensics++ without training on the source and target faces.

  • The additional comparison is conducted on FaceForensics++.
  • SimSwap is compared directly with DeepFakes.

B VIDEO RESULTS

The supplementary video results show SimSwap outputs and specify how source and target identities are represented in the accompanying videos.

  • Five face-swapping videos generated by SimSwap are provided.
  • Video filenames use s_X_t_Y.avi, where X denotes the source identity.
  • The additional FaceForensics++ comparison reports better results than DeepFakes without training on the source and target faces.
  • The source image appears in the upper-left corner and the target image in the bottom-left corner.

C ADDITIONAL RESULTS FOR SIMSWAP

Additional matrices show SimSwap results across male, female, and expressive identities, including fictional characters and exaggerated expressions.

  • The additional results include male and female face matrices generated by SimSwap.
  • The male matrix includes fictional characters in rows 1 and 7.
  • The female matrix includes fictional characters in rows 4 and 7.
  • The expression matrix includes identities with exaggerated expressions and reports decent face-swapping results.
Loading 2106.06340v1…