PaperlayerView exact revision on arXiv

Source-linked AI summary

One Shot Face Swapping on Megapixels

Yuhao Zhu, Qi Li, Jian Wang, Chengzhong Xu, Zhenan Sun

arXiv:2105.04932v2cs.CV

TL;DR

High-resolution one-shot face swapping is difficult because prior methods lose information, train unstably, and require substantial GPU memory. MegaFS uses hierarchical W++ encoding, nonlinear Face Transfer Modules, and fixed StyleGAN2 generation; it is presented as the first megapixel one-shot method, with benchmark effectiveness and a released megapixel database, though pose and expression errors remain inferior in some comparisons.

  • Problem

    High-resolution one-shot face swapping is limited by compressed representations, unstable adversarial training, and GPU memory constraints.

  • Method

    MegaFS hierarchically encodes faces in extended latent space W++, transfers identity through nonlinear FTM trajectories, and uses fixed StyleGAN2 for generation.

  • Results

    MegaFS is presented as the first megapixel one-shot face-swapping method, with effective benchmark results and a released megapixel face-swapping database.

  • Takeaways & Limitations

    The released database supports research on DeepFake detection and face image editing in the public domain.

  • Takeaways & Limitations

    MegaFS performs worse than FaceShifter and reenactment methods on pose and expression errors, with possible effects from less training data and StyleGAN2's smiling-face bias.

Abstract

from arXiv · show

Face swapping has both positive applications such as entertainment, human-computer interaction, etc., and negative applications such as DeepFake threats to politics, economics, etc. Nevertheless, it is necessary to understand the scheme of advanced methods for high-quality face swapping and generate enough and representative face swapping images to train DeepFake detection algorithms. This paper proposes the first Megapixel level method for one shot Face Swapping (or MegaFS for short). Firstly, MegaFS organizes face representation hierarchically by the proposed Hierarchical Representation Face Encoder (HieRFE) in an extended latent space to maintain more facial details, rather than compressed representation in previous face swapping methods. Secondly, a carefully designed Face Transfer Module (FTM) is proposed to transfer the identity from a source image to the target by a non-linear trajectory without explicit feature disentanglement. Finally, the swapped faces can be synthesized by StyleGAN2 with the benefits of its training stability and powerful generative capability. Each part of MegaFS can be trained separately so the requirement of our model for GPU memory can be satisfied for megapixel face swapping. In summary, complete face representation, stable training, and limited memory usage are the three novel contributions to the success of our method. Extensive experiments demonstrate the superiority of MegaFS and the first megapixel level face swapping database is released for research on DeepFake detection and face image editing in the public domain. The dataset is at this link.

1. Introduction

MegaFS targets one-shot face swapping at megapixel resolution, where limited representation, unstable adversarial training, and GPU memory constrain prior approaches. It combines hierarchical encoding, synchronized latent manipulation, and modular StyleGAN2-based generation, and reports benchmark effectiveness while releasing a megapixel face-swapping database.

  • Face swapping transfers source identity while preserving the target's facial attributes, supporting applications including entertainment, privacy protection, and theatrical production.
  • One-shot face swapping uses one source image and one target image for both training and testing, making it more challenging than subject-agnostic swapping with arbitrary identities.
  • High-resolution face swapping is hindered by compressed representations, unstable adversarial training, and GPU memory limits that previously confined methods to 256^2 resolution.
  • MegaFS addresses these issues with HieRFE in extended latent space W++, FTM for synchronized multi-attribute manipulation, and fixed StyleGAN2 decoding without discriminator optimization.
  • MegaFS is presented as the first method for one-shot face swapping at megapixel level.
  • Experiments on benchmark data show MegaFS effectiveness, and the authors release the first megapixel face-swapping database for DeepFake detection and face image editing.

2. Related Works

Prior work spans subject-specific and subject-agnostic swapping, GAN inversion, and latent-code manipulation. MegaFS builds on extended latent representations, hierarchical encoding, and nonlinear latent trajectories to support high-fidelity face generation.

  • Face Swapping: Subject-specific methods require training and testing on the same subjects, whereas subject-agnostic methods apply to arbitrary identities without additional training.
  • GAN Inversion: GAN inversion searches for latent codes that reconstruct images, with later methods extending W to W+ for better reconstruction and using encoders, optimization, or both.
  • Latent Code Manipulation: Latent-code manipulation edits semantic attributes through directions or trajectories, with earlier methods assuming linear StyleGAN directions and StyleFlow using nonlinear trajectories.
  • MegaFS Pipeline: MegaFS organizes its pipeline into Face Encoding, Latent Code Manipulation, and Face Generation, projecting two faces into W++ before manipulating high-level codes.
  • Hierarchical Representation Face Encoder: HieRFE combines a ResNet50 backbone, FPN-based feature pyramid, and eighteen lateral nonlinear mapping networks for hierarchical latent-code prediction.

3. Method

MegaFS divides megapixel face swapping into hierarchical encoding, latent-space identity transfer, and StyleGAN2 synthesis. HieRFE preserves complete face information in W++, while FTM transfers identity and target attributes through nonlinear latent manipulation.

  • 3.1. Hierarchical Representation for Face Swapping: MegaFS projects source and target faces into W++ to preserve complete facial information for subsequent swapping.HieRFE uses hierarchical encoding, with low-level topology represented by C and Llow and high-level semantics by Lhigh.
  • 3.1. Hierarchical Representation for Face Swapping: HieRFE combines a ResNet50 backbone, FPN-based feature refinement, and eighteen lateral nonlinear mapping networks for latent-code prediction.Each mapping network produces a 1×512 latent vector.
  • 3.2. Synchronized Control of Multiple Attributes: FTM follows a two-step nonlinear trajectory that removes irrelevant source semantics and incorporates a small amount of target semantic information.The first step scales multiplication coefficients through sigmoid activation, while the second shifts the latent representation toward target attributes.
  • 3.3. High-Fidelity Face Generation: StyleGAN2 generates the swapped face from target topology and low-level attributes together with the transferred source identity code.The method operates globally in W++, enabling nonlinear latent transformations without local feature-map distortions and avoiding explicit attribute disentanglement.
  • 3.4. Objective Functions: HieRFE and FTM are trained sequentially, with HieRFE supervised by reconstruction, perceptual, identity, and landmark objectives and FTM trained with four losses plus a stabilizing Lnorm term.The loss weights control the combined objectives, and each input and reconstruction is resized before loss computation.

4. Experiments

Experiments compare MegaFS with prior face-swapping methods on FaceForensics++ and CelebA-HQ, then evaluate its components through ablations. Results show stronger identity preservation and megapixel swapping, while pose and expression errors remain weaker than some alternatives.

  • Experiments on FaceForensics++: MegaFS is compared qualitatively with FaceSwap, DeepFakes, and FaceShifter using aligned FaceForensics++ frames.The evaluation also includes quantitative comparison settings following FaceShifter and reports ID retrieval, pose error, and expression error for face-swapping methods.
  • Experiments on FaceForensics++: FaceShifter and MegaFS produce more visually pleasant results than FaceSwap and DeepFakes, which show blending inconsistency, distortions, and artifacts.FaceShifter nevertheless produces unnatural source expressions and retains excessive target attributes in several rows.
  • Experiments on FaceForensics++: MegaFS achieves the highest ID retrieval on FaceForensics++, but performs worse than FaceShifter and reenactment methods on pose and expression errors.The paper attributes possible performance differences to training-data quantity and variation, as well as StyleGAN2’s FFHQ data bias toward smiling faces.
  • Experiments on CelebA-HQ: MegaFS swaps faces at 1024^2 resolution across varied expressions and poses while preserving source wrinkles, iris colors, eyebrow shapes, and nose shapes.The paper describes this as a megapixel-level capability not available in other compared methods except a subject-specific method requiring identity-specific decoders.
  • Ablation studies: HieRFE trained in W++ outperforms its W+ counterpart in ID similarity, pose and expression preservation, information preservation, robustness, and controllability.Qualitative results indicate that W+ fails on glasses, gaze, sunglasses, and complex lighting, while FTM achieves the strongest ID similarity among tested latent-code manipulators.

5. Conclusion

MegaFS addresses three key issues in high-resolution face swapping through a modular three-stage pipeline, and supports one-shot megapixel face swapping. It also enables a public megapixel face-swapping database for forgery detection and face-swapping research.

  • 5. Conclusion: MegaFS uses HieRFE, FTM, and StyleGAN2 in three stages to preserve facial information, transfer identity, and synthesize swapped faces.HieRFE operates in W++, FTM follows a non-linear trajectory without explicit feature disentanglement, and StyleGAN2 avoids unstable adversarial training.
  • 5. Conclusion: The modular design requires little GPU memory with negligible performance cost and performs comparatively with state-of-the-art methods at 256^2 resolution.
  • 5. Conclusion: MegaFS is presented as the first method for one-shot face swapping at megapixel resolution.
  • 5. Conclusion: A public megapixel-level face-swapping database is built for future forgery-detection and face-swapping research.
Loading 2105.04932v2…