PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Grained Face Swapping via Regional GAN Inversion

Zhian Liu, Maomao Li, Yong Zhang, Cairong Wang, Qi Zhang, Jue Wang, Yongwei Nie

arXiv:2211.14068v2cs.CV

TL;DR

Face swapping must preserve source identity, yet existing GAN inversion methods lack mechanisms for local shape and texture editing, and occlusion handling remains challenging. E4S uses Regional GAN Inversion to disentangle facial shape and texture; experiments report superiority across face swapping, editing, and extended applications.

  • Problem

    Face swapping must preserve source identity, but existing GAN inversion methods focus on global attributes rather than local shape and texture editing, while occlusion handling is also challenging.

    Source: §1 · pp. 1–2

  • Method

    E4S uses Regional GAN Inversion to disentangle facial-component shape and texture, projecting faces into regional style codes and injecting them into StyleGAN features according to masks.

    Source: §1 · p. 2, §3.2 · p. 4, §8 · p. 8

  • Results

    Experiments across face swapping, face editing, and extended applications report superiority; swapped faces achieve the best retrieval accuracy, with pose and expression comparable to SOTA methods.

    Source: §8 · p. 8, §6 · p. 8

  • Takeaways & Limitations

    The framework simplifies face swapping to exchanging corresponding style codes and masks.

    Source: §1 · p. 2

    When lighting differs, the result may not be fully adapted to the target; the authors leave disentangling light from texture for future work.

    Source: §6 · p. 7

Abstract

from arXiv · show

We present a novel paradigm for high-fidelity face swapping that faithfully preserves the desired subtle geometry and texture details. We rethink face swapping from the perspective of fine-grained face editing, \textit{i.e., ``editing for swapping'' (E4S)}, and propose a framework that is based on the explicit disentanglement of the shape and texture of facial components. Following the E4S principle, our framework enables both global and local swapping of facial features, as well as controlling the amount of partial swapping specified by the user. Furthermore, the E4S paradigm is inherently capable of handling facial occlusions by means of facial masks. At the core of our system lies a novel Regional GAN Inversion (RGI) method, which allows the explicit disentanglement of shape and texture. It also allows face swapping to be performed in the latent space of StyleGAN. Specifically, we design a multi-scale mask-guided encoder to project the texture of each facial component into regional style codes. We also design a mask-guided injection module to manipulate the feature maps with the style codes. Based on the disentanglement, face swapping is reformulated as a simplified problem of style and mask swapping. Extensive experiments and comparisons with current state-of-the-art methods demonstrate the superiority of our approach in preserving texture and shape details, as well as working with high resolution images. The project page is http://e4s2022.github.io

1. Introduction

The introduction frames face swapping as a fine-grained editing problem: preserve source identity details while retaining target attributes and handling occlusion. E4S addresses this through explicit component-level shape and texture disentanglement, enabled by Regional GAN Inversion.

1. Introduction

  • Existing identity extraction can miss informative facial details, contributing to in-between results, while target-preserved skin color can look unrealistic across distinct skin tones.The introduction also identifies accurate shape recovery from a single image as difficult.Source: §1 · pp. 1–2
  • Occlusion is difficult because hair may cover face regions; prior inpainting can blur results, while refinement can reintroduce target identity.Source: §1 · p. 2
  • E4S reframes swapping as replacing local facial-component shape and texture, with masks enabling occlusion handling without a dedicated module.A face parser's pixel-wise labels guide the generator in filling missing pixels with swapped texture features.Source: §1 · p. 2
  • RGI introduces a regional-wise Wr+ latent space, using a mask-guided multi-scale encoder for component style codes and mask-guided injection to manipulate generator features.In this representation, style codes encode texture and masks encode shape.Source: §1 · p. 2
  • The framework targets high-fidelity swapping with identity preservation and occlusion handling, and supports global and local feature swapping with user-controlled partial extent.The stated experiments cover face swapping, face editing, and extension tasks.Source: §1 · p. 2

2. Related Work

Related work spans GAN inversion and face-swapping approaches, with prior methods often limited in local control or identity-detail preservation. E4S positions itself as source-oriented fine-grained swapping based on explicit component shape and texture disentanglement.

2. Related Work

  • GAN inversion methods include learning-based, optimization-based, and hybrid approaches; optimization often improves inversion performance, while learning-based methods cost less time.Source: §2 · p. 2
  • Existing inversion methods support global edits but not precise local component control, motivating RGI's local-editing focus.Source: §2 · p. 3
  • Face-swapping approaches are broadly source-oriented or target-oriented, and identity extraction through recognition models or 3DMMs may miss details and produce an in-between effect.Source: §2 · p. 3
  • Among StyleGAN-based methods, StyleFusion leaves regional shape and texture entangled, while region-aware projector and HiRes methods do not support fine-grained selective swapping.Source: §2 · p. 3
  • E4S is source-oriented and treats swapping as editing shape and texture across facial components, explicitly disentangling them rather than relying on recognition models or 3DMMs.Source: §2 · p. 3

3. Methodology

E4S reframes face swapping as separate texture-code and mask recomposition, while Regional GAN Inversion (RGI) supplies regional style codes and mask-guided StyleGAN generation. The framework reenacts the source toward the target’s pose and expression, and uses facial masks to guide swapping and handle occlusion.

3.1. Editing-for-Swapping (E4S) Framework

  • E4S reenacts the source toward the target’s pose and expression, then extracts regional texture codes from the driven and target faces for swapping.The pipeline estimates masks for both faces and aggregates the parser’s 19 categories into 12.Source: §3.1 · p. 3
  • Texture swapping exchanges selected regional codes, including face skin, while shape swapping recomposes masks from driven inner-face regions and target background, neck, hair, and accessories.When occlusion leaves gaps between skin and hair or neck, the method fills them with face skin without an extra occlusion sub-network.Source: §3.1 · pp. 3–4
  • After mask and texture-code recomposition, StyleGAN synthesizes the swapped face, which is then blended with the target image.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2. Disentanglement of Shape and Texture

  • RGI uses a multi-scale encoder to extract mask-guided regional texture features, maps them to StyleGAN style codes, and combines those codes with masks for local editing.The resulting codes occupy StyleGAN’s Wr+ space.Source: §3.2 · pp. 4–5
  • The mask-guided StyleGAN extends style modulation regionally by combining feature maps according to downsized segmentation masks.The method applies the mask-guided block to the first K layers, setting K = 13 by default.Source: §3.2 · p. 5, Figure 4 · p. 5

3.3. Training Objective

  • RGI training uses reconstruction as a proxy task, so it does not require paired face swaps during training.After training, the encoder produces regional texture codes for input faces, enabling swapping in Wr+.Source: §3.3 · p. 5

4. Experimental setup

The experiments use CelebAMask-HQ and FFHQ for training and testing, with face masks available from the first dataset and generated by a parser for the second. The setup includes quantitative reconstruction ablations and qualitative comparisons of ablation settings.

Experimental setup

  • CelebAMask-HQ provides 30K images split into 28K training and 2K testing images, while FFHQ contains 70K images and receives masks from a pretrained face parser.Source: §4 · p. 5
  • Training runs for 200K iterations on CelebAMask-HQ and 300K on FFHQ, using eight NVIDIA A100 GPUs and batch size 2 per GPU.The learning rate starts at 10^-4; it decays by 0.1 at 100K and 150K iterations, respectively, and images are randomly flipped with probability 0.5.Source: §4 · pp. 5–6
  • The ablation study compares reconstruction performance across RGI configurations quantitatively and presents qualitative comparisons of different ablation settings.Source: Table 1 · p. 6, Figure 5 · p. 6

5. Ablation study

The ablations test the roles of reenactment, RGI, identity loss, and StyleGAN fine-tuning. They show that these choices affect alignment, detail preservation, identity, and hair quality in different ways.

Re-enactment

  • Without reenactment, the swapped result is misaligned with the target, indicating pose information is also embedded in per-region texture style codes.The authors use a pre-trained face reenactment model before shape and texture swapping.Source: §5 · p. 6

SEAN vs. RGI

  • Replacing RGI with SEAN yields visually pleasant results, but RGI preserves more eye and face-skin details and supports 1024^2 rather than 256^2 output.This comparison tests whether another per-region encoder and generator can be used within E4S.Source: §5 · p. 6

Identity loss

  • Without the ID loss, reconstruction performance is comparable to the baseline, but some identity information is missing.The missing identity information is also visible in the qualitative comparison.Source: §5 · p. 6

Pre-trained vs. fine-tuned StyleGAN

  • Fine-tuning the first 13 StyleGAN layers improves hair quality while maintaining other inner facial textures, whereas freezing the generator yields better SSIM, PSNR, and RMSE but poorer FID.The fine-tuned model addresses hair-texture preservation; configuration (B) freezes the generator and trains the texture encoder and subsequent MLPs.Source: §5 · p. 6

6. Face swapping results

The face-swapping comparisons report realistic, detailed results, including in occlusion cases, and the best source-identity retrieval accuracy. Pose and expression remain comparable to state-of-the-art methods, with reenactment accuracy affecting their preservation.

Qualitative results

  • Qualitative comparisons report sharper, more realistic swapped faces with better texture retention, including filled source-face skin and preserved target glasses in occlusion cases.The authors report that target-oriented methods may retain some target identity, and that theirs is the only compared method to preserve source skin color well.Source: §6 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7

Lighting limitation

  • When source skin color does not adapt fully to target lighting, the authors suggest applying lighting transfer beforehand and leave disentangling light from texture for future work.Source: §6 · p. 7

StyleGAN-based comparisons

  • Among StyleGAN-based methods, MegaFS results are described as blurry mixtures of source and target, StyleFusion as oversmoothed, and HiRes as artifact-prone despite 1024^2 output.The comparison reports that this method generates more realistic and higher-quality faces.Source: §6 · p. 7, Figure 7 · p. 7

Quantitative results

  • Our method achieves the best source-identity retrieval accuracy, while pose and expression preservation remains comparable to state-of-the-art methods.The comparison evaluates source identity with retrieval accuracy and target pose and expression with estimated distances; target-oriented methods generally preserve pose and expression better, while tending to preserve source identity less fully.Source: §6 · pp. 7–8, Table 2 · p. 7

7. Face editing results

RGI supports fine-grained face editing through regional masks and style swapping. The reported comparisons find improved reconstruction metrics and preserved details in edited results.

Fine-grained editing

  • RGI can edit a specific region by changing its mask or swapping its style with a reference, supporting face beautification, hairstyle transfer, and control of swapping extent.The method is evaluated on the CelebAMask-HQ test set against fine-grained editing methods and SofGAN.Source: §7 · p. 8

Quantitative results

  • RGI outperforms the compared methods on SSIM, PSNR, RMSE, and FID for CelebAMask-HQ test-set reconstruction.The comparison also includes RGI-Optim., which applies style-code optimization; its reconstructions preserve details such as hair curl, beard thickness, dimples, and background.Source: §7 · p. 8, Table 3 · p. 8

Qualitative results

  • Qualitative editing results are reported as more high-fidelity and natural, while maintaining details in other components and overall identity information.The mask modifications include hair, eyebrows, and chin.Source: §7 · p. 8

8. Conclusion

The paper presents E4S, which disentangles facial-component shape and texture and recasts face swapping as texture and shape swapping. Its RGI method uses regional style codes and mask-guided feature-map injection, and experiments report superiority across face swapping, editing, and extended applications.

8. Conclusion

  • E4S explicitly separates facial-component shape and texture, reformulating face swapping as texture and shape swapping.Source: §8 · p. 8
  • RGI projects input faces into per-region StyleGAN style codes with a multi-scale mask-guided encoder, then uses mask-guided injection to manipulate generator feature maps.Source: §8 · p. 8
  • Experiments on face swapping, face editing, and extended applications report the method's superiority.Source: §8 · p. 8

Related papers

Loading 2211.14068v2…