PaperlayerView exact revision on arXiv

Source-linked AI summary

RSGAN: Face Swapping and Editing using Face and Hair Representation in Latent Spaces

Ryota Natsume, Tatsuya Yatagawa, Shigeo Morishima

arXiv:1804.03447v2cs.CVcs.GR

TL;DR

Training deep generative face swapping is challenging without before-and-after swapped-image pairs. RSGAN learns separate face and hair latent representations and produces natural-looking swaps across tested differences in expression, orientation, and lighting.

  • Problem

    Deep generative face swapping is challenging because real before-and-after swapped-image training pairs are difficult or impossible to prepare.

    Source: §1 · p. 3, §3 · p. 5

  • Method

    RSGAN variationally learns separate latent spaces for face and hair appearances, then swaps faces by composing their latent representations.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    RSGAN achieves high-quality, natural-looking swaps across tested differences in face orientation and lighting, with identities, expressions, facial-part shapes, and shading naturally presented.

    Source: Face swapping · p. 10, §5 · p. 15

  • Takeaways & Limitations

    The system supports editing swapped images through visual-attribute changes and combinations of real-image regions with randomly generated face or hair parts.

    Source: Face swapping · p. 12

    The system edits images only at 128×128 resolution, the resolution used for its training data.

    Source: §4.2 · p. 15

Abstract

from arXiv · show

In this paper, we present an integrated system for automatically generating and editing face images through face swapping, attribute-based editing, and random face parts synthesis. The proposed system is based on a deep neural network that variationally learns the face and hair regions with large-scale face image datasets. Different from conventional variational methods, the proposed network represents the latent spaces individually for faces and hairs. We refer to the proposed network as region-separative generative adversarial network (RSGAN). The proposed network independently handles face and hair appearances in the latent spaces, and then, face swapping is achieved by replacing the latent-space representations of the faces, and reconstruct the entire face image with them. This approach in the latent space robustly performs face swapping even for images which the previous methods result in failure due to inappropriate fitting or the 3D morphable models. In addition, the proposed system can further edit face-swapped images with the same network by manipulating visual attributes or by composing them with randomly generated face or hair parts.

1 Introduction

RSGAN swaps faces by learning separate latent representations for face and hair regions, then uses them to synthesize images. The same system supports appearance editing and random face-part synthesis without additional runtime fine-tuning.

Introduction

  • Face swapping serves applications including photomontage, virtual hairstyle fitting, privacy protection, and machine-learning data augmentation,, [13],.Source: §1 · p. 1
  • 3DMM methods swap and re-render estimated face textures but can fail at geometry or lighting estimation, while database matching cannot handle arbitrary input pairs,.Source: §1 · p. 2
  • Deep-learning alternatives can require hundreds of portraits per person or fine-tuning on several tens of images, making them impractical for many users,, [13].CVAE-GAN and FakeApp require hundreds of images for each target person; the style-transfer approach fine-tunes a pretrained network using several tens of images of one person.Source: §1 · p. 3
  • RSGAN learns separate variational latent spaces for face and hair, then uses a GAN to synthesize natural images from representations encoded from real samples.The paper evaluates identity preservation with OpenFace and swap consistency using absolute difference and MS-SSIM after swapping faces twice between two inputs.Source: §1 · p. 3
  • The integrated system supports face swapping and appearance editing, uses a single DNN without runtime fine-tuning, and handles differing face orientations and lighting conditions.Source: §1 · p. 4

2 Related Work

Prior face-swapping methods use 3D models, image retrieval, or deep networks, with limitations including manual alignment, restricted image-pair choices, and substantial data requirements. Face-editing work ranges from manipulating estimated face geometry to direct DNN editing, including attribute changes, paint-guided synthesis, and image completion.

2.1 Face swapping

  • Face-swapping methods span 3DMM texture replacement, image retrieval, and identity-conditioned or style-transfer DNNs, but face fitting, arbitrary-pair swapping, and data demands remain limitations.3DMM methods replace texture maps and re-render using estimated lighting, but require manual alignment. Retrieval-based methods cannot freely select input images or handle arbitrary pairs; CVAE-GAN and neural style transfer [13] require users to collect dozens of images. Earlier work also swaps only facial parts such as eyes, nose, and mouth.Source: §2.1 · p. 4

2.2 Face image editing

  • Face editing has shifted from manipulating estimated 3D geometry and parts to end-to-end DNNs for attribute changes, paint-guided synthesis, and image completion.Autoencoder-based DNNs [29] [30] modify attributes such as expression and hair color; paint-based editing [31] follows user strokes, while completion methods [33] fill painted regions, making their results difficult to estimate because they produce plausible training-data-like content.Source: §2.2 · p. 5

3 Region-Separative GAN

RSGAN separates face and hair appearances into distinct latent spaces and recomposes them, enabling face swapping without requiring paired images of people before and after surgery-like face replacement.

3 Region-Separative GAN

  • RSGAN uses separate face and hair VAEs whose latent codes feed a GAN composer, with global and patch discriminators judging reconstructed and randomly sampled outputs.Random latent inputs train the composer to generate natural images rather than overfit; a classifier estimates continuous visual attributes for editing.Source: §3 · pp. 5–7, Figure 2 · p. 6

3.1 Training

  • Training combines reconstruction, KL-divergence, adversarial, classifier, and attribute-preservation losses, with background-weighted reconstruction to emphasize foreground detail.The background reconstruction error is halved using β = 0.5.Source: §3.1 · p. 7

3.2 Dataset

  • From CelebA, the authors extracted face and hair crops for 195,361 of 202,599 images, using 180,000 for training and 15,361 for testing.They estimated foreground masks with PSPNet and facial landmarks with Dlib, then used the landmarks to define crops and assigned the face region higher resolution.Source: §3.2 · pp. 8–9

3.3 Face swapping with RSGAN

  • For two inputs, RSGAN combines the first image’s face codes with the second image’s hair codes; optional gradient-domain stitching composites the swapped face region onto the target image.The paper reports results using RSGAN without stitching unless otherwise specified.Source: §3.3 · p. 9

4 Results and Discussion

On held-out test images, RSGAN supports face swapping, region-specific visual attribute editing, and synthesis that combines real face or hair appearances with randomly generated counterparts.

Face swapping:

  • Face swaps naturally preserved identities, expressions, facial-part shapes, and shading despite differences in expression, orientation, and lighting; RSGAN-GD was compatible with and had more natural part proportions than.The comparison used input images selected for similar layouts, which were more favorable to that method.Source: Face swapping · pp. 10–11, Figure 5 · p. 10
  • RSGAN can add visual attributes to both regions or only the face or hair, with edits confined to the selected region even for attributes such as “Male” and “Aged.”The system can also apply attribute editing to face-swapped images by supplying two input images and modified attribute vectors together.Source: Face swapping · pp. 11–12, Figure 6 · p. 11
  • Random face or hair appearances can be combined with the real counterpart region, preserving the real input appearance across a significant range of generated parts.The paper identifies random face changes for privacy protection and random hair changes for face-recognition data augmentation as potential uses.Source: Face swapping · p. 12, Figure 7 · p. 12

4.1 Experiments

  • RSGAN surpassed other generative networks in identity preservation and “Swap ×2” consistency, while Nirkin et al. [17] scored better on identity but handled only 81.7% of pairs.Nirkin et al. also outperformed generative networks on absolute-error and MS-SSIM consistency scores, partly because it changes only the face region while generative networks synthesize both face and hair.Source: Table 1 · p. 13, §4.1 · p. 14

4.2 Discussion

  • Although a non-variational RSGAN prototype reproduced images slightly better, the authors favored the variational model for its broader applicability, including random face-part sampling.The paper notes that prior approaches often use non-variational methods when reproducing real image appearances is the priority.Source: §4.2 · p. 14
  • Adding an LSTM to track synthesized regions made training harder and slower without significantly improving face-swapping or other application results.The LSTM outputs were fed by the two image encoder networks.Source: §4.2 · p. 15
  • RSGAN’s editing is limited to 128×128 resolution because it was trained on images of that size; the authors propose higher-resolution training with progressive input-resolution increases.They suggest this approach could be applied to RSGAN to address the limitation.Source: §4.2 · p. 15

5 Conclusion

RSGAN is an integrated face-image editing system that achieves high-quality face swapping across orientation and lighting differences, while enabling appearance changes through face and hair latent representations.

5 Conclusion

  • RSGAN is an integrated face-image editing system built around a novel generative neural network.The system is proposed for editing face images.Source: §5 · p. 15
  • The system achieves high-quality face swapping across different face orientations and lighting conditions.Face swapping is the main scope of the study.Source: §5 · p. 15
  • Manipulating latent-space representations of face and hair appearances enables modification of image appearances.Source: §5 · p. 15
  • The authors infer that RSGAN's architecture and training method suggest deep generative models can obtain image classes absent from the training dataset.They view their results as a key to generating images that are difficult to prepare in a training dataset.Source: §5 · p. 15

A Table of contents · 1. RSGAN’s training algorithm (Algorithm A1)

The contents list RSGAN’s training algorithm (Algorithm A1) and four additional-result topics: face swapping, visual attribute editing, random face-parts sampling, and face-parts interpolation.

1. RSGAN’s training algorithm (Algorithm A1)

  • The contents list face swapping as an additional result and identify its results as Fig. A1, not as a training algorithm labeled Algorithm A1.Source: §1 · p. 19

A Table of contents

  • The additional results include face swapping, shown in Fig. A1.Source: §1 · p. 19
  • The contents group face swapping, visual attribute editing, random face parts sampling, and face parts interpolation under additional results.Source: §1 · p. 19
  • The listed additional results include visual attribute editing, shown in Fig. A2.Source: §1 · p. 19
  • The listed additional results include random face parts sampling, shown in Fig. A3.Source: §1 · p. 19
  • The listed additional results include face parts interpolation, shown in Fig. A4.Source: §1 · p. 19

3. RSGAN with LSTM

The RSGAN-with-LSTM architecture inserts a bidirectional LSTM after the two separators. The section also provides its training pipeline and face-swapping results.

3. RSGAN with LSTM

  • Face-swapping results using the LSTM architecture are presented in Fig. A6.Source: §3 · p. 19
  • Training samples real-data batches, computes reconstruction, KL, adversarial, and classification losses, then updates network parameters.The specified weights are λrec = 4000, λKL = 1, λadv-g = 20, λadv-p = 30, λC = 1.0, and λGC = 50.Source: §3 · p. 20
  • A bidirectional LSTM is inserted after the two separators in the RSGAN architecture.The architecture is illustrated in Fig. A5.Source: Figure A5 · p. 25

RSGAN with LSTM

The section visually compares RSGAN results with and without LSTM.

RSGAN with LSTM

  • The figure places results without LSTM above results with LSTM; the former are the same results shown in Fig. 4.Source: Figure A6 · p. 26

Related papers

Loading 1804.03447v2…