Source-linked AI summary
RSGAN: Face Swapping and Editing using Face and Hair Representation in Latent Spaces
Ryota Natsume, Tatsuya Yatagawa, Shigeo Morishima
TL;DR
Face swapping and related editing tasks lack readily available paired images and can be difficult when fitting or alignment is unreliable. RSGAN addresses this by learning separate latent representations for faces and hair, enabling integrated swapping, attribute editing, and random-part synthesis. The system produces natural swaps across differences in expression, orientation, and lighting, while its main stated limitation is 128×128 image resolution.
Problem
Face swapping with deep networks is difficult because paired before-and-after face images are hard to prepare, while existing approaches can depend on fitting or extensive image collections.
Method
RSGAN variationally learns separate latent spaces for face and hair, generates images from their representations, and supports swapping by replacing the face representation.
Results
RSGAN achieves natural face swapping across differences in expression, orientation, and lighting, and supports attribute editing and random face or hair synthesis.
Takeaways & Limitations
A single trained network integrates face swapping with further appearance editing without additional runtime fine-tuning.
Takeaways & Limitations
The implementation is limited to 128×128 image editing because the training dataset uses that resolution.
Abstract
from arXiv · showhide
In this paper, we present an integrated system for automatically generating and editing face images through face swapping, attribute-based editing, and random face parts synthesis. The proposed system is based on a deep neural network that variationally learns the face and hair regions with large-scale face image datasets. Different from conventional variational methods, the proposed network represents the latent spaces individually for faces and hairs. We refer to the proposed network as region-separative generative adversarial network (RSGAN). The proposed network independently handles face and hair appearances in the latent spaces, and then, face swapping is achieved by replacing the latent-space representations of the faces, and reconstruct the entire face image with them. This approach in the latent space robustly performs face swapping even for images which the previous methods result in failure due to inappropriate fitting or the 3D morphable models. In addition, the proposed system can further edit face-swapped images with the same network by manipulating visual attributes or by composing them with randomly generated face or hair parts.
1 Introduction
Face swapping motivates an integrated editing system that avoids difficult fitting and extensive person-specific training. RSGAN separates face and hair representations so one network supports swapping and further appearance editing.
- The paper frames face-image manipulation as a response to demand for more attractive drawings and photographs, motivating research in face analysis and editing.
- Face image manipulation supports applications including photomontage, virtual hairstyle fitting, privacy protection, and machine-learning data augmentation.
- 3D morphable-model methods swap fitted texture maps and re-render them under estimated target lighting, but require accurate fitting.
- Database-based and deep-learning approaches can require similar input layouts or hundreds of images for each person, limiting generality and preparation efficiency.
- RSGAN variationally learns separate latent spaces for face and hair regions, then generates face-swapped images from representations encoded from real samples.
- The proposed system integrates face swapping with appearance editing, uses a single trained DNN without runtime fine-tuning, and handles different orientations and lighting conditions.
2 Related Work
Related work spans traditional part-based and 3D-model face editing, database composition, and neural approaches to appearance manipulation. These methods establish the application scope and the fitting and data challenges addressed by RSGAN.
- Face swapping: Face swapping has been applied to photomontage, virtual hairstyle fitting, privacy protection, and data augmentation, including methods that replace individual facial parts.
- Face swapping: 3DMM-based swapping estimates geometry, texture, and lighting before re-rendering, but accurate results require manual alignment of the model.
- Face appearance editing: Traditional editing estimates face geometry and part arrangements, whereas DNN-based methods pursue face appearance manipulation without explicitly relying on those underlying representations.
3 Region-Separative GAN
RSGAN separates face and hair appearances into distinct latent representations, then composes them with a generative adversarial architecture. It is trained on automatically extracted face, hair, and background-region data to support reconstruction, editing, and face swapping.
- Architecture: RSGAN uses two variational separator networks to encode face and hair appearances separately, plus a composer network to reconstruct images.The architecture also uses global and patch discriminators to evaluate reconstructed images.
- Latent representations: Face and hair representations, together with visual-attribute representations, are encoded into latent spaces and decoded to reconstruct corresponding regions and full images.The model samples latent variables through encoder networks and combines them in the composer network.
- Latent representations: Random latent samples are fed to the composer so RSGAN learns to synthesize natural face images rather than overfitting training data.The generated image is defined as ˆx′ = G(ˆzxf, ˆzcf, ˆzxh, ˆzch).
- Training: Training combines region and image reconstruction losses, KL losses, adversarial losses, and classifier-based visual-attribute losses.The background mask reduces the reconstruction penalty on background pixels, with β = 0.5 halving the background least-square errors.
- Dataset: The training data are generated from CelebA by estimating foreground masks, extracting facial landmarks, and defining face and hair regions.Face and hair region images, full images, and background masks are sampled from these processed portraits.
- Face swapping: For face swapping, RSGAN encodes the face from one image and the hair from another, then composes them; optional gradient-domain stitching can correct hair or background artifacts.The paper distinguishes the direct method as RSGAN and the stitched variant as RSGAN-GD.
4 Results and Discussion
RSGAN produces face-swapping results across differing expressions, orientations, lighting conditions, and appearances, while supporting region-specific attribute editing and random face or hair synthesis. Experiments compare its practical performance with prior methods and examine design choices, including variational representations, recurrent region memorization, and image-resolution limits.
- Face swapping: RSGAN achieves natural face swapping despite large differences in facial expression, face orientation, and lighting condition.The results preserve face identities, expressions, facial-part shapes, and shading in the swapped images.
- Face swapping: Compared with prior methods, RSGAN produces compatible results to and more natural facial-part proportions than.The paper attributes losses in to sensitivity to landmark detection and 3DMM fitting.
- Visual attribute editing: RSGAN edits attributes independently in face and hair regions, so changing one region leaves the other unchanged.The same network can apply attribute editing to face-swapped images while receiving two input images and modified attribute vectors.
- Random face parts synthesis: RSGAN samples independent latent representations to combine real face or hair appearances with randomly generated counterparts.The system supports applications including random face changes for privacy protection and random hair changes for face-recognition data augmentation.
- Quantitative evaluation: RSGAN performs face swapping on all 1,000 test pairs, whereas Nirkin et al.'s method succeeds on 81.7% because of frequent 3DMM-fitting failures.RSGAN has slightly worse identity preservation than Nirkin et al. but achieves the best swap consistency among the compared generative networks.
- Discussion and limitation: The variational RSGAN is preferred over a non-variational prototype because it supports random face-part sampling despite slightly weaker self-reproducibility.Adding an LSTM made training harder and slower without significantly improving visual quality, while the system remains limited to 128×128 image editing.
5 Conclusion
RSGAN is an integrated face-image editing system that supports face swapping and appearance modification through latent representations of faces and hairs.
- RSGAN performs high-quality face swapping across different face orientations and lighting conditions.
- Manipulating face and hair latent representations enables modification of image appearances.
- The results suggest deep generative models can generate image classes not explicitly prepared in the training dataset.
Supplementary Materials: RSGAN: Face Swapping and Editing via
The supplementary material identifies the paper, its authors, and their academic affiliation.
- The supplementary material is titled “Region Separation in Latent Spaces.”
- The paper lists Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima as authors.
- The authors are affiliated with the Graduate School of Advanced Science and Engineering.
1. RSGAN’s training algorithm (Algorithm A1)
The supplementary materials enumerate additional visual results for RSGAN’s editing capabilities.
- Additional results are provided for face swapping.
- Additional results demonstrate visual attribute editing and random face-parts sampling.
- Additional results show face-parts interpolation.
3. RSGAN with LSTM
The supplementary materials cover additional RSGAN outputs and an LSTM-based architectural variant, including its training procedure and example results.
- Additional results: The supplementary material includes face-swapping results with the LSTM variant and labeled face examples.
- Training algorithm: Algorithm A1 trains RSGAN iteratively by computing loss functions and updating network parameters until convergence.
- Additional results: The supplementary results include face swapping, visual attribute editing, random face-parts sampling, and face-parts interpolation.
- RSGAN with LSTM: The LSTM variant inserts a bidirectional LSTM after the two separators in the RSGAN architecture.
RSGAN with LSTM
Figure A6 compares RSGAN results without LSTM against results with LSTM.
- RSGAN with LSTM: Figure A6 presents an image-group comparison of RSGAN results with and without LSTM.The upper group shows results without LSTM, while the lower group shows results with LSTM.