Source-linked AI summary
StyleGAN2 Distillation for Feed-forward Image Manipulation
Yuri Viazovetskyi, Vladimir Ivashkin, Evgeny Kashin
TL;DR
Real-image editing with StyleGAN2 commonly relies on slow latent-code optimization, while paired training lacks suitable matched data for tasks such as age manipulation. The paper distills StyleGAN2 manipulations into image-to-image networks trained on synthetic paired datasets, and reports fast, high-quality performance on face transformations comparable to backpropagation and strong unpaired methods.
Problem
Real-image StyleGAN2 editing requires slow backpropagation-based embedding, while paired datasets for transformations such as age manipulation are difficult to collect.
Method
The paper generates paired synthetic face datasets from StyleGAN2 manipulations and trains a student image-to-image network to perform those transformations.
Results
The approach performs on real-world images on par with StyleGAN backpropagation and current state-of-the-art methods trained on unpaired data.
Takeaways & Limitations
The technique provides fast inference for gender swap and also applies to aging or rejuvenation, style transfer, and face morphing.
Takeaways & Limitations
StyleGAN2’s latent space is not perfectly disentangled, so the transformations are not perfectly pure.
Abstract
from arXiv · showhide
StyleGAN2 is a state-of-the-art network in generating realistic images. Besides, it was explicitly trained to have disentangled directions in latent space, which allows efficient image manipulation by varying latent factors. Editing existing images requires embedding a given image into the latent space of StyleGAN2. Latent code optimization via backpropagation is commonly used for qualitative embedding of real world images, although it is prohibitively slow for many applications. We propose a way to distill a particular image manipulation of StyleGAN2 into image-to-image network trained in paired way. The resulting pipeline is an alternative to existing GANs, trained on unpaired data. We provide results of human faces' transformation: gender swap, aging/rejuvenation, style transfer and image morphing. We show that the quality of generation using our method is comparable to StyleGAN2 backpropagation and current state-of-the-art methods in these particular tasks.
1 Introduction
The paper addresses the difficulty of collecting paired data and the slow backpropagation needed to edit real images with StyleGAN2. It distills StyleGAN2 manipulations into paired image-to-image networks trained on synthetic data, achieving real-image performance comparable to established methods.
- Motivation: Paired datasets for face manipulation are difficult to collect because corresponding images require matching identity, pose, and expression across ages.Existing paired datasets have different expressions and orientations and, to the authors’ knowledge, were not used for paired training.
- Motivation: StyleGAN2 supports synthetic paired data because its latent space enables controlled operations for expression transfer, morphing, and style transfer.StyleGAN2’s path-length regularization makes its latent space more suitable for manipulation.
- Motivation: Backpropagation-based embedding makes StyleGAN2 editing slow, costly in data centers, and difficult to run on devices.Feed-forward methods had mainly been reported as initialization for latent-code optimization.
- Approach: The proposed distillation generates paired synthetic data and trains a flexible student image-to-image network to reproduce a StyleGAN2 manipulation.The method extracts facial appearance and transformation information from StyleGAN2 into a single image-to-image translation model.
- Results: On real images, the resulting network performs on par with StyleGAN backpropagation and current state-of-the-art methods trained on unpaired data.The introduction reports this result across the paper’s evaluated manipulation setting.
- Scope: The method covers gender swap, aging or rejuvenation, style transfer, and face morphing, while applying synthetic-data training to real-world images.The contributions include studying qualitative and quantitative performance and publishing the paired datasets.
2 Related work
Related work spans StyleGAN2’s improved latent-space disentanglement, latent-code manipulation and inversion, paired and unpaired image translation, and synthetic-data training. The paper combines these strands to address high-resolution paired face editing without collecting matched real-image pairs.
- Unconditional image generation: StyleGAN2 improves StyleGAN by reducing artifacts and enhancing disentanglement through perceptual path-length regularization.Its architecture also uses mapping networks, adaptive normalization, constant input, noise, and mixing regularization.
- Latent space: StyleGAN’s mapping network transforms Z into a less entangled W space, with W+ providing layer-specific latent codes for multiscale changes.Varying codes at different generator layers changes image details at different scales.
- Latent code manipulation: Linear latent-space operations have enabled interpretable image manipulation across multiple GAN architectures.Prior work studied interpretable directions and latent-space walks in BigGAN, StyleGAN, and DCGAN.
- Latent code manipulation: Editing real images in StyleGAN typically requires backpropagation-based embedding, including optimization in the extended W+ space.These methods use non-trivial losses to obtain embeddings that are both close and perceptually good.
- Paired translation: Paired translation models support high resolutions, but collecting paired data for age manipulation is especially difficult.Pix2pixHD and SPADE are representative paired approaches, while SPADE is limited to segmentation-map translation.
- Unpaired translation: Unpaired translation methods use cycle consistency and generally operate at no more than 256x256 resolution for human faces.The related methods include single-mode, multimodal, latent-guided, and reference-guided systems.
- Synthetic data: Synthetic datasets must address appearance and content gaps to perform well on real-world images.Prior work also shows that synthetic data can support analysis tasks and knowledge distillation, including generative models.
3 Method overview
The method constructs synthetic paired datasets by manipulating StyleGAN2 latent codes, filtering generated images with a pretrained classifier, and training an image-to-image student network. It uses pix2pixHD as a unified translation framework while acknowledging architecture-specific artifacts and resolution difficulties.
- Training framework: The experiments use StyleGAN2 config-f on FFHQ with disentangled image codes, while pix2pixHD serves as the unified student architecture.The approach is intended to remain flexible across image-to-image models, but the experiments focus on pix2pixHD.
- Other manipulations: Style mixing and face morphing use two latent inputs, with style mixing combining codes at different scales and morphing using linear interpolation.Each task’s samples contain two source images and a target image.
- Attribute directions: A pretrained face classifier supplies gender and age predictions because these attributes are not explicitly encoded in StyleGAN2’s latent spaces.The classifier also provides face-detection confidence used to filter generated samples.
- Synthetic paired dataset: The pipeline generates paired data by sampling latent vectors, applying attribute directions, selecting suitable image pairs, and training a paired image-to-image network.Dataset generation proceeds from random latent vectors through attribute-based image sets and classification-based filtering before student-network training.
- Attribute directions: Attribute directions are estimated from class centers in intermediate latent space and applied to generate controlled gender-swap and aging/rejuvenation examples.For gender swap, the method generates images along a five-position sequence from w−∆ to w+∆; aging and rejuvenation use corresponding attribute vectors.
- Training framework: Pix2pixHD is retained despite known repeated-pattern and blob artifacts, and the authors leave improving or replacing it for future work.Most experiments use 512x512 resolution, with 1024x1024 additionally tested for gender swap.
4 Experiments
Experiments distill StyleGAN2 manipulations into paired image-to-image models and evaluate gender transformation against unpaired methods and latent-code optimization, alongside aging, style mixing, and morphing demonstrations.
- Experimental setup: Evaluation focuses on gender transformation using FID and human studies, with separate surveys measuring transformation quality and realism.Human comparisons use 1,000 questions per baseline, answered by 10 people each, with low-confidence examples filtered.
- Experimental setup: The method generates paired synthetic face datasets from StyleGAN2 and trains task-specific pix2pixHD models for gender transformation, aging, style mixing, and face morphing.Gender translation uses separate models for each direction; aging uses vectors corresponding to two age bins.
- Gender transformation: The distilled method produces significantly better visual quality and more stable gender transformations than unpaired image-to-image approaches, and wins on CelebA-HQ despite training only on FFHQ.The cross-dataset result is reported in Table 1b, where the model uses no CelebA samples during training.
- Gender transformation: Human evaluation finds the method performs better in transformation quality than StyleGAN-based optimization methods, while StyleGAN Encoder is better in realism but generates backgrounds unconditionally.FID is avoided for StyleGAN comparisons because all methods use the same StyleGAN model and FID does not assess personality preservation.
- Gender transformation: Pix2pixHD preserves more transformed-image detail than the encoders, likely because it passes unchanged content through the network instead of encoding all information into one vector.The authors characterize this as an easier task for pix2pixHD than for encoder-based methods.
- Limitations: A limitation is that the gender vector is not perfectly disentangled, so female translations can also introduce a smile because of FFHQ attribute bias and latent-space correlations.The experiments also report repeated patterns, light blobs, and difficulty fine-tuning pix2pixHD at 1024x1024 resolution.
- Other transformations: The approach also demonstrates aging/rejuvenation, style mixing, and face morphing using FFHQ-derived examples and StyleGAN2's coarse, middle, and fine style controls.Style-mixing models receive two source images concatenated into six channels, while latent-code interpolation supports morphing.
5 Conclusions
The paper distills StyleGAN2 manipulations into a fast image-to-image translation model with strong quality across several face-editing tasks. Its limitations include imperfect disentanglement and reliance on separate pix2pixHD models for each transformation.
- Conclusions: The technique combines unconditional generation with paired image-to-image GANs to distill StyleGAN2 manipulations into a single translation model.It reports fast inference and high image quality, including stronger FID and user-study or inference-time results than specified baselines on gender swapping.
- Conclusions: The approach applies beyond gender swapping to aging or rejuvenation and style transfer.
- Limitations: Imperfect disentanglement in StyleGAN2 causes transformations to contain impurities rather than remain perfectly pure.The paper characterizes these impurities as not severe.
- Limitations: The framework uses only pix2pixHD, although different tasks may benefit from different architectures, and each transformation requires a separate model.The paper identifies a universal model as a future research opportunity.