Source-linked AI summary
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, Daniel Cohen-Or
TL;DR
The paper addresses the difficulty of accurately embedding real or out-of-domain inputs for StyleGAN-based image manipulation and translation. It introduces pSp, an encoder that directly produces W+ style vectors for a pretrained StyleGAN generator, and demonstrates a generic framework supporting diverse translation tasks and multimodal synthesis. The approach also extends beyond the human facial domain, while remaining limited to images representable by the pretrained StyleGAN and potentially weaker at preserving fine details.
Problem
Prior StyleGAN workflows invert images before editing, but real-image reconstruction can be inaccurate and some translation inputs do not belong to the StyleGAN domain.
Method
pSp uses a novel encoder to directly map input images into W+ style vectors, which are passed to a fixed pretrained StyleGAN generator for image-to-image translation.
Results
pSp directly reconstructs real images without optimization and solves diverse image-to-image translation tasks with one framework, including multimodal synthesis and facial applications.
Takeaways & Limitations
StyleGAN-based translation simplifies training by removing the need for an adversary and supports inputs outside the StyleGAN domain through direct encoding.
Takeaways & Limitations
The method is limited to images that can be generated by StyleGAN and may struggle to preserve fine input details such as earrings or backgrounds.
Abstract
from arXiv · showhide
We present a generic image-to-image translation framework, pixel2style2pixel (pSp). Our pSp framework is based on a novel encoder network that directly generates a series of style vectors which are fed into a pretrained StyleGAN generator, forming the extended W+ latent space. We first show that our encoder can directly embed real images into W+, with no additional optimization. Next, we propose utilizing our encoder to directly solve image-to-image translation tasks, defining them as encoding problems from some input domain into the latent domain. By deviating from the standard invert first, edit later methodology used with previous StyleGAN encoders, our approach can handle a variety of tasks even when the input image is not represented in the StyleGAN domain. We show that solving translation tasks through StyleGAN significantly simplifies the training process, as no adversary is required, has better support for solving tasks without pixel-to-pixel correspondence, and inherently supports multi-modal synthesis via the resampling of styles. Finally, we demonstrate the potential of our framework on a variety of facial image-to-image translation tasks, even when compared to state-of-the-art solutions designed specifically for a single task, and further show that it can be extended beyond the human facial domain.
1. Introduction
StyleGAN offers high-quality synthesis and a semantically useful latent space, but accurately inverting real images and translating inputs outside its domain remain challenging. pSp addresses these challenges with a direct W+ encoder and a generic StyleGAN-based image-to-image translation framework.
- StyleGAN combines high-resolution visual quality with a disentangled W latent space that supports control and editing.
- Prior methods typically invert an image into StyleGAN’s latent space before applying semantic edits.
- A single 512-dimensional W vector often reconstructs real images inaccurately, motivating the extended W+ space of 18 layer-specific vectors.
- pSp directly encodes arbitrary images into W+ using a feature pyramid and style vectors inserted into a fixed pretrained StyleGAN generator.
- Unlike dedicated task-specific architectures, pSp uses one framework across many translation tasks without an adversarial discriminator.
- Resampling intermediate styles enables multimodal outputs for ambiguous inputs without changing the architecture or training process.
2. Related Work
Related work covers GAN inversion, semantic latent-space manipulation, image-to-image translation, and the pSp architecture. pSp differs from prior inversion-and-editing methods by directly encoding translation inputs into StyleGAN’s latent representation.
- GAN Inversion: GAN inversion seeks the latent vector that most accurately reconstructs a known image with a pretrained GAN.
- Latent Space Manipulation: Latent-space manipulation methods learn semantic directions or paths for edits such as age, expression, zoom, rotation, and local attributes.
- pSp Architecture: pSp extracts styles from three feature-pyramid scales and feeds them to StyleGAN according to its coarse, medium, and fine detail hierarchy.
- pSp Architecture: Unlike the usual invert-first-edit-later procedure, pSp directly encodes input images into the latent representation for translation.
- Image-to-Image: Image-to-image translation learns a conditional generation function mapping an input from a source domain to a corresponding target-domain image.
3. The pSp Framework
pSp encodes images into StyleGAN’s W+ space using hierarchical style vectors and trains the encoder with complementary reconstruction, perceptual, regularization, and identity objectives. Operating through the style domain enables global, correspondence-free translation and supports multimodal outputs through style mixing.
- Framework: pSp uses a pretrained StyleGAN generator with an encoder that matches input images to accurate W+ latent representations.The framework is built around StyleGAN’s representational capacity and W+ space.
- Framework: A feature-pyramid encoder extracts styles at coarse, medium, and fine scales before feeding them into corresponding StyleGAN layers.The hierarchy addresses the bottleneck of learning all 18 style vectors from one 512-dimensional vector.
- Loss Functions: The model combines pixel-wise L2, LPIPS perceptual, latent regularization, and recognition losses to train more accurate task-specific encodings.The loss weights are tunable according to the nature of each encoding task.
- Loss Functions: Latent regularization improves image quality without harming fidelity, especially for ambiguous tasks.The regularizer encourages style vectors toward the pretrained generator’s average latent vector, analogously to StyleGAN’s truncation trick.
- Loss Functions: Recognition loss preserves facial identity by measuring cosine similarity between the output and source images with a pretrained ArcFace network.This objective targets a common challenge in facial-image encoding.
- Benefits of the StyleGAN Domain: Style-domain translation operates globally without pixel-to-pixel correspondence, avoiding the locality bias associated with direct spatial inputs.The pretrained StyleGAN generator is governed by styles rather than residual spatial feature maps.
4. Applications and Experiments
pSp is evaluated across inversion, frontalization, conditional synthesis, and additional applications, using a shared StyleGAN-based encoding framework. The experiments report strong reconstruction, identity preservation, diverse conditional outputs, weakly supervised frontalization, and extension beyond human faces.
- StyleGAN Inversion: pSp preserves identity and fine details in StyleGAN inversion, including lighting, hairstyle, and glasses.
- StyleGAN Inversion: pSp’s W+ encoder improves over simpler W and expanded W+ ablations, while identity loss contributes to reconstruction quality.The W+ extension improves results, but the full architecture better preserves fine details.
- StyleGAN Inversion: pSp better preserves original images than other encoders in both perceptual similarity and identity evaluations.Similarity is evaluated with CurricularFace to remain independent of the training loss.
- Face Frontalization: In face frontalization, pSp generates realistic frontal faces with weak supervision and offers a fast alternative to the specialized R&R approach.R&R outperforms pSp quantitatively, but pSp does not require R&R’s 3DMM fitting and inpainting modules.
- Conditional Image Synthesis: For sketches and segmentation maps, pSp generates high-quality outputs across varied inputs and supports multiple fine-style outputs through style mixing.On sketches, the method retains finer details and produces more diverse outputs than the compared methods; on segmentation maps, it avoids the similar artifacts reported for competing pix2pixHD-based methods.
- Extending to Other Applications: With minimal training changes, pSp also supports super-resolution, inpainting, interpolation, local patch editing, and StyleGAN inversion and sketch translation for AFHQ cats and dogs.The AFHQ experiments report accurate, high-quality reconstruction and multi-modal synthesis through style mixing.
5. Discussion
The framework’s generated-image quality is constrained by StyleGAN’s training domain and by its global treatment of image information. These constraints can make unusual poses or expressions difficult and reduce preservation of fine local details.
- Generated outputs are limited to images that the pretrained StyleGAN can generate.Faces that are not close to frontal or contain expressions absent from StyleGAN’s training data may be challenging.
- The global pSp approach can struggle to preserve fine details such as earrings and background content.This limitation is especially relevant to inpainting and super-resolution, where local information can otherwise be propagated.
6. Conclusion
pSp directly maps real images into StyleGAN’s W+ space without optimization and uses the same encoder–decoder framework for varied image-to-image translation tasks. It can encode inputs outside the StyleGAN domain and support diverse problems with minimal training changes.
- pSp directly maps real images into W+ without optimization, using hierarchical styles fed into a fixed StyleGAN generator.The encoder extracts styles hierarchically and supplies them to corresponding generator inputs.
- pSp directly encodes translation tasks into StyleGAN, including inputs that do not reside in the StyleGAN domain.This differs from the prior “invert first, edit later” approach.
- The framework solves a wide variety of translation problems with one architecture and minimal changes to training losses and methodology.
A. Implementation Details
The implementation uses a recognition-pretrained ResNet-IR encoder with a fixed FFHQ-trained StyleGAN2 generator. Only the pSp encoder is trained for each translation task, with task-specific loss weighting.
- The backbone is a ResNet-IR network pretrained on face recognition, while StyleGAN2 is fixed and trained on FFHQ.Only the pSp encoder network is trained on the given translation task.
- For inversion, the loss weights are λ1 = 1, λ2 = 0.8, and λ3 = 0.1.
- Face frontalization increases identity-loss weight to λ3 = 1 and reduces L2 and LLPIPS weights, with separate values for inner-face and outer regions.The inner-face weights are λ1 = 0.01, λ2 = 0.8; elsewhere they are λ1 = 0.001, λ2 = 0.08.
- Conditional image synthesis uses the inversion-task constants but omits identity loss, setting λ3 = 0.
B. Dataset Details
Experiments use CelebA-HQ for training and testing, FFHQ for inversion and frontalization, and filtered, simplified images to construct sketch datasets. The sketch construction is also applied to AFHQ cat and dog data.
- CelebA-HQ contains 30,000 high-quality images, with approximately 24,000 used for training under a standard split.
- FFHQ contains 70,000 face images and is used for StyleGAN inversion and face frontalization.
- Sketch datasets are constructed by applying a pencil-sketch filter and then a sketch-simplification method to source images.The same procedure is used for CelebA-HQ and AFHQ cat and dog datasets.
C.1. Super Resolution
pSp reconstructs high-resolution images from low-resolution inputs using supervised training, while preserving realistic detail and supporting multiple plausible outputs through style-mixing.
- pSp is trained with random bicubic down-sampling from ×1 through ×32, using the original image as the high-resolution target.
- Style-mixing generates multiple super-resolution outputs from one low-resolution image by altering medium-level facial-feature styles.The method mixes a randomly sampled w vector on layers 4-7 with α = 0.5.
C.2. Inpainting
For inpainting, pSp encodes masked images into realistic face latents, enabling accurate reconstruction of occluded regions with fewer artifacts than pix2pixHD.
- The inpainting task reconstructs missing or occluded image regions, and pSp is trained against pix2pixHD using symmetric triangular masks.Both models are trained in a supervised fashion.
- pSp accurately reconstructs occluded regions with high-quality outputs and no artifacts, whereas pix2pixHD produces reconstruction errors and many artifacts.The comparison is shown in Figure 16.
C.3. Local Editing
pSp supports local editing, patch compositing, latent interpolation, and diverse conditional synthesis by manipulating encoded styles and latent codes.
- Local editing: Changing localized attributes in sketches or segmentation maps produces corresponding local edits in generated images.Examples include eyes, smiles, and hair.
- Local editing: pSp can seamlessly merge a desired local patch into a real face image.
- Image interpolation: Interpolating between two encoded W+ codes generates intermediate images using w′ = αw1 + (1 − α)w2 for 0 ≤ α ≤ 1.
- Beyond human faces: pSp extends these editing and translation capabilities beyond human faces to cat and dog datasets.The reported tasks include super-resolution, inpainting, and sketch-based image generation.
- Multi-modal synthesis: The framework produces diverse outputs from sketches and segmentation maps through multi-modal conditional synthesis.