Source-linked AI summary
Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains
Justin N. M. Pinkney, Doron Adler
TL;DR
GANs are limited to the domains represented by their training data, while creative use also requires control over the generated output. This paper interpolates StyleGAN generators by resolution, producing controllable images from a novel domain, including cartoon-structured faces with photorealistic rendering.
Problem
GANs generate images from their training domain, creating a need for methods that support novel domains and greater artistic control.
Method
The method combines base and transfer-learned StyleGAN weights with a resolution-dependent function, using layer swapping to select parameters from either model.
Results
The interpolated model generates cartoon-structured faces with photorealistic rendering while largely preserving identity for the same latent input.
Takeaways & Limitations
Resolution-dependent model interpolation provides control over which domain features are combined when generating images from a novel domain.
Abstract
from arXiv · showhide
GANs can generate photo-realistic images from the domain of their training data. However, those wanting to use them for creative purposes often want to generate imagery from a truly novel domain, a task which GANs are inherently unable to do. It is also desirable to have a level of control so that there is a degree of artistic direction rather than purely curation of random results. Here we present a method for interpolating between generative models of the StyleGAN architecture in a resolution dependent manner. This allows us to generate images from an entirely novel domain and do this with a degree of control over the nature of the output.
1 Introduction
The paper proposes resolution-dependent interpolation between StyleGAN models to control which domain features are combined. This extends model interpolation beyond uniform weight blending, using StyleGAN’s resolution-specific feature organization to support novel visual domains.
- 1 Introduction: The paper builds on transfer-learned models, whose weights remain related to the original base model and can produce outputs intermediate between their learned domains when interpolated.
- 1 Introduction: Resolution-dependent interpolation selects features from different generators according to layer resolution, enabling controlled combinations such as ukiyo-e structure with photorealistic rendering.Lower resolutions can control pose and head shape, while higher resolutions can control lighting and rendering.
- 1 Introduction: The approach addresses a limitation of linear interpolation across all model parameters, which does not exploit StyleGAN’s separation of feature roles across resolutions.
- 1 Introduction: Figure 2 depicts layer swapping by composing resolution-level blocks from the two input models into one interpolated model.
- 1 Introduction: Figure 3 illustrates cartoon-like structure with photorealistic rendering while largely preserving identity for the same latent input.
2 Method
The method creates an interpolated StyleGAN by combining weights from a base model and a transfer-learned model according to convolutional-layer resolution. A swap threshold determines where the source model changes.
- 2 Method: The interpolated model combines weights from the base and transferred generators using a resolution-dependent function, here implemented as binary layer swapping.
- 2 Method: The resulting weights pinterp are used to construct a new generator after combining the two input models’ resolution-specific parameters.
- 2 Method: The transition between models occurs at a chosen resolution level rswap, which determines which generator supplies weights on each side of the threshold.
3 Results - Toonification
The authors demonstrate that resolution-dependent interpolation can produce faces with cartoon structure and photorealistic appearance. Encoding an input face into the base model enables a corresponding toonified output from the interpolated model.
- 3 Results - Toonification: Resolution-dependent interpolation produces photorealistic faces with cartoon characteristics, including large eyes and a small chin, while preserving broad identity characteristics.The demonstration combines high-resolution FFHQ layers with low-resolution layers from a model transferred to animated character faces.
Appendices
The mathematical formulation represents image generation through a generator and combines base and transfer-learned parameters with a resolution-dependent function. The resulting parameters generate images from a novel intermediate domain.
- Appendices: A generator G maps a latent vector z and learned parameters to an image, with pbase defining the original image domain.
- Appendices: Transfer learning starts from pbase and produces ptransfer, whose parameters generate images from a new dataset domain.
- Appendices: The interpolated parameters pinterp combine pbase and ptransfer through a function that depends on the resolution block of each convolutional layer.
- Appendices: Using pinterp in the generator produces images from a novel domain qualitatively between the base and transferred datasets.
B.1.1 Ukiyo-e
The Ukiyo-e portrait model was trained through transfer learning from FFHQ using a museum-image dataset, then interpolated with resolution-dependent layer swapping.
- B.1.1 Ukiyo-e: The Ukiyo-e dataset contains approximately 5000 aligned face images collected from online museum images of Japanese ukiyo-e prints.Lower-resolution images were upscaled with ESRGAN trained on Manga109.
- B.1.1 Ukiyo-e: Transfer learning initialized a 1024x1024 FFHQ model, trained it for 312 thousand images, and used its exponentially weighted average for interpolation.Training used a learning rate of 0.002 with mirror augmentation and default StyleGAN2 FFHQ settings.
- B.1.1 Ukiyo-e: The interpolation used rswap = 16 for panels c and d and rswap = 32 for panel e, swapping high- and low-resolution layers between the base and transferred models.Panels c and e used base-model high-resolution layers and transferred-model low-resolution layers; panel e reversed this arrangement.
B.1.2 Toonification
The Toonification model was trained from a small aligned cartoon-face dataset and combined with the base model through resolution-dependent StyleGAN layer interpolation.
- B.1.2 Toonification: The Toonification dataset contained approximately 300 online images whose faces were detected and aligned using dlib and the FFHQ procedure.
- B.1.2 Toonification: Transfer learning initialized a 1024x1024 FFHQ model, trained it for 32 thousand images, and used its exponentially weighted average for interpolation.Training used a learning rate of 0.002 with mirror augmentation and default StyleGAN2 FFHQ settings.
- B.1.2 Toonification: Interpolation divides StyleGAN layers by activation resolution from 4x4 to 1024x1024 while retaining the base model’s mapping-network parameters.The authors report little difference when using transferred-network parameters for the mapping network.
- B.1.2 Toonification: Latent vectors for arbitrary faces were obtained with an adapted StyleGAN projector, although other StyleGAN image-encoding methods could also be used.
B.4 Uncurated results
The paper presents uncurated samples from four interpolated models, illustrating the generated outputs for the blended configurations.
- B.4 Uncurated results: Figure 4 shows 24 uncurated examples from the blended model presented in Figure 1c.
- B.4 Uncurated results: Figure 5 shows 24 uncurated examples from the blended model presented in Figure 1d.
- B.4 Uncurated results: Figure 6 shows 24 uncurated examples from the blended model presented in Figure 1e.
- B.4 Uncurated results: Figure 7 shows 24 uncurated examples from the blended model presented in Figure 3.
C Further work
The authors identify the interpolation function as an open design choice and propose smoother, resolution-targeted, multi-model, and channel-dependent extensions.
- C Further work: The demonstrated layer-swapping scheme is only one arbitrary choice for the resolution-dependent interpolation function.
- C Further work: Future schemes could vary smoothly with resolution, target a specific resolution, or interpolate among more than two models.
- C Further work: Channel-dependent interpolation could vary weights by resolution and convolutional-channel index to target particular image regions or features.The proposal parallels conditional interpolation between latent vectors but applies interpolation between models.