Source-linked AI summary
Image Processing Using Multi-Code GAN Prior
Jinjin Gu, Yujun Shen, Bolei Zhou
TL;DR
Applying pretrained GANs to real-image processing is limited by poor reconstructions from single-code inversion through optimization or learned encoders. mGANprior uses multiple latent codes and adaptive channel importance to compose intermediate feature maps, improving reconstruction and supporting diverse processing tasks with pretrained GANs.
Problem
Existing inversion methods based on back-propagation or additional encoders reconstruct real images poorly, limiting the use of pretrained GANs for image processing.
Method
mGANprior jointly optimizes multiple latent codes and adaptive channel-importance scores to compose intermediate generator feature maps for target-image reconstruction.
Results
mGANprior significantly improves reconstruction quality over existing approaches and achieves high-fidelity reconstructions across several image-processing applications.
Takeaways & Limitations
Pretrained GANs can serve as priors for colorization, super-resolution, inpainting, denoising, semantic manipulation, and style mixing without retraining or modifying the GAN model.
Abstract
from arXiv · showhide
Despite the success of Generative Adversarial Networks (GANs) in image synthesis, applying trained GAN models to real image processing remains challenging. Previous methods typically invert a target image back to the latent space either by back-propagation or by learning an additional encoder. However, the reconstructions from both of the methods are far from ideal. In this work, we propose a novel approach, called mGANprior, to incorporate the well-trained GANs as effective prior to a variety of image processing tasks. In particular, we employ multiple latent codes to generate multiple feature maps at some intermediate layer of the generator, then compose them with adaptive channel importance to recover the input image. Such an over-parameterization of the latent space significantly improves the image reconstruction quality, outperforming existing competitors. The resulting high-fidelity image reconstruction enables the trained GAN models as prior to many real-world applications, such as image colorization, super-resolution, image inpainting, and semantic manipulation. We further analyze the properties of the layer-wise representation learned by GAN models and shed light on what knowledge each layer is capable of representing.
1. Introduction
GANs offer strong image synthesis and support many processing tasks, but applying fixed models to real images is difficult because inversion-based reconstructions are inadequate. mGANprior addresses this by combining multiple latent codes with adaptive channel importance to improve reconstruction and support diverse applications.
- GANs produce high-quality images and have been applied to semantic editing, super-resolution, and image-to-image translation.
- Task-specific GAN methods often require specialized network structures or loss functions, limiting their generalization ability.
- Standard GANs synthesize from random noise, so real-image processing requires first inverting the target image into a latent code.
- Back-propagation and encoder-based inversion produce poor reconstructions, especially for high-resolution images, making them unsuitable for subsequent processing.
- mGANprior composes feature maps generated from multiple latent codes using adaptive channel importance to improve reconstruction and enable processing with pretrained GANs.
2. Related Work
Prior GAN inversion methods use optimization or learned encoders to map real images into a single latent code, while GAN-based processing methods are usually task-specific. mGANprior instead composes multiple latent-code feature maps with adaptive channel importance.
- GAN inversion reverses image generation by optimizing a latent code or learning an encoder that maps image space back to latent space.
- Single-code inversion often reconstructs poorly when the input lies outside the generator’s synthesis space, especially with a large domain gap from training data.
- GAN-based image-processing methods span denoising, inpainting, super-resolution, colorization, style mixing, and semantic manipulation but commonly use task-specific architectures or losses.
- mGANprior composes generative feature maps from multiple latent codes at an intermediate generator layer, weighted by adaptive channel-importance scores.
- The method jointly optimizes the latent codes and channel-importance scores to recover the target image.
3. Multi-Code GAN Prior
mGANprior reconstructs real images by jointly optimizing multiple latent codes and adaptive channel importance over composed intermediate generator features, then uses the reconstruction as a prior for image processing tasks.
- GAN inversion seeks a latent code that allows a pretrained generator to recover a target image.
- Single-code inversion is difficult because the latent code may lack sufficient expressiveness to recover all image details.
- Feature Composition: mGANprior composes intermediate feature maps generated from multiple latent codes rather than directly fusing their output images.
- Adaptive Channel Importance: Adaptive channel importance assigns each latent code channel weights intended to align composed features with different visual semantics.
- Optimization Objective: The method jointly optimizes 2N parameter sets, comprising the latent codes and their channel-importance scores.
- Optimization Objective: The reconstruction objective combines pixel-wise error with an l1 distance between perceptual features to use low-level and high-level information.
- Image Processing Applications: The reconstructed image serves as a multi-code GAN prior for colorization, super-resolution, and inpainting through task-specific post-processing constraints.
4. Experiments
Experiments evaluate mGANprior against inversion baselines, analyze how latent-code count and composition layer affect reconstruction, and apply the method across real-image processing tasks. The results show improved reconstruction quality, specialized latent codes, and strong performance across colorization, super-resolution, inpainting, denoising, and manipulation.
- Comparison with Other Inversion Methods: mGANprior outperforms single-code optimization, encoder-based inversion, and encoder-initialized optimization on PSNR and LPIPS across PGGAN models trained for bedroom, church, and face images.The evaluation inverts 300 real images for each of three PGGAN models.
- Analysis on Inverted Codes: Increasing the number of latent codes improves reconstruction, but adding codes beyond 20 produces no significant improvement.With 10 codes at the 6th PGGAN layer, optimization uses 20 times the original latent-space dimension.
- Analysis on Inverted Codes: Higher feature-composition layers generally improve inversion, while different tasks favor different layers according to their representation needs.Bedroom inversion is sufficient at the 4th layer, other tested models require the 8th, colorization performs best at the 8th, and inpainting at the 4th.
- Analysis on Inverted Codes: Individual latent codes specialize in reconstructing distinct meaningful image regions, whose combination composes the whole image.The roles are identified by comparing reconstruction difference maps with segmentation maps using IoU.
- Image Processing Applications: mGANprior produces competitive colorization and super-resolution results and convincingly repairs corrupted images while supporting semantic facial manipulation.Colorization is comparable to a task-specific method; super-resolution is comparable to or better than learning-based competitors; inpainting and denoising repair meaningful content.
- Knowledge Representation in GANs: Layer-wise analysis finds that lower layers encode abstract semantics, whereas higher layers represent content details and low-level pixel information.This explains why reconstruction tends to benefit from higher composition layers, while task-optimal layers differ for colorization and inpainting.
5. Conclusion
mGANprior uses multiple latent codes to reconstruct real images with a pre-trained GAN, enabling the GAN to serve as a prior for diverse image-processing tasks.
- mGANprior employs multiple latent codes to reconstruct real images with a pre-trained GAN model.