PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, Ping Luo

arXiv:2003.13659v4eess.IVcs.CV

TL;DR

Existing image priors do not fully capture rich semantics needed for versatile image restoration and manipulation. DGP exploits a GAN trained on large-scale natural images by progressively fine-tuning its generator with discriminator feature regularization, achieving faithful restoration and diverse manipulation across tasks.

  • Problem

    Existing image priors capture limited image statistics, while fixed-generator GAN inversion struggles to faithfully reconstruct complex real-world images.

  • Method

    DGP jointly optimizes the latent vector and generator parameters, using discriminator feature matching and progressive reconstruction to preserve the generative prior.

  • Results

    DGP generalizes across restoration and manipulation tasks, restoring missing information and producing visually realistic effects; colorization reaches 62.8% ResNet50 accuracy.

  • Takeaways & Limitations

    A GAN trained on massive natural-image datasets can serve as a generic image prior for versatile restoration and manipulation.

  • Takeaways & Limitations

    Fixed-generator inversion is limited by gaps between the learned and actual natural-image manifolds, causing mismatches for complex images.

Abstract

from arXiv · show

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig.1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN-inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature image, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.

1 Introduction

The paper develops Deep Generative Prior (DGP), using a GAN trained on large-scale natural images to capture rich semantics for general restoration and manipulation. It addresses fixed-generator GAN inversion by fine-tuning the generator progressively with discriminator-based regularization.

  • GANs trained on large-scale natural images capture color, spatial coherence, textures, and high-level concepts useful for broader restoration and manipulation.
  • Existing GAN inversion optimizes only the latent vector with a fixed generator, often producing mismatched details and contents for complex real-world images.
  • DGP jointly optimizes the latent vector and generator parameters on the target image, lifting the constraint of the training distribution.
  • Discriminator feature matching regularizes reconstruction because the discriminator preserves the generator's pretrained parameter structure better than separate perceptual metrics.
  • Progressive reconstruction with DGP restores missing color, patches, and details while preserving existing semantics across restoration and manipulation tasks.

2 Related Work

Related work includes conventional image priors, deep priors, task-specific restoration and manipulation systems, and GAN inversion. DGP instead studies a generic prior from a GAN trained on large-scale natural images while adapting the generator to each image.

  • Traditional priors model particular natural-image statistics, including neighboring-pixel correlations, dark channels, and total variation.
  • Deep image prior and SinGAN learn image statistics from a single image and support low-level restoration or manipulation tasks.
  • Task-specific restoration and manipulation methods target individual applications with specialized models, losses, or pipelines.
  • Prior GAN-based approaches use pretrained generators for manipulation or restoration, while multi-code GAN prior keeps the generator fixed.
  • GAN inversion reconstructs images by finding latent vectors for a fixed generator, either through direct optimization or an additional encoder.

3 Method

The method exploits a pretrained GAN as a rich image prior by jointly optimizing its latent code and generator parameters, while discriminator features and progressive block tuning preserve natural, semantically consistent reconstructions.

  • Deep Generative Prior: DIP mainly uses statistics already present in the degraded input, limiting tasks requiring general image statistics such as colorization and manipulation.A GAN trained on large-scale natural images can encode color, spatial coherence, textures, and high-level concepts.
  • Deep Generative Prior: Standard GAN inversion optimizes only the latent vector with a fixed generator, often producing mismatched reconstructions for complex or out-of-distribution images.The limitation arises from gaps between the generator’s approximated manifold, the training manifold, and the actual natural-image manifold.
  • Deep Generative Prior: DGP jointly optimizes the latent vector and generator parameters on the target image, making fitting a single image more achievable while retaining the generator’s natural-image statistics.The updated objective is θ*, z* = arg min L(x, G(z; θ)), with the generator fine-tuned on-the-fly.
  • Discriminator Guided Progressive Reconstruction: The method uses discriminator feature matching as a distance metric because the discriminator is trained with the generator and is aligned with its parameter structure.The metric computes L1 distances between discriminator features across selected blocks and can produce vivid natural intermediate outputs.
  • Discriminator Guided Progressive Reconstruction: Progressive reconstruction fine-tunes shallow generator blocks before deeper ones, controlling global configuration first and details later.This strategy addresses information lingering and better preserves consistency between missing and existing semantics than simultaneous tuning.

4 Applications

DGP applies a pretrained BigGAN prior to reconstruction and demonstrates broad image restoration and manipulation capabilities. Across tasks, it improves reconstruction fidelity, preserves spatial coherence, sharpens super-resolution, and supports flexible semantic editing.

  • Applications: DGP uses a pretrained BigGAN and progressively reconstructs images with discriminator feature loss.The method is evaluated on ImageNet and applied across restoration and manipulation tasks.
  • Applications: DGP achieves high-fidelity image reconstruction, outperforming other GAN-inversion methods in PSNR and SSIM while reducing visible mismatches.The reconstruction error is described as almost visually imperceptible.
  • Image Restoration: DGP colorizes grayscale images competitively without task-specific training, reaching 62.8% ResNet50 accuracy versus 51.5%–56.2% for reported baselines.The method is visually better or comparable to Autocolorize despite not being directly optimized for color prediction.
  • Image Restoration: For inpainting, DGP preserves spatial coherence across large missing regions and outperforms DIP and other GAN-inversion methods quantitatively.Its learned generative prior helps recover missing content in harmony with surrounding context.
  • Image Restoration: DGP produces sharper, more faithful 4× super-resolution, with loss weighting allowing trade-offs between perceptual quality and PSNR/RMSE.Biasing toward discriminator loss improves NIQE, while higher MSE weighting improves PSNR and RMSE.
  • Image Manipulation: DGP extends beyond restoration to adversarial defense, diverse colorization, hybrid restoration, out-of-distribution images, random jittering, morphing, and category transfer.Fine-tuned generators retain natural-image structure while enabling pose, content, and category changes.

5 Conclusion

The paper presents DGP as a generic image prior from a GAN trained on massive natural-image datasets, supporting restoration and realistic manipulation through progressive discriminator-guided reconstruction.

  • DGP restores missing information in degraded images by progressively reconstructing them under a discriminator metric.
  • DGP supports multiple visually realistic image-manipulation effects while addressing challenges in GAN inversion.

Appendix

The appendix adds qualitative results, implementation details, and links to restoration and manipulation videos.

  • The appendix provides additional qualitative results and implementation details for the experiments.
  • Restoration and manipulation videos are available through the project’s GitHub repository.

A Qualitative Examples

The appendix extends the main paper with qualitative examples spanning restoration, reconstruction, manipulation, and comparisons of losses and optimization methods.

  • Restoration: Figures 14–18 extend qualitative restoration examples for colorization, image reconstruction, inpainting, super-resolution, and varied restoration tasks.
  • Losses and optimization: Figure 19 compares MSE, VGG perceptual, discriminator feature matching, and progressive reconstruction in colorization and super-resolution.
  • Losses and optimization: Figure 20 compares loss types and optimization techniques for fine-tuning the generator to restore images.
  • Manipulation: Figures 21–24 present qualitative results for random jittering, image morphing, and category transfer.
  • Manipulation: In image morphing, latent-vector optimization fails to reconstruct accurately, while DIP and MSE or VGG-based joint optimization do not produce realistic interpolation; discriminator feature matching performs better.

B Implementation Details

The implementation uses BigGAN models trained on ImageNet, discriminator-feature initialization, progressive joint fine-tuning, task-specific losses, and explicit class-condition assumptions.

  • Architectures: BigGAN models at 128^2 and 256^2 resolutions are trained on ImageNet, with Inception scores of 103.5 and 94.5, respectively.
  • Initialization: Initialization samples 500 GAN images and selects the target’s nearest neighbor under the discriminator feature metric as the starting latent vector.
  • Class condition: BigGAN reconstruction assumes the class label is given except for adversarial defense; otherwise, both the latent vector and class condition are randomly sampled.
  • Fine-tuning: Fine-tuning jointly updates the generator and latent vector, uses tracked batch-normalization statistics, and proceeds across generator blocks with task-specific learning settings.
  • Task-specific settings: Inpainting and super-resolution combine discriminator and MSE losses, while instance normalization enables higher learning rates and better PSNR in specified settings.
  • Task-specific settings: The appendix specifies separate fine-tuning settings for colorization, super-resolution, adversarial defense, inpainting, and manipulation tasks.
Loading 2003.13659v4…