PaperlayerView exact revision on arXiv

Source-linked AI summary

PiiGAN: Generative Adversarial Networks for Pluralistic Image Inpainting

Weiwei Cai, Zhanguo Wei

arXiv:1912.01834v2cs.CVcs.LG

TL;DR

Large missing regions make image inpainting uncertain, while existing methods often return only one completion. PiiGAN uses an extractor and consistency losses to learn multiple style-conditioned results; the authors report diverse, natural outputs with higher quality across several datasets.

  • Problem

    For large missing regions, inpainting is uncertain, but existing methods typically seek one “optimal” result rather than offering multiple plausible completions.

    Source: §1 · pp. 1–2

  • Method

    PiiGAN extracts style vectors from ground-truth images and uses consistency losses to train a generator to produce outputs matching those styles and image semantics.

    Source: §1 · pp. 2–3

  • Results

    The authors report multiple plausible, semantically consistent outputs and higher quality and diversity across evaluated datasets.

    Source: §1 · pp. 2–3, §5 · p. 11

  • Takeaways & Limitations

    For a single image with missing regions, PiiGAN can generate numerous diverse results, particularly for images with large missing areas.

    Source: §5 · p. 11

    The extractor assumes an approximate Gaussian posterior with diagonal covariance for the latent representation.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

The latest methods based on deep learning have achieved amazing results regarding the complex work of inpainting large missing areas in an image. But this type of method generally attempts to generate one single "optimal" result, ignoring many other plausible results. Considering the uncertainty of the inpainting task, one sole result can hardly be regarded as a desired regeneration of the missing area. In view of this weakness, which is related to the design of the previous algorithms, we propose a novel deep generative model equipped with a brand new style extractor which can extract the style feature (latent vector) from the ground truth. Once obtained, the extracted style feature and the ground truth are both input into the generator. We also craft a consistency loss that guides the generated image to approximate the ground truth. After iterations, our generator is able to learn the mapping of styles corresponding to multiple sets of vectors. The proposed model can generate a large number of results consistent with the context semantics of the image. Moreover, we evaluated the effectiveness of our model on three datasets, i.e., CelebA, PlantVillage, and MauFlex. Compared to state-of-the-art inpainting methods, this model is able to offer desirable inpainting results with both better quality and higher diversity. The code and model will be made available on https://github.com/vivitsai/PiiGAN.

1 Introduction

Large missing regions make inpainting uncertain, yet existing methods typically produce only one “optimal” completion. PiiGAN uses a style extractor and consistency losses to learn multiple plausible completions, which the authors report are diverse and high quality across datasets.

1 Introduction

  • Large missing regions increase uncertainty, so a single reasonable completion may not meet expectations when other plausible outcomes exist.For faces, alternatives can include different eye directions or the presence of glasses.Source: §1 · p. 1
  • Existing methods generally seek one “optimal” result, while some CVAE-based approaches are limited to targeted attributes and may produce unreasonable images.The passage describes limited variation when a masked image serves as a label with usually only one matching training label.Source: §1 · p. 2
  • The proposed approach extracts style noise from ground-truth images and trains the generator with consistency loss to approximate those images.The paper also uses consistency loss to make style noise extracted from generated images approximate the input noise.Source: §1 · p. 2
  • A new extractor obtains style vectors from training samples, while consistency loss guides the generator to learn varied styles matching input-image semantics.During iterative training, extracted ground-truth style noise and the ground-truth image are input to the generator; random noise and a masked image are also used, with consistency losses and KL loss.Source: §1 · pp. 2–3
  • PiiGAN is reported to produce multiple plausible completions consistent with image semantics, with higher quality and diversity across evaluated datasets.The paper evaluates CelebA, Agricultural Disease, and MauFlex.Source: §1 · pp. 2–3

2 Related Work

Related work spans traditional completion, deep generative inpainting, and conditional image generation. The cited work describes limits in traditional methods’ scale or semantic reasoning, and in existing deep methods’ ability to produce varied plausible results.

2.1 Computer vision in IoT

  • In IoT computer vision, prior manual algorithms are described as accurate in laboratory and simulated settings but vulnerable when lighting or camera views depart from design assumptions.The passage also notes that IoT-related image-inpainting research had only just emerged.Source: §2.1 · p. 3

2.2 Image inpainting by traditional methods

  • Diffusion-based methods are limited to small missing areas, while patch-based methods can produce distorted structures and unreasonable textures.Other patch methods extract only low-level texture and structure, limiting semantically reasonable completion when matching texture is absent.Source: §2.2 · p. 3

2.3 Image inpainting by deep generative models

  • Deep generative methods can produce semantically consistent completions, but CNN-based approaches have been associated with boundary shadows, distortion, and blurred textures, and realistic results from residual latent features remain difficult.Spatial-attention methods use surrounding features as references; the architecture shown uses global and local discriminators to assess whole-image and inpainting-region consistency.Source: §2.3 · p. 4, Figure 2 · p. 4

2.4 Conditional Image Generation

  • Conditional image-generation approaches can yield diverse outputs, but cited attribute-conditioned facial restoration is limited to specific domains and requires specified attributes.Source: §2.4 · p. 4

3 Proposed Approach

PiiGAN targets the limitation of deterministic inpainting methods, which produce a single result, by adding an extractor and training a generator to produce pluralistic inpaintings.

3.1 Extractor

  • The extractor maps training images to style latent vectors, outputting mean and covariance for a Gaussian posterior and sampling a vector for the generator.Its architecture has four convolutional layers, a flattened layer, and two parallel fully connected layers.Source: §3.1 · pp. 4–6

3.2 Pluralistic Image Inpainting Network: PiiGAN

  • PiiGAN feeds masked images with either extracted style vectors or random vectors to the generator, and re-extracts style vectors from generated images.The L1 consistency loss brings generated images toward ground truth and extracted fake-image styles toward their input vectors.Source: §3.2 · p. 6

3.2.1 Consistency Loss

  • The consistency loss uses an extractor-derived style vector from a ground-truth image to guide the generator toward a completed image close to that ground truth.The paper adjusts perceptual loss to address consistency between generator and extractor feature maps.Source: §3.2.1 · p. 6

3.2.2 Adversarial Loss

  • For the local discriminator, gradient penalty is applied only to missing-region pixels by multiplying the gradient by the input mask.The text specifies mask value 0 in missing regions and 1 elsewhere.Source: §3.2.2 · p. 6

3.2.3 Distributive Regularization

  • The KL divergence adjusts the learned importance-sampling functions for ground-truth and fake images toward fixed potential priors, both described as Gaussian..Source: §3.2.3 · p. 7

3.2.4 Objective

  • The overall diversity-inpainting objective combines KL, consistency, and adversarial losses using tradeoff parameters.Source: §3.2.4 · p. 7

3.3 Training

  • Training uses an extracted ground-truth style vector with a masked image, then trains a random-vector path by matching the generated image’s extracted style to its input vector.The procedure updates the generator using L1 losses for image-to-ground-truth and style-to-input matches.Source: §3.3 · p. 7

4 Experiments and Results

The experiments evaluate PiiGAN across multiple datasets using quantitative and qualitative comparisons, with the reported results covering completion quality, diversity, and realism. The evaluations include a single-sample quantitative comparison and qualitative comparisons of outputs and competing methods.

4 Experiments and Results

  • The evaluation used CelebA, PlantVillage, and MauFlex, with a 15% test split; comparisons used centering-hole masks.The reported dataset sizes were 200k, 45k, and 25k images, respectively.Source: §4 · p. 7, §4.1 · p. 8

4.1 Implementation Details

  • The network had 47 million trainable parameters and was trained on one NVIDIA 1080 GPU with batch size 12.Training took roughly three days for CelebA, two days for Agricultural Disease, and one day for MauFlex.Source: §4.1 · p. 8

4.2 Quantitative comparisons

  • PiiGAN outperformed GL, SH, and CA on the reported CelebA quantitative tests, using one selected result per image for comparison.The comparison used 10,000 test images and a 64 × 64 center mask; Table 1 reports the results.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Qualitative Comparisons

  • On CelebA, PiiGAN produced clearer, plausible completions with complementary facial attributes; GL’s results were natural, while SH’s were blurry and lacked detail.Figure 4 presents qualitative comparisons for large missing regions.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
  • On MauFlex and the agricultural disease dataset, PiiGAN produced varied reasonable completions, while CA produced only a single result.Figures 5 and 6 show the comparisons on those datasets.Source: §4.3 · pp. 8–9, Figure 5 · p. 9, Figure 6 · p. 10

4.4 Other comparisons

  • In the Figure 7 comparison, BicycleGAN’s output was unnatural, while StarGAN produced variety but required targeted attributes for training.The figure places PiiGAN at the top, StarGAN in the middle, and BicycleGAN at the bottom.Source: §4.4 · p. 10, Figure 7 · p. 10
  • PiiGAN achieved the highest reported realism score; the authors associate its realism with KL divergence loss on the extractor’s latent vector.The authors caution that BicycleGAN’s relatively high diversity score may reflect unreasonable outputs.Source: §4.4 · pp. 10–11, Table 2 · p. 10, Table 3 · p. 11

5 Conclusion

The conclusion presents PiiGAN as a pluralistic inpainting model with a style extractor. It reports diverse, natural results, particularly for large missing areas, and identifies possible applications and future work.

5 Conclusion

  • PiiGAN combines a newly designed style extractor with a generative adversarial network for pluralistic image inpainting.The conclusion says one masked input can yield numerous diverse results with plausible content.Source: §5 · p. 11
  • Experiments across various datasets showed diverse and natural results, especially for images with large missing areas.Source: §5 · p. 11
  • The authors identify art restoration, facial micro-shaping, and image augmentation as potential applications, and large irregular missing areas as future work.Source: §5 · p. 11

A More Comparisons Results

Additional quantitative comparisons cover AgriculturalDisease and MauFlex, alongside comparisons with CA, SH, and GL. The authors report that PiiGAN is superior to current state-of-the-art methods on multiple datasets.

A More Comparisons Results

B Network Architecture

The extractor architecture is presented in Table 6, with ELU activations after each convolutional layer and notation for its architectural parameters and batch size.

B Network Architecture

C More diverse examples using the CelebA, AgriculturalDisease, and MauFlex datasets

Qualitative comparisons report greater diversity from the model than existing methods on MauFlex, CelebA, and AgriculturalDisease. The examples span varied trees, faces, and diseased leaves.

MauFlex

CelebA

  • On CelebA, the model’s direct outputs show more valuable diversity than existing methods.Images were randomly cropped to 178 x 178 and resized to 128 x 128 for training and evaluation; Figure 9 shows variation in faces, gaze angles, and glasses.Source: C More diverse examples using the… · p. 14, Figure 9 · p. 14

AgriculturalDisease

Related papers

Loading 1912.01834v2…