Source-linked AI summary
PD-GAN: Probabilistic Diverse GAN for Image Inpainting
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao
TL;DR
Image inpainting must fill uncertain holes while preserving realistic context, yet existing approaches often struggle to provide both diversity and realism. PD-GAN uses noise-driven generation with spatially probabilistic modulation and a perceptual diversity loss, and experiments report diverse, high-quality reconstructions.
Problem
Existing diverse inpainting methods struggle to generate content that is both varied and realistic for uncertain hole regions.
Method
PD-GAN progressively modulates random noise with a coarse reconstruction and mask using hard and soft SPDNorm, then applies perceptual diversity loss.
Results
Experiments across benchmark datasets show that PD-GAN produces diverse and high-quality image-inpainting reconstructions.
Takeaways & Limitations
PD-GAN balances contextual guidance near hole boundaries with greater generation freedom toward hole centers.
Takeaways & Limitations
A conventional diversity loss was found unsuitable because it alters contextual regions and makes training unstable.
Abstract
from arXiv · showhide
We propose PD-GAN, a probabilistic diverse GAN for image inpainting. Given an input image with arbitrary hole regions, PD-GAN produces multiple inpainting results with diverse and visually realistic content. Our PD-GAN is built upon a vanilla GAN which generates images based on random noise. During image generation, we modulate deep features of input random noise from coarse-to-fine by injecting an initially restored image and the hole regions in multiple scales. We argue that during hole filling, the pixels near the hole boundary should be more deterministic (i.e., with higher probability trusting the context and initially restored image to create natural inpainting boundary), while those pixels lie in the center of the hole should enjoy more degrees of freedom (i.e., more likely to depend on the random noise for enhancing diversity). To this end, we propose spatially probabilistic diversity normalization (SPDNorm) inside the modulation to model the probability of generating a pixel conditioned on the context information. SPDNorm dynamically balances the realism and diversity inside the hole region, making the generated content more diverse towards the hole center and resemble neighboring image content more towards the hole boundary. Meanwhile, we propose a perceptual diversity loss to further empower PD-GAN for diverse content generation. Experiments on benchmark datasets including CelebA-HQ, Places2 and Paris Street View indicate that PD-GAN is effective for diverse and visually realistic image restoration.
1. Introduction
Existing inpainting methods commonly produce one restored image and diverse methods struggle to combine variation with realism. PD-GAN addresses this by modulating random noise with contextual priors and spatially varying constraints, supported by a perceptual diversity loss.
- Encoder-decoder inpainting methods typically target a single restored result for each input image.
- Diverse inpainting methods remain less effective at generating content that is simultaneously diverse and realistic.
- PD-GAN builds on a vanilla GAN, injecting a coarse reconstruction and region mask into noise-driven decoding through SPDNorm.
- SPDNorm uses spatially varying modulation to enhance diversity toward the hole center while reducing it near the hole boundary.
- The perceptual diversity loss separates outputs generated from shared prior information but different noise vectors in feature space.
- Experiments on benchmark datasets indicate that PD-GAN generates diverse and visually realistic image-inpainting content.
2. Related Work
Related work distinguishes single-solution and diverse inpainting, while prior diverse methods commonly condition sampling variation on the masked image. PD-GAN instead combines noise-based generation with spatially probabilistic normalization to balance contextual guidance and diversity.
- Single-solution methods produce one result per masked image, whereas diverse methods generate multiple results for each corrupted image.
- Conditional-VAE inpainting methods sample from a Gaussian distribution conditioned on the masked image to obtain diverse results.
- Their diversity is limited because the masked image controls the distribution dispersion.
- PD-GAN samples a standard-Gaussian latent vector and maps it directly to an image with a single decoder.
- Other diverse image-generation methods are not designed for inpainting and may produce artifacts in inpainted results.
- SPDNorm extends spatially adaptive denormalization with hard and soft probabilistic maps that control prior-information confidence.
3. PD-GAN
PD-GAN combines a vanilla GAN decoder with SPDNorm, which injects coarse predictions and masks while spatially balancing contextual consistency near hole boundaries against diversity toward hole centers. A perceptual diversity loss further encourages semantically diverse outputs while preserving context.
- PD-GAN architecture: PD-GAN modulates a random latent vector through a single decoder, injecting coarse predictions and masks at multiple resolutions via SPDNorm residual blocks.The coarse prediction comes from a pre-trained Partial Convolutional encoder-decoder.
- Soft SPDNorm: Soft SPDNorm learns an adaptive diversity map from the prior information and input features, with background elements fixed to 1.Its foreground values tend to change smoothly and remain near 0.5, so it is paired with hard SPDNorm.
- Hard SPDNorm: Hard SPDNorm uses a fixed mask-derived map whose diversity probability increases from the hole boundary toward the center.The map is built through iterative mask dilation, with k empirically set to 4; background values remain 1.
- SPDNorm residual block: The SPDNorm residual block combines hard and soft normalization so fixed spatial constraints and learned prior-conditioned behavior complement each other.Each block operates at a different scale, requiring the prior information and mask to be downsampled accordingly.
- Qualitative comparisons: Qualitative comparisons present diverse outputs from PIC and PD-GAN alongside inputs, prior information, and original images on Paris Street View and Places2.The referenced comparisons organize PD-GAN outputs in panels (j)-(l) and PIC outputs in panels (g)-(i).
- Perceptual diversity loss: The perceptual diversity loss compares masked VGG-19 feature activations, encouraging diverse hole content while avoiding contextual changes and trivial all-black or all-white solutions.The loss omits latent-vector distances and is reported to stabilize training further.
4. Experiments
PD-GAN is evaluated against single- and diverse-solution inpainting methods on Paris StreetView, CelebA-HQ, and Places2, using qualitative, realism, diversity, and ablation comparisons. Across these evaluations, PD-GAN produces multiple visually natural results and achieves relatively high diversity, with 83% user preference over PIC for diversity.
- Datasets: Experiments use Paris StreetView, CelebA-HQ, and selected Places2 scene categories, retaining original splits for the first two datasets.Places2 uses nine scene categories and follows Shift-Net's training and validation setup.
- Compared Methods: The comparison includes RFR, PC, GC, EC, PIC, CVAE, and BicycleGAN for single-solution quality and diverse-result generation.The study compares both conventional inpainting approaches and methods specifically evaluated for diversity.
- Qualitative Comparisons: PD-GAN generates multiple results with higher naturalness than competing methods on CelebA-HQ and Places2 qualitative comparisons.Compared outputs include PC prior information, PIC diverse outputs, and PD-GAN diverse outputs; competing methods are described as blurry, insufficiently diverse, or semantically unreasonable in these settings.
- Ablation Study: The ablation study visualizes no-diversity-loss, hard-only, soft-only, conventional-loss, SPADE, and full PD-GAN variants.Figure 7 labels the outputs by the removed or replaced component.
- Quantitative Comparisons: PD-GAN obtains relatively higher diversity scores than existing methods using LPIPS on 5K generated pairs from 1K Places2 images.The diversity evaluation compares full-image and mask-region outputs.
- Quantitative Comparisons: 83% of user-study cases favored PD-GAN over PIC for diversity across 10 result sets evaluated by 10 volunteers.Each subject selected the method with more diverse results.
5. Ablation Study
The ablations show that hard and soft SPDNorm serve complementary roles, while perceptual diversity loss improves diversity without the degenerate outputs associated with conventional diversity loss. The full method therefore provides a favorable balance between diverse details, meaningful content, and visual quality.
- SPDNorm: Using both hard and soft SPDNorm achieves favorable diversity and quality, whereas hard-only and soft-only variants expose different weaknesses.Soft-only SPDNorm retains meaningful content but reduces diversity; hard-only SPDNorm produces diverse details with obvious artifacts.
- SPDNorm: Replacing SPDNorm with SPADE makes training unstable and produces worse results because SPADE unconditionally relies on coarse prior information.The paper characterizes SPADE as a degenerated form of SPDNorm with both diversity maps set to 1.
- Perceptual Diversity Loss: Removing perceptual diversity loss reduces diversity, while conventional diversity loss increases diversity but can produce all-black or all-white recovered content.The proposed perceptual diversity loss addresses these failure modes and achieves favorable numerical performance in the full method.
6. Conclusion
PD-GAN combines progressive prior modulation, soft and hard SPDNorm, and perceptual diversity loss to produce diverse, high-quality inpainting results.
- PD-GAN progressively modulates random noise with prior information to generate diverse inpainting results.
- Soft and hard SPDNorm control the probability of producing diverse results during modulation.
- A perceptual diversity loss further boosts PD-GAN’s diversity.
- Experiments across varied datasets show that PD-GAN produces diverse predictions and high-quality reconstruction content.