Source-linked AI summary
Patch-Based Image Inpainting with Generative Adversarial Networks
Ugur Demir, Gozde Unal
TL;DR
Large-hole inpainting remains difficult because existing neural methods can introduce artifacts and must reconcile local texture with global image structure. The paper proposes PGGAN, which combines G-GAN and PatchGAN through shared layers and split adversarial pathways. Its evaluations report considerable visual and quantitative improvements over current inpainting methods, while the problem remains open to further progress.
Problem
Large missing regions challenge inpainting systems because they require realistic reconstruction despite substantial information loss, artifacts, and the need for both local and global consistency.
Method
PGGAN combines G-GAN and PatchGAN in a discriminator with shared early layers and split paths producing separate adversarial losses.
Results
The framework achieves considerable visual and quantitative improvements over current inpainting methods.
Takeaways & Limitations
Combining local patch assessment with global image evaluation can aid local texture continuity while conforming to holistic image structure.
Takeaways & Limitations
The inpainting problem remains open to further progress.
Abstract
from arXiv · showhide
Area of image inpainting over relatively large missing regions recently advanced substantially through adaptation of dedicated deep neural networks. However, current network solutions still introduce undesired artifacts and noise to the repaired regions. We present an image inpainting method that is based on the celebrated generative adversarial network (GAN) framework. The proposed PGGAN method includes a discriminator network that combines a global GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network layers between G-GAN and patchGAN, then splits paths to produce two adversarial losses that feed the generator network in order to capture both local continuity of image texture and pervasive global features in images. The proposed framework is evaluated extensively, and the results including comparison to recent state-of-the-art demonstrate that it achieves considerable improvements on both visual and quantitative evaluations.
1. Introduction
Large-hole inpainting remains difficult because missing information requires both local texture continuity and global image harmony. The paper proposes PGGAN, combining PatchGAN and G-GAN pathways to address these requirements.
- Large missing regions are challenging because they remove substantial information and require both local and global image conformity.
- GANs use adversarial discrimination between real and generated images to train generators toward realistic outputs.
- PatchGAN evaluates local patches, while the proposed discriminator combines PatchGAN with G-GAN to capture local continuity and global structure.
- PGGAN shares early network layers before splitting into two paths with separate adversarial losses for local and holistic image features.
- The architecture adds dilated and interpolated convolutions to ResNet within an end-to-end network for high-resolution image inpainting.
- The study includes ablation analyses and detailed comparisons with recent state-of-the-art inpainting methods.
2. Related works
Prior inpainting methods addressed texture, global structure, or local details separately, but retained limitations in preserving compatibility across the completed image. PGGAN is positioned as a PatchGAN-based approach that explores local regions while incorporating global dependencies.
- Global GANs assess whole images, whereas PatchGAN slides a window across inputs to score local patches and preserve local continuity.
- PGGAN accommodates PatchGAN for inpainting and explores local regions together with dependencies among them.
- Early single-image methods copied texture from nearby regions but lacked global structural information.
- Context-Encoder and related methods used CNNs, adversarial losses, Euclidean losses, or style transfer, but reported implausible results or added correction stages.
- Prior arbitrary-region methods added a local discriminator over filled regions, but mask-location conditioning was observed to produce disharmony with surrounding unmasked texture.
3. Proposed Method
The proposed method uses a ResNet-based generator and a PGGAN discriminator to complete arbitrary large holes while evaluating both whole-image realism and local texture consistency. Training combines reconstruction and two adversarial losses through shared and split discriminator pathways.
- Generator network: The generator uses down-sampling, residual blocks, and up-sampling, with equal numbers of down-sampling and up-sampling layers to preserve input dimensions.Down-sampling uses strided convolutions without pooling, and residual blocks preserve activation-map width and height.
- Generator network: Dilated convolutions increase receptive-field size without additional sub-sampling or excessive convolution layers.Receptive-field size controls dependencies between distant regions and affects texture generation.
- Generator network: Interpolated convolution resizes feature maps before convolution to address checkerboard artifacts caused by transposed convolution.This operation maps low-resolution feature maps to the desired size before applying convolution.
- Discriminator architecture: PGGAN combines a global GAN structure with PatchGAN by sharing initial discriminator layers and splitting later layers into global and local paths.The global path produces a whole-image real/fake decision, while the second path evaluates local texture details.
- Objective function: Training jointly optimizes pixel-wise L1 reconstruction loss and adversarial losses from both PGGAN paths using Adam.The reconstruction term guides coarse colors and low-frequency details, while the adversarial terms are computed by the discriminator’s two paths.
- Objective function: The joint objective weights reconstruction, global adversarial, and local adversarial terms with coefficients λ, assigning corresponding losses to shared and unshared layers.Generator parameters use the joint loss; unshared global and patch paths use their respective adversarial losses, while shared layers use their sum.
4. Results
The evaluation combines ablation studies, quantitative comparisons, perceptual scoring, and visual inspection across 256x256 and 512x512 image settings. PGGAN blends local and global structure, removes checkerboard artifacts with interpolated convolution, improves reported measures, and has millisecond-scale computation comparable to GLGAN.
- Ablation study: Ablation results show that PGGAN blends local and global structure, producing more visually plausible inpainting than G-GAN or PatchGAN alone.The comparison uses the same ResNet generator with different discriminator architectures.
- Ablation study: Interpolated convolution clears checkerboard artifacts caused by transposed convolution, although transposed convolution sometimes produces more consistent structures.The authors state that analyzing both layers further is outside the study’s scope.
- Comparative evaluation: PGGAN is compared with CE-Context-Encoder, GLGAN, and Neural Patch Synthesis on Paris Street View images at 256x256 and 512x512 resolutions.The compared PGGAN variants are PGGAN-Res and PGGAN-DRes.
- Comparative evaluation: PGGAN and GLGAN have computation times on the order of miliseconds, whereas NPS takes several seconds because of its local texture constraint.The speed comparison concerns end-to-end texture generators versus NPS.
- Comparative evaluation: PGGAN achieves an improvement in all reported measures for both 512x512 and 256x256 images.The reported measures are PSNR, SSIM, mean L1, and mean L2 loss.
- Perceptual evaluation: Perceptual evaluation found slightly better naturalness scores for PGGAN, while CE trained for 128x128 images performed poorly on 256x256 test images.The evaluation used 12 voters scoring 500 randomly sorted and blinded Paris validation images.
5. Conclusion
The results suggest that merging PatchGAN with a traditional GAN helps preserve local texture continuity while respecting holistic image structure. The inpainting problem remains open to further progress.
- Low-level merging and high-level splitting of PatchGAN with a traditional GAN improves local texture continuity and holistic image conformity.
- The merger produces visually and quantitatively better results than current inpainting methods.
- Figure 8 compares PGGAN-RES with NPS on the 512x512 Paris Street View Dataset.
- Figure 9 presents non-cherry-picked results from PGGAN-DRes.
1. Additional visual results
Additional visual results show PGGAN outputs across ImageNet, Google Street View, and Places2 datasets under varied image sizes and masking settings.
- The visual results include input-output examples from ImageNet, Google Street View, and Places2 datasets.
- High-resolution ImageNet experiments use 512x512 inputs with randomly located cropped regions, and the model fills the blank areas.
- Google Street View experiments scale images to 256x256, extract 128x128 center patches, and reconstruct whole images without the mask location.
- Places2 training uses 8 millions images, 256x256 inputs, and randomly sized masks.