PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

Yanhong Zeng, Jianlong Fu, Hongyang Chao, Baining Guo

arXiv:1904.07475v4cs.CV

TL;DR

High-quality image inpainting must satisfy both visual realism and semantic plausibility, whereas existing approaches commonly emphasize one. PEN-Net combines pyramid-context attention transfer with multi-scale decoding and adversarial training, producing visually and semantically coherent results across evaluated settings.

  • Problem

    Image inpainting needs visually realistic and semantically reasonable completions, but existing approaches typically emphasize either detailed patch copying or semantic coherence.

  • Method

    PEN-Net extends U-Net with pyramid-context encoding, cross-layer Attention Transfer Networks, a multi-scale decoder, pyramid losses, and adversarial training.

  • Results

    PEN-Net generated semantically reasonable and visually realistic results, ranked better than other models in 82.10% of Facade preference votes, and appeared real in 82.23% of 32 × 32-mask judgments.

  • Takeaways & Limitations

    Pyramid filling and cross-layer attention transfer support completions with clear textures and consistent structures with surrounding context.

Abstract

from arXiv · show

High-quality image inpainting requires filling missing regions in a damaged image with plausible content. Existing works either fill the regions by copying image patches or generating semantically-coherent patches from region context, while neglect the fact that both visual and semantic plausibility are highly-demanded. In this paper, we propose a Pyramid-context ENcoder Network (PEN-Net) for image inpainting by deep generative models. The PEN-Net is built upon a U-Net structure, which can restore an image by encoding contextual semantics from full resolution input, and decoding the learned semantic features back into images. Specifically, we propose a pyramid-context encoder, which progressively learns region affinity by attention from a high-level semantic feature map and transfers the learned attention to the previous low-level feature map. As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured. We further propose a multi-scale decoder with deeply-supervised pyramid losses and an adversarial loss. Such a design not only results in fast convergence in training, but more realistic results in testing. Extensive experiments on various datasets show the superior performance of the proposed network

1. Introduction

Image inpainting must produce content that is both visually realistic and semantically reasonable, but existing approaches typically emphasize one requirement. PEN-Net addresses both by filling regions at image and feature levels with pyramid-context attention transfer.

  • Image inpainting supports applications including old photo restoration and object removal.
  • Patch-based methods preserve image details by copying full-resolution patches but often lack high-level semantic understanding.
  • PEN-Net uses a U-Net backbone with a pyramid-context encoder, multi-scale decoder, and adversarial training loss.
  • The Attention Transfer Network learns region affinity from high-level feature maps and transfers relevant features into missing regions at higher-resolution layers.
  • Compared with methods emphasizing either detail or semantic coherence, PEN-Net is presented as satisfying both semantic and visual requirements.
  • Pyramid filling repeats attention transfer from deep to shallow layers, restoring increasingly fine-grained details while targeting visual and semantic coherence.

2. Related Work

Prior image-inpainting methods use patch-based copying or deep generative feature completion, while PEN-Net combines contextual encoding, attention transfer, multi-scale decoding, and adversarial training.

  • Patch-based methods sample and paste similar patches from databases or undamaged surroundings using patch-distance metrics.
  • Deep generative inpainting models encode images into latent features, fill missing regions at feature level, and decode the features back into images.
  • PEN-Net extends a U-Net with a pyramid-context encoder, multi-scale decoder, and adversarial training loss.
  • Its Attention Transfer Network learns affinity from high-level feature patches and uses it to transfer contextual features into missing regions.

3. Pyramid-context Encoder Network

PEN-Net combines pyramid-context encoding, multi-scale decoding, and adversarial training to fill missing regions at image and feature levels. Cross-layer attention transfers semantic affinity from deep features to shallower, higher-resolution features before decoding.

  • 3. Pyramid-context Encoder Network: PEN-Net uses a U-Net backbone with a pyramid-context encoder, multi-scale decoder, and discriminator for image inpainting.The backbone encodes damaged images from full-resolution pixels into compact latent features and decodes them into images.
  • 3.1. Pyramid-context encoder: The pyramid-context encoder repeatedly applies ATNs from deep to shallow layers to fill missing regions using higher-resolution features with richer details.Feature maps are denoted from deep to shallow as φL, φL−1, ..., φ1, and ATNs construct features across these levels.
  • 3.1. Pyramid-context encoder: Cross-layer attention transfer and pyramid filling are designed to preserve both visual and semantic coherence in missing regions.ATNs can also aggregate multi-scale contextual information through dilated convolutions with different rates to refine filled features.
  • 3.1. Pyramid-context encoder: Each ATN learns patch affinity from high-level semantic features and transfers attention-weighted context into adjacent low-level features.The method extracts patches, computes similarities, applies softmax attention, and uses the resulting scores to fill holes in the neighboring feature map.
  • 3.2. Multi-scale decoder: The multi-scale decoder combines ATN-reconstructed skip features with latent features, enabling detailed reconstruction and synthesis of novel objects.Transposed convolution and feature concatenation produce decoder features, while latent features support objects unavailable in the undamaged context.
  • 3.2. Multi-scale decoder: Deeply supervised pyramid L1 losses refine predictions at each scale, while Patch-GAN adversarial training selects realistic completions for the ill-posed task.The overall PEN-Net objective minimizes adversarial and pyramid L1 losses.

4. Experiments

The experiments evaluate PEN-Net across four datasets using quantitative, qualitative, user-study, and ablation analyses. Results indicate strong visual and semantic coherence, with pyramid losses and attention mechanisms contributing to performance.

  • Settings: Experiments cover Facade, DTD, CELEBA-HQ, and Places2, using baselines including PatchMatch, GL, CA, and PConv.The datasets vary from structured facades and textures to faces and natural-world scenes.
  • Quantitative comparisons: The Places2 evaluation uses L1 loss, MS-SSIM, Inception Score, and FID under randomly masked 128 × 128 square holes.L1 and MS-SSIM compare reconstructed content, while IS and FID assess generated-image quality and distributional distance.
  • Qualitative comparisons: PEN-Net produces semantically reasonable and visually realistic results with clear textures and structures consistent with surrounding context.Qualitative comparisons include regular and irregular masks across multiple datasets.
  • User study: 32.70% of votes considered outputs with challenging 128 × 128 masks real, compared with 82.23% for 32 × 32 masks.The single-image study collected 1,425 valid votes from more than 25 volunteers.
  • Component analysis: Pyramid L1 loss helps decode compact latent features layer by layer, while attention-transfer visualizations and ablations analyze the proposed components.The reported analysis links performance improvements on a U-Net backbone to the attention-transfer and pyramid-filling mechanisms.

5. Conclusion

The paper concludes that PEN-Net improves U-Net-based image inpainting through pyramid-context encoding and multi-scale decoding. It targets semantically reasonable and visually realistic results, while identifying higher-resolution refinement as future work.

  • Conclusion: PEN-Net boosts the encoding and decoding effectiveness of a vanilla U-Net using a pyramid-context encoder and a multi-scale decoder.The conclusion emphasizes cross-layer attention transfer and pyramid filling from high-level semantic features to lower-level features.
  • Conclusion: The proposed network is designed to generate semantically reasonable and visually realistic image-inpainting results.This conclusion is stated as the paper’s central objective and outcome.
  • Future work: Refining the network for higher-resolution images is identified as future work.The stated scope boundary concerns resolution rather than the current datasets or mask settings.

Supplementary Material

The supplementary material adds architectural details, training information, qualitative comparisons, and visual results beyond the main paper.

  • Supplementary Material: Supplementary material presents additional network and training details, qualitative comparisons, and visual results.It also includes more examples for object removal in natural scenes.

A. Network architectures and training details

The implementation follows baseline settings with standardized image and mask sizes and reports direct model inference without post-processing. It also describes comparison implementations and runtime.

  • Baselines: GL and PConv are implemented for comparison, with performance matching their reported results.This supports the stated fairness of the baseline comparisons.

B. More qualitative comparisons and visual results

The paper provides qualitative comparisons across facade, texture, face, and natural-scene images, including irregular and square masks, plus object-removal examples. The architecture descriptions specify the pyramid-context encoder, multi-scale decoder, and discriminator components.

  • Architecture details: The supplementary material documents the architectures of the pyramid-context encoder, multi-scale decoder, and discriminator used by the model.The decoder produces predictions at each scale, while the discriminator uses spectrally normalized convolutions.
  • Qualitative comparisons: Figures 9 and 10 compare PatchMatch, GL, GntIpt, PConv, and the proposed model on Facade and DTD images with square or irregular masks.The comparisons assess whether textures and semantic structures remain consistent with surrounding context.
  • Qualitative comparisons: Figure 11 compares the proposed model with PatchMatch, GL, GntIpt, and PConv on CELEBA-HQ and Places2 images with square masks.The stated comparison focuses on naturalness for human faces and natural scenes.
  • Object removal: Figure 12 presents object-removal results on natural-scene images, highlighting semantically reasonable and visually realistic completions.The paper identifies these results as promising for user scenarios.
Loading 1904.07475v4…