PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Image Inpainting with Contextual Attention

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang

arXiv:1801.07892v2cs.CVcs.GR

TL;DR

Large-hole inpainting methods can produce distorted structures or blurry textures because CNNs do not explicitly borrow distant contextual information. This paper introduces a feed-forward generative network with contextual attention, and reports higher-quality results across faces, textures, and natural images than existing methods.

  • Problem

    CNN-based inpainting often produces distorted structures and blurry textures because it is ineffective at explicitly using distant contextual information.

  • Method

    A unified feed-forward generative network combines contextual attention, a parallel convolutional pathway, and end-to-end reconstruction and Wasserstein GAN training.

  • Results

    Experiments across faces, textures, and natural images demonstrate higher-quality inpainting results than existing methods.

  • Takeaways & Limitations

    The framework explicitly matches and attends to relevant background patches while preserving a pathway for synthesizing novel image contents.

  • Takeaways & Limitations

    Image inpainting lacks good quantitative evaluation metrics, and Inception score is not suitable because inpainting mainly focuses on background filling.

Abstract

from arXiv · show

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting.

1. Introduction

Image inpainting must synthesize realistic, semantically plausible pixels coherent with existing regions. The paper combines contextual attention with generative inpainting to borrow distant known features while retaining the ability to create novel content.

  • Image inpainting fills missing pixels while requiring visual realism, semantic plausibility, and coherence with existing image regions.
  • Patch-based methods copy background patches effectively but cannot hallucinate novel content for complex, nonrepetitive structures.
  • CNN-based methods can generate plausible structured content but often produce boundary artifacts, distorted structures, and blurry textures.
  • The unified network combines a contextual-attention pathway with a parallel convolutional pathway so it can use references while hallucinating novel contents.
  • Experiments on faces, textures, and natural images report higher-quality inpainting results than existing methods.
  • The proposed contextual attention layer explicitly attends to related feature patches at distant spatial locations.
  • The training improvements reduce the reported training time from two months to one week.

2. Related Work

Related work spans traditional diffusion or patch-based propagation and learning-based prediction with deep neural networks. Prior methods address texture transfer, large-hole completion, attention, and optimization, but face practical limitations such as expensive computation or slow inference.

  • Traditional approaches propagate background information using variational algorithms or patch similarity, while learning-based methods train CNNs to predict missing pixels.
  • Patch-based methods work well for stationary textures but remain limited on non-stationary natural images.
  • Context Encoders introduced deep-network completion of large holes, and later work added global and local adversarial losses.
  • Face-inpainting methods use latent-space search or face-parsing losses, but typically require post-processing steps.
  • Multi-scale neural patch synthesis preserves contextual structures and adds high-frequency details but is very slow because it relies on optimization.
  • Spatial transformer networks model global affine transformations, making them unsuitable for patch-wise attention.

3. Improved Generative Inpainting Network

The improved network combines coarse-to-fine prediction, weighted reconstruction, and global/local WGAN-GP supervision to improve training efficiency and inpainting quality. It supports variable hole configurations and removes the need for image blending post-processing.

  • Coarse-to-fine network architecture: The network accepts an image and binary mask, allowing variable hole sizes, shapes, and locations during inference, including multiple holes.Training uses 256 × 256 images with randomly sampled rectangular missing regions, while inference supports different image sizes.
  • Coarse-to-fine network architecture: The generator uses a two-stage coarse-to-fine architecture in which the first network predicts coarse content and the second refines it.The coarse network uses dilated convolutions, while the refinement network receives the coarse prediction and produces the final result.
  • Global and local Wasserstein GANs: Global and local WGAN-GP losses are attached to the refinement network to enforce consistency at both image-wide and local scales.The framework combines these adversarial losses with reconstruction loss for refinement supervision.
  • Spatially discounted reconstruction loss: Spatially discounted reconstruction weights each pixel by γ^l, with γ = 0.99, reducing the influence of more ambiguous pixels farther from known regions.Here l is the distance to the nearest known pixel, and boundary pixels receive stronger supervision because they are less ambiguous.
  • Training improvements: 120 GPU-hours reduced training from 11,520 GPU-hours on Places2, an almost 100× speedup, and eliminated the prior image-blending post-processing step.The improved baseline also produced more accurate inpainting results than the referenced prior model.

4. Image Inpainting with Contextual Attention

Contextual attention lets the inpainting network borrow feature patches from distant known regions while retaining convolutional generation. The layer matches, normalizes, propagates, and reconstructs patch information within a coarse-to-fine architecture.

  • Contextual attention: Contextual attention addresses CNN limitations in borrowing features from distant spatial locations by attending to related background patches.The layer is differentiable and fully convolutional, allowing integration into deep models and testing at arbitrary resolutions.
  • Match and attend: The attention layer extracts 3 × 3 background patches as convolutional filters and matches them with foreground patches using normalized inner-product similarity.The matching compares features in missing regions with surrounding known features.
  • Match and attend: Scaled softmax converts patch similarities into attention scores, after which the extracted background patches are reused as deconvolutional filters to reconstruct foreground patches.Overlapping reconstructed pixels are averaged.
  • Attention propagation: Attention propagation applies left-right and top-down operations to encourage coherent shifts between foreground and corresponding background patches.The propagation is implemented efficiently as convolution with identity-matrix kernels and improves testing results while enriching training gradients.
  • Memory efficiency: Memory overhead from extracting 12,288 filters for a 64 × 64 hole in a 128 × 128 feature map is addressed through strided patch extraction or foreground downscaling.The attention map is upscaled after propagation when using downscaled foreground inputs.
  • Network integration: Two parallel encoders separately hallucinate content and attend to background features before merging their outputs in a single decoder.The resulting attention maps use colors to indicate the relative locations of the most attended background patches.

5. Experiments

Experiments compare the baseline and full inpainting models qualitatively, assess contextual attention against alternative attention modules, and examine losses and evaluation metrics. The full model leverages surrounding textures and structures to produce more realistic results with fewer artifacts, while quantitative metrics remain imperfect for this task.

  • Datasets: The framework is evaluated on Places2, CelebA, CelebA-HQ, DTD textures and ImageNet, with examples including results and attention maps.The experiments cover faces, textures and natural images.
  • Qualitative comparisons: Qualitative comparisons evaluate the baseline and full models on 512 × 680 Places2 images using direct model outputs without post-processing.The displayed columns include the original image, input image, baseline result, full-model result, and an upscaled attention map.
  • Qualitative comparisons: The full model uses contextual attention to leverage surrounding textures and structures, producing more realistic results with fewer artifacts than the baseline.Attention maps indicate that the model adaptively borrows information from surrounding areas during synthesis.
  • Quantitative comparisons: Learning-based methods perform better on mean ℓ1 error, mean ℓ2 error and PSNR, while raw patch-copying methods achieve lower total variation loss.These metrics are reported on the Places2 validation set for reference because reconstruction metrics do not fully capture the multiple valid inpainting solutions.
  • Efficiency: The full model contains 2.9M parameters and runs at 0.2 seconds per 512 × 512 frame on GPU and 1.5 seconds on CPU.The reported hardware uses a GTX 1080 Ti GPU and an Intel Xeon E5-2697 v3 CPU.
  • Ablation study: Contextual attention is compared with appearance flow and spatial transformer attention; appearance flow can converge to similar attention maps, while STN attention is too coarse.The authors report that additional appearance-flow attention aggregation and patch-based attention ideas did not improve results sufficiently.
  • Ablation study: WGAN-GP improves convergence behavior, whereas DCGAN sometimes collapses to limited modes and LSGAN produces worse results.The authors describe WGAN-GP as yielding faster and more stable convergence for this framework.
  • Ablation study: The ℓ1 reconstruction loss remains essential because it captures content structures and regularizes GAN training, despite tending to make results blurry.Perceptual, style and total variation losses did not provide noticeable improvements and were excluded.

6. Conclusion

The paper presents a coarse-to-fine inpainting framework with contextual attention for matching and attending to relevant background patches. It reports improved inpainting results and identifies extensions to high-resolution and related image-generation applications as future directions.

  • Conclusion: The proposed framework combines coarse-to-fine generation with a contextual attention module for explicitly matching relevant background patches.The paper introduces both a baseline model and a full model containing contextual attention.
  • Conclusion: The contextual attention module significantly improves image inpainting results by learning representations that attend to relevant background patches.
  • Future work: Future work includes very high-resolution inpainting, while the framework may also apply to conditional generation, image editing and computational photography tasks.

A. More Results on CelebA, CelebA-HQ, DTD and ImageNet

Additional qualitative results show the full model applied to CelebA-HQ, CelebA, ImageNet and DTD textures. The CelebA-HQ and CelebA experiments resize images to 256 × 256 for training and evaluation.

  • CelebA-HQ: CelebA-HQ results use images resized from 1024 × 1024 to 256 × 256 for training and evaluation.
  • CelebA: CelebA results resize images to 315 × 256 and randomly crop 256 × 256 regions to roughly unalign face landmarks.
  • ImageNet and DTD: Additional full-model results are shown for ImageNet and DTD textures.

B. Comparisons with More Methods

The paper provides qualitative comparisons with Photoshop Content-Aware Fill, Image Melding and StructCompletion using each method’s default hyperparameter settings.

  • Comparisons with more methods: Qualitative comparisons include Photoshop Content-Aware Fill, Image Melding and StructCompletion.The comparisons are presented in Figures 14 and 15.
  • Comparisons with more methods: All compared methods use their default hyperparameter settings.

C. More Visualization with Case Study

The authors visualize the input regions receiving maximum attention when holes occur at different locations, showing contextual attention borrowing information from distant spatial locations.

  • Maximum-attention regions are highlighted and overlaid on the input image to identify which areas provide information for pixels inside holes.The visualization uses the highest attention scores for pixels in missing regions.
  • Across holes placed at different locations, the visualizations demonstrate contextual attention borrowing information from distant spatial locations.

D. Network Architectures

The supplementary architectures describe the network components and notation used for the inpainting model, including stacked encoder-decoder networks, critics, and a contextual-attention branch. Qualitative figures show direct outputs across faces, natural images, and textures.

  • Network Architectures: Architecture notation uses K for kernel size, D for dilation, S for stride size, and C for channel number.
  • Inpainting network: The inpainting network stacks two encoder-decoder architectures together, with each encoder-decoder following the listed convolutional and resizing sequence.
  • WGAN-GP critics: The local and global WGAN-GP critics use convolutional layers followed by a fully connected output to produce their respective critic scores.The local critic uses Leaky ReLU with α = 0.2.
  • Contextual attention branch: The contextual attention branch applies a contextual attention layer within a convolutional feature pathway before concatenating the resulting features.
  • Qualitative results: Qualitative results present direct outputs without post-processing for masked faces, ImageNet images, DTD textures, and comparison examples.The figures include attention maps for several datasets and visualization examples.
  • Qualitative results: Attention visualizations show the input image, inpainting result, and highlighted attended regions for examining where the model draws information.
Loading 1801.07892v2…