PaperlayerView exact revision on arXiv

Source-linked AI summary

Coherent Semantic Attention for Image Inpainting

Hongyu Liu, Bin Jiang, Yi Xiao, Chao Yang

arXiv:1905.12384v3cs.CV

TL;DR

Image inpainting methods can produce blurry textures and distorted structures when they ignore semantic relevance and feature continuity in hole regions. This paper proposes coherent semantic attention within a two-step rough-to-refinement generative model, supported by consistency loss and feature-patch discrimination. Experiments on CelebA, Places2, and Paris StreetView report higher-quality results than existing state-of-the-art approaches.

  • Problem

    Existing inpainting methods often generate blurry textures and distorted structures because they ignore semantic relevance and feature continuity in hole regions.

  • Method

    The paper uses a coherent semantic attention layer in a two-step rough and refinement model, with consistency loss guiding CSA and its decoder counterpart toward ground-truth VGG features.

  • Results

    Experiments on CelebA, Places2, and Paris StreetView show higher-quality inpainting results than existing methods and state-of-the-art comparisons.

  • Takeaways & Limitations

    The approach produces more coherent textures and supports both irregular and centering masks while allowing the two-stage network to be trained end to end.

Abstract

from arXiv · show

The latest deep learning-based approaches have shown promising results for the challenging task of inpainting missing regions of an image. However, the existing methods often generate contents with blurry textures and distorted structures due to the discontinuity of the local pixels. From a semantic-level perspective, the local pixel discontinuity is mainly because these methods ignore the semantic relevance and feature continuity of hole regions. To handle this problem, we investigate the human behavior in repairing pictures and propose a fined deep generative model-based approach with a novel coherent semantic attention (CSA) layer, which can not only preserve contextual structure but also make more effective predictions of missing parts by modeling the semantic relevance between the holes features. The task is divided into rough, refinement as two steps and model each step with a neural network under the U-Net architecture, where the CSA layer is embedded into the encoder of refinement step. To stabilize the network training process and promote the CSA layer to learn more effective parameters, we propose a consistency loss to enforce the both the CSA layer and the corresponding layer of the CSA in decoder to be close to the VGG feature layer of a ground truth image simultaneously. The experiments on CelebA, Places2, and Paris StreetView datasets have validated the effectiveness of our proposed methods in image inpainting tasks and can obtain images with a higher quality as compared with the existing state-of-the-art approaches.

1. Introduction

Image inpainting must synthesize plausible missing regions while preserving global semantic structure and realistic texture details. The paper addresses discontinuity and semantic gaps with a two-step model centered on coherent semantic attention.

  • Image inpainting synthesizes missing or damaged regions for tasks including object removal, occlusion completion, and image restoration.
  • Traditional patch-based methods struggle with high-level semantics and locally unique patterns, limiting reconstruction of meaningful missing content.
  • Spatial-attention methods improve contextual consistency but tend to produce pixel discontinuities and semantic gaps, especially for rectangular holes.
  • The proposed CSA layer initializes unknown feature patches from similar known patches, then optimizes adjacent-patch consistency to preserve global semantics and local coherence.
  • The model uses rough and refinement networks, embeds CSA in the refinement encoder, and applies consistency, reconstruction, and adversarial losses.
  • Experiments on CelebA, Places2, and Paris StreetView report higher-quality inpainting than existing methods, including Contextual Attention, Shift-net, Partial Conv, and Gated Conv.

2. Related Works

Related work spans non-learning propagation or patch-copying methods and learning-based semantic prediction methods. The paper positions CSA as an approach for feature relationships and irregular-hole restoration.

  • The proposed architecture combines rough inpainting with refinement and places CSA at 32×32 resolution in the refinement network.
  • Non-learning approaches propagate neighboring information or copy background patches, but they cannot generate semantically meaningful content for large missing regions.
  • Learning approaches use deep networks and GANs to infer missing-region pixels, with partial and gated convolutions designed to handle irregular holes.
  • Partial and gated convolution methods do not explicitly model correlations between valid features, which can produce color inconsistency in completed images.
  • Spatial-attention methods search contextual patches or shift encoder features to restore holes, but their limitations include pixel discontinuity and semantic gaps.

3. Approach

The approach performs rough inpainting followed by refinement with a CSA layer that models correlations among generated and contextual feature patches. Consistency, reconstruction, and adversarial objectives support feature learning and final image synthesis.

  • Overall framework: The system first produces a rough prediction and then uses a refinement network with CSA to generate the final result.The refinement network receives the rough prediction conditioned on the input, while patch and feature-patch discriminators operate on the refined output.
  • Coherent Semantic Attention: CSA initializes each hole feature patch from its most similar known-region patch, then combines that reference with previously generated patches.Its search phase finds contextual matches, while its generative phase uses adjacent generated patches to preserve feature continuity.
  • Coherent Semantic Attention: The CSA generation process iteratively accumulates contextual information from earlier patches and reconstructs the hole feature map through attention-weighted patch synthesis.Attention maps encode contributions from contextual similarity and adjacent generated patches before the extracted patches are reused as deconvolutional filters.
  • Consistency loss: The method introduces consistency loss to align both the CSA output and its corresponding decoder feature with a VGG-16 feature target.The selected VGG-16 4−3 layer supplies the target feature space, and L2 distances are computed at locations in the missing region.
  • Feature Patch Discriminator: A feature patch discriminator inspects VGG-16 feature maps to discriminate completed from original images, while a 70×70 patch discriminator inspects pixel values.The feature-map discriminator captures Markovian patch statistics, and relativistic average least-squares adversarial loss is used for training stability.

4. Experiments

Experiments evaluate the method on standard datasets, centered and irregular masks, quantitative metrics, and ablations of CSA, its position, consistency loss, and feature patch discrimination. The results report higher-quality inpainting and identify a 32×32 CSA placement as a favorable efficiency–performance trade-off.

  • Experimental setup: Experiments use Places2, CelebA, and Paris StreetView, with comparisons on both centered and irregular holes.The evaluation uses original dataset splits and compares against CA, SH, PC, and GC.
  • Qualitative comparison: Our model generates visually pleasing results with more coherent textures than competing methods for centered and irregular masks.Comparisons describe distorted structures, confusing colors, blur, missing details, discontinuous lines, and artifacts in competing methods.
  • Quantitative comparisons: Our method outperforms all other methods on L1, L2, PSNR, and SSIM for both irregular and centered masks.The comparison uses 500 randomly selected CelebA validation images with generated centered and irregular holes.
  • Effect of CSA layer: Replacing CSA with conventional convolution or contextual attention produces unreasonable content or lacks fine details and background consistency.The CSA ablation supports constructing global semantic structure and local coherency.
  • CSA position: At the 3rd down-sample position with 32×32 feature maps, CSA achieves a better efficiency–performance trade-off at 0.82 seconds per image.The 2nd position performs well but takes longer, while the 4th is efficient but produces coarser details.
  • Additional ablations: Consistency loss reduces distorted hole-center structures, while the feature patch discriminator produces finer details and more reasonable structure than discriminator alternatives.The feature patch discriminator is also reported to process each image 0.2 seconds faster than the conventional feature discriminator.

5. Conclusion

The paper proposes a deep generative image-inpainting approach centered on coherent semantic attention, consistency loss, and a feature patch discriminator. Experiments verify the effectiveness of these components, while future work considers extending the method to style transfer and single-image super-resolution.

  • Conclusion: The proposed approach uses Coherent Semantic Attention to learn relationships between features in missing image regions.Consistency loss supports CSA learning and training stability, while a feature patch discriminator improves predictions.
  • Conclusion: Experiments verify the effectiveness of the proposed methods for image inpainting.The conclusion identifies consistency loss and feature patch discrimination as additional components of the approach.
  • Future work: Future work will extend the method to style transfer and single-image super-resolution.

A. Definition of Masked Region in Feature Maps

The method defines masked feature-map regions differently for centering and irregular masks before applying CSA.

  • For centering masks, the 16×16 masked feature region corresponds to a half-image mask at CSA’s 32×32 resolution.
  • For irregular masks, a unit-width convolutional network estimates feature-map mask coverage from the input masked image.Values above 5/16 are assigned to the masked region.

B. Network Architectures

The rough network follows pix2pix, while the refinement network changes channels and spatial resolution through specified convolutions.

  • The rough network uses the same architecture as pix2pix.
  • The refinement network uses 3×3 convolutions to double channels and 4×4 convolutions to halve spatial size.

C. Quantitative Comparison of Ablation Study

Ablation studies evaluate CSA, its placement, consistency loss, and feature patch discrimination using Places2 validation images and reported tables.

  • Feature patch discriminator: The feature patch discriminator architecture begins with VGG 4−3 features followed by convolutional layers.
  • Effect of CSA layer: The CSA layer outperforms a conventional 3×3 layer and contextual attention in the reported Places2 ablation.
  • Effect of CSA layer at different positions: Embedding CSA at the third down-sample position provides a better efficiency–performance tradeoff than other tested positions.
  • Effect of consistency loss: The consistency loss improves the model’s reported performance in the Places2 ablation.
  • Effect of feature patch discriminator: The feature patch discriminator performs better than the other discriminator configurations tested.

D. More Comparisons Results

Comparisons across Places2 and Paris StreetView evaluate centering and irregular masks against existing approaches. The authors report stronger structural consistency, detail richness, and local pixel continuity for their model.

  • The comparisons cover centering masks on Places2 and irregular masks on Paris StreetView, with additional results on CelebA.
  • The model outperforms state-of-the-art approaches in structural consistency and detail richness for the reported comparisons.
  • The authors attribute improved local pixel continuity to CSA modeling semantic relevance between hole features.

E. More Results on CelebA, Paris StreetView, Places2

Additional qualitative results are reported for the full model across CelebA, Paris StreetView, and Places2 using centering or irregular masks. All images are resized to 256×256 for training and evaluation in the described experiments.

  • CelebA: CelebA results include full-model examples with both centering and irregular masks.The model is trained on CelebA, with images resized to 256×256 for training and evaluation.
  • Paris StreetView: Paris StreetView results show the full model evaluated with irregular masks.Images are resized to 256×256 for both training and evaluation.
  • Attention-map comparisons: Attention-map comparisons visualize pairs of adjacent pixels and separate vertical from horizontal neighboring relationships.Across CelebA, Places2, and Paris StreetView, the first three rows represent up/down adjacent pixels, while the fourth and fifth rows represent left/right adjacent pixels.
  • Places2: Places2 results show full-model examples with centering masks for canyon and butte scene categories.The model is trained on Places2, and images are resized to 256×256 for training and evaluation.
  • Additional examples: Additional qualitative examples are provided for CelebA, Paris StreetView, and Places2 under the corresponding mask settings.The figures cover CelebA centering and irregular masks, Paris StreetView irregular masks, and Places2 centering masks.
Loading 1905.12384v3…