Source-linked AI summary
Image Inpainting with Learnable Bidirectional Attention Maps
Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, Errui Ding
TL;DR
CNN-based inpainting methods struggle with irregular holes, while partial convolution relies on handcrafted feature re-normalization and forward mask-updating. LBAM learns bidirectional attention maps end to end, and experiments report sharper, more coherent, and visually plausible results than state-of-the-art methods.
Problem
CNN-based methods indistinguishably treat valid pixels and holes, while partial convolution uses handcrafted feature re-normalization and only forward mask-updating, limiting irregular-hole handling.
Method
LBAM learns feature re-normalization and mask-updating through attention maps, adding reverse attention maps so the U-Net decoder focuses on filling holes.
Results
Experiments on Paris StreetView and Places show that LBAM performs favorably against state-of-the-art methods, generating sharper, more coherent, and visually plausible inpainting results.
Takeaways & Limitations
Learnable bidirectional attention maps support irregular-hole adaptation and enable adversarial-loss training for improved visual quality.
Takeaways & Limitations
LBAM struggles with high-frequency details when damaged areas are too large or background objects are too complex, and may not recover objects whose shape is largely masked.
Abstract
from arXiv · showhide
Most convolutional network (CNN)-based inpainting methods adopt standard convolution to indistinguishably treat valid pixels and holes, making them limited in handling irregular holes and more likely to generate inpainting results with color discrepancy and blurriness. Partial convolution has been suggested to address this issue, but it adopts handcrafted feature re-normalization, and only considers forward mask-updating. In this paper, we present a learnable attention map module for learning feature renormalization and mask-updating in an end-to-end manner, which is effective in adapting to irregular holes and propagation of convolution layers. Furthermore, learnable reverse attention maps are introduced to allow the decoder of U-Net to concentrate on filling in irregular holes instead of reconstructing both holes and known regions, resulting in our learnable bidirectional attention maps. Qualitative and quantitative experiments show that our method performs favorably against state-of-the-arts in generating sharper, more coherent and visually plausible inpainting results. The source code and pre-trained models will be available.
1. Introduction
Image inpainting must fill complex, irregular holes with plausible content, yet existing convolutional and exemplar-based methods remain limited. The paper introduces learnable bidirectional attention maps and reports sharper, more coherent results than prior methods.
- Motivation: Image inpainting fills holes with plausible content coherent with surrounding known regions, but irregular patterns create substantial difficulty.Applications include distracting-object removal and occluded-region completion.
- Prior limitations: Exemplar-based and hybrid methods can struggle with complex structures, computational cost, or irregular holes despite using known-region information.Hybrid methods are better at rectangular holes and perform poorly on irregular holes.
- Motivation: Standard convolution treats valid pixels and holes indistinguishably, limiting irregular-hole handling and increasing color discrepancy and blurriness.This limitation affects many CNN-based inpainting methods.
- Prior limitations: Partial convolution improves irregular-hole handling but relies on hard 0-1 masks, handcrafted feature re-normalization, and forward-only mask updating.It absolutely trusts all filling-in intermediate features.
- Method: The proposed learnable attention module jointly learns feature re-normalization and mask updating end to end, adapting to irregular holes and convolutional propagation.The module is used for feature re-normalization on both U-Net encoder and decoder.
- Method: Learnable reverse attention maps let the U-Net decoder focus on filling holes rather than reconstructing both holes and known regions.Together with forward maps, they form learnable bidirectional attention maps and empirically facilitate adversarial-loss training.
- Results: Experiments on Paris StreetView and Places show favorable performance against state-of-the-art methods, producing sharper, more coherent, and visually plausible results.The method is reported to hallucinate cleaner semantic structures and more realistic textures than PConv in qualitative comparisons.
2. Related Work
Related work spans exemplar-based propagation, deep CNN architectures, multi-stage models, and mask-guided methods. The paper positions LBAM against limitations of prior approaches for irregular holes and visual artifacts.
- Exemplar-based methods: Exemplar-based methods progressively search and paste patches from known regions, with results depending strongly on propagation order and patch priority.Patch priority combines confidence and data terms to guide processing order.
- CNN-based methods: Early CNN inpainting methods targeted small, thin holes, while later encoder-decoder models addressed larger holes using reconstruction and adversarial losses.Context-encoders were used to recover semantic structures.
- CNN-based methods: Deep CNN work added global and local discriminators, multi-column architectures, and diversified reconstruction objectives to recover plausible structures and realistic details.These approaches include Global&Local and confidence-driven reconstruction with ID-MRF.
- Multi-stage methods: Multi-stage methods ease training by progressively generating content, predicting edges before images, or exploiting dependencies across phases.Examples include progressive generative networks and EdgeConnect.
- Hybrid methods: Hybrid exemplar-CNN methods refine results using known-region features, holistic content constraints, or local texture constraints, but can incur high computational cost.Contextual attention and patch-swap were developed to reduce the cost of multi-scale neural patch synthesis.
- Mask-guided methods: PConv combines mask convolution, feature re-normalization, and mask updating to improve irregular-hole handling.Its limitations include handcrafted re-normalization and forward-only mask updating, which remain associated with color discrepancy and blurriness.
3. Proposed Method
The method replaces PConv’s handcrafted, hard-mask processing with learnable attention maps and extends them bidirectionally across U-Net encoder and decoder features. This design targets irregular-hole inpainting by adapting feature renormalization, mask updating, and decoder focus end to end.
- Revisiting Partial Convolution: PConv combines mask convolution, feature re-normalization, and mask updating, but relies on handcrafted filters and activation functions.Its mask-updating function is nondifferential, complicating end-to-end learning and adversarial training.
- Learnable Attention Maps: The attention map multiplies convolution features element-wise, providing a flexible, end-to-end trainable alternative to PConv’s handcrafted re-normalization.This formulation is described as effective for adapting to irregular holes and convolution-layer propagation.
- Learnable Attention Maps: The learnable attention map replaces PConv’s fixed mask filter with layer-wise learnable filters and learns feature re-normalization and mask updating.The updated-mask activation uses α = 0.8, while the attention map uses an asymmetric Gaussian-shaped activation with learnable parameters.
- Learnable Bidirectional Attention Maps: Learnable reverse attention maps process decoder features from the hole mask and propagate updated masks toward earlier decoder layers.Unlike forward maps, reverse maps consider encoder and decoder features and apply updates to the former decoder layer.
- Learnable Bidirectional Attention Maps: Forward and reverse maps form LBAM, letting the decoder focus on filling irregular holes while supporting adversarial-loss training.Forward maps start from the valid-pixel mask, whereas reverse maps start from its complement; six encoder and six decoder layers use these maps.
- Loss Functions: LBAM combines pixel reconstruction, perceptual, style, and adversarial losses to recover texture details and semantics.The output is defined as Iout = Φ(Iin, Min; Θ), with Θ denoting learned model parameters.
4. Experiments
Experiments evaluate LBAM on Paris StreetView, Places, real-world object removal, feature and mask visualizations, ablations, and quantitative comparisons. LBAM performs favorably against competing methods, particularly for irregular holes and larger mask ratios.
- Experimental setup: LBAM is evaluated on Paris StreetView and Places using resized and randomly cropped 256×256 images, augmented during training with flipping.The experiments use random-shape masks and masks from prior work for training and testing.
- Comparison with state-of-the-arts: LBAM produces finer-detailed and more realistic textures than Global&Local, PatchMatch, Context Attention, and PConv in qualitative comparisons.The competing methods are described as limited by meaningless textures, poor complex-structure recovery, blur, artifacts, or over-smoothing.
- Comparison with state-of-the-arts: LBAM performs favorably in PSNR, SSIM, and mean ℓ1 loss on Places, especially when the mask ratio exceeds 0.3.The comparison covers mask-ratio intervals from (0.1, 0.2] through (0.4, 0.5].
- User study: LBAM achieves a 63.2% chance of winning the user study, compared with 15.2% for PConv and lower percentages for the other methods.Thirty-three volunteers assessed coherency, semantic structure, and fine details across 30 images.
- Real-world object removal: Real-world object-removal results show that LBAM produces realistic and coherent contents using both global semantics and local textures.The object area is masked with either contour-shaped or rectangular bounding boxes.
- Ablation studies: Forward masks gradually reduce encoder holes, while reverse masks gradually reduce known regions across decoder layers.The feature visualizations and mask updates compare unlearned, forward-only, and full LBAM variants.
- Ablation studies: Adversarial loss generally benefits visual quality, although removing it improves PSNR and SSIM in the reported ablation.The full model is described as better at fine-detailed structures and capturing global semantics.
5. Conclusion
The paper proposes learnable bidirectional attention maps for image inpainting, combining learnable attention and reverse attention mechanisms. Experiments report sharper, more coherent, and fine-detailed results than state-of-the-art methods.
- Conclusion: LBAM learns attention maps to adapt feature renormalization and mask updating to irregular holes and convolutional propagation.Reverse attention maps additionally let the U-Net decoder focus on filling holes.
- Conclusion: Experiments show that LBAM performs favorably against state-of-the-art methods in generating sharper, more coherent, and fine-detailed results.
Supplementary Material
Supplementary material details LBAM’s architecture, activation and mask-updating functions, visual comparisons, datasets, discriminator, and failure cases. It also reports qualitative performance on benchmark and real-world object-removal images.
- Architecture and functions: LBAM uses an asymmetric Gaussian-shaped activation for attention maps and a modified ReLU-based function for mask updating.These functions are applied within the bidirectional attention-map implementation.
- Architecture and functions: The model takes a damaged image, the input mask, and its reverse mask, using a 14-layer U-Net with seven encoder and seven decoder layers.Features are normalized through element-wise products with learnable bidirectional attention maps.
- Discriminator: The discriminator uses two parallel columns to process remained regions and missing holes for adversarial training.The parallel features merge after the sixth layer.
- Qualitative comparisons: On Paris StreetView and Places, LBAM is reported to produce better global consistency and fine-detailed structures than the compared methods.The qualitative comparisons include PatchMatch, Global&Local, Context Attention, and PConv.
- Ablation studies: Ablation visualizations compare the full model with unlearned, forward-only, and no-adversarial-loss variants, as well as alternative activations and mask-update filters.The full model benefits from adversarial loss for fine-detailed structures and global semantics.
- Real-world object removal: On real-world object-removal images, LBAM handles varied objects, backgrounds, contexts, shapes, and some large masked regions.These results are presented as evidence of practicability and generalization ability.
- Failure cases: LBAM struggles with high-frequency details when damaged areas are too large or background objects are too complex.Recovering an object’s original shape remains difficult when a large portion of that object is masked.