Source-linked AI summary
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations
Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, Chao Yang
TL;DR
Existing deep inpainting systems often recover structures and textures separately, leaving limited joint use of encoder features for complete hole filling. This paper introduces a mutual encoder-decoder that fills multi-scale structure and texture features and equalizes them before decoding. Experiments on benchmark datasets show effective recovery of both structures and textures with favorable performance against state-of-the-art approaches.
Problem
Existing methods commonly use separate encoder-decoders for structures and textures, limiting their joint consideration during image inpainting.
Method
A mutual encoder-decoder fills deep structure features and shallow texture features in separate multi-scale branches, then fuses and equalizes them through channel reweighing and bilateral propagation.
Results
The method effectively recovers structures and textures and performs favorably against state-of-the-art inpainting approaches on benchmark datasets.
Takeaways & Limitations
Joint feature equalization correlates recovered structures with textures for consistent image-content representation during inpainting.
Abstract
from arXiv · showhide
Deep encoder-decoder based CNNs have advanced image inpainting methods for hole filling. While existing methods recover structures and textures step-by-step in the hole regions, they typically use two encoder-decoders for separate recovery. The CNN features of each encoder are learned to capture either missing structures or textures without considering them as a whole. The insufficient utilization of these encoder features limit the performance of recovering both structures and textures. In this paper, we propose a mutual encoder-decoder CNN for joint recovery of both. We use CNN features from the deep and shallow layers of the encoder to represent structures and textures of an input image, respectively. The deep layer features are sent to a structure branch and the shallow layer features are sent to a texture branch. In each branch, we fill holes in multiple scales of the CNN features. The filled CNN features from both branches are concatenated and then equalized. During feature equalization, we reweigh channel attentions first and propose a bilateral propagation activation function to enable spatial equalization. To this end, the filled CNN features of structure and texture mutually benefit each other to represent image content at all feature levels. We use the equalized feature to supplement decoder features for output image generation through skip connections. Experiments on the benchmark datasets show the proposed method is effective to recover structures and textures and performs favorably against state-of-the-art approaches.
1 Introduction
Deep inpainting methods face difficulty recovering complete missing content, while separate structure and texture recovery can leave their features inconsistent. The paper proposes a mutual encoder-decoder that jointly fills and equalizes structure and texture features, with favorable benchmark performance.
- Motivation: Existing encoder-decoder methods struggle to reconstruct completely empty hole regions without sufficient image guidance.Two-stage methods address this by separately recovering structures and textures, but their intermediate outputs can remain inconsistent.
- Proposed approach: The proposed mutual encoder-decoder jointly learns structure and texture representations from deep and shallow encoder features, respectively.Deep features represent structure semantics, whereas shallow features represent texture details.
- Proposed approach: Separate structure and texture branches fill their corresponding CNN features at multiple scales before fusion and feature equalization.Each branch uses parallel streams with different kernel sizes for multi-scale hole filling.
- Proposed approach: Feature equalization reweighs channels and applies bilateral propagation to make structure and texture features consistent across spatial regions and feature levels.The equalized features are also made coherent with uncorrupted content around hole boundaries.
- Results: The method reduces blur and artifacts caused by inconsistent structure and texture features and performs favorably against state-of-the-art inpainting approaches.These outcomes are reported from experiments on benchmark datasets.
2 Related Works
Related image-inpainting work includes empirical methods based on diffusion or patch matching and deep-learning methods that use semantic or perceptual guidance. Empirical approaches are limited for large or structurally complex holes, motivating deep models.
- Empirical Image Inpainting: Diffusion-based empirical methods propagate neighboring appearances but may fail to generate meaningful structures and mainly handle small background holes.Their operations consider surrounding pixels of the missing region.
- Empirical Image Inpainting: Patch-match methods fill missing regions by transferring similar patches from remaining image areas into the holes.The passage presents patch matching as another empirical inpainting strategy.
- Deep Image Inpainting: Deep image-inpainting methods use generative adversarial networks and related semantic guidance to produce more meaningful hole-filling results.Examples include local and global discriminators, dilated convolutions, salient-edge prediction, and segmentation guidance.
3 Proposed Algorithm
The proposed mutual encoder-decoder jointly fills structure and texture features, then equalizes them across channels and spatial locations before supplementing decoder features for image generation.
- Mutual Encoder-Decoder: A six-layer encoder and five-layer decoder, connected by four dilated residual blocks, perform end-to-end image generation for hole filling.Dilated convolutions enlarge the receptive field for encoder features.
- Mutual Encoder-Decoder: Deep encoder features represent structures, while shallow features represent textures after resizing, transforming, and concatenating feature maps.The resulting structure and texture features are denoted Fst and Fte.
- Decoder and Training: Equalized features supplement decoder features through skip connections, and visualization shows progressively reduced holes before output generation.The training objective combines pixel reconstruction, perceptual, style, and relativistic average least-squares adversarial losses.
- Multi-scale Feature Filling: Two same-architecture branches fill structure and texture features independently using three parallel partial-convolution streams with different kernel sizes.The streams provide multi-scale hole filling within CNN feature space.
- Feature Equalization: Feature inconsistency between the branches can cause blur and artifacts, so their filled outputs are concatenated and fused with a 1 × 1 convolution.The fused feature is then equalized to make structure and texture representations consistent.
- Feature Equalization: Channel equalization reweights fused features using squeeze-and-excitation attention, while bilateral propagation equalizes spatial responses using global spatial and local feature similarities.The bilateral operation uses a global region for spatial distance and a 3 × 3 local region for feature-channel similarity.
4 Experiments
Experiments evaluate the method on multiple benchmark datasets, hole-mask settings, numerical metrics, and human judgments. Visual and reported evaluations indicate favorable performance against existing inpainting approaches.
- Experimental Setup: The method is evaluated on Paris StreetView, Place2, and CelebA using established dataset splits and training augmentation.Training uses flipping augmentation and dataset-specific optimization schedules.
- Experimental Setup: Comparisons cover six state-of-the-art methods and both 128×128 center holes and irregular masks with varying hole ratios.Irregular-hole comparisons use Paris StreetView and Place2 validation datasets.
- Visual Evaluations: For center holes, competing methods produce distorted structures, blurry textures, or unreasonable semantics, whereas the proposed method generates realistic textures.The comparison includes CE, CA, and other visual results shown with ground-truth images.
- Visual Evaluations: For irregular holes, competing results contain noisy patterns, missing details, distorted structures, and visible artifacts, while the proposed method produces visually pleasing contents.Irregular holes are described as more challenging than center holes.
- Quantitative and Human Evaluations: The evaluations on center and irregular holes indicate favorable performance against existing hole-filling approaches.The paper also reports numerical evaluations using SSIM, PSNR, and FID, plus human-subject judgments.
- Quantitative and Human Evaluations: A human-subject evaluation uses over 35 image experts who select the most realistic result among four methods without knowing the hole region.The evaluation covers CelebA, Place2, and Paris StreetView with 20 questions per subject.
5 Ablation Study
Ablation studies examine the structure and texture branches and the feature-equalization components. Joint branch use and feature equalization improve the visual quality of recovered contents.
- Structure and Texture Branches: Using only the texture branch preserves structure information but misses textures, while using only the structure branch does not provide the same combined content quality.The ablation compares separately trained branches after matching their output sizes.
- Feature Equalizations: Removing feature equalization produces visually unpleasant contents and visible artifacts, while equalization generates more realistic and visually pleasing contents.The equalization ablation evaluates the feature-equalization components by removing them from the pipeline.
- Structure and Texture Branches: Joint utilization of the structure and texture branches improves the recovered content quality.The study is summarized in the branch ablation figure and Paris StreetView ablation table.
- Feature Equalizations: The proposed bilateral propagation addresses local coherency and global feature-distance considerations that the Non-local block ignores.The paper reports similar favorable performance numerically in Table 5.
6 Concluding Remarks
The proposed mutual encoder-decoder correlates structure and texture features during image inpainting. Its multi-scale branches and feature equalizations recover both regular and irregular hole regions effectively against state-of-the-art approaches.
- The mutual encoder-decoder correlates filled structure and texture features during image inpainting.Deep and shallow encoder features are reorganized as structure and texture features, respectively.
- Separate texture and structure branches fill holes at multiple CNN feature scales before fusing their outputs.The branches process shallow and deep layer features and combine them through feature equalizations.
- Feature equalization first aligns channel attentions, then propagates information across spatial feature regions using bilateral propagation activation.
- Experiments show effectiveness against state-of-the-art approaches for filling both regular and irregular hole regions.