PaperlayerView exact revision on arXiv

Source-linked AI summary

Shift-Net: Image Inpainting via Deep Feature Rearrangement

Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, Shiguang Shan

arXiv:1801.09392v2cs.CV

TL;DR

Image inpainting must produce semantically plausible completions without sacrificing sharp structures and fine textures, a trade-off that can leave CNN-based results blurry. Shift-Net addresses this by adding a guided shift-connection layer to U-Net, shifting known-region encoder features into missing regions and training end-to-end; experiments on Paris StreetView and Places report fast, sharp, fine-detailed, and visually plausible results.

  • Problem

    Existing CNN inpainting methods can produce semantically plausible but blurry completions because missing parts are predicted from surrounding convolutional features.

  • Method

    Shift-Net adds a shift-connection layer to U-Net, shifts known-region encoder features into missing regions, and uses guidance, reconstruction, and adversarial losses for end-to-end training.

  • Results

    Experiments on Paris StreetView and Places report favorable results with sharp, fine-detailed, visually plausible, and photo-realistic inpainting, including about 80 ms processing for a 256 × 256 image.

  • Takeaways & Limitations

    Combining CNN-based semantic prediction with exemplar-like feature rearrangement provides a fast inpainting approach that preserves detailed textures and plausible structure.

Abstract

from arXiv · show

Deep convolutional networks (CNNs) have exhibited their potential in image inpainting for producing plausible results. However, in most existing methods, e.g., context encoder, the missing parts are predicted by propagating the surrounding convolutional features through a fully connected layer, which intends to produce semantically plausible but blurry result. In this paper, we introduce a special shift-connection layer to the U-Net architecture, namely Shift-Net, for filling in missing regions of any shape with sharp structures and fine-detailed textures. To this end, the encoder feature of the known region is shifted to serve as an estimation of the missing parts. A guidance loss is introduced on decoder feature to minimize the distance between the decoder feature after fully connected layer and the ground-truth encoder feature of the missing parts. With such constraint, the decoder feature in missing region can be used to guide the shift of encoder feature in known region. An end-to-end learning algorithm is further developed to train the Shift-Net. Experiments on the Paris StreetView and Places datasets demonstrate the efficiency and effectiveness of our Shift-Net in producing sharper, fine-detailed, and visually plausible results. The codes and pre-trained models are available at https://github.com/Zhaoyi-Yan/Shift-Net.

1 Introduction

Image inpainting must recover both global semantic structure and fine textures, yet existing exemplar-based and CNN-based methods each address only part of this challenge. Shift-Net combines their strengths through a shift-connection layer in U-Net, guided by decoder features and trained end-to-end.

  • Motivation: Image inpainting seeks plausible completions for missing regions, including object removal, damage repair, and occlusion completion.The task has applications in removing distracting scene elements and filling the resulting gaps.
  • Challenge: Existing methods must balance global semantic structure with fine detailed textures, but exemplar-based and CNN-based approaches have complementary weaknesses.Exemplar methods synthesize high-frequency textures but may miss global structure, while CNNs produce plausible semantics but can yield blurry results.
  • Approach: Shift-Net adds a special shift-connection layer to U-Net, shifting known-region encoder features to estimate missing parts under decoder-feature guidance.The design combines CNN-based prediction of structure and semantics with exemplar-like feature replication from known regions.
  • Learning: Guidance, reconstruction, and adversarial losses train Shift-Net end-to-end while encouraging accurate reconstruction and detailed textures.The guidance loss brings decoder features closer to ground-truth encoder features, while ℓ1 and adversarial losses support reconstruction and texture restoration.
  • Reported outcome: Shift-Net is reported to achieve state-of-the-art results and generate fine-detailed, visually plausible inpainting results.The paper evaluates the method on Paris StreetView and Places and compares it with Content-Aware Fill, context encoder, and MNPS.

2 Related Work

Prior inpainting research includes exemplar-based patch synthesis, CNN-based prediction, and style-transfer formulations. These approaches respectively emphasize texture copying, semantic structure, or feature-based content and texture transfer, with limitations in structure preservation or efficiency.

  • Exemplar-based inpainting: Exemplar-based inpainting fills missing regions by searching and copying matching patches from known areas.PatchMatch accelerates patch search, but exemplar-based methods are generally stronger at textures than at preserving edges and structures.
  • CNN-based inpainting: CNN-based inpainting predicts missing parts from surrounding context and can capture semantics and global structure in a single forward pass.The context encoder uses an encoder-decoder network and adversarial loss, but its results can lack fine detail.
  • Style transfer: Style-transfer formulations estimate and transfer both content and texture from known regions using CNN-based optimization or local feature matching.Related methods use Gram-matrix objectives, MRF regularization, or convolution-layer matching followed by image reconstruction.

3 Method

Shift-Net extends U-Net with guidance and feature-shift operations to combine CNN-based semantic recovery with exemplar-like rearrangement of known-region features. Its objective jointly uses reconstruction, guidance, and adversarial losses, while the entire model remains end-to-end trainable.

  • Guidance loss on decoder feature: The guidance loss makes missing-region decoder features approximate ground-truth encoder features, while known-region encoder features already approximate their ground truth.Concatenating encoder and decoder features therefore supplies an approximation of the missing-region ground-truth encoder information.
  • Guidance loss on decoder feature: Feature visualization shows that guidance produces a reasonable semantic and structural estimate, but the decoder feature remains blurry compared with the ground truth.This motivates using the decoder feature to guide feature rearrangement for better fine-texture recovery.
  • Shift operation and Shift-Net: Shift-Net adds a shift-connection layer to U-Net, using decoder features to guide rearrangement of encoder features from known regions.The shifted encoder feature provides an updated estimate for missing-region encoder features before subsequent decoding.
  • Shift operation and Shift-Net: Unlike exemplar-based pixel or patch copying, the shift operation acts in a learned deep encoder-feature domain and computes all shift vectors in parallel.The shift-connection layer and subsequent network parameters are learned from training data in an end-to-end manner.
  • Model objective and learning: The model combines guidance, ℓ1 reconstruction, and adversarial losses to reconstruct missing regions and restore detailed, photo-realistic textures.The overall objective weights guidance and adversarial terms with λ_g and λ_adv, and training uses back-propagation with an adversarially trained discriminator.

4 Experiments

Experiments on Paris StreetView, Places, random masks, and real images evaluate Shift-Net against established inpainting methods. The results report favorable visual quality, quantitative performance, efficiency, and generalization across completion settings.

  • Comparisons with state-of-the-arts: On Paris StreetView, Shift-Net generally produces more visually pleasing results than Content-Aware Fill, context encoder, and MNPS.Content-Aware Fill recovers low-level textures, context encoder captures semantics but appears blurry, and MNPS generally performs better than both while using a multistage scheme.
  • Comparisons with state-of-the-arts: Shift-Net achieves the best numerical performance on Paris StreetView for PSNR, SSIM, and mean ℓ2 loss.The paper attributes this result to combining CNN-based and exemplar-based inpainting with end-to-end training.
  • Random mask completion: For random region completion, Shift-Net is especially effective in structural regions, producing context coherent with global content and structures.Both Shift-Net and Content-Aware Fill perform favorably on textured and smooth regions.
  • Inpainting of real world images: A Shift-Net trained on Paris StreetView generalizes to real images for central-region inpainting and object removal.Central-region results use a central mask, while object removal uses a model trained with random masks.

5 Ablative Studies

Ablations examine guidance loss, shift placement, shifted features, and shift selection. They show that guidance and correctly selected shifted features improve structure and detail, while design choices create quality–efficiency trade-offs.

  • 5.1 Effect of guidance loss: Guidance loss helps both U-Net and Shift-Net suppress artifacts and preserve salient structure.The comparison trains each architecture with and without guidance loss.
  • 5.1 Effect of guidance loss: The guidance-loss weight λg must be balanced: λg = 0.001 can leave artifacts, whereas λg ≥ 0.1 makes the constraint too excessive.The paper states that both overly small and overly large values may harm inpainting results.
  • 5.2 Effect of shift operation at different layers: Shift operation improves semantic preservation and detailed-texture recovery compared with U-Net using guidance loss.The paper uses this comparison to support the effectiveness of adding shift operation.
  • 5.2 Effect of shift operation at different layers: Placing shift-connection at the (L −2)-th layer produces visually pleasing results but requires about 400 ms per image.Earlier placement increases feature-map size and computation, while later placement may lose detailed encoder information.
  • 5.3 Effect of the shifted feature: Removing the decoder feature prevents central structures, while removing the encoder feature preserves general structure but lowers final quality.These zeroed-slice experiments indicate that the decoder subnet constructs main content and guidance loss models the decoder–encoder relationship.
  • 5.3 Effect of the shifted feature: Discarding the shifted feature produces a mixture of structures, indicating that it refines and enhances clear, fine details.Replacing nearest-neighbor searching with random shift-connection also produces more artifacts, distortions, and structure disconnections.

6 Conclusion

The paper concludes that Shift-Net combines fast image completion with fine details through deep feature rearrangement. Its guidance loss supports efficient shift operation and favorable inpainting quality.

  • 6 Conclusion: Shift-Net provides fast image completion with promising fine details through deep feature rearrangement.The architecture uses guidance loss to strengthen the relation between encoded known-region features and decoded missing-region features.
  • 6 Conclusion: Experiments show favorable comparison with state-of-the-art methods and effectiveness in generating sharp, fine-detailed, photo-realistic images.The paper identifies faster nearest searching, multiple shift-connection layers, and other low-level vision tasks as future directions.

A Definition of masked region in feature maps

The method defines masked feature regions by propagating an input mask through a simplified encoder and thresholding the resulting feature map. The chosen threshold is robust across the tested values.

  • A Definition of masked region in feature maps: A simplified width-one CNN propagates the input mask to obtain the l-th layer feature map used to define masked regions.The filters contain 1/16 values and nonlinearities are removed.
  • A Definition of masked region in feature maps: The masked feature region is defined as Ωl = {y|(Ψl(M))y ≥ T}, with 0 ≤ T ≤ 1.The input mask has value 1 inside the missing region and 0 elsewhere.
  • A Definition of masked region in feature maps: Shift-Net is robust to T = 4/16, 5/16, and 6/16, and the experiments use T = 5/16.The paper attributes this robustness to supplying shift, encoder, and decoder features to the next U-Net layer.

B.1 Architecture of generative model G.

The generative model G uses a U-Net architecture with instance normalization throughout most convolutional and deconvolutional layers, plus guidance loss and shift operation near the decoder.

  • G adopts a U-Net architecture with convolutional encoder and deconvolutional decoder layers.
  • Guidance loss and shift operation are added in the (L − 3)-th layer, producing concatenated features for the adjacent deconvolution.
  • Instance normalization follows each convolution or deconvolution layer except at the bottleneck.The bottleneck is excluded because its 1 × 1 activation map would be zeroed out by InstanceNorm with batchsize 1.
  • The encoder uses Convolution-InstanceNorm-LeakyReLU layers, while the decoder uses seven Deconvolution-InstanceNorm-ReLU layers.

B.2 Architecture of discriminative network D.

The discriminative network D is a five-convolution-layer network that reduces inputs to 30 × 30 and ends with sigmoid activation.

  • D uses five convolutional layers with 4 × 4 filters and varying strides to reduce spatial dimensions to 30 × 30.
  • The final output uses sigmoid activation, while earlier layers use leaky ReLU with slope 0.2.
  • InstanceNorm is omitted from the first convolutional layer.

C.1 Comparisons on Paris StreetView and Places datasets

Shift-Net is compared with four prior approaches on Paris StreetView and Places, with additional object-removal tests on real-world images. The authors report stronger structural consistency and detail richness, including for complex backgrounds.

  • Shift-Net is compared with Content-Aware Fill, context encoder, pix2pix, and MNPS on Paris StreetView and Places.The qualitative comparison figures arrange input and four baselines alongside the proposed method at 256 × 256 resolution.
  • Shift-Net outperforms the compared approaches in both structural consistency and detail richness.The authors report that competing methods either generate unclear details or lack global structural consistency.
  • The model preserves both global structure and fine details in the reported comparisons.
  • On real-world object-removal images with large distractor areas and complicated backgrounds, the model is reported to handle the cases well.The authors present this as evidence of effectiveness, applicability, and generality.
Loading 1801.09392v2…