Source-linked AI summary
Image Inpainting for Irregular Holes Using Partial Convolutions
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro
TL;DR
Image inpainting methods can produce artifacts because standard convolutions use placeholder values inside holes, while prior work often centers on rectangular masks. This paper uses masked, renormalized partial convolutions with automatic mask updates, achieving state-of-the-art results for irregular holes. The method is robust across hole shapes, sizes, locations, and border distances, though it struggles with some sparse structures and the largest holes.
Problem
Existing inpainting methods can condition predictions on placeholder hole values and often focus on rectangular holes, motivating robust evaluation and modeling for irregular masks.
Method
The paper replaces standard convolutions with masked, renormalized partial convolutions and automatically updates the mask during the forward pass.
Results
The model achieves state-of-the-art inpainting results and handles irregular holes of varying shapes, sizes, locations, and distances from image borders.
Takeaways & Limitations
Partial convolutions make the model agnostic to placeholder hole values while supporting robust irregular-hole inpainting without additional post-processing.
Takeaways & Limitations
The method fails on some sparsely structured images and, like most methods, struggles on the largest holes.
Abstract
from arXiv · showhide
Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.
1 Introduction
The paper targets image inpainting for irregular holes, addressing artifacts from conditioning on placeholder values and limited evaluation of irregular masks. It proposes partial convolutions with automatic mask updates to produce robust, semantically meaningful results without post-processing.
- Motivation: Deep inpainting methods condition convolutional filters on placeholder hole values, causing texture loss, color contrasts, and artificial edges.These artifacts can require expensive post-processing.
- Motivation: Prior methods often focus on centered rectangular holes, limiting evaluation and potentially overfitting models to that setting.The paper evaluates irregular masks of varying sizes and whether holes touch image borders.
- Method: A partial convolution masks and renormalizes each convolution so outputs depend only on non-hole regions.The layer also updates the mask whenever at least one valid input contributes, progressively shrinking masked regions.
- Method: Automatic mask updates make the model agnostic to placeholder hole values and support handling irregular masks through successive layers.The approach is designed to leave valid responses throughout the feature map as holes shrink.
- Results: Substituting partial convolutions and mask updates for typical convolutions in a U-Net achieves state-of-the-art inpainting results.The paper identifies this substitution as a step toward state-of-the-art performance and reports efficacy on irregularly shaped holes.
- Contributions: The authors introduce a large irregular-mask dataset intended to facilitate future training and evaluation of inpainting models.The dataset is proposed specifically for irregularly shaped holes.
2 Related Work
Prior inpainting methods use neighboring appearance statistics, latent representations, or generative networks, while masked or reweighted convolutions provide a route to condition outputs on valid inputs. The paper applies stacked partial convolutions and mask updates to image inpainting.
- Non-learning approaches: Non-learning methods propagate neighboring appearance information but generally handle only narrow holes with low color and texture variance.Large holes can produce oversmoothing or artifacts resembling Voronoi regions.
- Non-learning approaches: Patch-based methods search non-hole regions or other sources for relevant patches, relying on image statistics rather than learned semantic priors.The supplied passage introduces these methods as a related non-learning approach.
- Deep learning approaches: Deep learning methods commonly initialize holes with constant placeholder values before passing images through convolutional networks, often requiring post-processing for resulting artifacts.The ImageNet mean is given as an example placeholder.
- Deep learning approaches: Other approaches avoid direct placeholder conditioning through latent-space search or network-prior optimization, but may require per-image hyperparameters or multiple iterations.These methods differ from single feedforward processing.
- Masked convolutions: Masked or reweighted convolutions condition outputs on valid inputs and have been used in segmentation, autoregressive generation, and depth completion.The paper adapts this operation for inpainting with mask updating.
3 Approach
The approach replaces standard convolutions with stacked partial convolutional layers that use valid pixels and update masks during the forward pass. A U-Net-like architecture combines this mechanism with reconstruction, perceptual, style, and smoothing losses for inpainting.
- 3.1 Partial Convolutional Layer: Partial convolution layers mask and renormalize convolution responses so outputs depend only on valid inputs.The mask is updated after each operation when at least one valid input contributes.
- 3.1 Partial Convolutional Layer: Automatic mask updates mark locations as valid when their convolution used at least one valid input, and repeated layers can eventually fill the mask.The update is implemented as part of the forward pass.
- Network Design: The model uses a U-Net-like architecture that replaces convolutional layers with partial convolutional layers and propagates feature maps and masks through skip links.The final layer receives the original masked image and mask, allowing the model to copy non-hole pixels.
- 3.3 Loss Functions: The training objective combines valid- and hole-region L1 losses with perceptual, style, and total-variation terms.The losses target pixel reconstruction and smooth transitions between predicted holes and surrounding context.
- 3.3 Loss Functions: Perceptual loss compares raw and composited outputs with ground truth in higher-level VGG-16 feature spaces, while style loss applies Gram-matrix autocorrelation before L1 comparison.The perceptual loss uses the pool1, pool2, and pool3 layers; the total-variation term smooths a one-pixel dilation around the hole.
- 3.3 Loss Functions: The ablation study reports that style loss is necessary for plausible results, whereas small style-loss weights can fail to produce plausible outputs.Figure 3 compares removing style loss, reducing its weight, and removing perceptual loss against the full Ltotal model and ground truth.
4 Experiments
The experiments evaluate irregular-mask construction, comparisons across datasets and methods, and human judgments under varied hole sizes and border conditions. PConv outperforms competing approaches on reported quantitative and user-study evaluations.
- Irregular Mask Dataset: Masks are generated from arbitrary streaks and holes, then augmented through dilation, rotation, and cropping.The training and testing collections contain 55,116 and 24,866 raw masks, respectively.
- Irregular Mask Dataset: The test set contains 12,000 masks spanning six hole-to-image area ratios, with and without border constraints.Each category contains 1,000 masks for each border condition.
- Comparisons: The study compares PConv with PatchMatch, GL, GntIpt, and a typical-convolution baseline across ImageNet, Places2, and CelebA-HQ.Released pretrained models are used for GL and GntIpt, while PatchMatch uses a third-party implementation.
- Comparisons: Qualitative comparisons show that competing methods can copy semantically incorrect patches or produce artifacts after post-processing, whereas Conv exhibits artifacts from hole placeholder values.The comparisons include ImageNet, Places2, and CelebA-HQ results.
- Comparisons: PConv outperforms all other methods on the reported ℓ1 error, PSNR, SSIM, and inception-score measurements for irregular masks.ℓ1 error, PSNR, and SSIM are reported on Places2; inception score is reported on ImageNet.
- User Study: In human studies, PConv performs significantly better in unlimited-time comparisons and outperforms other methods in most limited-time conditions.Both experiments separately consider holes near image borders and holes at least 50 pixels from the border.
5 Discussion & Extension
The discussion reports robust behavior across hole shapes, sizes, locations, and border distances, while extending the framework to image super resolution. The authors also identify failures on sparsely structured content and the largest holes.
- Discussion: PConv robustly handles holes of any shape, size, location, or distance from image borders.The authors report that performance does not deteriorate catastrophically as holes increase in size.
- Discussion: The method fails on some sparsely structured images and, like most methods, struggles on the largest holes.Bars on a door are given as an example of a sparsely structured failure case.
- Super Resolution Extension: The framework is extended to super resolution by offsetting low-resolution pixels into a larger image and inserting holes elsewhere.For scale factor K, the constructed input has height K*H and width K*W, with source pixels placed at offset coordinates.
- Super Resolution Extension: The super-resolution extension is compared with SRGAN and MDSR+ at an up-scaling factor of K=4.Example input and output are shown for K=4.
Details of Network Architecture
The network uses partial convolutional layers throughout a U-Net-like architecture, with mask-aware skip connections and staged encoder and decoder blocks.
- Network Architecture: PConv1-8 form the encoder stage and PConv9-16 form the decoder stage.The table specifies each layer’s filter size, stride, filter count, batch normalization, nonlinearity, and skip-link configuration.
- Network Architecture: Skip links concatenate the previous nearest-neighbor upsampled feature and mask maps for the next partial convolution layer.The architecture therefore passes both feature and mask information through skip connections.
More Comparisons on Irregular Masks
Additional irregular-mask comparisons use a shared notation across figures and place the PConv results alongside competing methods and ground truth.
- More Comparisons on Irregular Masks: The comparison layout presents input, PatchMatch, GL, GntIpt, PConv, and ground-truth images for visual inspection.The listed ordering identifies the methods and reference image used in the comparison.
- More Comparisons on Irregular Masks: The irregular-mask comparison uses the same abbreviations as Figures 5 and 6.The notation includes the methods and ground truth listed as PM, GL, GntIpt, PConv, and GT.
More Comparisons on Regular Masks
The regular-mask comparison includes PConv alongside other methods, with ground truth shown as the reference.
- The comparison lists Input, PM, GL, GntIpt, PConv, and GT results.
- Figure 15 reports comparisons on regular masks using the paper’s established notation abbreviations.
More Comparisons On Image Super Resolution
The image super-resolution comparison presents Bicubic, SRGAN, MDSR+, PConv, and ground-truth results.
- The comparison includes Bicubic, SRGAN, MDSR+, PConv, and GT results.