Source-linked AI summary
Free-Form Image Inpainting with Gated Convolution
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas Huang
TL;DR
Free-form image inpainting must complete arbitrary masked regions while accommodating multiple plausible outcomes and user guidance. The paper addresses this with gated convolutions and SN-PatchGAN, and reports higher-quality, more flexible results than previous methods.
Problem
Vanilla convolutions treat masked and valid pixels alike, producing artifacts, while rectangular-mask GAN designs do not fit arbitrary free-form holes.
Method
The system uses gated convolutions for learnable per-channel, per-location feature selection and SN-PatchGAN with pixel-wise reconstruction and adversarial losses.
Results
The system achieves higher-quality free-form inpainting than previous state-of-the-art methods on Places2 and CelebA-HQ, including user-guided results.
Takeaways & Limitations
User sketches guide inpainting for removing distracting objects, modifying layouts, clearing watermarks, editing faces, and creating novel objects.
Takeaways & Limitations
Partial convolution is incompatible with additional user inputs and heuristically classifies locations as valid or invalid.
Abstract
from arXiv · showhide
We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting
1. Introduction
The paper develops a free-form image inpainting system that combines gated convolutions, SN-PatchGAN, and user sketch guidance. It targets realistic, semantically correct completion across arbitrary masks and reports higher-quality results than prior methods.
- Motivation: Image inpainting synthesizes visually realistic and semantically correct content in missing image regions.It supports removing distracting objects and retouching undesired regions, with extensions to several image-editing tasks.
- Problem: Vanilla convolutions treat valid, invalid, and boundary features alike, causing color discrepancy, blurriness, and edge artifacts on free-form masks.Masked regions contain invalid or synthesized features, unlike standard vision inputs where all pixels are valid.
- Method: Gated convolution learns feature selection separately for each channel and spatial location, supporting arbitrary masks and conditional inputs such as sparse sketches.Its output multiplies learned features by sigmoid gating values and can be stacked in an encoder-decoder with contextual attention.
- Method: SN-PatchGAN applies adversarial supervision densely across discriminator output-map locations and channels for free-form masks.The training objective combines pixel-wise reconstruction and adversarial losses, while SN-PatchGAN is described as simple, fast, and stable.
- Results: The system extends inpainting with user sketches and achieves higher-quality free-form results than prior methods on Places2 and CelebA-HQ.The authors demonstrate applications including object removal, layout modification, watermark clearing, and face editing.
2. Related Work
Prior inpainting work spans patch matching and deep generative models, alongside user-guidance methods and feature-wise gating. Patch-based approaches handle stationary textures, while learned models better exploit semantics in non-stationary images.
- Image Inpainting: Patch-based algorithms extend pixels near holes by matching low-level features and work well on stationary textures but often fail on non-stationary images.Later methods reduce search cost or improve matching with local features and patch statistics.
- Image Inpainting: Deep inpainting models predict masked pixels directly and learn adaptive semantic features for faces, objects, and other complex images.Their advantage is end-to-end synthesis of visually plausible contents from large-scale learned representations.
- User Guidance: User guidance has been explored through dots, lines, structures, transformations, distortions, and image exemplars.Example-based systems can search large photo databases and transfer corresponding regions from matched images.
- Conditional Generation: Conditional generative networks use additional inputs such as color cues or semantic label maps for user-guided image processing and synthesis.These approaches extend generative models beyond unconstrained image completion.
- Feature Gating: Feature-wise gating has been applied across vision, language, and speech to re-calibrate or regulate feature responses.Examples include Highway Networks, Squeeze-and-Excitation Networks, and WaveNets.
3. Approach
The approach combines gated convolutions for dynamic feature selection with SN-PatchGAN for free-form image inpainting. It also uses automatically generated masks, an encoder-decoder network, and optional user guidance.
- Gated Convolution: Vanilla convolution applies shared filters to valid, invalid, and boundary features, producing artifacts such as color discrepancy, blurriness, and edge responses around free-form holes.
- Gated Convolution: Partial convolution improves irregular-mask inpainting but heuristically classifies locations as valid or invalid, shares one mask across channels, and cannot naturally handle user guidance.
- Gated Convolution: Gated convolution learns a dynamic feature selection mechanism for each channel and spatial location instead of using a fixed binary mask.The gating values are learned from the input and multiplied with learned features.
- SN-PatchGAN: SN-PatchGAN applies adversarial losses to dense discriminator feature elements representing different patch locations and semantics, avoiding the need for a global discriminator.Spectral normalization is added to stabilize training.
- Network and Training: The inpainting objective contains pixel-wise L1 reconstruction and SN-PatchGAN losses, while the network uses coarse and refinement encoder-decoder stages with gated convolutions.The model is trained end-to-end and supports free-form holes at arbitrary locations and different input resolutions during inference.
- Mask Generation: Free-form masks are generated on the fly by repeatedly drawing randomized lines and circles to simulate users brushing an eraser over undesired regions.
4. Results
The system is evaluated quantitatively and qualitatively on free-form inpainting, user guidance, object removal, creative editing, and SN-PatchGAN ablations. It achieves strong visual and user-rated results across these settings.
- Quantitative Results: The model runs in 0.21 seconds per 512 × 512 image on a Tesla V100 and 1.9 seconds on a Xeon CPU, regardless of hole size.The network contains 4.1M parameters.
- Quantitative Results: On Places2 validation images, learning-based methods outperform PatchMatch in mean ℓ1 and mean ℓ2 error, while PartialConv performs worse than the proposed method.The comparison covers both center-rectangle and free-form masks, with the same training framework for PartialConv* and the proposed model.
- Qualitative Comparisons: Gated convolution produces more visually pleasing results without noticeable color inconsistency and follows sparse sketches with seamless boundary transitions.The comparison includes automatic and user-guided inpainting on Places2 and CelebA-HQ validation examples.
- Object Removal and Creative Editing: The system supports object removal and creative editing, including modifying image layouts, editing faces, and following user sketches in faces and natural scenes.The object-removal study compares the method with Photoshop and a previous generative inpainting model.
- User Study: 79.4% versus 20.6%: users preferred the proposed model over the official PartialConv model in pairwise inpainting comparisons.The naturalness ratings were 9.89 for ground truth, 7.72 for the proposed model, 7.07 for re-implemented PartialConv, and 6.54 for official PartialConv.
- SN-PatchGAN Ablation: SN-PatchGAN produces significantly better inpainting results than one vanilla global discriminator and, with pixel-wise ℓ1 loss, yields photo-realistic results.The ablation also supports improved stability and performance from spectral normalization.
5. Conclusions
The paper concludes that gated convolution with SN-PatchGAN forms an effective free-form inpainting system, while randomized on-the-fly masks support varied training examples. User sketch guidance extends the system to interactive editing.
- 5. Conclusions: The proposed system combines an end-to-end generative network with gated convolution, pixel-wise ℓ1 loss, and SN-PatchGAN.The conclusion describes this combination as the basis of the free-form image inpainting system.
- 5. Conclusions: The system improves free-form inpainting with user guidance and supports removing objects, changing layouts, clearing watermarks, editing faces, and creating novel objects.These uses are presented as applications of user sketch guidance.
- 5. Conclusions: Quantitative results, qualitative comparisons, and user studies are reported as evidence for the system’s superiority over previous methods.The conclusion states this evidence at the paper level without specifying individual metrics.
- A. Free-Form Mask Generation: Randomized free-form masks are generated on-the-fly by repeatedly drawing lines with varied angles and smoothing joints with circles.The algorithm uses hyper-parameters controlling vertices, lengths, brush width, and angle, with flips for additional variation.
- A. Free-Form Mask Generation: The mask generator provides varied samples with little computational overhead and no storage requirement.CPU mask generation can be hidden behind GPU network training.
B. Sketch Generation
Sketch guidance is constructed differently for faces and natural scenes: landmarks are connected for faces, while thresholded HED edge maps are used for natural scenes. The same sketch-based guidance supports inpainting with varied sketches.
- Sketch Generation: For faces, related facial landmarks are detected and connected to create the training sketch.The procedure is described as robust and useful for face editing.
- Sketch Generation: The system demonstrates image inpainting with the same mask paired with different sketches.This illustrates that sketch guidance can vary while the masked input remains fixed.
- Sketch Generation: For natural scenes, HED extracts edge maps and values above threshold 0.6 are converted to ones.This produces a binary sketch for natural-scene guidance.
C. The Effects of Sketch Input
The inpainting network can follow user sketches for creative editing, and an additional comparison tests different sketches while keeping the input mask unchanged.
- The Effects of Sketch Input: The network follows user sketches for creative editing of image layouts, faces, and other image content.The examples include both faces and natural scenes.
- The Effects of Sketch Input: Figure 11 compares outputs produced from the same input mask with different sketches.The comparison isolates the effect of changing sketch guidance while retaining the mask.
D. Visualization and Interpretation
The section visualizes and interprets learned gating values in the inpainting network, comparing gated convolution with PartialConv.
- Figure 12 visualizes learned gating values in the inpainting network.
- The visualization provides an interpretation of how gated convolution operates across the inpainting network.
- The study compares the learned gating values of gated convolution with those of PartialConv.
E. Ablation Study of SN-PatchGAN
The ablation study examines SN-PatchGAN for free-form masks, where rectangular-mask global and local GAN designs are not applicable. A global GAN without spectral normalization performs substantially worse.
- SN-PatchGAN is studied because free-form masks may occur anywhere in images with arbitrary shapes.
- Global and local GANs designed for a single rectangular mask are not applicable to free-form image inpainting.
- A global GAN without spectral normalization has significantly worse performance across all examples than SN-PatchGAN.
F. More Comparison Results
The comparison results evaluate the proposed gated-convolution system against established learning-based inpainting methods. Compared with PartialConv, the proposed system produces higher-quality results, while PartialConv retains visible artifacts and struggles with difficult structures.
- The evaluation compares Global&Local, ContextAttention, PartialConv, and the proposed gated-convolution method on separate scene and face models.Testing images are not in the training set.
- The proposed inpainting system generates higher-quality results than the PartialConv baseline.
- PartialConv produces observable color inconsistency or shadows and struggles with large holes spanning segment transitions or strong structural edges.
G. More Inpainting Results of Our System
The paper presents additional natural-scene and face results from the proposed free-form inpainting system. These examples illustrate object removal, layout modification, face editing, and interactive creation of novel objects.
- More Inpainting Results of Our System: Additional results cover both natural scenes and faces in Figures 16, 17, and 18.
- More Inpainting Results of Our System: The system helps users quickly remove distracting objects, modify image layouts, and edit faces.
- More Inpainting Results of Our System: The system also supports interactive creation of novel objects in images.
- Visualization and Interpretation: Figure 12 compares gated and partial convolution through visualizations and interpretations of learned gating values.
- Ablation Study of SN-PatchGAN: Figure 13 presents an SN-PatchGAN ablation comparing a global GAN with the proposed patch-based approach on free-form masks.
- More Comparison Results: Figures 14 and 15 provide additional comparison results for natural scenes and faces, respectively.
- More Inpainting Results of Our System: Figures 16 and 17 show further free-form inpainting results on natural images.
- More Inpainting Results of Our System: Figure 18 shows further free-form inpainting results on faces.