Source-linked AI summary
Region Normalization for Image Inpainting
Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu
TL;DR
Existing inpainting networks apply full-spatial feature normalization without accounting for corrupted regions, producing mean and variance shifts that limit training. The paper proposes Region Normalization, with mask-based and learnable variants for separate regional normalization, and reports improved performance over existing methods and across other inpainting networks.
Problem
Most image inpainting methods ignore how corrupted regions affect feature normalization, although full-spatial normalization can cause mean and variance shifts that limit network training.
Method
Region Normalization divides spatial pixels into regions and computes regional statistics, using Basic RN early and Learnable RN later to normalize corrupted and uncorrupted regions separately.
Results
RN enables a simple backbone to surpass current state-of-the-art image inpainting methods and yields consistent performance improvements when generalized to other inpainting networks.
Takeaways & Limitations
RN is presented as a plug-and-play normalization module that can be conveniently applied to other image inpainting networks.
Takeaways & Limitations
The paper leaves applying RN to other supervised vision tasks such as classification and detection for future work.
Abstract
from arXiv · showhide
Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e.g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-of-the-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements. Our code is available at https://github.com/geekyutao/RN.
1 Introduction
Image inpainting must reconstruct corrupted regions, but existing normalization can ignore their distinct statistics. The paper proposes Region Normalization to normalize corrupted and uncorrupted regions separately and reports improved inpainting performance.
- Image inpainting reconstructs corrupted regions with visually plausible content for applications including object removal, face editing, and image disocclusion.
- Most learning-based methods use advanced networks but treat corrupted and uncorrupted regions as a general spatially consistent image.
- Region Normalization divides spatial pixels into regions according to the input mask and computes separate means and variances for normalization.
- Basic RN normalizes corrupted and uncorrupted regions separately in early layers using the original inpainting mask, while Learnable RN detects potentially corrupted regions in later layers.
- Experiments on Places2 and CelebA show that RN enables a simple backbone to surpass current state-of-the-art methods and improves other inpainting networks consistently.
- The work presents RN as a spatially region-wise normalization method and proposes two RN variants for image inpainting.
2 Related Work
Image inpainting methods include traditional approaches that propagate or copy visual information and learning-based approaches that extract semantic information from data. The paper identifies normalization as an overlooked issue in existing inpainting methods.
- Traditional methods use diffusion to propagate neighboring information or patch-based methods to copy similar patches into corrupted regions.
- Learning-based methods extract semantic information through massive data training and directly map corrupted images to completed images.
- Existing learning-based models include GAN-based methods, partial convolution, gated convolution, and edge-based completion pipelines.
- Most existing inpainting methods ignore the impact of corrupted regions on normalization, despite normalization being important for network training.
- Feature normalization is widely applied to help train deep neural networks; BN and IN normalize activations across different spatial scopes.
3 Approach
The approach identifies mean and variance shifts from full-spatial normalization and introduces Region Normalization, which normalizes spatial regions separately. It combines mask-based RN-B in early layers with learnable RN-L in later layers to address changing corruption regions and improve feature fusion.
- 3.1 Motivation for Region Normalization: Full-spatial normalization mixes corrupted and uncorrupted pixels, shifting feature statistics toward nonlinear saturation and potentially causing vanishing gradients.Separate normalization reduces this internal covariate shift and preserves training efficiency.
- 3.1 Motivation for Region Normalization: Region Normalization divides spatial pixels into regions according to a mask and computes mean and variance separately for each region.RN merges the normalized regions after applying region-specific learnable affine parameters.
- 3.2 Basic Region Normalization: Basic RN normalizes corrupted and uncorrupted regions separately using the original inpainting mask, making it suitable for early layers with large corrupted regions.RN-B uses two regions and merges their normalized outputs after region-wise normalization and affine transformation.
- 3.3 Learnable Region Normalization: Learnable RN detects regions from feature-derived spatial responses, thresholds the response map, and applies pixel-wise affine parameters derived from global spatial information.RN-L is designed for later layers, where convolution progressively fuses corrupted and uncorrupted regions and the original mask becomes less accurate.
- 3.4 Network Architecture: The inpainting generator replaces its original instance normalization with RN-B in encoder layers and RN-L in residual-block and decoder layers.This placement follows the changing reliability of the original mask across network depth.
4 Experiments
Experiments evaluate RN against established inpainting methods, normalization alternatives, architectural placements, thresholds, mask sizes, and other backbones. RN improves quantitative and qualitative results, with gains extending across settings and networks.
- Quantitative Comparisons: On Places2, RN surpasses CA, PC, GC, EC, and the baseline on PSNR, SSIM, and l1 loss.Testing uses 36,500 validation images and irregular masks covering 0–60% of the image area.
- Qualitative Comparisons: RN achieves better subjective results than CA, PC, GC, EC, and the baseline on Places2 and CelebA examples with irregular and regular masks.The reported benefit is attributed to RN-B reducing normalization shifts and RN-L improving reconstruction of corrupted regions.
- RN and Architecture: Applying RN-B in early encoder layers and RN-L in later residual blocks and the decoder gives the strongest architecture among the tested placements.Applying RN-L in the encoder reduces performance because soft fusion mixes corrupted and uncorrupted information and can wash away uncorrupted-region information.
- Comparisons with Other Normalization Methods: RN outperforms no normalization, instance normalization, and batch normalization during training and at final convergence on Places2.The comparison uses the same backbone and examines PSNR curves over the first 10,000 iterations plus final results after about 225,000 iterations.
- Threshold, Mask Area, and Generalization: RN-L performs best at threshold t = 0.8, and its advantage becomes more significant as the masked area increases on CelebA.RN is also generalized to CA, PC, and GC, producing performance boosts of 2.52 and 0.5 dB PSNR for CA and PC respectively; the gain on GC is not impressive.
5 Conclusion
The work finds Region Normalization more effective than full-spatial normalization for image inpainting and provides plug-and-play modules applicable to other inpainting networks. The model also performs well in object removal, face editing, and image restoration, while extending RN to supervised vision tasks remains future work.
- Region Normalization is more effective for image inpainting than existing full-spatial normalization.
- The two RN variants are plug-and-play modules that can be conveniently applied to other image inpainting networks.
- The inpainting model works well in object removal, face editing, and image restoration use cases.
- Applying RN to supervised vision tasks such as classification and detection is left for future exploration.