Source-linked AI summary
Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize
Andrew Aitken, Christian Ledig, Lucas Theis, Jose Caballero, Zehan Wang, Wenzhe Shi
TL;DR
Checkerboard artifacts remain a problem in image and label outputs. The paper proposes convolution NN resize initialization for sub-pixel convolution, which avoids artifacts at initialization and outperforms resize convolution at equal complexity.
Problem
Checkerboard artifacts can arise in generated images and labels, including from strided convolution or pooling in loss networks, which this work does not address.
Method
The paper proposes convolution NN resize initialization for sub-pixel convolution by first convolving and then applying nearest-neighbor resizing.
Results
Convolution NN resize is artifact-free immediately after initialization, has more modelling power than resize convolution at equal complexity, and converges to smaller test errors.
Takeaways & Limitations
The proposed initialization provides a more reasonable training starting point than random initialization and converges faster to a better minimum.
Takeaways & Limitations
The method does not address checkerboard artifacts arising from strided convolution or pooling in networks used as loss functions.
Abstract
from arXiv · showhide
The most prominent problem associated with the deconvolution layer is the presence of checkerboard artifacts in output images and dense labels. To combat this problem, smoothness constraints, post processing and different architecture designs have been proposed. Odena et al. highlight three sources of checkerboard artifacts: deconvolution overlap, random initialization and loss functions. In this note, we proposed an initialization method for sub-pixel convolution known as convolution NN resize. Compared to sub-pixel convolution initialized with schemes designed for standard convolution kernels, it is free from checkerboard artifacts immediately after initialization. Compared to resize convolution, at the same computational complexity, it has more modelling power and converges to solutions with smaller test errors.
Section 1: Sub-pixel convolution and resize convolution
Sub-pixel convolution rearranges independently initialized sub-kernel outputs into HR space, which can create checkerboard artifacts because neighboring features depend on different sub-kernel sets. Nearest-neighbor resize convolution avoids deconvolution overlap and random-initialization effects, but sub-pixel convolution retains more trainable parameters at equal complexity.
- Sub-pixel convolution and resize convolution: For 2× upscaling, a (12, 64, 5, 5) kernel produces 12 channels that periodic shuffling reorganizes into 3 color channels.The 12 output channels use 5×5 filters before reorganization into the three output color channels.
- Sub-pixel convolution and resize convolution: Independent initialization of sub-kernel sets causes checkerboard artifacts because neighboring HR features depend on different sets applied to the same input.Each sub-kernel set generates only one subset of HR outputs, while each HR pixel depends on one set.
- Sub-pixel convolution and resize convolution: Nearest-neighbor resize convolution fills sub-pixel space with interpolated values before convolution, using stride 1 to eliminate deconvolution overlap.Unlike sub-pixel convolution, resize convolution cannot skip convolution with interpolated pixels.
- Sub-pixel convolution and resize convolution: At the same computational complexity, sub-pixel convolution has more trainable parameters because it skips convolution with zero-valued sub-pixel entries.Resize convolution activates convolution weights for every calculated HR feature, removing random-initialization effects.
Section 2: Initialization to convolution resize
The section contrasts NN resize convolution with sub-pixel convolution and derives an initialization that makes sub-pixel convolution equivalent to convolution followed by NN resize. Copying one kernel set across all sub-kernel sets yields reconstructions free of checkerboard artifacts immediately after initialization, while preserving trainable upsampling after training.
- Convolution resize: Convolution followed by nearest-neighbor resizing eliminates checkerboard artifacts after initialization but makes the upsampling kernel non-trainable.NN resize preserves the number of feature maps while increasing spatial resolution, unlike the reshuffling operator.
- Initialization principle: The proposed initialization seeks weights W′ that make sub-pixel convolution identical to convolution NN resize after initialization.This equivalence is the central condition used to eliminate checkerboard patterns without permanently replacing the sub-pixel convolution operation.
- Initialization construction: Setting every sub-kernel set W′n equal to the initialized W′0 prevents checkerboard artifacts after initialization.The construction initializes one kernel set and copies its weights to the remaining sub-kernel sets.
- Initialization results: Figure 5 demonstrates that repeat sub-kernel initialization produces super-resolved images free from checkerboard artifacts immediately after initialization.The figure compares original high-resolution images with results from repeat sub-kernel-initialized sub-pixel convolution.
Section 3: Experiments
Experiments trained super-resolution models on ImageNet to compare sub-pixel convolution, NN resize convolution, and ICNR initialization. ICNR removed checkerboard artifacts, improved the training starting point, and produced smoother early kernels while retaining sub-pixel convolution’s modelling advantages.
- Experimental setup: Models were trained on 350 thousand ImageNet images using normalized BGR inputs and outputs with random 96x96 HR crops.Each mini-batch used 16 random high-resolution sub-images from different training images.
- Comparative results: Sub-pixel convolution substantially outperformed NN resize convolution, while ICNR converged faster and reached a better minimum than standard initialization.Training losses and testing errors for the three methods are shown in Figures 7 and 8.
- Initialization effects: ICNR removed checkerboard artifacts and provided a more reasonable training starting point than random initialization.The result directly supports the proposed initialization scheme’s intended purpose.
- Kernel visualization: ICNR kernels appeared substantially smoother than orthogonally initialized kernels through 2 million iterations, but all three kernel sets converged to similar patterns after 8 million iterations.The comparison comes from visualized kernels in the Appendix.
Section 4: Discussion
The note proposes convolution NN resize as a sub-pixel convolution initialization that avoids checkerboard artifacts immediately after initialization. At equal computational complexity, it offers more modelling power than resize convolution and converges to smaller test errors, while leaving efficient nontrivial-upscaling initialization and loss-induced artifacts open.
- Main contribution: Convolution NN resize avoids checkerboard artifacts immediately after sub-pixel convolution initialization.It is compared against initialization schemes designed for standard convolution kernels.
- Main contribution: At the same computational complexity, convolution NN resize has more modelling power and converges to solutions with smaller test errors than resize convolution.
- Future directions: A future direction is replacing NN resize with a more powerful upscaling operation.The note develops a relationship between sub-pixel convolution and resize-based operations to motivate this direction.
- Future directions: For non-trivial upscaling kernels, the corresponding initialized sub-pixel weights have larger spatial dimensions than convolution NN resize.The authors are still exploring more efficient initialization approaches based on this relationship.
- Limitations: Loss networks containing strided convolution or pooling can introduce checkerboard artifacts, but this source is not addressed because experiments used only a simple MSE loss.The note identifies solving this problem as an open question.