Source-linked AI summary
Is the deconvolution layer the same as a convolutional layer?
Wenzhe Shi, Jose Caballero, Lucas Theis, Ferenc Huszar, Andrew Aitken, Christian Ledig, Zehan Wang
TL;DR
The note asks how deconvolution relates to the proposed subpixel layer and why superresolution networks should compute in low-resolution space. It shows that transposed, subpixel, and efficient subpixel formulations are closely related, while LR-only convolution offers greater representation power at the same speed. The note limits its broader implications to questions for future applications and architectures.
Problem
The note addresses the relationship among deconvolution variants and whether explicit upsampling is necessary when designing superresolution networks.
Method
It compares transposed convolution, subpixel convolution, and efficient subpixel convolution, interpreting the efficient layer as convolution performed in LR space.
Results
For superresolution, convolution exclusively in LR space has greater representation power than an HR network at the same runtime speed, making explicit bicubic or deconvolution upsampling unnecessary in the discussed setting.
Takeaways & Limitations
LR convolution can use channels of LR feature maps to represent HR feature maps through simple convolutions and periodic shuffling.
Takeaways & Limitations
The note leaves open whether explicit upsampling is necessary in other applications and whether networks can learn when and how much to upscale.
Abstract
from arXiv · showhide
In this note, we want to focus on aspects related to two questions most people asked us at CVPR about the network we presented. Firstly, What is the relationship between our proposed layer and the deconvolution layer? And secondly, why are convolutions in low-resolution (LR) space a better choice? These are key questions we tried to answer in the paper, but we were not able to go into as much depth and clarity as we would have liked in the space allowance. To better answer these questions in this note, we first discuss the relationships between the deconvolution layer in the forms of the transposed convolution layer, the sub-pixel convolutional layer and our efficient sub-pixel convolutional layer. We will refer to our efficient sub-pixel convolutional layer as a convolutional layer in LR space to distinguish it from the common sub-pixel convolutional layer. We will then show that for a fixed computational budget and complexity, a network with convolutions exclusively in LR space has more representation power at the same speed than a network that first upsamples the input in high resolution space.
Section 1: Transposed convolution and subpixel convolutional layers
The note distinguishes transposed and subpixel convolutions as upsampling operations related to stride-2 convolution. Their outputs can be identical when the subpixel filter’s weight indices are reversed.
- Section 1: Transposed convolution and subpixel convolutional layers: A stride-2 convolution downsamples a padded 1D signal, while transposed and subpixel convolutions with the corresponding fractional stride upsample it.The examples use a signal of size 8, a filter of size 4, and an output of size 5 for stride-2 convolution; the upsampling example reverses these sizes.
- Section 1: Transposed convolution and subpixel convolutional layers: The note reserves a distinct case for deconvolution with memorized max-pooling indices, which is not treated as equivalent to transposed convolution here.The subsequent discussion assumes pooling indices are not memorized.
- Section 1: Transposed convolution and subpixel convolutional layers: Transposed convolution is named for the transposed operation matrix and is also called backward convolution because it corresponds to backward propagation through convolution.The transposed operation changes padding into cropping in the illustrated construction.
- Section 1: Transposed convolution and subpixel convolutional layers: Subpixel convolution derives its name from imaginary fractional-index subpixels inserted between the original pixels.The note presents this as a naming distinction from transposed convolution, not as a different upsampling objective.
- Section 1: Transposed convolution and subpixel convolutional layers: The two operations differ only in the weight indices contributing to the output, and reversing the subpixel filter indices makes them identical.Thus, with learned filters, both operations can achieve the same result.
Section 2: Deconvolution layer vs Convolution in LR
The proposed LR-space convolution is equivalent to subpixel convolution and can reproduce the same HR output as a corresponding deconvolution layer. It achieves this by separating independently activated kernel weights and periodically shuffling LR output channels into the HR image.
- Equivalence to deconvolution: A convolution with output channels o·r^2, input channels i, and kernel size k×k in LR space is equivalent to a deconvolution layer with kernel size k·r.The equivalence is demonstrated by rearranging the deconvolution kernels into subpixel-convolution kernels.
- Subpixel convolution: The subpixel operation first creates a fractionally indexed, zero-padded subpixel image from the LR input, then convolves it in subpixel space.Different kernel-weight subsets are activated by different subpixel positions.
- LR implementation: Because the kernel-weight subsets operate independently, they can be split into smaller kernels, making the operation invertible and implementable as LR convolution.The split kernels produce separate output channels corresponding to the independently activated weight sets.
- HR reconstruction: Periodic shuffling reshapes the LR convolution’s output channels into the HR image, yielding the same result as the full subpixel convolution.The construction generalizes to arbitrary kernel shapes and rescale ratios.
- Kernel correspondence: The paper illustrates this equivalence by transforming trained LR convolution kernels into corresponding deconvolution kernels that produce identical outputs for 32-channel inputs.The correspondence is shown using 9×9 deconvolution kernels and 3×3 LR kernels with nine output channels per input channel.
Section 3: What does this mean?
Convolutions confined to LR space can match HR representations while retaining greater representation power at the same runtime, making explicit upsampling unnecessary for superresolution.
- Deconvolution equivalence: The deconvolution operation is equivalent to LR convolution producing r^2 channels that can represent one HR image or feature map.Periodic rearrangement of these channels creates the HR representation, while the channel-producing operation remains an ordinary convolution.
- Representation power: The LR network has greater representation power than an HR network at the same runtime speed.The comparison assumes equal runtime and considers networks with feature maps exclusively in LR versus networks that upsample before convolution.
- Representation power: LR and HR networks retain the same information and have equivalent receptive fields in the original LR input space.The LR network nevertheless uses more parameters under the stated complexity comparison.
- Implication: For superresolution, the authors conclude that explicit bicubic or deconvolution upsampling is not necessary.They cite later work using LR convolution for superresolution and style transfer as independent support.
- Open questions: The note leaves open whether networks can learn when and how much to upscale, or combine LR and HR features automatically in other tasks.These questions concern applications beyond the demonstrated superresolution setting.