Source-linked AI summary
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, Zehan Wang
TL;DR
Existing CNN super-resolution methods upscale inputs into high-resolution space, adding computational cost. This paper extracts features in low-resolution space and learns sub-pixel upscaling filters, achieving over 10× speed and gains of +0.15dB on images and +0.39dB on videos.
Problem
CNN super-resolution methods commonly process images in high-resolution space, creating computational costs and leaving efficient learned upscaling insufficiently explored.
Method
The paper extracts feature maps in low-resolution space and uses a learned sub-pixel convolution layer to reconstruct high-resolution outputs.
Results
+0.15dB on images and +0.39dB on videos, with over 10× speed improvement over the previous CNN approach.
Takeaways & Limitations
The model enables real-time super-resolution of HD videos on a single GPU while improving performance over the previous CNN approach.
Takeaways & Limitations
The paper does not yet extend ESPCN to spatio-temporal super-resolution using multiple neighboring video frames.
Abstract
from arXiv · showhide
Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.
1. Introduction
The paper proposes performing feature extraction in LR space and learning the LR-to-HR upscaling operation only at the network’s end. This design reduces computational and memory costs while improving the learned reconstruction mapping and achieving nearly order-of-magnitude speedups.
- Motivation: Increasing resolution before enhancement raises computational complexity, while bicubic interpolation adds no information to the ill-posed reconstruction problem.Convolutional processing speed depends directly on input resolution, making early upscaling especially costly.
- Proposed architecture: Upscaling occurs only in the final layer, so nonlinear feature extraction operates directly on LR inputs rather than the larger HR representation.The reduced resolution permits smaller filters for the same contextual area, substantially lowering computational and memory complexity.
- Proposed architecture: The efficient sub-pixel convolution layer learns the upscaling operation from LR feature maps for image and video super-resolution.It aggregates LR feature maps and constructs the SR image in a single final step.
- Learned upscaling: The network learns nL−1 upscaling filters for nL−1 feature maps instead of one filter for the input image.Without explicit interpolation, the network implicitly learns the SR processing and a more complex LR-to-HR mapping than fixed-filter upscaling.
- Evaluation: Nearly an order of magnitude faster performance was reported on images and videos compared with previously published methods.The approach was evaluated on publicly available benchmark datasets and reported state-of-the-art performance.
2. Method
The ESPCN avoids upscaling the LR input before reconstruction by extracting features in LR space and using a final sub-pixel convolution layer to produce the HR image. The layer periodically shuffles LR feature outputs into HR space, reducing the computational cost of resolution recovery.
- Problem formulation: SISR estimates an HR image from an LR image generated by Gaussian filtering and downsampling by the upscaling ratio r.The LR and HR images have sizes H × W × C and rH × rW × C, respectively.
- Network architecture: ESPCN applies the first L−1 convolutional layers directly to the LR image, then upscales LR feature maps with a final sub-pixel convolution layer.The final layer converts LR feature maps to the HR reconstruction without applying a nonlinearity to its outputs.
- Computational efficiency: r2, naive LR-to-HR implementations increase computational cost by a factor of r2 because convolution occurs in HR space.The proposed LR-space operation activates only the relevant periodically varying filter weights.
- Sub-pixel convolution: The sub-pixel convolution layer uses periodic shuffling to rearrange an H × W × C · r2 tensor into an rH × rW × C HR image.Its convolution produces r2C channels before the periodic shuffling operation.
- Computational efficiency: log2r2 times faster than deconvolution layer in training and r2 times faster than implementations using upscaling before convolution.Training can avoid explicit periodic shuffling by preshuffling the training data.
3. Experiments
Experiments on standard image and 1080p video benchmarks show that ESPCN benefits from sub-pixel convolution and tanh activation, achieves competitive or superior reconstruction quality, and substantially reduces runtime versus prior methods.
- Datasets: ESPCN was evaluated on Timofte, Set5, Set14, BSD300, BSD500, and Super Texture image datasets, using 50,000 randomly selected training images for final models.The benchmarks provide source images and established comparisons for single-image super-resolution.
- Ablation studies: Tanh activation outperformed relu for single-image super-resolution, while ESPCN’s sub-pixel convolution layer was evaluated against SRCNN’s standard 9-1-5 model.The comparisons used models trained on 91 images and on ImageNet images.
- Image super-resolution: ESPCN was significantly better than the SRCNN 9-5-5 ImageNet model and close to, or sometimes better than, the TNRD 7 × 7 5-stage model.The comparison used the best SRCNN 9-5-5 ImageNet model and TNRD as the strongest published baseline considered.
- Runtime: 4.7ms was ESPCN’s average time to super-resolve one Set14 image on a K2 GPU, with operation complexity 2.5 × r × r times lower than the compared approach.The model was reported to run an order of magnitude faster than previously published methods.
- Video super-resolution: 0.038s per frame was ESPCN’s runtime at 3× upscaling versus 0.435s for SRCNN, while at 4× it required 0.029s versus SRCNN’s 0.434s.These measurements were obtained on 1080p HD videos from the Xiph and Ultra Video Group databases.
4. Conclusion
The paper concludes that adaptive upscaling with feature extraction in low-resolution space improves efficiency and reconstruction over non-adaptive high-resolution processing. The proposed model achieves substantial speed and performance gains and enables real-time super-resolution of HD videos on one GPU.
- Conclusion: Feature extraction in LR space, combined with a sub-pixel convolution layer, replaces computationally costly HR processing with adaptive upscaling into HR space.The sub-pixel layer super-resolves LR data with little additional computational cost compared with deconvolution during training.
- Conclusion: +0.15dB on Images and +0.39dB on videos were achieved versus the previous CNN approach with more parameters.The comparison used an extended benchmark dataset with an upscaling factor of 4; the previous models used 9-5-5 layers versus the proposed 5-3-3 model.
- Conclusion: > 10× speed improvement was achieved versus the previous CNN approach, making the model capable of real-time SR of HD videos on a single GPU.The evaluation compared the proposed approach with the previous CNN model described as having more parameters.
5. Future work
Future work can exploit the redundancy shared across neighboring video frames for video super-resolution. Spatio-temporal networks are a relevant direction because they use temporal information from videos.
- Future work: Neighboring video frames generally share most scene content, creating redundancy that video super-resolution can exploit.Scene changes and objects appearing or disappearing are exceptions.
- Future work: Scene changes and sporadically appearing or disappearing objects limit the assumption of shared content across neighboring frames.
- Future work: Spatio-temporal networks are a relevant approach because they fully use temporal information from videos.The passage notes their popularity for human action recognition.