Source-linked AI summary
Accelerating the Super-Resolution Convolutional Neural Network
Chao Dong, Chen Change Loy, Xiaoou Tang
TL;DR
SRCNN offers strong super-resolution quality but remains too slow for real-time use. This paper introduces FSRCNN, a compact hourglass CNN that accelerates SRCNN by more than 40× while retaining or improving restoration quality and enabling real-time CPU performance.
Problem
SRCNN provides strong restoration quality, but its processing speed on large images remains far below the 24-fps real-time target.
Method
FSRCNN uses a compact hourglass CNN with direct LR-to-HR deconvolution, feature shrinking and expansion, and deeper mapping with smaller filters.
Results
More than 40× speedup is achieved with superior restoration quality, while FSRCNN-s reaches real-time performance above 24 fps on a generic CPU.
Takeaways & Limitations
Shared convolution layers allow fast training and testing across upscaling factors by fine-tuning or selecting only the corresponding deconvolution layer.
Takeaways & Limitations
The study cannot investigate every variable, so it assigns reasonable values to insensitive variables in advance and leaves sensitive variables unset.
Abstract
from arXiv · showhide
As a successful deep model applied in image super-resolution (SR), the Super-Resolution Convolutional Neural Network (SRCNN) has demonstrated superior performance to the previous hand-crafted models either in speed and restoration quality. However, the high computational cost still hinders it from practical usage that demands real-time performance (24 fps). In this paper, we aim at accelerating the current SRCNN, and propose a compact hourglass-shape CNN structure for faster and better SR. We re-design the SRCNN structure mainly in three aspects. First, we introduce a deconvolution layer at the end of the network, then the mapping is learned directly from the original low-resolution image (without interpolation) to the high-resolution one. Second, we reformulate the mapping layer by shrinking the input feature dimension before mapping and expanding back afterwards. Third, we adopt smaller filter sizes but more mapping layers. The proposed model achieves a speed up of more than 40 times with even superior restoration quality. Further, we present the parameter settings that can achieve real-time performance on a generic CPU while still maintaining good performance. A corresponding transfer strategy is also proposed for fast training and testing across different upscaling factors.
1 Introduction
The paper introduces FSRCNN, a compact hourglass CNN that learns directly from original low-resolution images to high-resolution outputs without bicubic preprocessing. It reduces computational cost through end-of-network deconvolution and low-dimensional, multi-layer mapping, achieving major speedups while preserving restoration quality and enabling real-time CPU inference.
- Motivation: SRCNN is limited by bicubic upsampling, whose convolutional cost grows n^2 times relative to the original low-resolution image for upscaling factor n.Learning directly from the original low-resolution image could provide an approximately n^2 acceleration.
- Motivation: SRCNN’s costly mapping projects patches into high-dimensional spaces, while wider mapping layers improve accuracy at the expense of runtime.SRCNN-Ex uses 57,184 parameters, reported as six times larger than SRCNN.
- FSRCNN design: FSRCNN replaces bicubic interpolation with a deconvolution layer placed at the network’s end, making complexity proportional to the original low-resolution spatial size.The network therefore learns an end-to-end mapping from original low-resolution to high-resolution images without preprocessing.
- FSRCNN design: The hourglass architecture shrinks and expands feature dimensions around several 3 × 3 mapping layers, restricting nonlinear mapping to a low-dimensional feature space.Its overall structure is symmetrical, thick at the ends, and thin in the middle.
- Results: 40×: FSRCNN achieves a speed-up of more than 40× with superior performance to SRCNN-Ex, while FSRCNN-s is 17.36 times faster than SRCNN and runs at 24 fps on a generic CPU.The small FSRCNN-s preserves similar restoration quality to SRCNN and achieves real-time performance.
- Transfer strategy: FSRCNN shares all convolution layers except deconvolution across upscaling factors, allowing another factor to be supported by fine-tuning only the deconvolution layer.The transfer strategy is intended to facilitate fast training and testing across different upscaling factors with almost no loss of mapping accuracy.
2 Related Work
Deep learning has been successfully applied to image super-resolution, beginning with SRCNN and extending to tasks such as face hallucination and depth map super-resolution. Related work also explored deeper structures and CNN acceleration, while FSRCNN operates on original low-resolution images and supports multiple upscaling factors through deconvolution-layer changes.
- Deep learning for SR: SRCNN pioneered deep learning for super-resolution and motivated state-of-the-art applications in face hallucination and depth map super-resolution.Deeper network structures were subsequently explored.
- Deep learning for SR: FSRCNN processes the original low-resolution image instead of a bicubic-upscaled input and uses a simpler, more efficient mapping layer.The passage contrasts FSRCNN with prior networks that process bicubic-upscaled low-resolution images.
- Deep learning for SR: FSRCNN requires changing only the deconvolution layer for different upscaling factors, enabling faster upscaling to several image sizes.Previous methods require training a completely different network for each specific upscaling factor.
3 Fast Super-Resolution by CNN
FSRCNN accelerates SRCNN by processing the original low-resolution image, shrinking and expanding feature dimensions, and using smaller filters with more mapping layers. Its hourglass design reduces computation while improving restoration quality and supports transfer across upscaling factors.
- Network reformulation: FSRCNN replaces SRCNN’s interpolated input with the original low-resolution image and adds a deconvolution layer for learned upsampling.The network comprises feature extraction, shrinking, mapping, expanding, and deconvolution parts.
- Network reformulation: The shrinking layer reduces LR feature dimension from d to s << d using Conv(1, s, d), substantially lowering mapping complexity.The filters perform a linear combination within the LR features, with s controlling the shrinking level.
- Computational efficiency: The complexity becomes O{(9ms2 + 2sd + 106d)SLR}, shifting dependence from HR-image size toward the smaller LR image.The middle layer contributes most to SRCNN’s parameters, motivating the focus on feature dimension and image size.
- Acceleration analysis: 8.7× speedup and roughly 0.12 dB gain result when SRCNN-Ex’s final convolution is replaced by deconvolution for upscaling factor n = 3.This change makes computation proportional to SLR instead of SHR.
- Acceleration analysis: FSRCNN (56,12,4) outperforms SRCNN-Ex by 0.23dB on Set5, showing that acceleration does not require performance degradation.The result is reported for average PSNR on Set5 using the 91-image training dataset.
- Transfer across factors: FSRCNN enables fast training and testing across upscaling factors because only the final deconvolution layer contains upscaling-factor information.Experiments found that convolution filters are almost the same for different upscaling factors.
4 Experiments
The experiments introduce the General-100 training dataset, evaluate FSRCNN’s sensitive architectural variables, and demonstrate strong quality, acceleration, and real-time performance. They also establish transfer across upscaling factors and fair restoration-quality comparisons using an augmented 91-image dataset.
- Training dataset: The General-100 dataset contains 100 BMP-format images, addressing limitations of the widely used 91-image training set and JPEG-format BSD500 images.The paper notes that 91 images are insufficient for best deep-model performance and that BSD500 JPEG images are not optimal for super-resolution.
- Controlling experiments: Twelve controlling experiments vary LR feature dimension d, shrinking filters s, and mapping depth m across d = 48, 56, s = 12, 16, and m = 2, 3, 4.The experiments use all 2 × 2 × 3 combinations of the three variables.
- Controlling experiments: m = 4 generally produces better Set5 PSNR than m = 2 and m = 3, while better results usually require more parameters.The paper analyzes the average PSNR values horizontally by mapping depth and vertically by feature dimension and shrinking-filter settings.
- Acceleration and quality: 32.87 dB is achieved by FSRCNN (48,12,2), exceeding SRCNN-Ex’s 32.75 dB with 8,832 parameters and 58.3× acceleration.The acceleration is computed from the reported 57,184 SRCNN-Ex parameters and 8,832 FSRCNN parameters, multiplied by the scale-factor term.
- Real-time implementation: 24.7 fps is reached by FSRCNN (32,5,1), which has 3,937 parameters and satisfies the 24 fps real-time requirement while outperforming SRCNN (9-1-5).The target configuration is derived from an estimated maximum of approximately 3,976 parameters for real-time implementation.
- Transfer across upscaling factors: FSRCNN supports learning and testing across upscaling factors by transferring convolution filters, demonstrated by training ×2 from a model trained for ×3.The default network for this demonstration is FSRCNN (56,12,4).
5 Conclusion
The paper concludes that redesigning SRCNN yields a substantially more efficient super-resolution network without sacrificing restoration quality. The proposed model achieves over 40× acceleration and can support real-time video super-resolution.
- Conclusion: More than 40 times acceleration is achieved through the redesigned SRCNN structure.The redesign targets high running speed while preserving restoration quality.
- Conclusion: The proposed method delivers satisfactory super-resolution performance while improving runtime.Extensive experiments support its restoration quality and speed advantages.
- Conclusion: The proposed model can be adapted for real-time video super-resolution.The conclusion connects the efficiency improvements to practical real-time video SR use.