Source-linked AI summary
Deeply-Recursive Convolutional Network for Image Super-Resolution
Jiwon Kim, Jung Kwon Lee, Kyoung Mu Lee
TL;DR
Image super-resolution needs broad contextual information, but increasing depth can add parameters and make training difficult. The paper proposes DRCN, which repeatedly reuses one convolution and adds recursive-supervision and skip-connection. The method reports state-of-the-art performance and outperforms existing methods by a large margin on benchmark images.
Problem
Image super-resolution benefits from large receptive fields, but adding depth increases parameters, while deeply recursive networks suffer from exploding or vanishing gradients.
Method
DRCN repeatedly applies the same convolutional layer and uses recursive-supervision plus skip-connection to ease training.
Results
DRCN demonstrates state-of-the-art performance on common benchmarks and outperforms existing methods by a large margin on benchmarked images.
Takeaways & Limitations
The network efficiently reuses weight parameters while exploiting a large image context for super-resolution.
Takeaways & Limitations
Training a deeply recursive network is very difficult because of vanishing and exploding gradients, and the basic recursive model did not succeed in training deeply.
Abstract
from arXiv · showhide
We propose an image super-resolution method (SR) using a deeply-recursive convolutional network (DRCN). Our network has a very deep recursive layer (up to 16 recursions). Increasing recursion depth can improve performance without introducing new parameters for additional convolutions. Albeit advantages, learning a DRCN is very hard with a standard gradient descent method due to exploding/vanishing gradients. To ease the difficulty of training, we propose two extensions: recursive-supervision and skip-connection. Our method outperforms previous methods by a large margin.
1. Introduction
DRCN addresses the need for large receptive fields in image super-resolution while avoiding parameter growth from added depth. Recursive-supervision and skip-connection ease training, and the method achieves state-of-the-art benchmark performance.
- Motivation: Large receptive fields provide contextual information for inferring missing high-frequency components in ill-posed super-resolution.More neighboring pixels can provide clues about information lost during downsampling.
- Motivation: Increasing receptive-field size through larger convolutions adds parameters, while pooling typically discards pixel-wise information.For image restoration, avoiding pooling preserves important image details, but added weight layers increase overfitting and storage concerns.
- Method: DRCN repeatedly applies the same convolutional layer, enlarging the receptive field without increasing the number of parameters.Its receptive field is 41 by 41, compared with SRCNN’s 13 by 13.
- Training extensions: Recursive-supervision trains every recursion through HR predictions and combines intermediate predictions for the final output.The reconstruction method is shared across recursions.
- Training extensions: A skip-connection directly links the input to reconstruction layers, preserving input information when input and output are highly correlated.This addresses attenuation of exact input copies across many forward passes.
- Results: DRCN demonstrates state-of-the-art performance on common benchmarks.The paper reports that its method outperforms existing methods by a large margin on benchmarked images.
2. Related Work
Prior work explored convolutional and recursive approaches to image vision tasks, but deeper recursion was limited. DRCN extends recursive convolution to 16 applications, widening receptive fields without increasing model capacity.
- Prior super-resolution methods: Learning-based super-resolution methods model mappings from low-resolution to high-resolution patches using techniques such as neighbor embedding and sparse coding.SRCNN demonstrated an end-to-end convolutional approach to super-resolution.
- Prior super-resolution methods: Simply stacking more weight layers can improve SRCNN, but increases parameters and data requirements for preventing overfitting.DRCN instead seeks to model long-range pixel dependencies with limited capacity.
- Recursive networks: Earlier recursive convolutional models for static images used only two recursions or encountered dimension-reduction constraints incompatible with full-sized super-resolution outputs.Super-resolution methods cannot reduce the output dimension.
- DRCN: The basic architecture combines embedding, recursive inference, and reconstruction, while the advanced model adds recursive-supervision and skip-connection.The recursive inference network is unfolded to show repeated application of the same layer.
- Recursive networks: Recursive convolutions previously produced worse performance than a single convolution in one model because of overfitting, while another approach improved performance up to three convolutions.These approaches used architectures or parameterizations different from DRCN’s deeper recursive design.
- DRCN: DRCN applies the same convolution up to 16 times, compared with the previous maximum of three, and reports improved super-resolution performance.The approach increases recursion depth while reusing convolutional parameters.
3. Proposed Method
DRCN uses one recursively reused convolutional layer to expand receptive fields without increasing parameters, but deep recursion creates severe training difficulties. Recursive supervision and skip-connections address these difficulties by providing intermediate training signals and direct access to the input.
- Basic Model: The model composes embedding, inference, and reconstruction subnetworks to map an interpolated input image x to a super-resolved estimate.The embedding network produces feature maps, the inference network performs recursive processing, and the reconstruction network generates the output image.
- Basic Model: Each inference recursion applies the same convolution and ReLU, widening the receptive field while reusing the same weights and biases.The recursive function is g(H) = max(0, W ∗ H + b), and the same W and b are used at every recursion.
- Model Properties: Deep recursion is difficult to train because exploding and vanishing gradients hinder long-range pixel dependencies and preserving the input through many recursions.The basic recursive model also faces the practical problem of selecting an appropriate recursion depth, which otherwise requires training multiple networks.
- Recursive-Supervision: Recursive supervision reconstructs a high-resolution prediction after every recursion and combines all intermediate predictions into the final output.Using shared reconstruction layers avoids introducing a distinct reconstruction mechanism for each recursion, while direct losses shorten some backpropagation paths.
- Skip-Connection: Skip-connection feeds the input directly to reconstruction, saving capacity otherwise needed to preserve the input and enabling its exact copy during prediction.This addresses the strong input-output correlation in super-resolution and the difficulty of learning a simple relation across many recursions.
- Training: The training objective weights intermediate-output supervision with α, applies weight decay through β, and decays α over training to emphasize the final output.A high initial α stabilizes training because early recursions converge more easily; α is then reduced as training progresses.
4. Experimental Results
Experiments evaluate DRCN across benchmark datasets, recursion depths, ensemble predictions, and qualitative examples. DRCN achieves the best reported PSNR/SSIM results across datasets and scale factors while producing sharper image details.
- Experimental Setup: The experiments train on 91 images and test on Set5, Set14, B100, and Urban100.Urban100 is described as containing challenging urban images that existing methods fail on.
- Study of Deep Recursions: Ensembling intermediate recursion predictions significantly improves performance, although no single recursion depth is best across all scale factors.The ensemble combines predictions from different recursion levels.
- Study of Deep Recursions: Increasing recursion depth raises PSNR on Set5 at scale factor ×3.The evaluated models use 1, 6, 11, and 16 recursions, with identical parameter counts apart from ensemble weights.
- Comparisons with State-of-the-Art Methods: DRCN outperforms all existing methods on every dataset and scale factor in both PSNR and SSIM.The benchmark covers Set5, Set14, B100, and Urban100 at scale factors ×2, ×3, and ×4.
- Comparisons with State-of-the-Art Methods: Qualitative examples show sharper edges and clearer fine structures in DRCN outputs than in competing methods.Examples include straightened lines, separated branches, sharp character edges, and visible three-line stripes.
- Comparisons with State-of-the-Art Methods: DRCN processes a 288 × 288 image in one second on a Titan X GPU.The reported comparisons use luminance components and a common public evaluation code, with matched border cropping for fairness.
5. Conclusion
The paper presents DRCN as a parameter-efficient super-resolution method that reuses weights while exploiting large image context. Recursive-supervision and skip-connection ease training, and the method outperforms existing methods on benchmark images.
- Conclusion: DRCN reuses weight parameters while exploiting a large image context for super-resolution.The conclusion describes the network as deeply recursive in nature.
- Conclusion: Recursive-supervision and skip-connection are used to ease training of the model.These are presented as the two improvements to the simple recursive network.
- Conclusion: DRCN outperforms existing methods by a large margin on benchmarked images.The conclusion states this as the demonstrated benchmark outcome.