Source-linked AI summary
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
Jiwon Kim, Jung Kwon Lee, Kyoung Mu Lee
TL;DR
Single-image super-resolution requires reconstructing high-resolution images from low-resolution inputs, but existing approaches are limited by small context, slow convergence, and single-scale operation. This paper uses a very deep residual-learning CNN with high learning rates and gradient clipping, outperforming existing methods by a large margin on benchmark images while supporting multi-scale super-resolution in one model.
Problem
Existing super-resolution methods use limited image context, converge slowly during training, and may require separate models for each scale.
Method
The method uses a 20-layer convolutional network with residual learning, high learning rates, adjustable gradient clipping, and a single model for multi-scale super-resolution.
Results
The method outperforms existing methods by a large margin on benchmarked images and surpasses previous methods across several datasets.
Takeaways & Limitations
Very deep convolutional networks can provide accurate super-resolution while training quickly and supporting multiple scale factors in one model.
Takeaways & Limitations
The approach is not valid when the required surrounding region is very large because cropping can make the final image too small for visual quality.
Abstract
from arXiv · showhide
We present a highly accurate single-image super-resolution (SR) method. Our method uses a very deep convolutional network inspired by VGG-net used for ImageNet classification \cite{simonyan2015very}. We find increasing our network depth shows a significant improvement in accuracy. Our final model uses 20 weight layers. By cascading small filters many times in a deep network structure, contextual information over large image regions is exploited in an efficient way. With very deep networks, however, convergence speed becomes a critical issue during training. We propose a simple yet effective training procedure. We learn residuals only and use extremely high learning rates ($10^4$ times higher than SRCNN \cite{dong2015image}) enabled by adjustable gradient clipping. Our proposed method performs better than existing methods in accuracy and visual improvements in our results are easily noticeable.
1. Introduction
The section frames single-image super-resolution as generating a high-resolution image from a low-resolution input and motivates a very deep convolutional approach. It addresses limited context, slow convergence, and single-scale operation through large receptive fields, residual learning with gradient clipping, and one multi-scale model.
- Problem: Single-image super-resolution generates a high-resolution image from a low-resolution image for applications including security, surveillance, and medical imaging.The task is needed when more image details are required on demand.
- Motivation: SRCNN introduced end-to-end CNN-based super-resolution without engineered features, but it uses small image regions, converges slowly, and supports only one scale.These limitations motivate the proposed method.
- Context: The proposed very deep network exploits contextual information over large image regions through a large receptive field, improving detail recovery when small patches are insufficient.This is particularly important for large scale factors, where the problem is ill-posed.
- Convergence: 104 times higher than SRCNN, the initial learning rate is enabled by residual learning and gradient clipping to accelerate training without exploding gradients.Residual learning models the difference between the high-resolution and low-resolution images, which share much of the same information.
- Scale Factor: A single convolutional network handles multi-scale-factor super-resolution, avoiding the impractical need to train and store separate models for arbitrary user-specified scales.The approach is presented as highly accurate, relatively accurate, and fast compared with state-of-the-art methods.
2. Related Work
The section contrasts the proposed very deep residual-learning SR network with SRCNN, emphasizing deeper models, larger receptive fields, faster convergence, and efficient multi-scale handling. It reports 20 weight layers, a 41 × 41 receptive field, and a three-fold parameter reduction across ×2, 3, and 4 scales.
- SRCNN comparison: SRCNN uses three layers for patch extraction/representation, non-linear mapping, and reconstruction, with 9 × 9, 1 × 1, and 5 × 5 filters.The proposed method is compared against SRCNN as a representative state-of-the-art deep learning-based SR approach.
- Depth and receptive field: 20 weight layers enable the proposed network to use a 41 × 41 receptive field, compared with SRCNN’s 3 layers and 13 × 13 receptive field.Each layer uses 3 × 3 filters; the authors argue that increasing depth significantly boosts performance.
- Training: Residual-image modeling avoids learning the shared low-frequency content carried through SRCNN, enabling faster convergence with better accuracy.SRCNN directly models high-resolution images, while the proposed network models residual images directly.
- Scale: A single network handles multiple-scale SR efficiently and compares favorably with a single-scale expert for each sub-task.For three scale factors (×2, 3, 4), the number of parameters is reduced by three-fold.
- Implementation differences: The proposed method pads zeros during training to preserve output size and uses the same learning rate for all layers, unlike SRCNN.SRCNN produces an output smaller than its input and uses different learning rates for different layers.
3. Proposed Method
The proposed method uses a very deep convolutional network that predicts residual image details from an interpolated low-resolution input. Zero-padding preserves image dimensions and boundary predictions, while adjustable gradient clipping enables fast training and shared parameters support multiple scales.
- Network structure: The network repeatedly applies 3 × 3 convolutional and nonlinear layers with 64 filters, transforming an interpolated low-resolution image into a residual-based high-resolution reconstruction.The first layer operates on the input image, and the last uses a single 3 × 3 × 64 filter for reconstruction.
- Image boundaries: Zero-padding keeps all feature maps and the output image the same size, allowing pixels near image boundaries to be predicted without cropping.Cropping becomes problematic when the required surround region is large because the resulting image is too small.
- Residual learning: Residual learning predicts image details r = y − x, which are added to the interpolated low-resolution input to produce the final high-resolution image.The loss is computed from the residual prediction or reconstructed image relative to the ground-truth high-resolution image.
- Training: Adjustable gradient clipping permits high learning rates while suppressing exploding gradients, making convergence substantially faster for very deep networks.The 20-layer network trains within 4 hours, whereas 3-layer SRCNN takes several days.
- Multi-Scale: The multi-scale model shares parameters across predefined scale factors by combining training datasets for those scales.This reduces the need to create and store a separate network for each scale factor.
4. Understanding Properties
The section shows that depth improves SR by expanding contextual information, residual learning accelerates convergence and improves final performance, and scale augmentation enables one network to handle multiple scales with reduced capacity.
- Depth: Deeper networks improve SR performance, with experiments over 5–20 weight layers showing performance generally increases and often improves rapidly with depth.For depth D, the receptive field is (2D + 1) × (2D + 1), increasing the contextual information available for predicting high-frequency components.
- Residual learning: Residual learning converges faster than standard CNN training and reaches higher final PSNR.With initial learning rate 0.1, residual learning reaches 36.90 within 10 epochs, while rate 0.001 reaches 36.52 after 80 epochs; after 10 epochs, residual and non-residual networks reach 36.90 and 27.42.
- Multi-scale training: Scale augmentation lets a single network perform comparably to single-scale models across trained scales while using less capacity than combined multi-network approaches.For training scales {2, 3, 4}, PSNR is 37.06 vs. 37.10 at ×2, 33.27 vs. 32.89 at ×3, and 30.95 vs. 30.86 at ×4.
- Multi-scale training: Training on multiple scales boosts performance at larger scales compared with single-scale models.At test scale 3, models trained on (×2, 3), (×3, 4), and (×2, 3, 4) achieve PSNRs 33.22, 33.24, and 33.27, versus 32.89 for ×3.
5. Experimental Results
Experiments benchmark the method across four standard datasets using a 20-layer network and a shared evaluation framework. Quantitative and qualitative results show that the method outperforms previous methods, with relatively fast execution and clearer reconstructions.
- Experimental setup: The benchmark trains on 291 images with rotation or flip augmentation and evaluates against several state-of-the-art SISR methods.The compared methods are A+, RFL, SelfEx, and SRCNN.
- Datasets: Benchmarking uses Set5, Set14, Urban100, and B100, including challenging urban and natural-image datasets.Set5 and Set14 are commonly used benchmarks, while Urban100 contains challenging images and B100 contains natural images from the Berkeley Segmentation Dataset.
- Training: The final model uses network depth 20, batch size 64, momentum 0.9, and weight decay 0.0001.Weights are initialized using the procedure described by He et al. for networks using rectified linear units.
- Training: Learning starts at 0.1, decreases by a factor of 10 every 20 epochs, and stops after 80 epochs; training takes roughly 4 hours on a Titan Z GPU.All experiments use 9960 iterations with batch size 64, and the learning rate is decreased three times.
- Results: The method outperforms all previous methods quantitatively across the benchmark datasets and is relatively fast.The evaluation follows Huang et al.’s publicly available framework, while comparisons use the same procedure and boundary cropping for fairness.
- Results: Qualitatively, only the proposed method perfectly reconstructs the line in Figure 6, while its contours remain clean and vivid in Figure 7.Other methods produce severely blurred or distorted contours in Figure 7.
6. Conclusion
The paper presents a very deep-network super-resolution method that accelerates and stabilizes training through residual learning, extremely high learning rates, and gradient clipping, achieving a large margin over existing methods on benchmark images.
- 6. Conclusion: The method uses residual learning and extremely high learning rates to optimize very deep networks quickly.Very deep networks otherwise suffer from slow convergence.
- 6. Conclusion: Gradient clipping ensures training stability while maximizing convergence speed.
- 6. Conclusion: The method outperforms existing methods by a large margin on benchmarked images.