Source-linked AI summary
Enhanced Deep Residual Networks for Single Image Super-Resolution
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee
TL;DR
Super-resolution networks remain sensitive to architectural choices, while different upscaling factors are typically handled by separately trained models. This paper introduces optimized single-scale and compact multi-scale residual networks, achieving state-of-the-art benchmark performance and first- and second-place NTIRE2017 rankings.
Problem
Super-resolution networks have non-optimal, architecture-sensitive designs, while different scale factors usually require independently trained networks.
Method
EDSR removes unnecessary residual-network modules and uses residual scaling, while MDSR shares most parameters across scale-specific modules.
Results
The single- and multi-scale networks achieved state-of-the-art PSNR and SSIM across benchmark datasets, ranking first and second, respectively, in the NTIRE2017 challenge.
Takeaways & Limitations
A compact multi-scale model can handle multiple super-resolution scales with comparable performance to a single-scale model while reducing model size and training time.
Abstract
from arXiv · showhide
Recent research on super-resolution has progressed with the development of deep convolutional neural networks (DCNN). In particular, residual learning techniques exhibit improved performance. In this paper, we develop an enhanced deep super-resolution network (EDSR) with performance exceeding those of current state-of-the-art SR methods. The significant performance improvement of our model is due to optimization by removing unnecessary modules in conventional residual networks. The performance is further improved by expanding the model size while we stabilize the training procedure. We also propose a new multi-scale deep super-resolution system (MDSR) and training method, which can reconstruct high-resolution images of different upscaling factors in a single model. The proposed methods show superior performance over the state-of-the-art methods on benchmark datasets and prove its excellence by winning the NTIRE2017 Super-Resolution Challenge.
1. Introduction
The introduction frames SISR as reconstructing a high-resolution image from one low-resolution input, while highlighting architectural and training limitations in existing deep networks and the inefficiency of separate scale-specific models. It presents simplified single-scale and parameter-sharing multi-scale networks, evaluated as state of the art and ranked first and second in the NTIRE 2017 Super-Resolution Challenge.
- Problem setting: SISR reconstructs a high-resolution image from a single low-resolution image, whose degradation may include bicubic downsampling, blur, decimation, or noise.Many studies assume bicubic downsampling, but the low-resolution formation process can vary by situation.
- Limitations: Deep neural networks improve SR PSNR, but performance is sensitive to architectural changes, initialization, and training techniques.The introduction therefore motivates carefully designed architectures and sophisticated optimization methods.
- Multi-scale SR: Existing methods often train separate networks for different scale factors, while VDSR jointly handles scales but requires bicubic-interpolated inputs with higher computation and memory costs.This exposes a tradeoff between joint scale handling and efficient scale-specific upsampling.
- Single-scale network: The proposed method simplifies SRResNet by removing unnecessary modules and uses an appropriate loss function and careful training modifications to improve results.The motivation is that directly applying classification-oriented ResNet architecture to super-resolution can be suboptimal.
- Multi-scale network and results: The multi-scale model transfers knowledge from low-scale pretraining, shares most parameters across scales, uses significantly fewer parameters than multiple single-scale models, and achieves comparable performance.The authors also report state-of-the-art PSNR and SSIM across benchmark and DIV2K datasets, with first- and second-place NTIRE 2017 rankings for the single- and multi-scale networks, respectively.
2. Related Works
Prior super-resolution research progressed from interpolation and natural-image-statistics methods to learned mappings, self-similarity techniques, and deep neural networks. Studies also explored efficient late upsampling and alternative loss functions beyond widely used MSE/L2.
- Classical methods: Early super-resolution methods used sampling-theory interpolation, but struggled to predict detailed, realistic textures; later methods incorporated natural image statistics.These approaches aimed to reconstruct better high-resolution images.
- Learning-based methods: Learned super-resolution methods modeled mappings between ILR and IHR image pairs using neighbor embedding, sparse coding, and clustered patch-space functions.Other methods exploited image self-similarities and geometrically transformed patches to enlarge internal dictionaries without external databases.
- Deep architectures: Deep neural networks produced dramatic improvements in super-resolution, following early CNN-based methods and residual networks for training much deeper architectures.Skip connections and recursive convolution were reported to reduce the burden of carrying identity information.
- Upsampling strategies: Late-network upsampling can reduce computation without losing model capacity because feature maps remain smaller than in methods that first bicubically upsample inputs.Many algorithms instead upsample the input before feeding it into the network, while other approaches train upsampling modules at the network’s end.
- Loss functions: MSE or L2 loss is widely used for image restoration and is also a major PSNR measure, but L2 training does not guarantee better PSNR and SSIM than alternatives.The cited experiments reported improved performance for a network trained with L1, although the supplied passage is truncated before specifying the full comparison.
3. Proposed Methods
The proposed methods simplify residual super-resolution networks to improve performance and computational efficiency, then develop single-scale EDSR and multi-scale MDSR architectures for different upscaling factors. EDSR uses enlarged capacity with stabilized training, while MDSR shares parameters across scales to achieve comparable performance with fewer parameters.
- EDSR: Removing batch normalization improves the network while reducing training memory usage by approximately 40% versus SRResNet.The authors argue that batch normalization limits feature-range flexibility and consumes memory comparable to preceding convolutional layers.
- EDSR: EDSR expands the baseline to B = 32 and F = 256 with residual scaling factor 0.1, while omitting ReLU activations outside residual blocks.Residual scaling stabilizes training when using many feature maps.
- EDSR: Pre-training the ×4 EDSR model from a ×2 network makes training converge much faster than random initialization and improves final performance.The same strategy is used when training models for upsampling factors ×3 and ×4.
- MDSR: MDSR uses a shared main branch with B = 16 residual blocks and scale-specific modules to exploit correlations among multiple super-resolution scales.Its preprocessing modules use two residual blocks with 5 × 5 kernels to reduce input-scale variance while keeping scale-specific processing shallow.
- MDSR: 3.2 million parameters in the baseline multi-scale model compare with 4.5M across three single-scale baselines, while MDSR maintains comparable performance.Each single-scale baseline has about 1.5M parameters.
- MDSR: MDSR requires only 2.5 times more parameters despite having approximately 5 times more depth than its baseline multi-scale model, and performs comparably to scale-specific EDSRs.The parameter efficiency comes from residual blocks being lighter than scale-specific parts.
4. Experiments
Experiments evaluate EDSR and MDSR on DIV2K and four standard benchmark datasets, using controlled training procedures, ablations, and self-ensemble inference. Results show improvements from L1 loss, architectural modifications, larger models, and geometric self-ensemble, while avoiding batch-normalization memory costs.
- Datasets: EDSR and MDSR are evaluated on DIV2K and the Set5, Set14, B100, and Urban100 benchmark datasets.DIV2K provides 800 training, 100 validation, and 100 test images; comparisons use the validation set because test ground truth is unreleased.
- Training setup: Training uses 48×48 RGB low-resolution patches, augmentation, DIV2K mean subtraction, ADAM, minibatches of 16, and a learning rate initialized at 10^-4.The learning rate is halved every 2 × 10^5 minibatch updates, and the optimizer uses β1 = 0.9, β2 = 0.999, and ϵ = 10^-8.
- Ablation study: Ablation experiments show that L1 loss slightly improves SRResNet over L2, while architectural changes and larger final models produce larger gains.The final EDSR+ and MDSR+ models use geometric self-ensemble, and omitting batch normalization substantially reduces GPU-memory requirements.
- Self-ensemble and multi-scale evaluation: Self-ensemble averages outputs from flipped and rotated inputs, retains the original model parameter count, requires no separately trained models, and achieves approximately the performance gain of conventional ensembles.Methods using this strategy are denoted with a “+” postfix; MDSR training randomly selects among ×2, ×3, and ×4 scales per update.
- Benchmark results: Final-model evaluation compares EDSR+ and MDSR+ with A+, SRCNN, VDSR, and SRResNet using PSNR and SSIM on public benchmarks and DIV2K validation data.The models are trained for 10^6 updates with batch size 16, while other baseline settings are retained for comparison.
5. NTIRE2017 SR Challenge
The NTIRE2017 Super-Resolution Challenge sought the highest-PSNR single-image super-resolution system across bicubic and unknown degradation tracks at ×2, ×3, and ×4 scales. EDSR and MDSR demonstrated robustness to different downsampling conditions, with EDSR+ and MDSR+ taking first and second place, respectively.
- Challenge objective: The challenge aimed to develop a single-image super-resolution system with the highest PSNR.The work was initially proposed for participation in NTIRE2017.
- Challenge structure: The challenge included bicubic and unknown-degrader tracks, each evaluated at ×2, ×3, and ×4 downsampling scales.Unknown-track inputs were also severely blurred.
- Challenge results: EDSR and MDSR were submitted to each competition and proved robust to different downsampling conditions.The models successfully reconstructed high-resolution images from severely degraded inputs in the unknown-downsampling track.
- Challenge results: EDSR+ won first place, while MDSR+ won second place.These results were reported for the NTIRE2017 Super-Resolution Challenge.
6. Conclusion
The paper presents compact enhanced single-scale and multi-scale super-resolution models that achieve strong performance through architectural simplification, residual scaling, and shared multi-scale processing. Both models achieve top ranks on standard benchmarks and DIV2K, while the multi-scale model handles multiple scales with reduced model size and training time.
- Enhanced single-scale model: Removing unnecessary ResNet modules improves results while making the single-scale model compact, and residual scaling stabilizes training of large models.The single-scale model surpasses current models and achieves state-of-the-art performance.
- Multi-scale model: The multi-scale network uses scale-dependent modules and a shared main network to handle multiple super-resolution scales in one framework.It reduces model size and training time, remains compact compared with separate single-scale models, and delivers comparable performance.
- Overall results: Both the single-scale and multi-scale models achieve top ranks on standard benchmark datasets and the DIV2K dataset.This result supports the reported effectiveness of both proposed model types across benchmark settings.