PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks

Yifan Wang, Lijun Wang, Hongyu Wang, Peihua Li

arXiv:1607.07680v1cs.CV

TL;DR

Single-image SR must reconstruct HR detail from ill-posed LR observations, but prior CNN methods separate hand-designed upsampling from learned reconstruction. EEDS jointly learns feature extraction, feature-space upsampling, and multi-scale reconstruction, while training deep and shallow networks together; it consistently outperforms competing methods across test sets and upscaling factors.

  • Problem

    Prior SR methods decouple hand-designed image-domain upsampling from CNN reconstruction, while deeper networks are difficult to train.

  • Method

    EEDS jointly learns feature extraction, latent feature-space upsampling, and multi-scale HR reconstruction, while jointly training deep and shallow CNNs.

  • Results

    EEDS consistently outperforms other methods across three test sets and all upscaling factors, improving over CSCN in PSNR and SSIM.

  • Takeaways & Limitations

    Joint feature-space processing and deep–shallow training provide a fully end-to-end SR approach with faster convergence and superior reported performance.

Abstract

from arXiv · show

One impressive advantage of convolutional neural networks (CNNs) is their ability to automatically learn feature representation from raw pixels, eliminating the need for hand-designed procedures. However, recent methods for single image super-resolution (SR) fail to maintain this advantage. They utilize CNNs in two decoupled steps, i.e., first upsampling the low resolution (LR) image to the high resolution (HR) size with hand-designed techniques (e.g., bicubic interpolation), and then applying CNNs on the upsampled LR image to reconstruct HR results. In this paper, we seek an alternative and propose a new image SR method, which jointly learns the feature extraction, upsampling and HR reconstruction modules, yielding a completely end-to-end trainable deep CNN. As opposed to existing approaches, the proposed method conducts upsampling in the latent feature space with filters that are optimized for the task of image SR. In addition, the HR reconstruction is performed in a multi-scale manner to simultaneously incorporate both short- and long-range contextual information, ensuring more accurate restoration of HR images. To facilitate network training, a new training approach is designed, which jointly trains the proposed deep network with a relatively shallow network, leading to faster convergence and more superior performance. The proposed method is extensively evaluated on widely adopted data sets and improves the performance of state-of-the-art methods with a considerable margin. Moreover, in-depth ablation studies are conducted to verify the contribution of different network designs to image SR, providing additional insights for future research.

I. INTRODUCTION

Single-image SR is ill-posed, while prior learning-based methods separate hand-designed upsampling from learned reconstruction. The proposed method jointly learns feature extraction, feature-space upsampling, multi-scale reconstruction, and deep–shallow training in an end-to-end CNN.

  • Single-image SR must recover high-frequency HR detail from an LR observation even though multiple HR images can produce the same LR image.
  • Prior deep SR methods first bicubically upsample the LR image and then apply CNN reconstruction, preventing joint optimization of both steps.
  • Existing approaches often use local patches, limiting explicit integration of the short- and long-range context needed for SR.
  • The proposed CNN extracts LR features, learns feature-space upsampling, and reconstructs HR images with multi-scale context in one jointly trained model.
  • The deep–shallow ensemble trains faster and improves restoration by combining the shallow network’s low-frequency reconstruction with the deep network’s high-frequency details.

II. RELATED WORK

Learning-based SR progressed from hand-designed features and shallow models to deep CNNs, but existing methods retain decoupled upsampling and face deep-training challenges. The paper addresses these limitations with jointly trained deep and shallow networks.

  • Very deep SR networks remain difficult to train, so prior work often uses relatively shallow architectures.
  • The proposed training strategy jointly trains deep and shallow networks, with the shallow network stabilizing training and the deep network improving HR reconstruction.
  • Deep CNNs improve feature learning, but many SR methods still upsample in the image domain with hand-designed operations decoupled from the network.

III. ARCHITECTURE

EEDS is an ensemble of a complex deep CNN and an easier-to-converge shallow CNN. The deep network is organized into feature extraction, upsampling, and multi-scale reconstruction modules.

  • The EEDS architecture combines a 13-layer deep CNN with a simpler shallow CNN to balance detailed reconstruction and training stability.
  • The deep CNN comprises feature extraction, upsampling, and multi-scale reconstruction modules for restoring detailed HR content.

A. Feature Extraction

The feature-extraction module replaces hand-designed filters on bicubic-upsampled images with learned convolutional mappings applied directly to the LR image. Shortcut connections support gradient flow through the feature extractor.

  • Prior SR methods often extract features after bicubic interpolation, whose filters are not specifically designed for the SR task.
  • The feature extractor uses three convolutional layers with ReLU nonlinearities to learn mappings directly from the original LR image.
  • Each feature-extraction layer uses 3×3 kernels and 64-channel feature maps with zero padding to preserve spatial size.
  • A shortcut adds the first layer’s feature map to the third layer’s output, facilitating gradient flow through the feature-extraction stack.

B. Upsampling

The method performs learned upsampling in feature space and reconstructs HR images using multi-scale context within an end-to-end CNN. It also combines deep and shallow networks to address training difficulty while retaining high-frequency restoration capacity.

  • B. Upsampling: Learned upsampling increases feature-map spatial size using trainable operations instead of hand-designed interpolation.The module connects feature extraction and HR reconstruction, and deconvolution kernel size affects spatial consistency and computational cost.
  • B. Upsampling: Multi-scale convolutions explicitly encode short- and long-range context for HR reconstruction.Convolutions with 1 × 1, 3 × 3, 5 × 5, and 7 × 7 kernels produce features that are concatenated and fused.
  • B. Upsampling: The 13-layer deep network directly maps original LR images to HR images through jointly learned feature extraction, upsampling, and reconstruction.Shortcut connections are used to alleviate vanishing and exploding gradients, although direct restoration from LR inputs remains challenging.
  • B. Upsampling: The shallow network mirrors the three-module design with feature extraction, deconvolution-based upsampling, and HR reconstruction.It uses three trainable layers, including a 3 × 3 convolution, a deconvolution layer, and a final 5 × 5 convolution.
  • B. Upsampling: Deep and shallow networks independently process the same LR input, and their HR outputs are combined as an ensemble prediction.The networks do not share weights; the shallow network captures major components while the deep network restores high-frequency details and corrects errors.

E. Training

The deep and shallow networks are jointly optimized on paired LR-HR images using Euclidean loss with weight decay. Training uses mini-batch stochastic gradient descent with scheduled learning-rate reductions.

  • E. Training: Joint training minimizes Euclidean error between predicted and ground-truth HR images over N training pairs.The predicted HR image is computed from the ensemble, with weight decay applied to network parameters.
  • E. Training: Optimization uses mini-batch stochastic gradient descent with batch size 256, momentum 0.9, and weight decay 0.005.Convolution filters use Gaussian initialization, while deconvolution filters use bilinear interpolation kernels.
  • E. Training: The learning rate starts at 1e−4 and is reduced by a factor of 0.1 when validation loss stabilizes.

A. Setup

The evaluation follows widely adopted SR protocols across three scale factors and three public datasets. Performance is measured with PSNR and SSIM, while luminance is super-resolved in YCbCr space.

  • A. Setup: Training uses 91 images, augmented by rotation and flipping to produce 728 training images and a 200-image validation set.Randomly cropped 96 × 96 patches are used for each upscaling factor.
  • A. Setup: The model is evaluated at upscaling factors 2, 3, and 4 on Set5, Set14, and BSD100 using PSNR and SSIM.The datasets contain 5, 14, and 100 images, respectively, and inference accepts LR images of arbitrary size.
  • A. Setup: Only the luminance channel is super-resolved in YCbCr, while chrominance channels are upsampled with bicubic interpolation for display.

B. Comparison with state-of-the-arts

EEDS is compared with bicubic interpolation and learning-based SR methods across three datasets and scale factors 2, 3, and 4. It consistently outperforms the compared methods, with sharper edges and fewer artifacts in sampled results.

  • B. Comparison with state-of-the-arts: The comparison includes bicubic interpolation and nine learning-based methods spanning sparse representation, anchored regression, forests, and deep learning.
  • B. Comparison with state-of-the-arts: ImageNet-trained ESPCN and SRCN-L results are reported for reference but excluded from performance ranking.
  • B. Comparison with state-of-the-arts: EEDS consistently outperforms the other methods across all three test sets and all upscaling factors.The paper reports a considerable PSNR and SSIM margin over the second-best CSCN method.
  • B. Comparison with state-of-the-arts: Sampled EEDS reconstructions are perceptually more plausible, with relatively sharp edges and little artifacting.

C. Architecture Analysis

The architecture analysis shows that EEDS benefits from jointly trained deep and shallow networks, learned feature-space upsampling, and multi-scale reconstruction. These components improve convergence, high-frequency restoration, and reported PSNR relative to corresponding alternatives.

  • Deep Network vs Shallow Network: EED restores high-frequency details but suffers illumination changes, whereas the shallow network provides more accurate overall magnitudes.The reported discrepancy is attributed to the complexity of directly mapping LR images to HR images and possible convergence to a local minimum.
  • Deep Network vs Shallow Network: EEDS jointly trains shallow and deep networks, converging faster than EED while achieving the best reported performance.The shallow network learns major image components early, while the deep network subsequently focuses on high-frequency details and error correction.
  • Upsampling Analysis: EEDS and EED considerably improve average PSNR over their bicubic-preupsampling variants, supporting learned upsampling in feature space.The comparison uses EEDS-ND and EED-ND, which replace learned upsampling with bicubic interpolation in the original color space.
  • Upsampling Analysis: The analysis varies deconvolution kernel sizes and reports that increasing kernel size can improve performance, while larger kernels add computational overhead.The tested alternatives use kernel sizes 7, 21, and 25 instead of the default 15; the selected size 15 balances effectiveness and efficiency.
  • Multi-scale Analysis: Fusing four reconstruction scales improves Set5 average PSNR by 0.20dB over EEDS-SS7, combining short- and long-range context.The single-scale results also indicate that larger kernels perform better than smaller ones, although larger kernels increase computational overhead.

V. CONCLUSION

The paper presents an end-to-end single-image super-resolution method that learns feature extraction, latent-space upsampling, and multi-scale reconstruction jointly. Jointly training shallow and deep networks accelerates convergence, and experiments report superior performance over state-of-the-art methods.

  • The proposed method extracts LR features, learns upsampling in latent feature space, and reconstructs HR images using short- and long-range context.
  • Jointly training shallow and deep networks accelerates convergence by assigning major image components to the shallow network and residual details to the deep network.
  • Extensive experiments report superior performance over state-of-the-art methods, while ablations provide empirical knowledge about CNN architecture design for image SR.
Loading 1607.07680v1…